Common problem when someone removes a file that a process is using. The rm command simply unlinks files, it doesn't actually remove them, so if a process is still accessing it, then free space will not return.
To work out the processes that are hogging space:-
host01:/var/tmp# lsof +aL1 /usr/local COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME ns-httpd 919 nobody 5u VREG 85,18 1263031805 0 329450 /usr/local (/dev/md/dsk/d18) ns-httpd 919 nobody 11u VREG 85,18 624088105 0 329264 /usr/local (/dev/md/dsk/d18) ns-httpd 2321 nobody 5u VREG 85,18 35870416 0 447248 /usr/local (/dev/md/dsk/d18)
Now you have that information, the important columns are the PID and the FD handle number. Because the file no longer has a link within the normal filesystem, we have to access it through the proc interface that the process is using. This can be found in: /proc/<pid>/fd/<fd>
host01:/proc/919/fd# ls 0 1 10 11 12 2 260 3 4 5 6 7 8 9 host01:/proc/919/fd# cat /dev/null > 5
The last cat command should erase the contents of the deleted log, thereby freeing up space. BEWARE, doing this will still cause the logs to be written to nowhere, and will be unavailable. If you *really* had to, you could cat the file descriptor to a real file...
A nice way to find the unlinked files if you don't have lsof installed:-
find /proc/*/fd -links 0 -type f