If anything, this is a note to self. I tend to keep a session of multitail open on any server I like keeping an eye on. Typically it will include the security logs, http logs, error logs, and audit logs. In a few instances when 'logrotate' has kicked in overnight while a 'multitail' session was open, watching the files that were getting moved, all hell breaks loose.
Since the file handle/files that multitail watches suddenly disappear/become invalid, the spawned 'tail' children go into a psuedo-fork-bomb/zombie loop.
For instance, this morning I awoke to about 3000 rapidly spawning and dying 'tail' processes that made it difficult to near impossible to pull up a process list. When that failed, I ended up browsing /proc to see if something bad was going on. I also peeked at /proc/loadavg which was actually pretty normal. Since I have some sane ulimits set on the machine in question, the overall problem didn't become a system-wide out-of-memory problem/crash.
When I get a moment I'll see if I can hack a way to make multitail a little more friendly to files getting purged from under it.
If you have no idea what I'm talking about with multitail, please check out the website for it:
http://www.vanheusden.com/multitail/
It is like tail on steriods.