Load average and memory usage very high on wiley recently, due to webalizer.
djce@wiley:/usr/local/webstats/zaphod-nginx$ ls -lh tmp/carl/
rw-r r- 1 root root 4.1G 2012-07-01 07:13 musicbrainz-full-combined.log-20120701
Typical daily log size.
- suck it up
- use something other than webalizer
- find a way to make webalizer use less resources (feed the log in in chunks? adjust webalizer.conf ?)
- process less data (only feed in, say, 10% of the log at random)
- process the data elsewhere (not on wiley)