Hi,
Is it possible that some of the load here is also caused - at least temporarily during some peak time - by the so-called "hidden traffic" of some crawlers and bots? (whould show within Apache's access log and/or some statistic software). (At my sites the hidden traffic is about 40% of the overall traffic daily). If so they can be throttled down so that they do not crawl so frequently and aggressively, just with an entry into the robots.txt:
User-agent: *yahoobot* //just an example, wildcards can be used
Crawl-delay: 60
//that's seconds, so it would crawl one URL after another only every 60 seconds instead of crawling several at the same time as the Googlebot for example does.
http://en.wikipedia.org/wiki/Robots_exclusion_standard
This entry is respected by all major bots (Google, Yahoo).
And for really bad ones (which do not respect the robots.txt at all) set an environment variable within apache-conf which blocks those spiders.
//code_start
SetEnvIfNoCase User-Agent really_bad_spider BAD_BOT
Order allow,deny
Allow from all
deny from env=BAD_BOT
//code_end
(The related Apache_mods have to be enabled first).
And one other thing: do you use Apache file-header caching for images, js-, css-files and any "static" content?
That would be maintained with mod_expires, e.g.
//code_start
<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 15 minutes"
ExpiresByType text/plain "access plus 15 minutes"
ExpiresByType image/gif "access plus 1 day"
ExpiresByType image/x-icon "access plus 1 day"
ExpiresByType image/jpg "access plus 1 day"
ExpiresByType image/jpeg "access plus 1 days"
and so on
//code_end
That would tell the browser to compare the file's timestamp on the server with its own in its browser cache during http-header handshake. That reduces bandwidth and response time too, besides of "real" file caching by Apache (two different caches). The browser does not download the file if it's local copy did not expire. The above settings are NOT default btw.
This should further reduce bandwidth consumption and CPU-load.
All the best,
Robert