Drupal and the crashing server

If you have an Apache server and it's a page that's accessible to anonymous visitors, you could try the Boost module. Boost will cache such a page in "fully rendered" form in a file outside of the Drupal database, and will cause Apache to serve the page from that file directly, without even bootstrapping Drupal. Boost won't help you at all for logged-in users, but for anonymous users it makes an incredible difference. Boost has excellent documentation and is quite easy to set up.

You should probably consult http://drupal.org/node/2601 and/or http://groups.drupal.org/high-performance, as I'm far from the world's expert on this, but here's what I can contribute:

Speeding up Drupal for logged-in users is harder, but the single most effective measure we've found is to implement PHP opcode caching, if you don't have this already. This basically precompiles the PHP source code that Drupal loads, for itself and contrib modules, and saves the compiled information in memory, so it doesn't have to be recompiled on every new page request. It saves CPU time, not disk time or bandwidth. It requires shared memory (which may be limited on virtual servers) for the opcode cache. There are several PHP opcode caches available, but the most common one is the APC cache, which is available as a PHP extension. Setting this up requires some degree of Linux sysadmin access and expertise. Some studies on the Web show a 3X speedup in Drupal page load times with APC and other caches, and our experience is consistent with this.

Another classic tool is Memcache. This is a general-purpose in-memory caching tool that can "cache anything", but there is a Drupal module http://drupal.org/project/memcache that hooks Memcache into the Drupal caching system (the one you're using, that caches blocks, CSS/JS compressed files, etc.). Drupal ordinarily stores the cached information in its database, but with Memcache, these read/write requests are redirected to the in-memory cache, which is faster and lightens the load on the database server. Memcache requires extra conventional memory, but not shared memory. The only caveat here is that I recently learned from a real expert -- Barry Jaspan at Acquia -- that they've temporarily stopped using Memcache because they found that it doesn't seem to be speeding up recent Drupal sites, and may actually slow things down. Last I heard, they were trying to figure out why this was so. In general, Memcache should help speed things up considerably. Note that Memcached is the common name of the daemon process that maintains the Memcache in-memory cache.

Some great advice above :D

I wholly agree with the opcode caching. It is one of those low hanging fruit which yield some terrific results. Even though it requires shared memory, your overall memory size per request will go down significantly (I've typically seen in the range of 4x - 5x less memory usage per request) so your site will be able to handle more page requests (as a note, your site will use UP TO 128 megs of memory per apache request, not allocate that whole amount in one go). With that said...

I also recommend you take a look at some of the articles by Khalid Baheyeldin (http://2bits.com/contents/articles) - he has talked a lot about backend performance at camps and cons - he knows his stuff. But the most important advice he gives is to DIAGNOSE your issue. As a start of us, what kind of server environment are you in (shared, vps, dedicated, cloud, etc - this is a fairly important question as the kind of help that can be offered will vary)? What kind of access to logs do you have? Caching could be the issue or it could be happening somewhere else. Devel will help out with what happens to logged in users (or I like to see it as a worst case scenario tool) on the number of queries occurring on the page, seeing problematic queries, seeing your memory usage for a page load, etc. Its also worthwhile to see your apache and mysql logs on what is happening. From everything you look at, you might find it is not related to any of the above at all (I have seen instances where it was faulty hardware that was cause of the issue(s) - bad ram and/or slow disks will not really solve any other things you do to the server). If could be that your db layer requires tuning (if you are able to do it), but no easy way to tell without some initial diagnoses.

And regarding memcache - it is a fantastic tool. But contrary to what is normally said, it actually isn't all that fast (it can be faster than the db, but its the connecting to memcache that eats up a chunk of time and depending on the setup, that is where issues could come up)). There are 2 areas that make memcache the amazing beast that it is: it first takes the load off the database layer and (where it really shines) is that it scales horizontally so you can have multiple memcache servers. The cache will be placed across them and retrieved from the right server smartly (so you could have a lot of really slow servers with a lot of ram acting as memcache servers for your site(s) if you wanted). In the case of acquia, it depends on their setup where memcache could be a hinderance (see the connection time chewup above). If you can actually get it working nicely, cacherouter (http://drupal.org/project/cacherouter) and using the apc backend is very fast (will use apc as the caching backend). APC directly talks to memory (no connection handshake necessary). Its drawback would be it doesn't scale (has to be on the web server and will only store the cache from that server so there isn't a pool of caching that can be called on). But for a single server instance (and if you get cacherouter working the way you want - frankly, I've found it finicky at times), apc will make it blaze.

@davemaxg - have you tried to simply crawl the site? Or are you referring to preloading the cache for logged in users (which can also be done by associating a session with your site crawler). Also, what is your current cache expiry set to? If you have cron running, it will clean out expired content (exception would be boost which handles things differently). You could also take a look at modules like object cache (http://drupal.org/project/object_cache) which would help with caching content on node_save/update/delete but does require you to do some programming (it is an API).

@davemaxg one other thing I guess I would do is something via drush for your logged in users (I'm a bit confused because you say that your site won't get a lot of users so your content has to be in cache at all times? And are users logging into your site or are they anon. users? authcache works best with logged in users while boost/varnish with anon. users.). If you are using views and panels to generate content views on your site, you should look at the cache settings for each view/panel you set up - having those be far in the future will help you out a ton.

For the object caching, I mentioned that because you could use it with something like display suite and it will cache the node display so your node content would effectively get cached for logged in users as well.

And under your circumstances, I don't know if you are already using pressflow or not. If you are not, then benchmark with pressflow and see if it yields you positive results.

Regarding crawling site via an authenticated session - check out http://drupal.lu/developing-drupal/advanced-usage/drupal-test-suite. I'm sure something could be automated to get you a authenticated session (or you could even try something out with drush).