Towards a Generalized Drupal Object Caching Mechanism

In my never-ending quest for greater Drupal Glory, I've been spending the past year boning up on the various ways to improve site performance and address issues of scalability. Today, doing some noodling with Amazon EC2 instances (and remaining unconvinced about their raw performance as potential master database servers) I had a thought:

What would it take to extend the static node cache in node_load() beyond the individual drupal bootstrap?

Like just about everyone else, I've been loving how much memcached helps speed site performance. It simply rocks, and everyone looking to reduce server load and speed page responses should be looking into it. One of the better things about it is that it can store and return data objects natively, meaning not only are you letting PHP pull something out of a lightning-fast memory cloud, you also don't incur the CPU overhead of having to unserialize() a string into an object or array.

This let me to my thought. If you want a massively scalable interactive drupal site, you need ready access to tons of nodes. Inevitably you will hit the wall with logged-in requests for these from your database. But what if we were able to take the performance boost we get from node_load()'s static cache, and make it work persistently across an entire site, rather than just for one pageload?

This presents an attractive roadmap for high-performance drupal architecture: using the database as the persistent data store, with tables optimized around high-performance searching/sorting using the methods outlined in David Strauss's excellent DNA module, and maintaining a memcached-based cloud of fully-built node objects on top of that.

Outlining the code here is pretty simple. A module with a low weight would implement hook_nodeapi() to handle clearing the node cache on insert and update, and filling it on load. Then a simple wrapper around the default node_load() would check said cache first before going to the database.

In a perfect world, this would be baked into core, and such a thing might be possible for 7.x. However, for experienced Drupal developers, the kind of patching needed to implement this under 6.x would be even simpler than the serialization diffs associated w/memcache under 5.x. It could literally be a couple lines after the static check in node_load().

Should such a project emerge and mature, it could develop into an overall object-cache for all the core drupal elements (e.g. users, taxonomy terms, etc). Along with the new schema architeture (which lays the groundwork for application segmentation/sharding) and testing framework, a high-performance object-cache would position Drupal 7 to take the enterprise by storm.

Any responses?