The drupal caching problem

I know there are a lot of contrib caching modules available. But as I stated they either cache or do nothing at all. When using a CCK and Views setup with in example a pollfield an place that in a block on the homepage you can't use pagecaching or block caching. When does it need to cache the block? the drupal caches are reletively stupid, the granularity is limited to role or user selections. There are no other means to define in wich cases caching should take place without programming something yourself.

In my idea the module will be able to define(hooks) sertain states of the objects they deliver. Ie the pollfield would have 2 states: unvoted and voted. In the settings tab you will be able to define seperatly wich states get cached and by what criteria(in what context). The context filters are also defined by modules(hooks) ie. the user module could define/mediate the user/role based context but via the same api the cck_pollfield could provide other context templates like when a vote has been cast.

Thats more along the lines I was thinking. More depth, making it more extensible. You don't have to program the context templates over and over again only the way your module reacts to them.

Thanks for your reaction. "content aware caching" is a nice term for what Iam trying to achieve, but its also only one aspect of what would be ultimately possible when implementing these features.

As for wich modules I would like to support, those would be CCK, views, block and node as these modules will cover alot of the drupal websites in production. For those modules wishing to either leverage the caching system or provide context filters will have an api trough wich to do so.

Also theme context filters would be a great feature thanks for the suggestion.

Iam currently looking into the posibilities of the trigger/action system and if it can be leveraged to trigger the specific events.

I hope to get as much information from you(my fellow developers) before I start producing a workable proposal.

Thanks for the usefull reply, any other ideas/critism would be welcome.

I have to say, that I can completely see where Joost is coming from.

One only has to take 2 simple scenarios to show how limited caching is in Drupal:

• A block somewhere on a page that should show random ads. These ads should rotate on every page refresh.
• The random image block.

Now, even with the most basic level of cache turned on, the randomness of both the above is broken for users who are not logged in (which is the majority in just about any site). So for truly random content (without having to resort to AJAX and the like) one has to disable all caching!

I've read about Boost and while it sounds clever, I'm sorry, don't like it. Relying on a browser to perform one or more AJAX requests to fill a page seems silly when the server could integrate that dynamic content before returning the page.

IMHO a better approach would be that cached pages should include special markup. Something like "". The caching system can build pages and store this markup, and retrieve it as now. However, it should be fed through what I guess is a filter that would replace this markup with appropriate HTML (ads or images in the examples given above) before sending it out the door to the browser.

Can anyone comment on if/why they think this would be good or bad?

@raintonr:
You don't have to completely disable anonymous page cache. You can do it only for pages where your dynamic block is showed. And as always, "there's a module for that" :)

Yes, Drupal lacks out-of-the-box granular caching. But that's not because Drupal is bad at this, but rather because granular cache invalidation in generic way is very hard. To effectively cache individual content pieces one would need to flush cache for these pieces. Then there's problem noone knows what your content pieces are, and how do you use them. Because of this, out-of-the box granular caching is not possible.

You can look at the Panels module. Using Panels, it's possible to divide pages into separate blocks, and cache these blocks differently and separately. Then it's possible to flush your caches, using context system. Context system of Panels furtunately is very useful for caching, because it answers simple but important for caching questions in generic way: "What is inside this block ? How does it depend on external data ?". Then, because you know the answers, you know relationship to the external data, and can track that external data, and propagate invalidation of cache using context system. That's what Advanced Panels Cache module does for node context.

@crea... when you put ads and/or random images just about everywhere it's hardly worth the effort to figure out when not to cache pages.

I've seen the panels module but it's not useful when the dynamic bits of the pages you want to refresh on each load are in blocks on the side regions of the page.

But then isn't a bit off that you are forced to use panels and context to build your site so you can have decent caching?

Wouldn't it be better to just have a caching module that works with what you build your site with?
Seems to me there is a genuine need for something like this.

Before summing up all the comments I would like to point out that javascript on demand loading of content is outside the scope of what I am trying to achieve. Let me sum up what I would like to create:

A general cache control module responsible for:
- Handling knowledge of wich cachable objects and associated states exist. These may be defined by one or multiple hooks.
- Presenting a backend for the site administrator to define the context that will meet the criteria for when to clear the caches. In example: "Added new node for content type X" or "Vote cast on pollfield attatched to node".
- Center control for context aware caching. It will dispatch all context aware caching requests for it will hold all the information concerning the object(&states), the configured triggers and actions. It will match the object(state) being loaded against the provided contexts and handle approriatly.

A contrib module would be able to integrate this by:
- Defining ie: hook_[mymodulename]state(). In wich you could define all your objects you want to be configurable by the cache.
- Defining ie: hook[mymodulename]_context(). Wich will allow you to define custom contexts for your module.
- Allowing perhaps different optional proccessors by the module for when the cache handling is too specific. This is just a tought.

Now what this project won't do:
- Handle cache backends, I would much rather like to see integration with the cache router project.

The goal of this project:
- Adding depth and granularity to the caching of whatever objects you would like to see cached.
- Easy to use hooks for the programmers since it will be designed to make their life's easier and removing redundant custom cache handling. Thus removing the need to otherwise develop custom modules that would have been achievable by combining others IE cck&views.
- A way for site administrators(non-programmers) to use this functionality provided by other modules to speed up their website.

Essential for the success of this project:
- This module will only create the general API + I will personaly integrate some frequent used modules like node,cck, and views (maybe later the block module too). But for this module to become a success it must become a everyday tool in the module developers workset. The point being is that I can't build all the context, so the more context modules will be build the more this module will grow.