Hi all,
I want to build a module that handels caching and is able to deal with altering pieces of content based on some variables.
For a client we've developed a D7 site a couple of years back. The site is high-traffic and serves a lot of visitors but only anonymous users. The site uses views, panels and panels everywhere for it's layout, general pagecaching for cache and all media comes from a CDN. Everything works great!
The client wants to take the site to "the next level" and implement personalisation. The site should change pieces of content based on behaviour and data we gathered about the user. This means that pagecaching becomes a problem, but turning it off would immediately kill all servers due to the high traffic (well, Amazon AWS would boot a lot of extra webservers to keep up with demand :) ).
so to re-iterate; I want to build a module that handels caching and is able to deal with altering pieces of content based on some variables.
I've looked into and done tests with panel cache, panel content cache, panel hash cache, etc. to see if i could cache independent panes of content and write a custom CTools cache plugin that would add the necessary checks to the caching and create a couple of versions of cache for a single pane.
But no caching method comes remotely close to just plain ol' page caching (200 - 300 ms). The only thing that does works a bit (roughly 4 - 5 times slower then page caching, arnoud 800 - 1000ms per page) is using panel content cache and caching entire panels (not panes). But this doesn't fit my needs as i'm not able to change the content of a single pane of the panel. Caching single panes comes in at around 2500 - 3000 ms per page.
The other method i researched was altering the drupal page cache to add some custom checks and create different versions for pages based on the user data, but there don't appear to be any hooks that which i can implement to accomplish this.
The only other method which could maybe work is something like Varnish, but i don't have any experience with Varnish so it's hard for me to say for sure.
Has anyone here have an idea how to go about adding in caching in the above situation so the site remains at usable speeds but with alterations in content based on some variables (user data in this case)?
Any help or suggestions are greatly appreciated!
Comments
I would recommend looking at
I would recommend looking at Varnish ESI. Can't use Authcache since your users are anonymous, so assuming you are using cookies to track your users? Would probably still have to write custom code since your not using drupal accounts/sessions to track your users though.
This sort of thing (custom pages for anon users) is new to me but I hope that helps get you moving in the right direction. I'm also interested in your results as I may have to do the same in an upcoming project. :)
Happy hunting!
-Andre
There seems to be a
There seems to be a misunderstanding. Authcache happily handles anonymous requests (with and without an open session). You can follow the tutorial in order to get a feeling of what it does and how it works.
What about using lazy_pane
What about using lazy_pane for panes you want to personalize?
I would send a fully cached
I would send a fully cached page 'skeleton ' (i.e. All non-personalized content) on the initial request then populate the personalized content with cache busted Ajax callbacks. It will mean your server will have to process more requests, but to the user it will still be very snappy.
Agreed, same concept as ESI,
Agreed, same concept as ESI, except using AJAX calls instead of ESI calls. Good call.
-Andre
AJAX support in the ESI module
The ESI module has support for AJAX-ifying the ESI fragments, so you could run the module without an ESI processor. That's more useful for testing though; the benefit of ESI really comes into its own when the fragments are cached outside Drupal (typically in Varnish/Akamai/etc).
There's a hook that allows you to define how content changes: e.g. you can specify that pane X changes according to geographical location; pane Y changes according to the time of day, etc. The scope of personalisation is sent as a cookie, which allows that content to be cached as an ESI fragment in Varnish (or other ESI-capable proxy) then reassembled with the page.
It does need a decent amount of Varnish VCL-fu to confugure though.
Acquia Lift might also be relevant and worth exploring.
--
Marcus Deglos
Founder / Technical Architect @ Techito.
Drupal 8
I'd take a good look at Drupal 8 using big pipe https://dev.acquia.com/blog/drupal-8-module-of-the-week/drupal-8-module-...
Thanks!
Thanks all for your response! Didn't expect so many answers in such a short timespan.
To list them all:
- Varnish with ESI
- Authcache
- Lazy panes
- Ajax approach with the dynamic blocks loaded on pageview
I think Varnish might be the sure-fire way of doing something like this and keep it really fast, but i've to work on a set budget regretably so this will probably be a bit overkill for now. Maybe if nothing else works :)
Authcache seems promising as wel, although if i understand correctly, all the different versions of content are stored on a role basis. I will need to be able to hook into this to set my own variables to differentiate the caches on. Will look into this.
Lazy panes could work, but the downside is that content will be gotten after the page loads which is not ideal, but will test anyway.
The ajax approach is as far as i can see what Lazy Panes is trying to do. Will look at this together with lazy panes.
You all gave some good advice. Will report back with my findings!
Authcache
You can tune Authcache and create your own "groups", not depending on roles. There are two mechanisms
1) Global (in your settings.php)
$conf['authcache_key_generator'] = 'custom_auth_key';
function custom_auth_key(){
global $base_root;
return $base_root;
}
2) in a module
function YOURMODULE_authcache_key_properties_alter(&$properties) {global $user;
$user=user_load($user->uid);
$properties['gid']=array();
if (@is_array($user->og_user_node['und']))
foreach($user->og_user_node['und'] as $gid){
$properties['gid'][]=$gid['target_id'];
}
if (isset($properties['base_root'])) unset($properties['base_root']);
}
Varnish vs budget
"I think Varnish might be the sure-fire way of doing something like this…but i've to work on a set budget"
Are you referring to a hardware budget, or a "time to learn all this and do it" budget?
Varnish is pretty lightweight: I run my blog on a very small VM with Varnish, and the overhead of running Varnish is more than compensated by the reduction in load on Drupal (even for anonymous users, fully cached page, with Memcache and APC).
So you would essentially get Varnish "for free" in terms of hardware costs.
But if you're looking at the costs of setting up/configuring Varnish, learning about it and about VCLs, and especially learning about configuring and using ESIs, that can be quite a time-sink.
I've talked about Varnish, ESI and the impact on performance at a few Drupalcamps…if you're interested in taking a look, you might find some of these slidedecks useful for getting started:
--
Marcus Deglos
Founder / Technical Architect @ Techito.
Yeah i was talking about the
Yeah i was talking about the time to setup/learn Varnish. We've enough servers to fit varnish in somewhere :)
If we go the varnish way, i will definately check those links out!