First-class support for ESI in Drupal: Lets Do It!

Quick Summery/Plan

• Support the following replacement methods
  Server Side
  ESI - http://varnish-cache.org/wiki/ESIfeatures (get this working first, add in SSI, AJAX later)
  SSI - http://httpd.apache.org/docs/trunk/mod/mod_include.html

  Client Side
  AJAX - Steal code from boost to do this

• Use esi.php either as a file or menu hook. File is so we can cache it (implement with a cache lifetime of 0 first, work on storage later).
• Replacement of Blocks, Panels, Theme Hooks, Regions
• Need to agree on a standard for calling esi.php and the security concerns with making a generalized way of getting chunks of html out of Drupal. Eventual support for JSON output.

I see the first alpaha of this module will have ESI, no cache, and the replacement of blocks or panels working.

Current interesting projects/code
http://drupal.org/project/authcache
http://drupal.org/project/ajaxify_regions
http://drupal.org/project/remoteblocks
http://drupal.org/node/657826#comment-2372308

I wanted to share some of my findings and testing in D6.

My goal was to test without patching / hacking core and focusing on blocks only. More or less, I was after a proof of concept, so this method is likely not suitable for anything future / scalable / deployable to users who don't really know why they want ESI. BUT it might spark some ideas or discussions on the future implementation.

The use case is for anonymous traffic with the page cached / varnished; therefore, as long as you rewrite the content of the block to an ESI tag when the page gets built, store some basic info to rebuild the object later in the menu callback, and be creative with the menu callbacks so you can target in your varnish default.vcl -- then once the page is cached everything is driven by varnish calling the esi callback paths.

Method

1. Storing Data
   • hook_form_{block_add_block_form}_alter, hook_form_{block_admin_configure}_alter
   • store in db table: block bid, ttl
   • store in cache: block ttl and basic info to rebuild block on menu callback
2. Page building
   • Override theme_blocks()
     • Override an entire region or target individual blocks
     • rewrite content with <esi:include src = "esi / {ttl} / {identifying info}"/>
     • Normal processing if not anonymous
   • Implement hook_preprocess_page()
     • Flag a page as having any esi, so you can target in vcl file to run esi; and disable any page compression [server or drupal] - see compression issue below
     • Collect all AJAX (ttl = 0) blocks into one $.getJSON call
3. ESI Callback
   • hook_menu(): path includes ttl and some identifying - "esi / {ttl} / {identifying info}"
   • esi callback:
     1. Bootstrap Full
     2. Load block with identifying info
     3. Output content

Results

I added 3 blocks that output the current time - AJAX, ttl = 5s, and ttl = 15s (short times for dev). Hitting refresh a few times, you can see the magic take place as times change / don't change.

Caveats / Issues

• Multiple Bootstraps - I believe this is the biggest hurdle to overcome. In my test above, there is a min of 1 bootstrap and a max of 3 - when the 5s and 15s blocks align. Using entire regions can help minimize the number of calls. Having different ttl blocks / regions could complicate any intermediate cache layer to minimize the calls.
• Page compression with esi - this is currently not supported, so you have to turn off any compression when processing any esi - see http://varnish-cache.org/wiki/PostTwoShoppingList#a6.Gzipsupport

Hope this gets some discussions going. Playing around with the blocks, you can really see the power and possibilities that esi brings to the table.

drupal.org does not use ESI. Patches welcome :)

I know that this module was born just a month ago at Drupalcon and holds some of the most recent ESI work - http://drupal.org/project/esi_api