Cache Warmer Customizations

the parallel mode works by POSTing the URIs to be crawled to the lua location. So you need to POST the options and extract them at the Lua level. This needs to be fixed, so that you can set header options. Can you try replacing the cache_warmer_requests.lua script furnished with cache_warmer by this: https://gist.github.com/2870811 and report back.

because http_build_query percents encodes the URIs. So the URI to be hit has a lot of % signs. And I guess boost doesn't handle url encoded URLs. Could you check the server logs?

We had no lucky with this one. It behaves the same with that code change.

this is the crawler vhost access log

1127.0.0.1 - - [05/Jun/2012:00:11:29 +0100] "POST /cache-warmer HTTP/1.1" 200 76 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:11:43 +0100] "POST /cache-warmer HTTP/1.1" 200 51 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:13:09 +0100] "POST /cache-warmer HTTP/1.1" 200 76 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:13:23 +0100] "POST /cache-warmer HTTP/1.1" 200 51 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:13:51 +0100] "POST /cache-warmer HTTP/1.1" 200 51 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:13:51 +0100] "POST /cache-warmer HTTP/1.1" 200 51 "-" "-"
127.0.0.1 - - [05/Jun/2012:00:14:45 +0100] "POST /cache-warmer HTTP/1.1" 200 39 "-" "-"

nothing to post in the crawler vhost error log.

Is extrange because we just see in the nginx wiki the ngx.unescape_url command and it seem to do in paper exactly what we need :(

First of all, thanks for keep helping us with this...

Here is a portion of the crawler vhost error log:

we put this:

drush_log(http_build_query($post_data, '', '&'), 'error');

in the funtion cache_warmer_crawl_multiple inside the main loop and we get this kind of log on the console:

data0=&data1=catalogo&data2=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20&data3=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D1&data4=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D2&data5=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D3&data6=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D4&data7=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D5&data8=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D6&data9=catalogo%3Fprice%5Bmin%5D%3D%26price%5Bmax%5D%3D%26populate%3D%26sort_by%3Dtitle%26sort_order%3DASC%26items_per_page%3D20%26page%3D7&base_uri=http%3A%2F%2Fes.gmidos.nx

It seem that the...

catalogo_brands[0]=1247%26sort_by=title%26sort_order=ASC.html

...generated pages, were badly generated. All are views pages and after boost caches them it seem that all files sizes are the same and the html generated has problems (views empty results) so that piece of code dont work.

microcaching as opposed to Boost? It simplifies your setup and makes it faster. One less thing on the drupal side. Move the cache to the server layer.

Ok, we made our first try in setting up the microcaching system... with no luck :)

We have Boost uninstalled and we are using your nginx configuration with microcaching activated for anon users in drupal 7.

The only change we made in the files (only for debugging purposes) was to change this 2 lines:

fastcgi_cache_valid 200 301 60m;
fastcgi_cache_valid 302 60m;

We want to use this in conjuntion with "cache_warmer" and not using microcache per se but use cachewarmer frecuently enough and Nginx cache with a big TTL. Before we tried to test cache warmer, we tryed to browse directly.

When browsing normally the site as anonymous and viewing the headers in Firebug we get "X-Micro-Cache:MISS" in the Response Header for all pages.

Also if we go to the directory where cache files should be saved? /var/cache/nginx/microcache there are no files inside.

Truth is that we have some custom Cookies. Maybe this is too much but we will try to explain some of them:

• saw_welcome:
  This Cookie is set to "0" for all users (anonymous and authenticated) the first time he visits any of our pages.
  In every page load we look at this Cookie and if the user is ANONYMOUS and the Cookie is "0" (or is not set) the user is redirected to a welcome page.
  If the user submits the form in the welcome page, this Cookie is set to "1" and the user is redirected to the page he was trying to see.
  All subsequent page loads will see that the user has this Cookie set to "1" and will not redirect him again.

• has_qtip:
  We use the jQuery QTips plugin so we can use custom tooltips in our links.
  This Cookie is set to "1" for all users (anonymous and authenticated) the first time he visits any of our pages.
  In every page load we look at this Cookie and if the user is AUTHENTICATED and the Cookie is "1", we add a
  special class "qtip" to all links in our page so JavaScript can get those links and apply jQuery QTips to them.
  The point of this Cookie is that we have a TOOGLE link on top of every page that the user can click and ENABLE/DISABLE this feature.
  So everytime the user clicks on this link, JavaScript toggles the Cookie value from "1" to "0" or from "0" to "1".
  With the Cookie at "0" the user will see the normal browser's tooltip.
  This TOOGLE link is only presented to AUTHENTICATED users so anonymous will always see the jQuery QTips tooltips (as they have the Cookie set as "1").

We have 3 more Cookies (very similar to "has_qtip" Cookie. They are TOGGLES for accesibility/usability purposes) but you get the idea.
This was all working good in Boost, as Boost doesn't care about Cookies and all cached files were created successfully.

So our question are:

1) Is there a way of configuring microcaching so pages are cached and served even if there are Cookies?
So for example, in the case of the "has_qtip" Cookie, the pages should be cached and served with the special class "qtip" in all links. We could get rid of all of our custom Cookies except the "saw_welcome" one, but we think this will not be any solution as we can see other Drupal Cookies (and Google Analytics).

2) We are trying to get it work like boost so is there a way to tell the microcache which pages we want to be cached or which wont using something like wildcards inside your nginx config?

We are trying a lot of things by reading the "HttpFastcgiModule" help pages (http://wiki.nginx.org/HttpFcgiModule) but this is all brand new to us.

Thanks in advance, you have been very helpful already.

microcache_fcgi_auth.conf instead. It handles the Cookie and Set-Cookie headers differently. They're not analysed at all, they're passed untouched to the backend.

After learning a lot of what your config codes do, we have managed to microcache our commerce site for Anonymous users.

Now we want to limit the cache to only specific pages as we had in Boost (so pages like "/admin" or the Shopping Cart, CheckOut pages, Profile pages, etc... will remain uncached).

We have the following code in "microcache_fcgi.conf":

The only problem we have is that we can't seem to cache the HOME page.
We have tried a lot of possible combinations for calling this page:
""
"/"
"/home"
"$document_root"
"^/"
"^/$"
""
...etc...

We have also tried with the following conditional without luck:

We have also tried using $request_uri instead of $uri with the same luck.

In Drupal (/admin/config/system/site-information) we have configured the front page to be "/home".

So is there an easy way of checking the front page $uri?
Thanks in advance.

Thank you again perusio and sorry for the late response. We are using "map" now to make all of our conditions inside "map_cache.conf" and all is working good.

Well, of course we are having more troubles :) This time are the form submits and Drupal messages.

A couple of days ago we were having troubles login in the site with the login block (it's on every page on our site). After login, the page was refreshed but you still were anonymous until you go to an un-cached page. So we end up redirecting this form to the user's profile page (which is not cached).

All good until we tried to use othe forms in our site, like the contact form, the "subscribe to our newsletter" block (simplenews), the "send this product to a friend" form (forward), etc... After submiting any of these forms, a message is sent (for example the "Your message was sent succesfully" in the contact form).

The problem is that:
- If you submit the form in a already cached page, no message is visible until you reach an un-cached page (where it then displays).
- If you submit the form in a non-cached page, the message is displayed, but the page is cached so other anonymous that go to the same page will see that message.

We understand it's a POST problem? We have a map as the following:

Do we have to alter all forms to add a "?nocache=1" query to the redirection and use the $arg_nocache variable?

Will try tonight some more tests but we were wondering if there's another way. What we need is to bypass saving/serving a cache if a form was submitted.
Thanks again :)

Hi again.

We still having problems trying to crawl URIs like the following with the PARALLEL crawler:
catalogo?brands[0]=1500&sort_by=title&sort_order=ASC
Those URIs get crawled with only the first parameter:
catalogo?brands[0]=1500

This does not happend in the SINGLE crawler as stated before, so we were using the SINGLE for these types of URIs and the PARALLEL for the normal ones.
Till here we are good, we don't mind using the 2 crawlers.

The problem is that cache files created in the SINGLE crawler are empty. Well, not really empty, but only with the headers. If you go to that crawled page, you see a blank page.

The only thing we came up is modifying the cUrl options array and setting the CURLOPT_NOBODY option to FALSE:
CURLOPT_NOBODY => FALSE
After this cache files are full with the headers AND the html, and if you go to the crawled page you see it as it should be.

We thought that all was good, but yesterday we started creating log files.
Our log file (adding > path/log_file.log to the Drush command) is getting filled with HTML from all crawled pages, so it ends up being 90Mb.
Inside the log, if we scroll down to the end, we can see the JSON responses returned by cache_warmer_execute.

Setting again CURLOPT_NOBODY => TRUE makes the log with only the JSON responses, but makes our cache files empty.

Any ideas what could be the cause for the response HTML getting inside our logs?
Thanks in advance

What I'm curious about is why the URIs with several & are not handled properly by the lua client. Could be some lua socket quirk though. That's code that is getting long in the tooth. I would like to move it to the Nginx Lua cosocket API as soon as possible.

It won't be able to do massive parallelization (thousands of requests in parallel) with lua socket.

This is work in progress.

I'll have more time to deal with after devdays. It will be cool to compare the resource usage and speed of httprl and Lua cosocket. I don't think PHP can compare with Lua. No JIT, much larger language, I don't think it goes beyond select() at the I/O event notification layer. My main interest in having httprl is that is an alternative to following the Lua route.

OTOH I think that httprl will probably beat cURL multi easily in terms of performance.

it can handle thousands of simultaneous connections. The weak link is l luasocket. With cosocket yes becauses it uses the event loop made available by the Nginx API.