Lately, I've been having some performance issues on a high traffic Drupal 5 site I maintain. Eventually, I'd like to get on Pressflow 6 with Varnish and Memcache, but as an interim step I decided to drop PF5 on our staging site to see what would happen. As it's supposed to be drop in drop out compatible, it seemed worth a shot. After doing some JMeter performance testing, it looked like there were marginal gains to be had with just PF over stock Drupal. Everything seemed ok, so I decided to push to production while I installed Varnish on the staging site and got into configuring it there. That's where my problems began.
Unfortunately, almost immediately I started getting reports of bloggers sporadically seeing "Validation error, please try again. If this error persists, please contact the site administrator." on form submissions. This is the error you get when the hidden form token doesn't match against (I believe) your session. I asked them to log out and log back in because I thought it might be that PF was generating session tokens differently than stock Drupal. They still (sporadically) got the error. That didn't seem good so I reverted the site back to stock D5 and figured I'd test it out more in staging. The problem is, the bloggers were still seeing this error. I considered that maybe they now had different PF generated tokens in their session so I blew away the session table and had everyone log back in. That didn't work. Looking in the apache error log, the only thing I see is:
PHP Warning: unserialize() expects parameter 1 to be string, array given in /var/www/vhosts/www.maximumpc.com/httpdocs/includes/bootstrap.inc on line 423
It comes in the variable_init when the variable is read from the cache. Thinking that maybe PF stuck some different formatted stuff in the cache, I restarted memcached. Another error that some of the users saw was the "Date Timezone module requires you to set the site timezone name." This seemed like it could be related to a problem reading a variable out of the cache and that Drupal error wasn't occurring before the upgrade to PF. Now, I'm not sure whether this Apache error occurred before PF or not but I thought I'd point it out in case it was relevant.
So, here I am, running the same code I was yesterday but now I have reports of sporadic form validation errors (which I was able to reproduce once in staging) and an error in the Apache error log. I've cleared the session table and reset memcache but I still see these errors. Does anyone know what might be causing them and how Pressflow might be involved? Any suggestions on what to check?
Comments
Form validation issues on
Form validation issues on sites using memcached are most commonly caused by the form cache sharing a memcached bin with other, frequently cleared caches. That issue also affects the standard Drupal core.
Pressflow 5 makes no changes to your schema or site data. If an issue persists after moving back to standard Drupal core, it is, quite simply, not a Pressflow-related issue. Your unserialization issue may be related to the patch bundled with the Drupal 5 memcache module; it changes how and when core serializes cache items.
You need to know which
You need to know which variable is causing the issue. In variable_init() change:
<?phpwhile ($variable = db_fetch_object($result)) {
$variables[$variable->name] = unserialize($variable->value);
}
?>
to
<?phpwhile ($variable = db_fetch_object($result)) {
$variables[$variable->name] = unserialize($variable->value);
var_dump($variable->name);
}
?>
make sure display_errors is on. Now you can search for the Warning and below will be the name of the offending variable.
Some of mystery just cleared up
Ok, so I figured out why reverting back to D5 from Pressflow didn't seem to affect my errors. It's because I thought I had reverted back but I really hadn't (deploy script failed but I didn't notice). However, that still leaves me with a Pressflow install that has some strange intermittent errors with form validation and variable retrieval.
The cache bins I'm using are default, filter, and page so the form cache gets lumped into (I'm assuming) default. I don't believe that cache is being flushed especially often as there are three 512MB instances of memcache running (one on each web head and one on the db) so there should be plenty of space. Furthermore, I don't see these same errors now that I've properly reverted the site back to standard D5.
I'll take a look at tracking down which variable is having an issue tomorrow and see what pops up.
memcache patch
David,
Can you expand on the memcache module issue, as I'm pretty confident that is the culprit. When we were evaluating using Pressflow, we noticed the drupal.org infrastructure issue (http://drupal.org/node/466444) referenced on https://wiki.fourkitchens.com/display/PF/Comparison. This pointed out changes in cache.inc and it's potential effects on memcache.inc.
We have been running the memcache module on our Drupal 5 installation, and definitely rely on it. I expected to be able to use both pressflow reverse proxy support and the memcache module. Do we need to apply a specific patch to our memcache module in order for it to work correctly? We were planning on rolling out Pressflow with memcache prior to getting Varnish configured and deployed in front of it.
Are there any other particular trouble spots that we should look out for as we switch from Drupal 5 to Pressflow 5? It seems like flushing the cache and potentially dropping sessions is needed when we roll it out on our live site.
Thanks for releasing Pressflow. I'm excited to get it working properly for our sites.
@cdoyle It's not a problem
@cdoyle It's not a problem with memcached running out of space. It's a problem with how the memcache module for Drupal performs wildcard clears: it empties the cache bin. That means your form cache is being erased during the invalidation of other cache items.
@mmarano There's no issue using memcached and Varnish together, nor should there technically be a reason to need to clear anything after switching to Pressflow 5. That said, it's never a bad idea to flush caches and sessions during a deployment. However, the memcache module itself has a patch for core that changes cache serialization behavior to make it work like Drupal/Pressflow 6. That patch can be used with Drupal 5 or Pressflow 5, and most sites using memcache apply it.
lightbulb
@David thanks for the follow up. We had applied the patches from memcache to our base installation of Drupal which we deploy, but those same patches were most likely NOT applied to the Pressflow core upon dropping it in. I imagine after applying those patches, and separating our form bin, these issues should be resolved.
Pressflow is up
We applied the patches from memcache.inc and that seems to have solved our problems. We've got Pressflow 5 running in production. Now, we're on to configuring Varnish...
Are there any baseline Drupal varnish configs, or good starting points for Drupal Varnish configs aside from what I've found on blog posts?
Mercury instructions, but has basic Varish config.
http://groups.drupal.org/node/25425/
I've used this as a baseline for configuring Ubuntu 9.10 on a non-virtual server. Pretty awesome.
propeller
We've been referring to Greg's great blog post at http://blogs.osuosl.org/gchaix/2009/10/12/pressflow-varnish-and-caching/, and are calling it the "propeller" config :)
Haha ... thanks
The config in that post is a bit dated. Here's a copy of my current running config, warts and all. It will cache even if a SESSION cookie is set. I'm running it in production, but I'd consider it a pretty early beta:
http://blogs.osuosl.org/gchaix/2010/01/23/varnish-config-defaultvcl/
-Propeller-head
subscribing
subscribing
I seem to have run into this
I seem to have run into this 'form validation error' on a Pressflow installation with memcache. Which are the core patches that have to be applied with memcache? I don't see that in the latest memcache release (6.x-1.9).
"Be the change you wish to see in the world", M. K. Gandhi.
Something like this?
http://drupal.org/node/795004
See solution here
http://drupal.org/node/795004#comment-4297750