I wonder if someone could point me in the right direction on this: I've just migrated a major UK site (600,000 users, 30,000 pages, typically 200 anonymous users accessing the site at any one time) to Drupal: http://www.ivillage.co.uk (not the US version of the site) and be prepared for possible embarrassment, this is a women's magazine website :-)
It's up and running and stable on 3 load balanced front-ends and DB server with backup. Akamai handles all assets ending with an extension (jpg, js, css and so on) but lets everything else through without caching. We have Varnish running to handle anonymous requests. And also MemCached.
Basically the site is fine, just getting tweaks and minor changes as we go along, except for quizzes.
This is my first launch of this size and I don't know enough to be able to sensibly debug the problem so I am hoping you may be able to offer some advice (the contract finishes in two days so I really need to get this resolved).
The quizzes are the problem. I built them from the Chaos Tools form wizard which is very nice and works beautifully everywhere except the Live site. Where it sort of works but doesn't accept the forms first time around but eventually does - and you can work through to the end and get a valid answer.
We are on Drupal 6/Pressflow. I have the "reverse_proxy" flag set in settings.php, the Forms cache is not included in the MemCache list (I didn't set this part up) so I assume forms aren't getting cached with MemCache. I do have the Chaos Tools/Forms Wizard patch for anonymous users in place (which does work on the Dev/Staging sites).
The Pressflow Cookie module is enabled.
Direct access to the site (i.e. not via Akamai) can be achieved by prefixing the domain with "origin-www" instead of just "www" (will be passwording this, but it's open at the moment).
Can anyone help on this? What else do you need to know?
Comments
memcache does cache forms
You have 2 options from my point of view
a) http://drupal.org/node/959200
b) http://drupal.org/node/795004#comment-3611924
I would opt for option a if possible. test, test, test!
Varnish header
Are you sure Varnish is caching? I don't see the standard two-timestamp header:
http://screencast.com/t/EC0xeLLdEaM
$ siege -g http://www.ivillage.co.uk/
GET / HTTP/1.1
Host: www.ivillage.co.uk
Accept: /
Accept-Encoding: gzip
User-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.67)
Connection: close
HTTP/1.1 200 OK
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_perl/2.0.4 Perl/v5.10.0
X-Powered-By: PHP/5.2.6-1+lenny9
X-Drupal-Cache: HIT
ETag: "1292817956-1"
Cache-Control: public, max-age=0
Last-Modified: Mon, 20 Dec 2010 04:05:56 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie,Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
X-Cacheable: NO: !beresp.cacheable
Content-Length: 16979
X-Varnish: 1505654513
Date: Mon, 20 Dec 2010 04:28:21 GMT
Connection: close
I didn't get an issue with the quiz at all. Maybe because cache isn't actually working? Or is there a subtler issue with using the quiz that I'm not seeing?
Thanks for the comments -
Thanks for the comments - I'll take a look at these. Maybe the quiz is working because Varnish has failed? (I wish I understood more of this stuff - still, this is a good way to learn, I guess.)
I'll let you know...
Steve
(Update: About quizzes - they can appear to work for a while, and then will fail on a later question, I just tried it and went through fine for 4 questions, then it re-asked the question, then did another couple fine, and then re-asked - it's not consistent so might have appeared to have worked if you did just a couple of questions - it seems time related, answer quickly and you're okay.)
Okay, I've implemented the
Okay, I've implemented the memcache bypass. Although we were using memcache.db.inc anyway - basically I extended the concept in the comment and applied it to both files: I added a function in dmemcache.inc to test whether a table is excluded from memcache; then used that in both memcache.db.inc (to skip memcache calls) and in memcache.inc to call the standard cache routines.
It may only be subjective but the problem seems to happen slightly less often.
But maybe I'm looking in the wrong place. The CTools form wizard uses its own table for storing form information, this is not a standard cache table so doesn't get put into memcache anyway.
Doing an anonymous quiz and checking the contents of this table after every question shows me that the session ID is changing, sometimes. And when it does it forgets it's done a question and asks again (then generates a "you have missed a question" validation error). But it manages to keep track of the form information so gives the right results at the end.
Yep, double and triple checked - it's somehow forgetting the session ID but not the form data and answers. Which suggests it's forgetting it during the page processing - so it might actually be my code. I'll check further. (EDIT: Nope, don't think it's my code - since I don;t touch session ID anywhere.)
Aha!
I've been beating my head against this for the last couple of days - got to grips with Varnish VCL in the process, figured out and cleaned up the unnecessary complexities the previous guy had left behind in the Varnish code - and increased the cache hit rate by a factor of 100 in the process.
But I had become fixated on the idea that the problem had to be Varnish, completely missing the fact that the errors occur using the Akamai route, but not using the non-Akamai route which still uses Varnish.
Thick as a brick.
Tomorrow is my last day on the project but it'll be enough time to brief the full-time techie on the situation.
Phew.