Varnish not caching anonymous requests on my site

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
anantagati's picture

I just installed Mercury on Linode server, but Varnish doesn't cache pages on site I transferred there.

Steps I did during site transfer:
- copied modules and themes
- copied files
- imported database (didn't change settings.php as I used same database as in settings.php)
- run update.php
- enabled Varnish, Cookie cache bypass modules
- changed Caching to External (there is following warning: 'The following enabled modules are potentially incompatible with aggressive- and external-mode caching and will not function properly: mollom, notifications, og, statistics, statistics_advanced')
- run update.php
- cleared cache

When I run:
ab -k -n 100000 -c 50 http://domain/

I can see in 'top' there many 'apache2' instances starting and server 'crashes' with full memory and swap.

When I tried same with fresh Mercury installed Drupal it got served by Varnish and could handle it without problems.

Do you know what could be problem? Or what is best way how to import site?

Comments

Try disabling the statistics

justintime's picture

Try disabling the statistics and statistics_advanced modules. From my previous experience, statistics and 'high performance' are mutually exclusive :)

On another note, did you use the stackscript on the Linode? If so, how big of a Linode is it? You shouldn't be able to cause the server to thrash like that and it indicates some tweaks need to be made in Mercury.

Yes, I used StackScript for

anantagati's picture

Yes, I used StackScript for Mercury. It is Linode 720.

Http Headers

joshk's picture

Also, please inspect your HTTP headers and see if session or other cookies are being set. This is the most common cause of Varnish "not working."

There is a problem as Drupal

anantagati's picture

There is a problem as Drupal behaves for anonymous users like they are authenticated and set 'Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0'.

Seems like some of the modules makes problem.

Cookie: __unam=3ee1f7b-125392ef976-7513e30e-2808; __utma=9268519.1055559740.1259384867.1273159855.1273181591.72; __utmz=9268519.1259384867.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=9268519.authenticated%20user; __utmb=9268519.64.10.1273181591; SESS00afa164d320a7d8da1c2b364b6c9545=19c6c522df9bd776e469e8456cbbd5b8; has_js=1; __utmc=9268519; NO_CACHE=Y; SESSf8f910eda8a6aa32ad7c48e70f891937=fa3c8a6861c5f6ea7a6d8e03cbffe89e

HTTP/1.1 200 OK
Server: Apache/2.2.12 (Ubuntu)
X-Powered-By: PHP/5.2.10-2ubuntu6.4
Last-Modified: Thu, 06 May 2010 22:12:10 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Etag: "1273183930"
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Content-Length: 3509
Date: Thu, 06 May 2010 22:12:10 GMT
X-Varnish: 1911746106
Age: 0
Via: 1.1 varnish
Connection: keep-alive

Sesssion

joshk's picture

SESS00afa164d320a7d8da1c2b364b6c9545=19c6c522df9bd776e469e8456cbbd5b8

This indicates you have an active session. Varnish will not cache content for an active PHP session.

Yup ... something in your

gchaix's picture

Yup ... something in your site config is setting not one but two session cookies and a "NO_CACHE" cookie. I'd take a look at what modules you have installed to see what's setting the cookie. There are several modules - hierarchical select and masquerade, for example - that set session cookies on anonymous users that prevents pages from caching.

There are several modules -

anantagati's picture

There are several modules - hierarchical select and masquerade, for example - that set session cookies on anonymous users that prevents pages from caching.

I am using both of them. I don't need Macquerade so much, but I cannot be without Hierarchical select (which is used only on edit forms and one more page).

ayalon's picture

Just to test the same scenario on my server...

Greg Coit's picture

Both varnishhist or varnishstat (command line tools that come with Mercury) can be used to verify that pages are being served form Varnish. Firebug for Firefox can also be used by clearing any related cookies from your browser, opening Firebug, clicking on the net tab and loading an anonymous page from the site. You should see that the page has has age other than 0.

Hope this helps,

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

Test Varnish

ayalon's picture

I disabled cookies in my browser but the "Age" is always 0 in my pages although I see no cookies. Maybe its Google Analytics, that always tries to add some cookies? Or what could be the reason?

I used the Web Developper Toolbar to disable all cookies but I'm quite sure that varnish is not caching the page.

What could be a more specific test?

Check out

Do you think it is cause by

anantagati's picture

Do you think it is cause by some modules? As I didn't change anything in Mercury configuration.

You could edit bootstrap.inc

stewsnooze's picture

You could edit bootstrap.inc and add debug lines to drupal_session_commit(). You should be able to then search your code for the variables created in the session to blame a specific module

Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.

Thanks, I will try.

anantagati's picture

Thanks, I will try.

@stewsnooze could you

mpaler's picture

@stewsnooze could you elaborate on this?

Blogged on the technique

stewsnooze's picture

I've since posted an article on my website on how you can find the module causing your anonymous cache misses.

http://www.stewsnooze.com/content/what-stopping-varnish-and-drupal-press...

Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.

I disabled Masquerade module

anantagati's picture

edit: Sorry, I wanted to post it to different thread.

Problem seems to be in

anantagati's picture

Problem seems to be in Masquerade module and modules to restrict access. After disabling those modules pages are served by Varnish.

Thanks to everybody for help.

Domain Access?

ayalon's picture

I had exactly the same situation. No request were cached. I debugged this for hours and finally I found out the following:

If I disable the Drupal Caching in Pressflow and restart Varnish, the caching is working. As soon as I use any other option than "Disabled", the caching is not working.

Is the the normal behaviour or did I miss any instruction to always deactivate the Drupal Cache?

Greg Coit's picture

To work properly, Drupal Caching in Pressflow should be set to external.

Greg

--
Greg Coit
Systems Administrator
http://www.chapterthree.com

do you mean...

mpaler's picture

that you are selecting "disabled" under Performance > Caching Mode?

Disabled
Normal (recommended for production sites, no side effects)
Aggressive (experts only, possible side effects)
External (experts only, possible side effects)

Views Exposed Filters

tomhung's picture

I just found that Views Exposed Filters sets a NO_CACHE=Y cookie, therefore causing Varnish to "MISS"

interesting stuff

HongPong's picture

Needs to be a more standardized way of looking at all this...

i am having this same issue

drupalninja99's picture

i am having this same issue

HTTP/1.0 200 OK
Date: Wed, 10 Nov 2010 22:40:42 GMT
Server: Apache
X-Powered-By: PHP/5.2.6-1+lenny9
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 10 Nov 2010 22:40:42 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Vary: User-Agent,Accept-Encoding
Content-Type: text/html
X-Cache: MISS from 097921
Connection: close


so its prob some contrib modules doing this - but i have a million contrib modules, how do i narrow it down?

Follow me on twitter: @drupalninja

this may be a dumb question

drupalninja99's picture

this may be a dumb question but will any page with a block that has BLOCK_NO_CACHE not be cached by varnish? i would assume so and i found out that the language switcher dropdown has that setting so if thats correct then theres at least 1 block keeping my pages from being cached.

Follow me on twitter: @drupalninja

did you get an answer to our

guenoz's picture

did you get an answer to our common question ? cheers

BLOCK_NO_CACHE should not

stewsnooze's picture

BLOCK_NO_CACHE should not affect the Varnish layer. It just tells the block cache not to cache a block

Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.

Mercury

Group organizers

Group categories

Post Type

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week