Everything going the wrong way

Posted by cookie235 on February 17, 2011 at 8:43pm

Hi everybody,

I am working on a Drupal site that needs to be quite fast. I must mention it is using Domain Access, and TAC. I've been trying to setup Memcache and Varnish in an effort to improve the general performance of the site, but things have been going downhill rather than uphill. I am using Pressflow, though it was originally an Acquia install (I just overwrote the files). Here are a few numbers :

Anonymous     : regular cache  : 41  #/s (cache normal in preferences)
Anonymous     : memcache       : 36  #/s (cache normal in preferences, with cache_inc = .../memcache.inc)
Anonymous     : external cache : 0.8 #/s (cache external in preferences, no memcache)
Authenticated : regular cache  : 0.4 #/s (cache normal in preferences)
Authenticated : regular cache  : 0.4 #/s (cache normal in preferences, with cache_inc = .../memcache.in)

First of all, using memcache is WORST than not using it. Does this really happen ? I installed memcache_admin and made sure it was actually using memcache (also monitored memcache with memcache.php).
Second thing : 0.4 #/s when setting the cache to external ? I have no clue how to diagnose this. I have used the varnish.vcl config from 4 Kitchen (https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow)

Here is my APC config too (just in case) :

extension=apc.so
apc.shm_size=64
apc.include_once_override=1

Any idea on what to do from here ? 0.4#/s for authenticated users will never work. I need to improve it, but not sure how.

Comments

Regarding varnish, it will

Posted by soyarma on February 17, 2011 at 9:14pm

Regarding varnish, it will only serve cached pages when you have no cookies on the page. You should use something like the net tab in firebug to determine if there are cookies. Culprits are things like google analytics or omniture. You need to write rules to strip their cookies (which is OK, because the javascript is all processed client-side).

I too have never had good luck with memcache. I find that it is only of use if you have multiple webservers. Otherwise I nix it and use APC only.

stripping cookies

Posted by alexander allen on March 29, 2011 at 3:11pm

According to this (not a very reliable source per se) http://stackoverflow.com/questions/871283/how-do-tracking-cookies-work, software like GA and Omniture rely on setting a multitude of cookies, which are used to count recurring visits (how many times the same visitor comes in the same day, week and month). Wouldn't stripping these cookies from user's request mean that every time the user visits the page he counts as a new visitor?

Stripping cookies in Varnish is safe

Posted by stackpr on March 29, 2011 at 4:51pm

When cookies are stripped using VCL, it means that those are removed from the request object that is passed along to the web server. It does not affect the cookies that the user sees, so GA continues to work correctly. It is kind of like marking certain cookies as "do not send to server", which is great for tracking cookies like that since it allows you to treat more requests as identical and therefore increase your cache hit rate.

However, the VCL linked to in fourkitchens does appear to remove the GA tracking cookies, so that should not be the source problem.

I already checked cookies,

Posted by cookie235 on February 17, 2011 at 9:30pm

I already checked cookies, but ended up finding the problem with Varnish : I didn't setup my settings.php properly :
https://wiki.fourkitchens.com/display/PF/Using+Pressflow+behind+a+revers...

I've heard though that Varnish (through external cache) is not compatible with Domain Access. I haven't witnessed any problem yet, but didn't do an extensive testing either.

Now it's working, and I get about 500 #/s on anonymous users, which is much better. What are you using for APC : apc, cacherouter ? One hasn't been updated since 2009 and the other has no 6.x version :/

We use varnish and domain access

Posted by mikeytown2 on February 17, 2011 at 9:46pm

Granted we are not using pressflow and have a custom vcl file so it is different then a typical drupal varnish install. But from a technical standpoint they work together.

The performance settings page

Posted by msonnabaum on February 17, 2011 at 11:41pm

The performance settings page will tell you which modules are incompatible with external cache. Should be anything that implements hook_init or hook_exit.

You need a better way of knowing whether you're getting cache hits or not than looking at r/s. Add something like this to your VCL and you'll get headers that tell you:

sub vcl_deliver {
  # Add an X-Cache diagnostic header
  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
    set resp.http.X-Cache-Hits = obj.hits;
  } else {
    set resp.http.X-Cache = "MISS";
  }
  return(deliver);
}

I would guess that sessions are being started for anonymous users. Use curl -I http://www.somesite.com/ to check if a Set-Cookie header is being sent for anonymous users.

All that said, if the bulk of your traffic is authenticated, I would spend my time elsewhere as varnish will only help you cache static resources. You should probably be profiling with something like xhprof to see where you can start optimizing.

Thanks for the tip, I added

Posted by cookie235 on February 18, 2011 at 9:18am

Thanks for the tip, I added the lines to my varnish config. When I do a curl -I, I get this :

HTTP/1.1 200 OK
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny9
Cache-Control: public, max-age=86400
Last-Modified: Fri, 18 Feb 2011 09:11:33 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie
ETag: "1298020293"
Content-Type: text/html; charset=utf-8
Content-Length: 49330
Date: Fri, 18 Feb 2011 09:11:47 GMT
X-Varnish: 1197393210 1197393200
Age: 13
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
X-Cache-Hits: 10

So no cookie set, and it's a hit on varnish. Yet, when I do an "ab -c 1 -n 10 ...", I get only 1 #/s :

Requests per second:    1.43 [#/sec] (mean)
Time per request:       696.929 [ms] (mean)

On a copy of this server, with the same setup, I get :

Requests per second:    274.92 [#/sec] (mean)
Time per request:       3.637 [ms] (mean)

I don't get how I can have a hit on varnish, and still have something as low as 1#/s. The curl -I on the copy is similar :

HTTP/1.1 200 OK
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny9
Cache-Control: public, max-age=21600
Last-Modified: Fri, 18 Feb 2011 08:37:34 +0000
Expires: Sun, 11 Mar 1984 12:00:00 GMT
Vary: Cookie
ETag: "1298018254"
Content-Type: text/html; charset=utf-8
Content-Length: 45015
Date: Fri, 18 Feb 2011 09:16:27 GMT
X-Varnish: 2067207966 2067207653
Age: 2331
Via: 1.1 varnish
Connection: keep-alive

Your headers look good, but

Posted by msonnabaum on February 18, 2011 at 9:34am

Your headers look good, but you aren't doing enough requests to get a useful r/s stat. Try something like ab -c 100 -n 1000.

Also, if you can, run ab from the server that's running varnish to eliminate the network latency. If all is well, you should see something around 2000-5000 r/s.

I was indeed running the 'ab'

Posted by cookie235 on February 18, 2011 at 12:27pm

I was indeed running the 'ab' test from a far away server, and not on localhost. This is why I was getting poor results. When doing the test localy, I got 5000#/s. Awesome :D

Thanks a lot for the help.

I disagree. Latency will not

Posted by dalin on February 19, 2011 at 2:19am

I disagree. Latency will not effect r/s as long as you are doing a sufficient number of requests. Doing benchmarking from a remote server can however reveal network issues. While r/s that a web server can serve to itself is useful to see the effect the network is having, it is not a useful metric by itself.

Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

I use cacherouter. It doesn't

Posted by soyarma on February 17, 2011 at 9:57pm

I use cacherouter. It doesn't have a lot of bells and whistles, but I have been able to get the following stats with cacherouter + apc + normal drupal cache

200 concurrent connections
30 ms average page generation time

i use cacherouter in file

Posted by ddorian on February 18, 2011 at 12:13pm

i use cacherouter in file mode and boost in shared hosting and everything is fine, which storage are you using for cacherouter?