Posted by alexus on October 11, 2012 at 4:41pm
in our environment even though we have varnish (frontend) w/ apache (backend) somehow logged-in users are able to bring apache to it's knees, it probably have something to do with following:
// do not serve cached pages for logged in users
if (req.http.Cookie ~ "SESS[a-f|0-9]+") {
return (pass);
}
so I was thinking if it's possible to do following:
- have pressflow to give out different cookie to a user based on their role.
- have varnish recognize that cookie and serve cached content to regular users and not server cached content to user with specific role (admin,publisher,etc).
Is it something possible to do? where would I start?
Comments
That rule is there for a
That rule is there for a reason.
Without knowing the architecture of your site, I can't say for sure if there are any reasonable strategies for caching authenticated requests, but I can say that it's unlikely and/or not likely going to be the best use of effort and time.
How well are you using cache? What cache backend are you on (is it still just the database)? Do you have custom code that needs to be refactored? Could you add more capacity? (Hardware is often cheaper than developer time.)
You haven't mentioned if it's the httpd, application or database that's bottlenecking, do you know?
Varnish isn't magic fast sauce (unless you have very large anonymous:authenticated).
as i mentioned in my initial
as i mentioned in my initial post on backend we have apache's httpd that's hosting pressflow (drupal6), we also have another caching mechanism (apc) to help php.
bottleneck is happening on apache httpd/php level as i'm seeing httpd using a lot of cpu, load average goes through the roof, and while httpd is going crazy, varnish on the other hand is nearly idle, so in my opinion we don't utilize varnish as we should as it pass too much to apache and apache is not capable to handle that many users at the same time.
adding capacity (hardware) may not be wise solution here as that's more like band-aid not a solution, we'd like to figure out and adjust varnish configuration to include more to cache, so there is less load to apache and more to varnish.
this is another part of varnish's default.vcl
so, it seems like it should cache following extensions, YET following from apache's:
grep -c .png access_log
155649
grep -c .jpg access_log
76065
let me go over an actual example: I just looked up one of jpg file "mary-wade.jpg" inside of access_log:
grep -c mary-wade.jpg access_log
142
so, apache's httpd served that file 142 times, now let's take a look at varnish
$ curl -I XXX/files/photo/mary-wade.jpg
HTTP/1.1 200 OK
Server: Apache
Last-Modified: Wed, 04 Aug 2010 15:39:35 GMT
Cache-Control: max-age=63113852, public
Expires: Sun, 12 Oct 2014 07:54:49 GMT
Content-Type: image/jpeg
Content-Length: 30129
Date: Thu, 11 Oct 2012 20:17:18 GMT
X-Varnish: 1716173621 1716173602
Age: 2
Via: 1.1 varnish
Connection: keep-alive
X-Varnish-Cache: HIT
X-Cache-Hits: 1
$
according to this output this file was cached, yet varnish served it only 1 time.
alexus.org && alexus.biz
your vcl?
I checked my varnish vcl and it's a bit more complicated about images - it actually makes sure that it's not an imagecache or system/files url before removing the cookie. So in fact, your setup looks like it should be caching more aggressively (too aggressively, possibly).
But that's just a possible symptom that your varnish vcl might not be working right. For examle - are you able to confirm that varnish is caching drupal pages when accessed anonymously?
The other thing to consider if you're using the varnish module, is how it might be clearing the varnish cache. I think it normally will clear using Drupal cache clearing rules, which will be frequent if your content is getting updated frequently. If you've using the cron cache clearing with Varnish and have a cron running every 5 minutes, then you won't get much either.
In any case you'll want to look at those settings on the performance page and know that they're confusing and not the same for varnish vs. the normal db page cache.
When I look at my stats, images are getting served only rarely, they're the source of most of varnish's benefit.
Using the right metrics
Lots of things can block the Apache/PHP threads, causing them to pile up and drive up CPU utilization: slow SQL queries, excessive lstat requests to the filesystem due to open_basedir/realpath cache issues, memcache timeouts, etc. Just because httpd seems to be eating CPU doesn't mean the problem is actually Apache or PHP.
Without knowing a lot more about your usual traffic - especially anonymous vs authenticated users - and how your site is configured - like whether you're using memcache - it's going to be very hard to tell what needs to be done to improve the site's performance.
Note that Varnish even under extremely heavy load does not commonly show much CPU utilization. I'd recommend looking at varnishstat and varnishhist instead of CPU utilization, as that will give you a much better picture of how much of the traffic Varnish is (or is not) caching. Running "varnishlog -c -o ReqStart your.ip.address.here" and hitting the site a few times with a browser tells you much more about what's being passed back to the backend and why since it shows the headers from both the request and the Apache response headers before Varnish has done anything to them.
Also, in my experience the X-Cache-Hits header is of limited usefulness and tends to under-report the actual number of hits. I really only pay attention to whether it's >0. I haven't dug into why it tends to under-report the number - whether it's being too pedantic about the object in question, or whether there's some sort of time window limiting its count - it really doesn't matter that much since there are better ways to see what is and is not being cached.
Yup. Exactly. Varnish can marginally help authenticated user traffic by offloading static assets like images and stylesheets, but generally it doesn't really help very much since most of those all have long expiries (if configured properly) and will be cached by the users' browsers locally. In addition, you quite often want the page to be generated instead of pulled from cache. When looking at improving performance for authenticated users, you're better off looking at how the internal caching is configured. Pushing things like menu, variables, theme, session, etc. into memcache or redis instead of the database will help far more than Varnish for authenticated users.
Without knowing a lot more
we have fair amount of anonymous users and those anonymous users are being handled by varnish just fine (at least that's what it seems like) yet authenticated users on another hand is a whole different issue.. since varnish doesn't cache them they all going against life server, which is often dramatically slows down from amount of users hitting box, do we use memcache? not really, actually right answer is not yet! we're looking towards implementing apc and memcache... BUT improving pressflow (apache) server is one thing, yet not putting that much load and utilize varnish is a whole different story) and that's what I'm trying to do. varnish and apache resides on two separate VMs (and I'm not looking at CPU utilization on varnish box as it's always very low, yet apache's load is always very high)
here is an example:
part from varnish's default.vcl:
// Cache things with these extensionsif (req.url ~ ".(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
return (lookup);
}
this is from apache server:
2510 /sites/all/themes/custom/uft/images/uft_content_corner_sprite.pngfirst number represent hits that apache served this image and following is from my workstation:
$ curl -I www.uft.org/sites/all/themes/custom/uft/images/uft_content_corner_sprite...
HTTP/1.1 200 OK
Server: Apache
Last-Modified: Tue, 11 Sep 2012 18:37:11 GMT
Cache-Control: max-age=63113852, public
Expires: Sat, 18 Oct 2014 03:33:41 GMT
Content-Type: image/png
Content-Length: 1419
Date: Wed, 17 Oct 2012 15:56:09 GMT
X-Varnish: 380113442
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Varnish-Cache: MISS
$
second time I get "HIT" (as it was cached). Why didn't it cached 2510 times before that? my guess is due to user was authenticated when image was requested to be pulled, so my question remains: How can I configure varnish to cache more? My goal is to be able to offload as much as possible from apache to varnish especially static content such as images.
alexus.org && alexus.biz
A couple of things: 1) It
A couple of things:
1) It appears you're not using APC. No wonder your Apache CPU utilization is so high. Go enable that right now and then come back to finish reading this post.
2) Apache serving static assets like images takes very little CPU. It's not what's driving your load up. It's much more likely that PHP is the culprit, not static assets. You can spend a lot of time and effort optimizing and tweaking your VCL for some very minor increases in static asset delivery while the majority of your page rendering time is still being eaten up in PHP.
3) As I said before, if you want to see why something appears to be un-cached don't look at the headers generated by Varnish on the client side, look at the conversation between Apache and Varnish. That will tell you if and why something is or is not being cached.
When you do your curl request, have a "varnishlog -c -o ReqStart ip.address.of.curl.machine" and a "varnishlog -b -o TxHeader ip.address.of.curl.machine" running in a couple of shell sessions on the Varnish box. That will show you the entire transaction (and filter for only traffic coming from your client IP address), the "-b" shows the request as passed from Varnish to the Apache backend and the "-c" shows the request between Varnish and the client (curl). That should show you exactly what's going on and should reveal why the request isn't being cached.
But first, enable APC on Apache. Now. You'll see an immediate and significant improvement in your server load. Then look at Memcache or Redis for the internal Drupal cache. Trying to get Varnish to cache content for authenticated users isn't going to help nearly as much (and has some significant downsides).
APC is in use...
Thanks gchiax, and everyone else for all the great information. It appears that there was some incorrect information posted as we do have APC loaded and we have a 98.8% hit rate. Here are some additional information on the apc.php page. We have the page only accessible from the internal network but it does appear to be working. As for memcache, that is not in Prod but we have it on our staging environment and will deploy soon.
Thanks so much.
General Cache Information
APC Version 3.1.6
PHP Version 5.2.17
APC Host xxxxxxxxxxxx
Server Software Apache
Shared Memory 1 Segment(s) with 128.0 MBytes
(mmap memory, pthread mutex locking)
Start Time 2012/10/14 04:03:59
Uptime 3 days, 18 hours and 43 minutes
File Upload Support 1
File Cache Information
Cached Files 557 ( 40.3 MBytes)
Hits 9695262
Misses 115946
Request Rate (hits, misses) 30.04 cache requests/second
Hit Rate 29.68 cache requests/second
Miss Rate 0.35 cache requests/second
Insert Rate 0.35 cache requests/second
Cache full count 0
User Cache Information
Cached Variables 87 ( 94.1 KBytes)
Hits 388
Misses 0
Request Rate (hits, misses) 0.00 cache requests/second
Hit Rate 0.00 cache requests/second
Miss Rate 0.00 cache requests/second
Insert Rate 0.00 cache requests/second
Cache full count 0
Going
in a completely different direction. Why don't you replace Apache with Nginx + php-fpm and try microcaching of authenticated users. Each users get's his/her own cache. It could work. If it does it saves a all lot of trouble. As a bonus you get a much better performing HTTP server - PHP handling.
Just an idea.
we're not looking to switch
we're not looking to switch away from apache to nginx just yet, but who knows maybe one day...
alexus.org && alexus.biz
Also check to see if
you're serving the static assets with a cookie. That would void your Varnish setup, since you're filtering for the presence of a cookie.
right, that's what I'm
right, that's what I'm talking about) I'm trying to understand if I can tweak varnish enough to even though user has cookie still utilize some of varnish instead of using 100% of apache.
alexus.org && alexus.biz
yes
Yes, you can.
But here's the thing: you haven't divulged your entire vcl file, so there's no way of answering you with any authority about your specifics, i.e. why yours doesn't already do what you want it to.
From the pieces you've provided, I think you are using Varnish's default vcl. This will not help you much, you need to use a Drupal-specific one, because the default one is really conservative and can't make any assumptions. Lullabot, four kitchens, and acquia all have good starting vcls for Drupal, which should cache images even for authenticated users, where appropriate (when using the imagecache module, or private file systems - there will be cases where you may not want to cache them for anonymous users).
If you think you already have a drupal-specific VCL, you'll need to provide it here to get a useful answer.
there you go
you're a absolutely correct, i apologize for not sharing it earlier, it's not entire config but rather following parts ( vcl_recv, vcl_deliver, vcl_hash ) so here it is: http://pastebin.com/8ueerjZu
alexus.org && alexus.biz
Varnish caching of
Varnish caching of authenticated user content is of limited usefulness. You don't usually want to Varnish to cache the actual PHP-generated page content for authenticated users, since they're often generating content (nodes, comments, etc. that they expect to show up immediately after submission and not after the cache expires) or accessing private content that should not appear in the public cache. You can (and almost always do) cache the images, CSS, etc. but the effort required for Apache to serve those static assets is trivial compared to the amount of work Apache needs to do to bootstrap and generate page content. A well-tuned Apache daemon can serve up static assets almost as quickly and with as little CPU as Varnish or Nginx - albeit with a larger memory footprint, but that's not the concern here.
If you really want to improve performance for authenticated users, your time is probably best spent optimizing (or even enabling?) the internal Drupal cache, PHP performance (APC), and database performance (slow queries, query cache, etc.) instead of a front-end cache. Varnish is best at helping with anonymous user traffic. The internal cache is what helps authenticated users, much more so than the front-end cache. Things like APC, Authcache, Redis, Memcache, Cacherouter will help your authenticated users much more than Varnish will.
I think you are wrong
I don't know what kind of websites, you usually run, but you are really talking about your specific experience as if it was universal truth.
I disagree
I disagree a lot. Just because someone is a registered user and therefore has access to certain articles and is able to write comments, doesn't mean that the front page or an article page looks in any way different from one authenticated user to another. It may look (very) different to an anonymous user, but as long as there are only 2-3 "editions" of a page, I see NO reason to re-generate this page again and again and again and again, just because the user happens to be authenticated.
Citation needed... this is news to me.
Of course you use a opcode cache like APC, that's been the standard setup for ANY PHP system for the past 10 years.
And of course you spend time optimising database performance - because no matter the reverse proxy cache, you still need to generate a page sometimes
But instead of frontend cache?? I cannot see the reasoning for that.
No wonder - any time anyone asks about anything not related to anonymous users, you tell them "don't bother".
Citation needed... Besides, the question is not whether to use EITHER Varnish OR Memcache, APC etc. but how to use BOTH Varnish AND Memcache, APC, etc.
SO in order to actually try to answer the question:
I am (evidently) in the same situation and my thoughts go in the direction of the vcl_hash function in Varnish.
My idea is to create a module that sets a "grouping" cookie for a user on every request made to the backend webserver. It takes user group memberships (possibly only a configurable selection of groups) and drops it in a cookie along with a hashed value of the group memberships combined with a secret key and possibly the date or time of day or something else that changes periodically, but not on every request.
Then in vcl_hash, Varnish can take the group memberships etc. and hash it with the same secret key as Drupal and if it passes validation, add it to the hash, otherwise send the user through to the backend where re-validation and re-cookie-setting will happen.
I still have some quirks to iron out - especially about how to tell Varnish that e.g. a user profile page is NOT to be cached, when the user views or edits his own profile - but the principle should work, I feel.
ESI
Check out ESI, it's close to what you're describing. http://drupal.org/project/esi
You're welcome to disagree,
You're welcome to disagree, Mithrandir. My observations and suggestions are based on quite a few years' experience, but I don't claim them to be some sort of "universal truth" to use your term. I'm simply sharing what's worked for me.
Regarding Varnish cache for authenticated users - note I said "limited" usefulness. Varnish very definitely does provide benefits for almost all user traffic. I wasn't trying to claim it didn't. I use Varnish for all my sites and wasn't suggesting that it not be used at all for authenticated users. What I was suggesting was twofold: 1) that greater gains can be found by looking elsewhere in the stack for performance tweaks and 2) very often in my experience authenticated users need to have much of the content they view (re)generated at the time of request instead of stale cached content. The last thing I want is to have a content author submit an edit on a node and then receive back a pre-edit cached version of the node and have them wonder why the system didn't accept their edit. Yes, caching of content can be useful, but it's easy to have unwanted side effects and often greater performance gains can be found and implemented more easily elsewhere. I never said, "don't bother" with regard to Varnish for authenticated users. Not quite sure how you got that from what I posted. Thanks for pointing it out, though. I'll try to be clearer in the future about that.
As for Apache serving up static assets - no, Apache not going to be as fast as Varnish or Nginx. Apache can easily serve up static files in the 10ms range with almost no CPU. While Varnish and Nginx can do the same asset in 2-3ms, when dealing with 500ms pageload times saving less than 10ms on a CSS file is not on the top of my list of things to optimize.
Regarding APC and other backend tuning - I think I was a bit unclear in my answer to the original poster. I'm not talking about using any of that instead of a frontend cache. I was talking about using that in addition to a frontend cache. I've lost count how many times I've run into people building sites who are trying to tweak their Varnish config when they're not using any backend optimization like APC or memcache. No amount of VCL tweaking is going to save a site from a 10sec Drupal bootstrap because it has 200 modules enabled and no opcode cache.
In a nutshell - yes, Varnish can help improve site performance for authenticated users. However, when troubleshooting a poorly-performing site for authenticated users often the greatest relative improvement can be made by optimizing things other than Varnish. While some gains can be made by optimizing Varnish for authenticated users, the ratio of effort to improvement is relatively poor. Definitely leave Varnish in place, but in cases like this I feel it's better to look at improving the backend's performance first where the potential savings are in the hundreds of milliseconds before spending a significant amount of time tweaking Varnish. Once the backend has been been optimized as much as possible, then start looking at the frontend cache for those tens of milliseconds improvements.
I apologize if I came across
I apologize if I came across as somewhat harsh. I just felt this annoying feeling when you shared what's worked for you. Maybe it was the wording of "You don't usually want to..." and similar that just made me tick a little.
I didn't get the feeling that the OP had a poorly performing site, just that it didn't cache enough.
You may have lost count of how many times you've met people tweaking the wrong place - and I agree that if you are NOT using APC and memcache and if you have no idea why your uncached page takes 50 seconds to load, then no Varnish optimization will help you (and you should probably think about finding another line of work), however if you DO use APC and memcache and generally have ok performance, but lack scalability, you should look at getting Varnish to cache more.
The hundred millisecond gains in the backend might be a larger number than the 5-10 milliseconds on the frontend, but if I can transform a considerable portion of the uncached pageviews (some hundred milliseconds generation time) into cached pageviews (milliseconds generation time) I will have improved the site experience for many users. I see no reason not to pursue that goal once the basics like APC and memcache have been covered.
I still hold that the idea of hashing a key when the user hits the webserver and validating that key in Varnish in order to use different cache hashes depending on the key, is a proper way to go and I feel encouraged by the ideas expressed by the Varnish backers https://www.varnish-cache.org/trac/wiki/VCLExampleCachingLoggedInUsers
I "just" need to figure out how to tell Varnish to NOT cache user-specific content (content specific to the user), but only user-dependant (content specific to the key/group).
Why do you think on caching
Why do you think on caching whole pages instead just parts of it? ESI for Varnish is might be your friend.
Aries
Wouldn't that just move the
Wouldn't that just move the problem to the ESI included parts? I agree that it would be a SMALLER problem, but we are discussing the situation where Varnish shall serve some content to anonymous users, other content to some users and (at least I am talking about) yet other content to other users.
Different caching TTL's for different content on the page does not change the fact that I for example want to show an article teaser and a promotion to create an account and buy a membership to anonymous users, a promotion to buy a membership to non-member authenticated users and the complete article to members.
Granted, ESI would mean that only the actual article body would need to be re-generated by the backend in every request, but that is still a Drupal bootstrap, calls to the database etc. - all to provide one of only three different versions of the content that rarely changes.
Edit: I now see that http://drupal.org/project/esi does indeed have this kind of feature on the drawing board. I'll take my ideas to them, I think.
Heads up
Mithrandir
Did you even see my above comment about ESI?
Yep. looks like you did now :)
Care to elaborate
Questions:
Thanks
Do you have setup with
Not immediately at hand. Basically when you're talking about serving a a CSS file or image, Apache can generally serve it up in 10ms or so. Sure, Varnish can do it in 3ms, but when we're looking at PHP generating pages in the hundreds of milliseconds, why put a ton of effort into saving less than 10ms when there are greater gains to be found elsewhere?
Not quite sure what you're asking here. When I say "static assets" I'm referring to CSS, javascript, theme images, etc.
As I said above - static assets can be easily served up by Apache in the 10ms range with almost no CPU utilization. Varnish and Nginx are definitely faster, usually in the 2-3ms range, but not so much so on the scale of a Drupal full bootstrap.
A well-tuned Apache daemon
I have been advocating the use of Apache in MPM Worker, so the threaded server would be very lighweight and PHP will be in FastCGI mode.
It works well for many sites, those with mainly anonymous traffic benefit the most, without varnish.
But I would not go as far as to say that Apache in threaded mode is the same speed as Varnish.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Sorry - I was unclear. I
Sorry - I was unclear. I never meant to imply Apache was as fast as Varnish. 10ms for Apache vs 2-3ms for Varnish is in the same neighborhood, hence my used of the word "almost".
New Relic or XHProf
One more thought - sign up for a free New Relic (http://newrelic.com) account or install xhprof (http://drupal.org/project/XHProf) and use one of them to profile your authenticated user traffic. New Relic has especially good tools to dig into the stack and show you exactly what is causing things to be so slow. Remember that while it looks like PHP is eating CPU, it's entirely possible that the bottleneck is really PHP tying up threads waiting on a slow database query or filesystem request. Tools like New Relic or XHProf will very likely quickly reveal the bottleneck.
The reason that you are not
The reason that you are not getting static assets cached for authenticated users is due to your vcl_hash() function. You are hashing by cookie, which means that each unique combination of cookie is getting its own cache. And since every authenticated user has a different session cookie, they get their own cache.
Overall this default.vcl is the type that was used early on in the days of Drupal + Varnish and there have been far better versions developed since.
Starting instead with this one from Lullabot will get you a whole lot further:
http://www.lullabot.com/articles/varnish-multiple-web-servers-drupal
(though that is written for Varnish 2.1 and it looks like you are using 2.0).
If you have a user role on your site for 'basic' users that don't really create content, and only log in to get access to private content, and contribute a lot to your scalability problems, then it is possible to configure Varnish to cache pages for these users. But you need to use ESI to swap out the parts of the page that are either unique for each user (ex. link to "My Account" page), or that need to be constantly up to date. But you need to do a lot of legwork and write a lot of custom code in default.vcl to ensure that only the proper pages are cached and are served to the correct people.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Dave is right. I really need
Dave is right. I really need to update that old blog post that's linked from the Varnish module page. The VCL posted there is definitely outdated and the Lullabot one is vastly better.
in regards to:The reason
Dave Hansen-Lange, in regards to:
The reason that you are not getting static assets cached for authenticated users is due to your vcl_hash() function. You are hashing by cookie, which means that each unique combination of cookie is getting its own cache. And since every authenticated user has a different session cookie, they get their own cache.it's been a while and varnish has been upgraded to from 2.0 to 2.1 and then to 3.x
# rpm -q varnishvarnish-3.0.4-1.el5.centos.x86_64
#
vcl_hash is actually commented out all together...
alexus.org && alexus.biz