Nginx Benchmarking
mikeytown2 and I hijacked my nginx built-in caching thread for some benchmarking discussions. Mostly Boost vs Varnish vs. Nginx caching. It needs its own thread.
My quick and dirty ab testing on my home "dedicated" server is at http://groups.drupal.org/node/26485#comment-101296 with a couple more using keepalives is at http://groups.drupal.org/node/26485#comment-101518.
I've also run a few tests with the same setup using "-H 'Accept-Encoding: gzip'" to see if serving gzipped files makes a difference. It didn't, but it did beg the question of whether the size of the data being sent might make a difference. Speeds changed with the size of the page being sent, and one proxy might be better or worse at large pages vs. small pages. So tests at 10 bytes and tests at 50,000 bytes might be worthwhile.
Another consideration is CPU load. I saw that running 50,000 ab requests from the same system with 200 concurrent connections on that last set of tests would often show ab consuming 50% of the CPU and varnish or nginx using the other 50%. On the Mercury thread they use another EC2 instance for the ab machine, which makes some sense. Here at home, a second machine is bottlenecked by my 100Mb network interfaces.
I haven't used EC2 but I'm familiar with using Rackspace cloud servers. I can provision two 512Mb RAM VPSes, one to run the ab instance and one to run the proxies. They only have the one Texas datacenter, and they give you internal, NON-METERED IP addresses for inter-server communication. I don't need a huge bill for a jillion dollars in bandwidth because I used the internet ip address for my 1,000,000 requests! Also I think I noticed a cap on the internet facing network card of about 2500 r/s which makes sense since you're sharing a machine with 31 other people. I didn't notice such a cap on the internal IP, but if there is one, and we hit it, then all the testing is for naught. If I put aside 2 hours for quick setup and some tests, that should cost me about 12 cents. Gotta love the cloud.
I can use ab and try siege also. Can anyone suggest any other parameters for the testing?
Now that I think about it,
Now that I think about it, maybe I should spring for the whole 96 cents and get 4Gb instances. With fewer of those VPSes per machine, there's less likelihood of being CPU or network bound.
What size represents the most useful test data?
Test Size
Ideally you test small (Blank garland page), medium (5 paragraphs) and large (big wikipedia article) web pages; find out if some setups work better then others. All of them gzipped; the vast majority of web browsers support gzip.
If you wouldn't mind spending the extra $ for the 4GB instance that would take away more unaccounted variables. Looking at CPU & Memory usage while the test is running would be nice. If you can get stats on the network card usage too that would be useful, because there is a good chance we might peg it.
Hmph. Might have to polish
Hmph. Might have to polish my munin skills before the test. I'm also looking into using EC2 and the Mercury image and just loading nginx on it. Good times.
Network interface is the bottleneck
Those are great numbers. In either case, the network interface is the bottleneck. We can probably tune varnish better in this respect, but I'm definitely willing to believe you could use other similar file caching systems and get comparable/slightly-better results.
I'm curious what kind of load you see on the nginex server under these tests?
Additionally, an advantage of Varnish is that it's a service that can be decoupled from the webserver, which provides some other upsides. I think it provides some other upsides: it can do load-balancing, service multiple backend data sources, etc. Better foundation for moving to clustered infrastructure.
Anyway, I'll have to look more at Nginex, as the key question for Mercury now is how to improve logged-in performance.
http://www.chapterthree.com | http://www.outlandishjosh.com
Nginx can act as an
Nginx can act as an independent proxy, do simple round-robin load balancing, proxy multiple upstream servers of different varieties, but it can't compete with the VCL. Varnish definitely has a more sophisticated and versatile control language of its own. The problem is that Drupal doesn't speak that language.
I know you've opened a project at http://drupal.org/project/varnish "which allows Drupal to dynamically invalidate cache entries." A module that speaks the Varnish language and says "sure go ahead and cache these pages for a day, but someone might come change them so I'll let you know whether to purge a particular page or not" would be huge.
The Boost module knows exactly how to do that today and it keeps all the control within Drupal. If a page changes, Boost can replace the file for that page and the new page instantly starts being served. It could also conceivably assign different expire times to different pages.
The question for Boost is whether it can provide comparable speeds to Varnish, while keeping all the cache-control at the Drupal level. It can't under Apache/mod_php. Not even close. But can it under Nginx?
This is not to diminish the Pantheon project at all. I love it. Prebuilt, tweaked server images for Amazon and Rackspace Cloud Servers are awesome and will be useful to me and lots of folks. But Nginx+Boost might be comparable to Varnish+Apache, especially for low memory 512MB instances. The beauty of your project is choice of server images.
I sincerely appreciate the visit and the open mind and welcome your input, josh.
As for logged in users.
As for logged in users. Picture a forum that is serving completely static cached files. As long as no one posts, that's fine. Now imagine someone posts a comment. Is Boost sophisticated enough to now dynamically expire the node page and also the Forum topics list and the Forum Container List last post, and also every page that contains a "Latest Forum Posts" views block? I don't think so yet. :)
But keeping all the control on the Drupal side, that is possible. There's no way Varnish can know which pages to invalidate. And if there was a module on the Drupal side that told Varnish which pages to expire there's no savings over that same module telling Boost which page to replace.
It's kinda neat.
I did some testing on an EC2
I did some testing on an EC2 Mercury instance and Varnish came out ahead of Nginx and Boost by about 17%. I'm still going over things I did wrong in the test or at least things I'd do differently the next time.