Posted by tomhung on May 12, 2010 at 7:45pm
Most of the conversation I hear about performance includes, Load Balancers, MySQL Database servers, Clusters, etc.
It just seems like Vanilla Ubuntu/Apache/MySQL/PHP isn't very optimized at all.
I would like to see what your opinions are for single server configurations.
Comments
Check out this video:
Check out this video: http://www.archive.org/details/2.4MillionPageViewsPerDay60MPerMonthOneSe...
Mike Milano
Thanks
Hey
Thanks for the link.
I gave that talk at DrupalCon San Francisco and it was very well received.
The title is a bit inaccurate though. The figures quoted (2.4M page views a day, 60M per month) were from the time the session was submitted. The week before the conference, the site hit new record numbers, and the presentation shows 2.8M page views per day and 70M page views per month.
If you review the talk, you will see that there is minimal use of patches, and none of the semi-drastic measures that we often see other sites use.
I always advocate simplicity is best, and only break that rule when really necessary, not just because I can.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Funny you should mention that ...
I'm just today working on benchmarking a vanilla Ubuntu Lucid virtual machine (2 virtual CPUs and 1G RAM) running the whole stack (preparing for a talk about it at Open Source Bridge next month).
Using the default Apache, PHP, APC, MySQL configs installed via apt-get, a stock Drupal 6.16 site with some randomly-generated content served approximately 30 req/sec at 50 concurrent users using ab:
$ /usr/sbin/ab -k -n 1000 -c 50 http://zigzag.osuosl.org:8080/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking zigzag.osuosl.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: Apache/2.2.14
Server Hostname: zigzag.osuosl.org
Server Port: 8080
Document Path: /index.php
Document Length: 22482 bytes
Concurrency Level: 50
Time taken for tests: 31.961 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 22839000 bytes
HTML transferred: 22482000 bytes
Requests per second: 31.29 [#/sec] (mean)
Time per request: 1598.071 [ms] (mean)
Time per request: 31.961 [ms] (mean, across all concurrent requests)
Transfer rate: 697.83 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.2 1 2
Processing: 463 1586 245.5 1567 2808
Waiting: 421 1428 231.2 1390 2643
Total: 465 1587 245.4 1568 2808
Percentage of the requests served within a certain time (ms)
50% 1568
66% 1614
75% 1662
80% 1695
90% 1800
95% 1963
98% 2314
99% 2399
100% 2808 (longest request)
During the test the server load spiked over 12. It was struggling, but it was at least marginally usable.
I added memcache, Varnish, and Pressflow 6.16.77 to the server - basically the Mercury stack sans Solr search - and benched it using the same test. 4759 req/sec:
$ /usr/sbin/ab -k -n 1000 -c 50 http://zigzag.osuosl.org/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking zigzag.osuosl.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: Apache/2.2.14
Server Hostname: zigzag.osuosl.org
Server Port: 80
Document Path: /index.php
Document Length: 22477 bytes
Concurrency Level: 50
Time taken for tests: 0.303 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1000
Total transferred: 22968865 bytes
HTML transferred: 22477000 bytes
Requests per second: 3301.15 [#/sec] (mean)
Time per request: 15.146 [ms] (mean)
Time per request: 0.303 [ms] (mean, across all concurrent requests)
Transfer rate: 74046.49 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 2
Processing: 1 3 10.7 2 107
Waiting: 1 3 10.5 2 104
Total: 1 3 10.9 2 109
Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 2
95% 3
98% 4
99% 105
100% 109 (longest request)
Zero change in the server load. The second test was so short it was practically done before it started. So I tried ramping it up tenfold - 10000 iterations with 500 users - and got similar results: 4993 req/sec, 100ms mean time/request. Note that the first hit primed the cache and every single subsequent hit was served from cache without every waking up an Apache thread. Yes, Pressflow and Varnish make a difference. :-)
With authenticated users, things get a bit trickier, but adding cache and lazy session initiation will still help by handling the static content (images, CSS, JS, etc.) and reducing the amount of work Apache has to do. Memcache and Authcache should help quite a bit as well.
I'm working up a set of more realistic tests (ab isn't a particularly good indicator of how many real browser sessions a server can support) using FunkLoad. I'll share more when I finish.
gchaix - have seen these
gchaix - have seen these similar improvements in various other posts and my first thought is always the same... aren't you simply testing the ability of varnish to serve its cache and not drupal? Anonymous traffic with static content is an easier solution - just put varnish in front of the stack. Purely authenticated traffic always seems to be the elephant in the room when dealing with a single server. Add user-level access controls and SSL to that and the elephant gets even bigger. I look forward to see your results with authenticated users to see how it compares with your first non-varnished tests.
Authenticated users
Sure, authenticated users are always more resource-intensive. Anonymous users are easy, and we can show the greatest gains in performance by optimizing the config to serve them from cache. That doesn't mean we're ignoring the authenticated users. It's a viable and realistic goal to serve as much content from cache as possible, even for authenticated users, saving Apache/PHP processor cycles for content that really has to be dynamically generated. A lot of the config that helps anonymous user traffic also benefits authenticated users. If we don't have to hit Apache (or worse, the database) for content we deliver to the user, that's a good thing, regardless of whether the user is logged in or not.
With a well-tuned Varnish/Drupal site, authenticated user traffic only hits Apache/PHP/MySQL for the actual PHP/dynamic HTML generation. All the images, CSS, scripts, etc. are served from Varnish, so we don't have to use CPU and memory-expensive Apache threads to handle them. On a typical page load (anonymous or authenticated), the browser will make 10 or more requests. The vast majority of those requests are for static content like CSS or images that can and should be cached. Offloading that static content onto Varnish significantly increases the number of sessions the server can handle. Throw in something like Authcache+Memcache and maybe even some per-session cookie hashes in Varnish, and authenticated user traffic hitting the database drops even more and concurrency goes up.
varnish is good, yes
I think my post was taken the wrong way... I never doubt the ability of varnish to serve static files. Static files could even be offloaded to a CDN. I'm just pointing out the comparison, as you stated also, appears so great when comparing the 2... as it should be.
My concern comes from my current use case...
All secure traffic and access control is down to the user level - meaning a user can see some global site content and only see nodes that they create. Therefore, user centric content is created fresh on every page. Current setup: Drupal + Apache + APC + memcache. Authcache (last time i looked) is role based cache and not user level.
@tomhung, the purpose of
@tomhung, the purpose of multiple servers is not only increased scaling but also redundancy. If a power supply dies and the CPU melts to the motherboard the site is still up due to there being multiple machines. Multiple machines can actually decrease performance due to the increased round-trip-time between PHP and MySQL.
@recrit there's nothing to say that you can't use Varnish for authenticated users too.
http://www.grammy.com/
does and I'm about to launch a site that does as well.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
varnish for authenticated users?
@dalin
Do you know of any other sites that use varnish for logged in users? To me it seems the logical way to do it with varnish + user specific customization is to use ESI and/or Ajax.
How did it go/how does it work on the new site you're launching? Can you share some tips?
We're going to be testing varnish for authenticated users for a big site (with currently over 50 000 000 comments and over several million user accounts).
I know that optimization is
I know that optimization is dependent on a per-situation basis. Yet, the more I look into single server optimization it seems that there are some general "good practices" for setting it up.
I wanted to know if there has been any mention in the community about publishing a Virtual Machine (http://www.vmware.com/appliances/) or Pre-packed ISO (http://www.turnkeylinux.org/) with lots of these generic optimizations set up? It seems like a great starting point for many Drupal sites.
Greg
Mercury
Checkout project Mercury:
http://getpantheon.com/mercury/what-is-mercury
http://groups.drupal.org/pantheon
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his