I've been reading articles all over the place for the last two days, about what should and shouldn't be done. But I don't think all of it applies to our current setup. I am new to the world of Drupal and Apache - I was a Windows/IIS admin - we are currently in a replatforming project for our site.
Here is our setup:
Live Servers (just application, MySQL is separate)
- MemTotal: 31.41638 gb
- MemFree: 21.47192 gb
- CPU: 4 6-core processors
cpu MHz : 2411.115
cache size : 512 KB
Running Pressflow Drupal 6.15.73 with the Jason Litka PHP 5.2.13
- /var/www/html/drupal - holds all the drupal application information
- /var/www/html/presentation - all the code that is consistently developed
- Presentation directory is stored on a SAN GFS2 mount
Currently our site is responding incredibly slow. I would like to tune Apache and PHP as much as I can. I am looking at installing/configuring memcached and APC.
Ran an Apache Benchmark - from what I can see is that it's not good:
$ ab -n 500 -c 50 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests
Server Software: Apache/2.2.3
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 60251 bytes
Concurrency Level: 50
Time taken for tests: 71.611596 seconds
Complete requests: 500
Failed requests: 46
(Connect: 0, Length: 46, Exceptions: 0)
Write errors: 0
Total transferred: 30397448 bytes
HTML transferred: 30125448 bytes
Requests per second: 6.98 [#/sec] (mean)
Time per request: 7161.160 [ms] (mean)
Time per request: 143.223 [ms] (mean, across all concurrent requests)
Transfer rate: 414.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.2 0 7
Processing: 1873 6813 5745.3 4181 23861
Waiting: 1530 6667 5739.3 4054 23677
Total: 1873 6813 5745.3 4181 23861
Percentage of the requests served within a certain time (ms)
50% 4181
66% 4627
75% 6081
80% 11708
90% 20570
95% 22181
98% 23072
99% 23276
100% 23861 (longest request)
I am basically looking for any information that any of you can provide. I'm all open ears right now. We are supposed to go live in about a month...
Comments
Tips
If you have more then 1 server use Memcache & APC http://drupal.org/project/memcache. If your only using 1 box try to only use APC as this will be faster most of the time; http://drupal.org/project/cacherouter OR http://drupal.org/project/cache
You should identify who will be coming to the site. If its all anonymous users then you can benefit greatly from a page cache. Drupal has one built in (use SQL for storage) & there are other options like Boost, Memcache, & Varnish. This will only help for anonymous users.
Upgrade pressflow to the latest version. https://launchpad.net/pressflow. 6.16 came with some important performance tweaks (locking) so I recommend upgrading.
We will have 3 Live Drupal
We will have 3 Live Drupal servers handling the entire site. Most of the site will be anonymous and authenticated - don't know which one is more, my guess is authenticated. I know we have Varnish installed and I think configured - third-party did that for us. I wish we could upgrade, but that is not an option since we are still developing everything and we have to use the "older" version until we are live.
Memcache is good
This is not necessarily true in all cases.
We run memcache on single server installations with excellent results. In fact, using memcache allows you to run PHP in more resource efficient configurations, such as FastCGI, and Apache as a threaded MPM server.
Don't hesitate to run memcache on single servers.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Mercury
Also take a look at http://getpantheon.com/get-mercury
Something seriously broken
You have 7 seconds response time there. This is unreal.
Are you sure you have caching set to "Normal" under admin/settings/performance?
Start with 10 or 20 users only and perhaps 2000 requests. See if that yields any better figures before increasing the number of users.
Does your application open a network connection to another server (e.g. some web 2.0 widgets, ...etc.)?
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Caching on
Caching on /admin/settings/performance is set to Disabled.
And I ran another benchmark:
$ ab -n 2000 -c 10 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests
Server Software: Apache/2.2.3
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 60248 bytes
Concurrency Level: 10
Time taken for tests: 988.454457 seconds
Complete requests: 2000
Failed requests: 1848
(Connect: 0, Length: 1848, Exceptions: 0)
Write errors: 0
Total transferred: 121585841 bytes
HTML transferred: 120497841 bytes
Requests per second: 2.02 [#/sec] (mean)
Time per request: 4942.272 [ms] (mean)
Time per request: 494.227 [ms] (mean, across all concurrent requests)
Transfer rate: 120.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1282 4936 4153.4 2790 22059
Waiting: 1197 4786 4150.1 2666 21880
Total: 1282 4936 4153.4 2790 22059
Percentage of the requests served within a certain time (ms)
50% 2790
66% 5082
75% 6472
80% 7588
90% 11528
95% 13897
98% 16655
99% 20445
100% 22059 (longest request)
And I believe our homepage (localhost) does make outside calls - but I would have to ask our developers for more information on that.
Still bad
Still bad.
Most of the requests have failed (1848 out of 2000).
Not surprising since you disabled the cache.
Set it to Normal, and repeat the tests.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
from where are you running ab?
Are you running ab from the server itself or perhaps from within the local network of the server?
Just checking.
From the server itself - I
From the server itself - I don't really have anywhere else I can run it. My laptop is Windows, and the only other linux boxes are within the same domain as the webservers.
I enabled the caching and
I enabled the caching and here are the new results:
$ ab -n 2000 -c 10 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests
Server Software: Apache/2.2.3
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 60249 bytes
Concurrency Level: 10
Time taken for tests: 862.641522 seconds
Complete requests: 2000
Failed requests: 215
(Connect: 0, Length: 215, Exceptions: 0)
Write errors: 0
Total transferred: 121585784 bytes
HTML transferred: 120497784 bytes
Requests per second: 2.32 [#/sec] (mean)
Time per request: 4313.208 [ms] (mean)
Time per request: 431.321 [ms] (mean, across all concurrent requests)
Transfer rate: 137.64 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 1205 4306 4832.3 2276 29602
Waiting: 1129 4158 4829.0 2124 29502
Total: 1205 4306 4832.3 2276 29602
Percentage of the requests served within a certain time (ms)
50% 2276
66% 2979
75% 4074
80% 5539
90% 9818
95% 15943
98% 21881
99% 26146
100% 29602 (longest request)
Better but not much better
The number of failed requests is less.
But the requests/second is still dismal.
You have something that is very wrong in your setup.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Authcache, Multi-Memcached, YSlow
Since you noted you have authenticated users, then I also recommend to look at using is the Authcache module (http://drupal.org/project/authcache). The Authcache module offers page caching for both anonymous users and logged-in authenticated users. This allows Drupal/PHP to only spend 1-2 milliseconds serving pages, greatly reducing server resources.
If you do look at using Memcached (as others have recommended), I recommend having your Memcached environment running upon another server rather than taking up the memory from the actual Drupal/PHP server locally. If possible, run a multi-memcached configuration as well (http://drupal.org/node/495478#comment-1819338) which will allow you to split up memory resources for each of the tables within Drupal.
Also, have you tested the website through the Firefox/Yahoo 'YSlow' add-on module (https://addons.mozilla.org/en-US/firefox/addon/5369/) to see what could be tuned?
Got Varnish running properly....
I am seeing a huge improvement now, finally got Varnish configured properly (it's not perfect right now - it won't allow login to /admin and issues with using the VIP - working on those). But this number is a huge improvement!!!
ab -n 2000 -c 10 http://perf-drupal1.dev.*****.pri/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking perf-drupal1.dev.goaaa.pri (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests
Server Software: Varnish
Server Hostname: perf-drupal1.dev.goaaa.pri
Server Port: 80
Document Path: /
Document Length: 0 bytes
Concurrency Level: 10
Time taken for tests: 0.436719 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Non-2xx responses: 2000
Total transferred: 444000 bytes
HTML transferred: 0 bytes
Requests per second: 4579.60 [#/sec] (mean)
Time per request: 2.184 [ms] (mean)
Time per request: 0.218 [ms] (mean, across all concurrent requests)
Transfer rate: 991.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.3 0 2
Waiting: 0 0 0.1 0 1
Total: 0 0 0.3 0 3
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 1
98% 1
99% 1
100% 3 (longest request)
Not a valid test
This is not a valid test. All the responses are non-2xx.
Repeat the test with a valid page.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
I tried hitting a page we
I tried hitting a page we have http://localhost/travel and I tried http://localhost/index.php both load locally from the server fine. But when I do the ab with that page I still get Non-2xx responses: 2000 - what does that actually mean? Not able to find much from google. Thanks.
check output from wget
do
wget http://localhost/traveland look at the html. is this what your expecting?Varnish only serves anon
Varnish only serves anon sessions. AB is only hitting the server from anon too.
I'm interested in seeing your results with logged in AB tests.
Greg
Help!!!!
[#########~]# ab -n 2000 -c 10 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests
Server Software: Apache/2.2.3
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 40179 bytes
Concurrency Level: 10
Time taken for tests: 51.493629 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Total transferred: 80717974 bytes
HTML transferred: 80358000 bytes
Requests per second: 38.84 [#/sec] (mean)
Time per request: 257.468 [ms] (mean)
Time per request: 25.747 [ms] (mean, across all concurrent requests)
Transfer rate: 1530.79 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.9 0 55
Processing: 30 255 117.6 238 1247
Waiting: 30 254 117.1 238 1247
Total: 30 256 117.6 239 1247
Percentage of the requests served within a certain time (ms)
50% 239
66% 278
75% 310
80% 333
90% 393
95% 461
98% 550
99% 660
100% 1247 (longest request)
my result
I each time I run this I get the same warning. Anyone have any ideas?
ab -n 2000 -c 10 http://www.serviidb.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.serviidb.com (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests
Server Software: Apache/2.2.14
Server Hostname: www.serviidb.com
Server Port: 80
Document Path: /
Document Length: 150658 bytes
Concurrency Level: 10
Time taken for tests: 1.274 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Total transferred: 303391630 bytes
HTML transferred: 302325913 bytes
Requests per second: 1570.30 [#/sec] (mean)
Time per request: 6.368 [ms] (mean)
Time per request: 0.637 [ms] (mean, across all concurrent requests)
Transfer rate: 232624.94 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.5 0 5
Processing: 2 6 2.2 5 24
Waiting: 0 2 1.8 2 19
Total: 2 6 2.2 6 25
WARNING: The median and mean for the initial connection time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 6
66% 7
75% 7
80% 8
90% 9
95% 10
98% 12
99% 14
100% 25 (longest request)
This looks like some kind of
This looks like some kind of rounding artifact to me. Median is zero, mean is 1, deviation is 0.5? The story is: This number has a few outliers, but usually it is really small, so small that it's hard to measure, and the reported statistics will depend critically on just when and how the individual measurements are being rounded off when those statistics are computed.
You probably just don't care. "So fast that it's hard to reliably measure or report" is a pretty good place to be.
longer test good numbers
I am getting 2.5-7 seconds on webpagetest.org.
ab -n 10000 -c 30 http://www.serviidb.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.serviidb.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: Apache/2.2.14
Server Hostname: www.serviidb.com
Server Port: 80
Document Path: /
Document Length: 150658 bytes
Concurrency Level: 30
Time taken for tests: 7.446 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1513930442 bytes
HTML transferred: 1508613008 bytes
Requests per second: 1343.08 [#/sec] (mean)
Time per request: 22.337 [ms] (mean)
Time per request: 0.745 [ms] (mean, across all concurrent requests)
Transfer rate: 198567.64 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 2.1 1 30
Processing: 2 20 9.5 17 88
Waiting: 0 6 5.9 4 50
Total: 4 22 9.8 19 90
Percentage of the requests served within a certain time (ms)
50% 19
66% 21
75% 23
80% 24
90% 33
95% 45
98% 55
99% 62
100% 90 (longest request)