Hi,
The short story: I just completed a project for a client. I put it online and 5 min later when the wave of people arrive the server is unreachable. Do I have the wrong server?
And now the details.
This is an event booking website using D6, UC2 and node_checkout module, beside some regular modules. In order to make a booking the user shall login, so in a way 90% of the traffic is from logged in users. There are only a couple of static pages, with event pages, booking form and the whole UC flow dynamic. I also have one view to list the event. There are 2 vocabularies, 100 nodes and it should grow to about 10 000 users and as many nodes in the future.
The first event promoted by my client on the site is very popular. When they published the info that the booking for it was opened with a link to the site a rush of connections arrived (probably around 500).
The service was unreachable with the server completely overloaded.
Server specs are:
2X Xeon 2.8 GHz
4 GB RAM
2 HDD in RAID 1
Zend optimizer
Plesk 9
PHP5.2
MySQL5
Drupal 6
When I contacted my hosting company, their diagnosis was a DDOS attack. Therefore I should put a dedicated firewall in front of it.
Sounds a bit strange to me that it happens just 5min after going live when the domain name is unknown and the service not an interesting target.
The next technician I got told me the RAM was deficient and had to be replaced.
The site is currently offline, my client upset but ready to give it another go if I can guaranty that it will work this time.
I tried to read all over the forums on some capacity requirements but cannot find much about it. I did apply all the performance recommendations I came accross (disable statistics, DB logging, any unrequired modules)
So I would appreciate if you could share your feeling on that one.
Do you think this server can hold the load or I need a more powerful box?
Thanks
Comments
more information needed
Please provide mysql config for review
also, do you use apache? - if yes, what is your ServerLimit and MaxClients settings?
Site down after start isn't look like DDOS, it just a simple "first wave" and you was out of RAM. It is better to set limits and be ready to process N request and drop other limit, than get site down(from my view point).
Views have caching - use it, even few mins cache life time can help. See what blocks you can cache. And sure use mencoder.
Install dev module and inspect most visited pages queries - you need to look for queries which don't use keys. I don't know logic of your site, but I pretty sure that you can cache some data and lower load.
10 000 - can be handled with described server with right configuration and balance.
Here is mysql config
Here is mysql config (my.cnf)
[mysqld]
set-variable=local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Apache settings read
ServerLimit 256
MaxClients 256
I was thinking to put a connection limit to avoid the site going down next time, but as I use Paypal for payment, a client would not be able to get back to the site after his payment.
In Drupal performance tab, I enabled caching and block caching as well as CSS/js.
I have been using Dev but didn't know what values to check besides loading time (1s in average).
My host said the RAM was not working. It was replaced and I doubled the amount to 4GB at the same time.
I just want to be sure that the server will not plunge again under that wave. Looking at Analytics on the link that sent the traffic, it was probably 800 incoming connections in that wave.
Look at the Mercury project
Look at the Mercury project here in the group.
make more caching with memcache or apc and tune your mysql.
Greatest website are under Drupal like MTV or the new site from the white house.
you are out of memory
@julient
this default values is too big:
ServerLimit 256
MaxClients 256
just image:
each apache process eat 20-40m. With loged in users without advanced caching it will around 30M. As result we have 30*256 = 7680M, it is maximum RAM consummation, but you have RAM for mysql, mail daemon and various system processes. As result your server start use swap and this is make it very slow.
May suggestion is to monitor how much RAM eat apache processes and set them to use NOT more 3G.
With around 30m per apache process:
ServerLimit 100
MaxClients 100
is accepted, but recomened values will be 70-80. To grantee your client that server will leave you can set
ServerLimit 75
MaxClients 75
Also what engine you use for mysql tables? for loged users when Drupal update often nodes, MyISAM engine will be problem, so InnoDB for node* tables as minimum recommendation.
Also you MUST monitor server(for example with munin) - this will give us more information for suggestions.
As I see you don't make advanced configuration for MySQL - you need to investigate what is MySQL load(dev module will help) and reserve enough memory via my.cnf settings.
Stron limits for memory usage will keep you server leave, may be you will loss some users (monitor apache logs) - with given information I can not make more suggestions for server configuration.
My apache.conf modifications
I've been tuning my apache as the default apache config for the worker module can be improved significantly.
Work out which worker module type I'm using:
sysadmin@WebSrv01:~$ apache2 -VServer version: Apache/2.2.11 (Ubuntu)
Server MPM: Prefork
These were the defaults for apache, installed from Ubuntu repos:
Timeout 300 <-- this holds the server in a wait of upto 300 seconds for a connection to timeoutKeepAlive On <-- keepalives allow for multiple requests per connection, holding open sessions on the web server
MaxKeepAliveRequests 100 <-- 100 KeepAlive requsts
KeepAliveTimeout 15 <-- which timeout after 15 seconds
<IfModule mpm_prefork_module>
StartServers 5 <-- start 5 servers
MinSpareServers 5 <-- keep 5 spare
MaxSpareServers 10 <-- to a max of 10 spare
MaxClients 150 <-- hand 150 clients per server
MaxRequestsPerChild 0 <-
</IfModule>
We are now running as follows:
Timeout 10 <-- connections timeout after 10 secondsKeepAlive Off <-- no keepalive, new connection per request
HostnameLookups off <-- don't wait doing DNS/RDNS lookups
<IfModule mpm_prefork_module>
StartServers 50 <-- start 50 servers
MinSpareServers 15 <-- keep 15 spare
MaxSpareServers 30 <-- to a max of 30 spare
MaxClients 225 <-- handle 225 clients per server
MaxRequestsPerChild 4000 <-- stop at 4000 requests per child (stops memory leaks)
</IfModule>
For reference: http://www.ibm.com/developerworks/linux/library/l-tune-lamp-2.html?ca=dg...
You won't have any idea of
You won't have any idea of the capacity of your site if you don't do load testing before launch. Check out Apache Bench for a simple load testing tool. But if the majority of your traffic is logged in you may need to test with something like JMeter.
Also disabling DB logging is a really bad idea unless you've setup sending Drupal logs to syslog and have a way to monitor that easily and effectively.
While I agree that your default Apache settings look like they could be a big contributor to the problem, you need to find out what is causing the problem before you start randomly changing settings. The odds are low that implementing a random "performance improvement" will actually help in your specific situation. Is Apache hogging all the memory? Are there some poorly written queries that are holding up MySQL? Is MySQL creating many temp tables on disk? etc. etc.
Be sure to benchmark before and after you make a change so that you know if that change helped or hindered. A slightly mis-configured "performance improvement" can do more harm than good.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
If you can get a monitoring
If you can get a monitoring tool like munin installed, you can watch the apache processes vs ram usage while hitting the site with ab or similar, and get a realistic picture of your true capacity and limitations.
Several places
Hi,
I think you should do several things...
1. Run a mysql tunning check script like this http://www.day32.com/MySQL/ . This will guide you how to set up your mysql configuration.
2. Run apache benchmarks against your server, and try different configuration by understanding very well the configuration, mostly the prefork mpm parameters, beware not having too many spare servers and have a max clients that match your memory, also consider lowering the time for the keep alive or disalowing it.
3. Use a CDN for the static files, keeping the static files out of your server will help you to focus your server only on the dinamic output and not overload it with static requests.
4. Use a good configuration of the drupal performance settings, focusing on server not client.
5. Use several caching strategies, for example you can use Boost module for anonymous pages and cache router module for inside pages, using APC or file for the engine for example.
6. Considering installing the zend server comunity edition, which compile and caches the php files and have several performance enhacements and is free.
I hope this helps you as a guide.
Nestor
Nestor Mata
http://nestor.profesional.co.cr/es
... and slow queries
And I forgot to point out too...
7. Log and check for low queries, and consider creating indexes, changing some table engines to InnoDB, and maybe some small and high used to memory (beware, this is for cache like tables, data gets lost when server goes down) and also consider modifying queries and views to be better.
8. Also find what would be the most viewed places or pages and consider altering them to be programmed more efficient.
Nestor Mata
http://nestor.profesional.co.cr/es
Thank you all for your
Thank you all for your inputs.
That gives me a lot of points to work on.
Curious about performance
I don't currently have any issues, but I am curious about the performance characteristics of Drupal...
The question was, can the server handle 500 simultaneous active user connections. I think the question is ambiguous because it entirely depends on how active each user is (ie in terms of page views per minute)
A more specific question would be can the above server handle for example 10 page views per second, which it would seem that it should? (500 users requesting a page approximately once per minute)
I'm curious about Alexandr Kosarev's response that the server configuration could handle 10,000 (presumably simultaneous users) - this corresponds to approximately 166 page views per second (using my rough estimates). Is this realistic ? (I really don't know, but would love to get an idea)
Also, I'm curious how julient got on with solving his clients performance issues ?
(ps: thanks for the info already provided on this discussion - very instructive :) )
cacherouter with APC as
cacherouter with APC as oppcode and object cache, optimized front end, also aggresive block and views caching - this will make miracles... (next step - also authcache)
drupal+me: jeweler portfolio