Posted by ktrev on May 12, 2014 at 5:10am
We have separate machines for database server and web server (each with 8GB RAM and 4 cores)
The user CPU% of web server is around 60-70% (when you run a jmeter test that login 100 users)
In database server its less than 5%.
We have Apache with php5-fpm.By top command we can see that each php-fpm process is taking more than 20% of CPU
Memcache and Apc are on web server itself.Why is the CPU usage this high ?
Comments
Which PM you are using -
Which PM you are using - Dynamic / static or OnDemand ? Also what are the thread settings in www.conf file ?
pm = static pm.max_children =
pm = static
pm.max_children = 15
pm.start_servers = 4
pm.min_spare_servers = 2
pm.max_spare_servers = 6
Also curious to hear if you
Also curious to hear if you find resolution. Ran some tests with Blitz.io to simulate 250 concurrent users are kept maxing out processor at 100 users. Similar spec'ed machine though I was running APC / Filecache for my performance management
Ex Uno Plures
http://elmsln.org/
http://btopro.com/
http://drupal.psu.edu/
If running Nginx, use
If running Nginx, use microcaching. I also typically see a 10-12% performance boost from ZO+ over APC.
I need to do some benchmarking because I seem to have lost the results from the last major load test that I did, but I recall being able to get over 200 authenticated requests per second with a basically out-of-the-box configuration on a 1vCPU/1GB/SSD cloud server.
Running similar numbers
I am seeing very similar CPU usage with php-fpm routinely taking 20 -30% per process. Very similar setup and settings although I am running mysql on the same server and have lower concurrent user count.
Any thoughts on this thread welcome. What is "normal" CPU usage for php-fpm?
Running Dynamic
Just noticed the "static" setting, I am running in Dynamic with generally lower server numbers. I am experimenting with " pm.max_requests = 10000" in www.conf
You definitely want to run
You definitely want to run static if running a single PHP application (and pool). Dynamic requires CPU cycles to spin up more processes as needed and kill stale processes, while a static configuration keeps the processes running until max_requests is reached. Basically, max_requests should be tuned as a balance between the frequency that processes are restarted and limiting memory leaks. For example, if your application uses 100MB per PHP-FPM child, that child is going to grow over time. Assuming that it grows to 110MB after 5000 requests, you could probably run 8 processes per GB of RAM with a max_requests of 5000. But if it grows to 180MB after 10000 requests, you can only run 4 processes per GB. But if you set it too low, you'll be wasting resources restarting those processes. A theoretical balance for that application could very well be 6 processes at 9000 max_requests. It's really something that you've got to tune to be sure.
Normal CPU for PHP
That's a hard one. It depends highly on the manner in which the site is constructed and the raw CPU available on the machine. If you're using CTools / Panels heavily and don't have a huge amount of CPU it can easily take up 20-30% of a core. We find we're typically CPU bound due to the entity rendering. As much as possible you want to avoid that, use Views Content Cache and -- if using Panels -- Panels Content Cache and other caching modules as much as you can, they'll reduce both the latency of the site as well as often the CPU.
It might be more helpful to
It might be more helpful to think of CPU usage as simply a byproduct of other performance issues. By cleaning up the performance bottlenecks the CPU usage will fix itself.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Excellent point about Panels
Excellent point on panels caching, I do lots of views caching. I am guilty of overusing panels. Although it is a bit off topic, I wonder how bad a performance hit it is to rely on panels pulled from the db for display information?
APC & memcached are a given, but should I go back and build these page layouts using php templates and CSS? I don't want to because, well, Panels is sooooo darn easy!
Thoughts?
Really only you can answer
Really only you can answer that question. Do some PHP profiling to see how much of an impact Panels is having. If it's a lot see if it's made manageable by adding caching. Otherwise you'll have to refactor.
As a side note we make heavy use of Panels on all the sites that we make and can always make the performance fast enough for our needs. Avoid the flexible layouts though.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Another excellent point, dalin
That is (another) excellent point, dalin. CPU (over) usage is always a symptom not the disease.
Just to call out something
Just to call out something that can sometimes get overlooked: any APC fragmentation?
sethviebrock.com
agreed; especially if using
agreed; especially if using APCu / user bin capability of APC. I do an apache restart nightly to flush this since I use user bins.
Ex Uno Plures
http://elmsln.org/
http://btopro.com/
http://drupal.psu.edu/
Our APC settings apc.shm_size
Our APC settings
apc.shm_size is 256M
apc.stat=0
and there is fragmentation :Fragmentation: 0.79%
Hit% is 98.8
We have installed devel module and it shows that
24 queies executed in 3.36ms , page execution time is 102.61ms .
I think the page execution time is high compared to the query execution time
I'm a bit confused. 102ms
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
That is when a single user is online
Executed 33 queries in 4.95 ms. Queries exceeding 5 ms are highlighted. Page execution time was 107.45 ms. Memory used at: devel_boot()=1.62 MB, devel_shutdown()=16.76 MB, PHP peak=17 MB
That is just for one user.When the no.of users increases it goes up to 10s.At that time still the query execution time is less , in milliseconds. Why should it take this much time for page execution ?
So you are saying that when
So you are saying that when the number of users on the site increases, then the page execution time reported by Drupal goes up to 10s, or is your browser is spinning for 10s but the page execution time is still reported at around 100ms?
If the latter then it's likely a webserver configuration problem. The web server is hitting a max number of concurrent requests and queuing additional requests until the active ones are complete.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
The page execution reported
The page execution reported by drupal is 10s
Solve this issue with one command line
I was having this issue for about 6 months without any resolve. I read countless forums on how to fix this but all it really took is to restart the MYSQL.
I realized that my MYSQL was stuck in a loop so once I restarted it, the CPU usage dropped form 100% to 1%
So just LOGIN to your SSH and put this command
"sudo service mysql restart"
I've had MySQL servers
I've had MySQL servers running for a year+ without every having this problem (on some pretty large, high traffic sites as well). I would try updating MySQL, as it sounds like your version might have a bug in it. If it keeps happening, go into the MySQL command line and do a "show processlist;" command to see if that helps track down the problem too.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.