Most users logged - Slow load time

cbccharlie's picture

Hello,

I have a Drupal with the majority of users logged

Approximately 25,000 pageviews per day.
Two front and two backend servers.
APC and Memcached installed.
Authcache for authenticated users.
Some 5,000 nodes and 20,000 reviews, about 20 types of content.
Approximately 100 modules.

The loading time of the web for a single user is about 3-5 seconds, but for 100 concurrent users are more than one minute.

What could be done?

Thank you very much.

Comments

Look for slow modules

kbahey's picture

We support a site that has ~ 160,000 page views a day, with peaks over 200,000. The number of concurrent logged in users on the site is 400, with peaks of 700. The site has 195 modules enabled. No authcache.

The site runs fine on a single server, with memcache, APC and varnish. But it only runs well after we discovered what is slow and remedied it. One module was responsible for slowness because it consumed so much CPU time, starving everything else. Even NewRelic would not pin point the module.

You need to find out what is eating up the page time, and then go about from there.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Hi Khalid Baheyeldin, First

cbccharlie's picture

Hi Khalid Baheyeldin,

First of all, thank you very much for everything.

I enabled the Xhprof to try to locate the module that slows down the page.

Overall Summary
Total Incl. Wall Time (microsec): 1,633,006 microsecs
Total Incl. CPU (microsecs): 1,413,786 microsecs
Total Incl. MemUse (bytes): 22,933,756 bytes
Total Incl. PeakMemUse (bytes): 23,136,844 bytes
Number of Function Calls: 99,504

How I can find out which module consumes CPU? I think it is the views module, but not sure. Could it be the cause?

Thank you very much!

Wall time

kbahey's picture

It is not really that simple, but you start by sorting by wall time, and see what takes the most time, relate it to a module, see what it does, and go from there.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Reviewing the function tree,

cbccharlie's picture

Reviewing the function tree, I have the following results (each line is a child function):

  • menu_execute_active_handler - 1,313,576 (78.4%)
  • drupal_deliver_page - 1,302,056 (99.1%)
  • drupal_deliver_html_page - 1,013,125 (77.8%)
  • drupal_render_page - 1,009,044 (99.6%)
  • block_page_build - 785,369 (77.8%)
  • block_get_blocks_by_region - 784,654 (99.9%)
  • block_list - 784,386 (100.0%)
  • _block_render_blocks - 764,686 (97.5%)
  • module_invoke - 745,187 (97.5%)
  • call_user_func_array - 751,661 (87.8%)
  • views_block_view - 525,422 (49.6%)

This is the complete graph: https://dl.dropboxusercontent.com/u/3476608/callgraph.php.png

Looks like a slow block

mikeytown2's picture

Think it has something to do with a slow views block.
Also looks like https://drupal.org/project/menu_position has a slow alter in it.

After fixing that slow views

dalin's picture

After fixing that slow views block you might want to investigate block caching. If you can't enable that due to node access rules you'll need
https://drupal.org/project/blockcache_alter

--
Dave Hansen-Lange
Technical Manager
Advomatic LLC
Great White North Office
Canada

Thanks. I tested this module,

cbccharlie's picture

Thanks.

I tested this module, but rendering times of these blocks do not vary. To view these load times, use the timer module block indicates jchin1968.

What do you ever use it?

Block Timer Module

jchin1968's picture

Try running this module (https://drupal.org/sandbox/jchin1968/2061005) on your development site. I created it for one of my recent projects to help pinpoint which views block (out of 20 or so on the home page) was rendering slowly.

Once you have identified your troublesome views block, try disabling it to see if it makes a difference. You should also enable views cache and reduce the number of relationships (under Views Advanced settings) if possible. I found if you have more then 2 or 3 relationships defined within a Views, it can really slows things down.

Another good performance debugging tool is the "Display Query Log" option in the Devel module which highlights slow database queries.

Hello, I found that in the

cbccharlie's picture

Hello,

I found that in the block table records still exist views that were renamed or deleted, but in page load still calling (200 blocks). I think it's a bug in the views module. This what I've discovered thanks to the module "block timer" (thanks for your input, jchin1968). Hope this helps more people, and that could happen to many.

Furthermore, mikeytown2, the module menu position, if I remove it, the low load time two tenths or so. I hope to find an alternative.

I'll run new with jmeter load testing and results will here.

Thank you very much.

I agree with Khalid that you

Jamie Holly's picture

I agree with Khalid that you most likely have a module causing the problem. Also try to identify exactly where the bottleneck is (disc, processor, memory, network), then what is causing that. Another common thing is a lot of lock waits on the database.

Just for reference, I've got one customer site that gets over 300,000 pageviews/day and has over 50,000 registered users. It's running on one web and one database DB with almost 100 modules and never goes above 500ms to deliver the HTML. As matter of fact, just this week one big link gave them over 120,000 page views in under an hour and the web server never went above a load of 4 and the database never over 1. The site is running APC and memcached, but no authcache.


HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

Khalid, I'm curious - how did

Sam Moore's picture

Khalid, I'm curious - how did you identify the module that was causing the trouble?
I've inherited a very active site that has some custom module development which I suspect is not very good. I'm planning on watching the slow-query log and working back from there, but are there any useful tricks to this?

When I do that, it's on a

Jamie Holly's picture

When I do that, it's on a local copy of the site. First thing is using the devel module to make sure there aren't any really nasty queries. After that I'll run it through XHProf to see which module functions are eating up the most time. It's not the perfect way, and Khalid might have a better suggestion, but I've had very good luck with this approach in the past with some very nasty sites I too have inherited over the years.

(FYI - when doing this stuff, I don't use any caching. Rather get the baseline from there, then add the cache back in later)


HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

Usually XHProf or XDebug profiling

kbahey's picture

We use a variety of tools for this kind of problem. We first take the site to a lab setup, and recreate the site, and validate that the problem is indeed reproducible and not due to the hosting environment. We then use XHProf or XDebug's profiler. In other cases, NewRelic can give a clue, but not in this instance.

For this particular site and problem it was a combination of deductive sleuthing, refactoring some code to use caching, then more caching, then re-architect the offending module altogether.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Great info, Jamie - thanks.

Sam Moore's picture

Great info, Jamie - thanks.

In addition to all the

gp177's picture

In addition to all the troubleshooting above, start running varnish.

In addition to speed it can act as a security layer, it can provide limited site content during downtime, and protect against the slashdot effect.

Hello, After deleting blocks

cbccharlie's picture

Hello,

After deleting blocks views that do not exist and remove the module menu FirstChild, which consumed too much, these are the results.

Frontpage (with caches disabled to simulate an authenticated user), 20 concurrent users load average 16 seconds.
A user only Page execution time was 1286.09 ms Memory used at: devel_boot () = 2.92 MB, devel_shutdown () = 21.94 MB, PHP peak = 23.25 MB.

Infrastructure where I do tests is a frontend with 4 CORES and 2 GB of RAM and two backend with the same resources, connected to a proxy mysql and Files folder on a GlusterFS.

In production, I have two frontends 8 CORES and 12 GB of RAM and two backends with 4 cores and 8 GB of RAM. A user is slow, but over 250 is already extremely slow, unusable.

Using Xhprof not find anything that could look over? modules? inconsistencies in database? infrastructure?

Thank you very much for the help.

High performance

Group events

Add to calendar

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week