Recently I was doing some consulting with a fortune 100 company (can't specify any more) who is running a large intranet site with upwards of 90,000 active users.
They are having a couple of issues that I thought the high performance group might have some thoughts on.
First, are there any good tools available for correlating logs to identify what user actions or system events precipitate spikes?
Essentially, they will see a spike in the resources used by their web heads or mysql servers and they want to correlate this with the load on the other servers in the cluster and, ideally, with the actual actions that were taking place on the Drupal site at the time. Right now they are digging through access logs of their 10 php heads and trying to correlate these with slow query logs, etc.
Are there any tools that help to pull this sort of data together?
Second, does anyone have a good method of identifying performance bottlenecks within a running production site?
As a company that runs most of their infrastructure on Java, they expressed frustration with the tools available to do ongoing monitoring of their Drupal stack to determine where the resources were being used. They described a number of enterprise java solutions that they have used to monitor the aggregate resources spent across different portions of the application.
They have a large user base (again, ~90,000 actives on the intranet who are all authenticated) and are serving highly targeted content to each of these users (determined by role, geographical location, department, etc.) and they would like to get some ongoing statistics to aggregate what percentage of their production site's resources are being spent on the bootstrap, theme layer, particular pages, etc.
Does anyone have any experience putting something like this together? It seems like attempts to do this in PHP could be computationally expensive, would likely require hacking core (depending on what tracking was necessary) and storing/aggregating the data could be a real challenge. Does anyone have any experience with putting something like this together?
Comments
Profiling
Mostly true; best way to see whats sucking your resources is to hit a dev server with Siege or JMeter after you have Xdebug (with webgrind) and/or XHProf up and running.
In terms of slow queries the DB Tuner module will help you identify views that might be causing them. In your case, run this module (dev version) on a dev box.
EDIT:
I'm assuming you know about Devel and it's performance logging features; It's good at IDing slow queries and the same query that runs multiple times in that request. Example of ID-ing issue and patch http://drupal.org/node/929082 (nodewords). Also be aware of modules that might cause the page to be rendered twice; I encountered this issue with the page title module when used with panels; blame might be on tokens though, token should have better cache control http://drupal.org/node/619892.
There's also the Performance
There's also the Performance Monitor module that is bundled with the Devel module. It can track page generation times.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Splunk == enterprise log correlation
The de-facto standard in log correlation and querying is Splunk. It's non-free, but everyone I've spoken to who owns it can't do without it. Personally, I've never used it, I just use rsyslogd.
Couple either of those with the syslog module in core, and you have your log correlation. NTP sync is a must for this as well.
Yeah, deve's performance
Yeah, deve's performance monitor doesn't help achieve what they're looking for. The problem they have had with this and with siege/jmeter as well as per-request php tools is that all of their users are authenticated and users are being served highly targeted content. Depending on their region of the country, department, position in the company, etc they get a significantly different collection of content. This means that each type of user (determined the combination of all of these variables) has a different profile. The goal is to determine the resources used in aggregate on the production cluster as a whole.
The individual test connections need to provide a good sample across all of these variations. It seems like the best way to achieve this might be to build a set of sophisticated scripts to be run with The Grinder.
I think the only way to get the detail that they are accustomed to is probably by scripting the user behaviors with as many of the variables as possible (combined with some existing usage statistics) logging the time spent on different operations in a db and then aggregating the results.
@justintime: thanks for the pointer at Splunk, I wasn't familiar with that and it looks very helpful!
Modify performance.module
You can modify performance.module to add columns for what you want, for example, the role of the user. Then you can analyze the data based on URL and those new fields as well.
The level of granularity you are asking for require instrumentation at a far lower level than what devel or performance offer though.
Something like this can be used as a base
http://code.google.com/p/instrumentation-for-php/
But you still have to go modify hundreds of functions in dozens of modules for the granularity you require.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Cacti
I graph everything with Cacti and compare the graphs. This includes the host, database, varnish, network traffic, memcache bins, APC cache, and even a ton of Apache Solr stats. None of that is Drupal-specific, but what I did with the Drupal side was to roll a quickie module (http://drupal.org/project/cacti) that allows me to also poll Drupal itself to create Cacti graphs. Now I can do things like compare the other graphs to the Drupal graphs so I can best determine what was happening at that instant. There's a patch in the issue queue to add hooks, so you can hook_cacti to expose just about anything to it for graphing. It might take a bit of coding to extend the Cacti data for your needs but it's a good way to correlate Drupal activity with the other layers.
EDIT: I also LOVE Splunk for log correlation. It's stupidly easy to setup and is free up to 500MB of logs per day.