Performance Profiling System

Posted by echoditto_tom on March 22, 2008 at 3:55am

Moved to official ideas list

We'd like to propose the development of a profiling module and possible API capable of collecting performance data for hook invocations and calls to enabled functions in addition to the theme and database timings reported by the Devel and Devel Themer modules. The output of this module could include full traces of hooks, theme functions, database queries, external API requests and other functions similar to the trace files generated by debugging tools like XDebug but tailored to provide information that is useful to module developers and site builders. Other useful reports might include ranked execution times to highlight problem code and statistical aggregates gathered multiple page loads.

Development of this module will involve an understanding of Drupal's inner workings and will provide an excellent opportunities to learn the details of how each page is built from the ground up. It will also provide a great service to the Drupal development community, Drupal integrators and systems administrators. Use of the devel module is fine for identifying runaway queries, but it can be difficult to tie this to a complete picture of the system's modules, particularly if a module is making too many API calls rather than just too many large database queries, is making use of suboptimal algorithms or is encountering systems-level bottlenecks.

In the past we've had to spend time examining Drupal modules on client sites to determine which are most responsible for page load times, database and overall system load. This involved process is lengthy and usually not pretty, involving numerous one-off hacks to core to enable needed reporting followed by hammering the site with a/b tests and tools not optimal for debugging the Drupal stack. This project would not only provide a vastly improved interface for the debugging and optimization of Drupal code and live installations, but would also provide the groundwork for an approach to full-stack PHP debugging which takes the needs and conditions of Drupal development as its initial focus.

A list of potential features includes:

selective activation -- process only fires for specified user roles, IP pools or for a percentage of page loads so as to not affect overall performance
complete trace dumps of Drupal page generation stack
"problem functions" report for system administrators and others evaluating code in the wild
collection of stats re: authenticated:unauthenticated session ratio
optional integration with Drupal's throttle mechanism based on the results of the profiling
display of stats through pretty graphs (Google Chart API or similar)

EchoDitto would be glad to provide mentorship to any students who end up undertaking this task.

Comments

Some existing pieces

Posted by kbahey on March 22, 2008 at 5:20am

The trace module does timings for hooks http://drupal.org/project/trace
The loadtest module does fine grained performance tests http://groups.drupal.org/node/5488

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

I would love to see this

Posted by gdd on March 22, 2008 at 4:46pm

I would love to see this happen. What would be really great is if this accepted plugins so developers could add new components (for release or just custom for their own needs.) I think getting the complete project done as specified above might be somewhat ambitious for SoC, but putting together the framework with a couple tests and the stats might be doable. The trace module appears to be abandoned, but the loadtest project looks like it could offer some code to build upon.

Cross-posting

Posted by gdd on March 23, 2008 at 1:30am

To High Performance and Performance Optimization groups for more feedback

Tom, this sounds

Posted by alex_b on March 24, 2008 at 5:21pm

Tom,

this sounds interesting. I've put a profiler together for monitoring one of our apps that i really like but it's way too messy to share. One particular thing I love is that I can plot the time consumption of a program part per page load or per cron run over a week with a couple of lines:

// pseudo code
$start = micro_time();
do_something();
monitor_add('variable_name', micro_time() - $start);

and then for outputting on a log page I plot a graph:

// pseudo code
$output .= monitor_graph('variable_name', 2436007);

This has been hugely useful in identifying and addressing performance issues on cron or on page loads.

http://www.twitter.com/lxbarth

I'd be interested in adding

Posted by moshe weitzman on March 24, 2008 at 7:43pm

I'd be interested in adding this sort of logging to devel package. what does monitor_graph do exactly? Also note that Drupal has a tiny timer API already - see timer_read().

The google chart API looks extraordinarily simple for adding charts to any page. Just specify the particulars in the URL. Check out the SRC for this img.

Only local images are allowed.