Cacti stats for a Drupal Site

Posted by raintonr on April 22, 2009 at 1:13am

I was thinking about trying to monitor several Drupal metrics using Cacti and went looking for someone who's done it before. Doesn't seem so, but I would have thought monitoring statistics on your site(s) is an essential part of any high performance setup so bring it up here.

Anyhow - found something similar only and added a comment over on a thread about monitoring Drupal:

http://groups.drupal.org/node/20271#comment-74657

Anyone here interested in such an effort and how it might be built?

Comments

Munin too?

Posted by kbahey on April 22, 2009 at 1:50am

Munin also outputs RRD files. Wriing and extension is trivial. I have one running less than 30 minutes.

The Munin plugins could query the database for number of nodes, users, comments, and pretty much anything else that can be gleaned from Drupal's database.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Drupal Module?

Posted by raintonr on April 22, 2009 at 7:13am

Cool... did you produce a Drupal module to output stats then? Or are you talking about a small script that just grabs stats you need out of a local database and runs as part of Munin?

What I had in mind would have to run as a module as it would have to be queried remotely based on the Drupal site name.

Outside of Drupal

Posted by kbahey on April 22, 2009 at 3:08pm

Munin works by executing plugins in a specific directory. They can be written as shell scripts, perl or anything else.

So, I have done them the Munin way.

If you want something from within Drupal, then check the Drupal Nagios integration module, which I wrote. It makes Drupal query-able from Nagios.

Another thing to watch is this Google Summer of Code project, which aims for something similar, within Drupal http://groups.drupal.org/node/20889

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

When having a (virtual) root

Posted by Alexander Langer@drupal.org (not verified) on April 22, 2009 at 2:26pm

When having a (virtual) root server you might easily be able to write your own Munin or Cacti extension, grabbing data directly out of the database.

A more general approach would be to write a Drupal module that delivers statistics upon requests and provides hooks for extensions. It would also have to do some authentication, at least by defining which IP address may grab statistics (similar to how munin's nodes are configured).

To not reinvent the wheel you'd watch out for something to build upon. Services and webservices modules come into my mind, though I haven't worked with them by now.

I'd be interested in helping out where I can. I'm using Munin on all of my servers and know Cacti only from some demos as well as from killes' going through some of drupal.org's stats, but as Khalid said both use a similar data format.

Alex

I use Cacti & Drupal

Posted by slurpee on April 26, 2009 at 2:01pm

Check out http://drupalcampchicago.com/cacti

I have been trying to setup as many graphs as possible for Drupal related services. The hardest part is enabling and configuring SNMP on the server. Most Drupal users don't have access or the knowledge how to set it up.

Google created some great MySQL graphs - http://code.google.com/p/mysql-cacti-templates/

I could on about this for awhile. Contact me if you need some help or want to work together.

Matthew@Lechleider.com

I'd love to follow your

Posted by stewsnooze on April 27, 2009 at 9:35am

I'd love to follow your work. Hopefully I can help at some point. We are attempting to do a similar thing

Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.

Wanted more Drupal Specific Stats

Posted by raintonr on April 27, 2009 at 1:19am

@slurpee - Yeah, those stats you show are all host/DB specific things and am already using most of them.

Useful as this is I was thinking that more Drupal specific stuff like nodes/users/comments/cache hits/etc/etc. would be good too and that's what I was posting about.

Cheers,

Rob

What stats do you want

Posted by webmasterkai on May 5, 2009 at 8:57pm

I'm working on some drupal/cacti graphs with Dmitri Muntean. What graphs are you interested in? I'd be happy to finish up what we have and submit it.

I've got things like:
guests on site now
authenticated on site now
total users
new users in last month

That sounds brilliant, i've

Posted by a_c_m on May 6, 2009 at 2:59am

That sounds brilliant, i've just spent 6 hours re-grok'ing cacti and once i saw how the plugins worked that was my first thought.

Perhaps also new nodes and new nodes of a given type, new users, new users by role, total users, emails sent, comments left ... the list goes on!

More stat ideas

Posted by raintonr on May 6, 2009 at 5:03am

Fantastic offer. Stats to consider:

Pages built (and loaded from cache)
Blocks built (and loaded from cache where block cache is there)
Views built (and loaded from cache) where views are enabled
Total page generation time. Divide by page views to get average.
Someone else has mentioned nodes created. What about comments when that module is in use?
CAPTCHAs displayed/passed/failed where CAPTCHA is enabled.
and on and on and on ;)

Did you end up submitting

Posted by a_c_m on May 18, 2009 at 9:43pm

Did you end up submitting the code?

I'm not going to get a chance to look at this again till next month, but will share any metrics / modules that come out of our work on it.

how to find drupal snmp data?

Posted by slurpee on May 19, 2009 at 4:30pm

How are you finding the snmp data to collect related to Drupal?

Which OIDs are you utilizing?

Do you have anything I can try via snmpwalk?

SNMP Drupal

Posted by raintonr on May 19, 2009 at 9:32pm

There won't be any. We'll have to make up our own custom OIDs.

For anyone thats interested,

Posted by a_c_m on June 9, 2009 at 1:22pm

For anyone thats interested, this is quite a good little tutorial on creating a custom graph - it gives a nice insight into creating your own graphs (and is a handy tool in its own right).

http://www.askaboutphp.com/tutorials/44/cacti-using-cacti-to-monitor-web...

@webmasterkai any updates?

I'm also very interested in

Posted by mrfelton on September 16, 2009 at 11:10am

I'm also very interested in this.

--
Tom
www.systemseed.com - drupal development. drupal training. drupal support.

Cacti

Posted by rjbrown99 on July 26, 2010 at 6:50am

Here's what I am doing for Drupal-related Cacti graphs at the moment.

PALMV3. I'm using the APC graphs from this package.

Memcache I'm using this multiport memcache since I split up the bins across numerous instances of memcache.

Apache Solr. I'm using their template and a new data collection script which I ported to work the Solr 1.4.

Varnish. I'm using the "version 3" template.

Better Cacti Templates. I'm using their Apache, InnoDB, and MySQL templates.

This has me covered at the MySQL, Memcache, HTTPD, Solr, APC, and Varnish levels for stats. What I have yet to find are released Drupal+Cacti templates as referenced above. I did send a note to webmasterkai asking for his started templates. I already have Google Analytics and log analysis, what I'm now looking for are direct-from-the-db stats such as:

Authenticated user count vs. guests (plotted on the same graph)
Total nodes in the system, plotted across content types (i.e. one line for each content type)
Total users + new users
Hits to a specific path I created in a custom module (a hook_menu path) plotted by authenticated user vs guest.
LOTS of stuff related to Feeds (items imported vs updated per feed node, etc)

The goal of some of this is to focus efforts on where specifically to performance tune the stack since authenticated vs. unauthenticated is a big part of that.

Not much activity here in the past year. Anyone else interested in helping roll some of this?

Still interested

Posted by slurpee on July 26, 2010 at 2:38pm

Maybe we can setup an IRC meeting to discuss these topics?

Sure

Posted by rjbrown99 on July 26, 2010 at 6:35pm

Sure, how about popping in to #drupal-cacti when you have a chance? I'll leave a client running and perhaps we can catch up.

Project

Posted by rjbrown99 on July 27, 2010 at 5:07am

I started on this today. Project is here:
http://drupal.org/project/cacti

As I write this, I committed into HEAD an initial module that acts as a Cacti data input method. Basically this is the part that runs on the Drupal site and provides raw output to enable it to be graphed. Still to-do are the client side and then templates. I'd love some help, please pop in to the issue queue. The goal is to enable this to be a flexible way to graph just about anything from a Drupal site. Right now I just started with logged in users vs. anonymous users.

Thanks