Proposal - Community analytics for Drupal

Events happening in the community are now at Drupal community events on www.drupal.org.
damien tournoud's picture

Overview: This proposal is about creating a community dashboard for any Drupal website. Do you want to know how healthy your community is? This dashboard is for you. It will display the key performance indicators of your community: number of new members, number of active users, number of new posts, of new comments, etc. The design will be pluggeable so that any module can publish statistical data (ie. sales amount from your Ubercart store, etc.).

Description: One of the key point of the project is to build an extensible and modular framework for collecting and displaying statistical data inside Drupal. Three main components (hear: modules) will have to be developed:

  • A statistical datastore. Modeled around the great RRDTool, but rewritten for SQL storage, this module will allow you to store historical statistical data and its aggregates. Similar to RRDTool, you will define the granularity and the number of data points of each data stream (along the lines: I want to store one week of data at a step of an hour, and one month of data at a step of a day, and 2 years of data at a step of one week).

  • A data extractor. This module will, on cron run, collect data about the activity of the Drupal site (number of nodes, number of comments, number of users, users active in the last hour, etc.). The design will be pluggeable so as to allow other modules to provide data.

  • A data display front-end. This module will generate graphs (using the Google Chart API), and display your data nicely. The exact perimeter of the data analysis enabled by this module remains to be determined.

Mentors:

Difficulty: Medium

Comments

This is also a great idea,

hypertext200's picture

This is also a great idea, technically strong and comprehensive. I like to work on this idea too, I'm interest to developing this, what is the google used in their analytics? Doesn't this be a good model to plot data?

Senior Drupal Developer at DrupalConnect

I like this one

snufkin's picture

I like this one

You will need more

Bojhan's picture

You will need more researched metric then new members, new posts ect - that's obvious data - you don't necessary need a dashboard for. Try to find some more interesting metrics.

This is a key aspect of this

damien tournoud's picture

I use "Dashboard" in the management and Business intelligence sense, ie. a set of key performance indicators (KPI) displayed in a readable and pretty way.

The question "what are the KPIs of my community?" is a difficult one. It is probably different from community to community. On drupal.org, we tend to split "code", "documentation" and "support". The KPIs of those different aspects of our community are different:

  • for "code", it might be: number of open critical issues, number of issues that get closed in the last month, number of commits, number of commiters, number of people mentionned in commit messages, number of lines of code modified, etc.
  • for "documentation", it might be: number of lines of documentation modified, number of different contributors, number of pages of documentation for novices, advanced users, programmers vs. themers and designers, etc.
  • for "support", it might be: number of new forum topics, number of unanswered forum topics, number of unique authors in follow-ups, mean time a forum topic remains unanswered, etc.

Damien Tournoud
http://drupalfr.org

Damien Tournoud

@Damien Tournoud : Are you

hypertext200's picture

@Damien Tournoud : Are you addressing to add some BI (Business Intelligent ) in to this?

Senior Drupal Developer at DrupalConnect

Could this integrate with

BrightLoudNoise's picture

Could this integrate with the work already done in Activity module? http://drupal.org/project/activity

I think this is a great idea

kyle_mathews's picture

I think this is a great idea -- I was already writing something similar for my own site (albeit much simpler at least initially). I for sure will help out on this project either as a mentor or interested community member.

Kyle Mathews

Kyle Mathews

Can we add..

Amila's picture

Can we add flex with this? flex with drupa?

Hell no.

damien tournoud's picture

Hell no.

Damien Tournoud

What about silverlight,

Bojhan's picture

What about silverlight, maybe silverlight? (joking)

I think this is very intresting, but if you want to work on something that has a visualisation end - you will need extensive time to make that work (just plotting google graphs) is unlikely going to correspond nicely. I envision a pipes like interface for this(dreaming), but apart from that sounds like quite a solid idea.

I am still a bit puzzeld on the name, as it doesnt actually describe at all what it does - or at the very least where the focus is upon.

About your idea...

kimkha's picture

User (mean: admin of his own site) can customize his indicators or not? Example: he installed some modules and want to view performance indicators of them, so he can set his own dashboard to store and view some components of these modules.

I'm student, and want to implement this idea of yours. But I don't know how to register GSoC and join your group... Can you help me?


KimKha's Weblog: http://kakalia.co.cc

This looks great

acouch's picture

I think this could have two key use cases, 1) large interactive sites such as social networks that want a quick snapshot of activity, 2) small sites for non-technical users who don't know what to make of the standard drupal admin section. wordpress has a real advantage for non-technical users because their dashboard is easier for a nube to understand and interact with.

it would be great to see it be flexible enough to have a clean and user-friendly blank state as well as flooded state.

good luck!

I like this idea and I am

mikey_p's picture

I like this idea and I am intrigued as I have been spending time setting up RRDtool systems like cacti and munin lately. Are you proposing to use RRDtool for the storage, or to replicate it's storage with PHP and MySQL?

Also, I am not sure that using Google Charts is the best option for this. Maybe if could be a selectable option with something like FLOT as the default.

I suppose the ultimate here would be to also expose all this data to Views 2 as well. Thoughts?

Looking for a co-mentor

damien tournoud's picture

Hey,

I'm looking for a co-mentor for this project. You love to help, and you have some more free time then I do? Please step in!

Damien Tournoud
http://drupalfr.org

Damien Tournoud

I'm interested in being a

kyle_mathews's picture

I'm interested in being a co-mentor on this project. I was planning on building something like this anyways for my own uses this summer so my interests and this project's are nicely aligned :)

Kyle Mathews

Kyle Mathews

I think this has overlaps

dldege's picture

I think this has overlaps with http://groups.drupal.org/node/20452 and they should be combined or at least discussed if they really differentiate.

Dan DeGeest
Lead Software Developer
iMed Studios
http://www.imedstudios.com

Dan DeGeest
Software Developer
Somewhere or Another

Interested in this project

haxney's picture

This project sounds intriguing, and would really help on the Drupal site I maintain. I'm a student applying for GSoC, so how would I get started on the proposal for this project? Should I simply copy over the existing proposal here, or rephrase it in my own words?

After half a year using Drupal as a webmaster, I'm excited to cross over to the development side.

Thanks!

I've submitted an official

mikey_p's picture

I've submitted an official proposal for this at the GSOC site. http://groups.drupal.org/node/20889

Use Cacti

raintonr's picture

I think it would be very useful to integrate monitoring of a site at the Drupal level into an external RRD based system like Cacti.

It would work like this:

  • Use existing machine/web server/database based statistics for underlying O/S, web server and DB monitoring. Eg. Load, memory usage, network traffic, Apache Stats (hits, thread scoreboard, etc). This runs on the physical (or virtual if you're on a VPS) host.
  • Create a new 'Drupal' template in Cacti. Such a template would need to be extensible or handle statistics that are extensible. As there can be many Drupal sites on a single web server each site to report on would have to be created as a 'host' in the Cacti system.
  • Create a Drupal module (named, 'Stats Dump'?) that returns current statistics as required for use by the Cacti template. Such a script could return stats such as number of users registered, logged in, sessions opened, nodes created (possibly split by node type), comments posted (possibly split by node type), watchdog errors or other log entry numbers, pages rendered, pages retrieved from cache, blocks rendered, etc, etc, etc.

Take an example scenario:

  • The host web1.example.com is a Linux box running Apache. Standard O/S level and Apache stats stuff is graphed for this box.
  • The Drupal sites forum1.example.com, cms2.eg.com and site3.etc.com run on this server. They are all configured in the Cacti monitoring system as 'Drupal Sites' and just have Drupal level statistics reported on.

As you can imagine when starting to think about this an 'Stats Dump' module would have to make use of hooks to allow other modules to report their statistic name and value. If used with Cacti such a module would be called every 5 minutes or so.

The point is that there are a few RRD based tools for statistics collection and display about - we wouldn't want to rebuild them and focus on just producing stats at a Drupal level for them to record.

Thoughts?

Simple stats

raintonr's picture

So... after proposing that Cacti would be a useful thing to use I went away and wrote something completely different (see attached). ;)

This is a simple bit of code one can place in any page with 'php' input format. You can see this in use here:

http://nobmob.com/node/9160

There's a similar graph of active users here:

http://nobmob.com/node/9161

I've made a few pages like this for content types and taxonomy terms for our site. The stats are simple but effective I think but are more for user interest than performance monitoring which is where the Cacti solution would come int.

Anyhow, hope this is useful to someone,

Cheers,

Rob

SoC 2009

Group categories

Admin Tags

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: