Memetracker Wiki Page

Project information

Project pages on drupal.org: Memetracker and MachineLearningAPI

Current status: Programming rough prototype

Description

I am writing two modules for Drupal as part of Google Summer of Code. One called Memetracker and the other called MachineLearningAPI. The Memetracker module will use algorithms in the MachineLearningAPI to intelligently filter and group content from designated content sources both internal and external. The module's purpose is to find and display to a community in real time the most interesting conversations and memes within the community as they emerge.

My project will emulate functionality of successful commercial memetrackers such as Techmeme, Google News, Tailrank, and Megite. It will be an open-source implementation of memetracking technology that can be easily plugged into Drupal-based community sites.

Read my full proposal

Project schedule

• 1 week (May 25-30) -- Write post on g.d.o on plans for architecture, hooks, use cases to support etc. Spend the rest of the week responding to other's ideas and investigating machine learning algorithms.
• 2 weeks (June 2-13) -- Start stubbing out code. Get something working as soon as possible that I can release.
• 1-2 weeks (Late June) - Set up public testbed for memetracker. I talked more about this in my proposal. I'll start a number of memetrackers to test algorithms / performance under real-world conditions.
• 2 weeks (early July) - Experiment with a huge range of different algorithm setups, other configuration testing all the while to discover optimal set-ups for different conditions.
• Until end of summer -- improve admin UI, test coverage, bug fixing, documentation, etc.

Status updates

• June 3rd, 2008:
1. What did you get done this week?

I planned the architecture for my module and discussed it with my mentors here.

2. What are you planning to do over the next week?

I will start building the memetracker using two simple machine learning algorithms to test the wisdom of different architectural decisions.

3. Is there anything you're blocked on?

Nothing at the moment.

• June 10th, 2008:
1. What did you get done this week?

I coded up the architecture discussed in the memetracker group.

2. What are you planning to do over the next week?

Identify a clustering library and implement it within my module. Make my initial release of code. Set up a live test-site.

3. Is there anything you're blocked on?

Finding the right clustering algorithm. There's a bunch of open-source libraries and I'm not sure which will be best. I guess I'll just start experimenting.

• June 17th, 2008:
1. What did you get done this week?

Last week I got most of the basic architecture laid out in code and everything was working except the cluster analysis part which is tricky as it requires integrating with a non-php 3rd party library. Progress has slowed a bit as I talked to my brother last week and he preached to me test-driven-development (TDD), i.e. write tests before code so I've stopped writing new code to study up on SimpleTest and to start writing a test suite for Memetracker.

2. What are you planning to do over the next week?

Write unit tests. Implement cluster code. Clean up code and documentation sufficient to make a first release.

3. Is there anything you're blocked on?

Not really.