Memetracker Wiki Page

Project information

Project pages on drupal.org: Memetracker and MachineLearningAPI

Current status: Writing tests, adding CRUD functionality, writing code to detect interlinking

Description

I am writing two modules for Drupal as part of Google Summer of Code. One called Memetracker and the other called MachineLearningAPI. The Memetracker module will use algorithms in the MachineLearningAPI to intelligently filter and group content from designated content sources both internal and external. The module's purpose is to find and display to a community in real time the most interesting conversations and memes within the community as they emerge.

My project will emulate functionality of successful commercial memetrackers such as Techmeme, Google News, Tailrank, and Megite. It will be an open-source implementation of memetracking technology that can be easily plugged into Drupal-based community sites.

Read my full proposal

Project schedule

• 1 week (May 25-30) -- Write post on g.d.o on plans for architecture, hooks, use cases to support etc. Spend the rest of the week responding to other's ideas and investigating machine learning algorithms.
• 2 weeks (June 2-13) -- Start stubbing out code. Get something working as soon as possible that I can release.
• 1-2 weeks (Late June) - Set up public testbed for memetracker. I talked more about this in my proposal. I'll start a number of memetrackers to test algorithms / performance under real-world conditions.
• 2 weeks (early July) - Experiment with a huge range of different algorithm setups, other configuration testing all the while to discover optimal set-ups for different conditions.
• Until end of summer -- improve admin UI, test coverage, bug fixing, documentation, etc.

Status updates

• June 3rd, 2008:

1. What did you get done this week?

I planned the architecture for my module and discussed it with my mentors here.

2. What are you planning to do over the next week?

I will start building the memetracker using two simple machine learning algorithms to test the wisdom of different architectural decisions.

3. Is there anything you're blocked on?

Nothing at the moment.

1. What did you get done this week?

I coded up the architecture discussed in the memetracker group.

2. What are you planning to do over the next week?

Identify a clustering library and implement it within my module. Make my initial release of code. Set up a live test-site.

3. Is there anything you're blocked on?

Finding the right clustering algorithm. There's a bunch of open-source libraries and I'm not sure which will be best. I guess I'll just start experimenting.

1. What did you get done this week?

Last week I got most of the basic architecture laid out in code and everything was working except the cluster analysis part which is tricky as it requires integrating with a non-php 3rd party library. Progress has slowed a bit as I talked to my brother last week and he preached to me test-driven-development (TDD), i.e. write tests before code so I've stopped writing new code to study up on SimpleTest and to start writing a test suite for Memetracker.

2. What are you planning to do over the next week?

Write unit tests. Implement cluster code. Clean up code and documentation sufficient to make a first release.

3. Is there anything you're blocked on?

Not really.

1. What did you get done this week?

Since my last update I've released three alphas, set up a demo site at memes.educon20.org, fixed loads of bugs, and added a number of optimizations.

2. What are you planning to do over the next week?

Write unit tests. Add pages showing list of feed sources and page showing clicks in past 72 hours, CRUD functions for feed sources, code to detect links.

3. Is there anything you're blocked on?

This feature request is very important to Memetracker. Memetracker needs a way to get content somehow from web pages that feed items link to. I'd appreciate help here: http://drupal.org/node/283607

1. What did you get done this week?

I've released another alpha with additions to the admin ui allowing easy creation of new memetrackers.

2. What are you planning to do over the next week?

Add code to detect interlinking between feed items and simplify memebrowsing UI.

3. Is there anything you're blocked on?

Nothing really.