Memetracker module proposal

Summary:
I want to write two modules for Drupal as part of Google Summer of Code. One called meme_tracker and the other called machine_learning_api. The meme_tracker module will use the machine_learning_api to intelligently filter and group content from both internal and external content sources. The module's purpose is to find and display to a community in real time the most interesting conversations and memes within the community as they emerge.

Hello Drupliers. My name is Kyle Mathews. I'm a grad student in Information Systems at Brigham Young University working as a research assistant to several faculty members here. We are studying how and where social software can be used in education.

In the past eight months, I've built a number of classroom websites using Drupal. I've learned a considerable bit about Drupal in the process, became very involved in the community, and am writing a module (writing_assignment -- still a work in progress btw). I've loved most everything about Drupal but have found Drupal is missing an important component for building the perfect social learning website. This itches. So, in the best open-source fashion, I'm applying to Google Summer of Code to scratch my itch.

What's the Problem?
Any online community generates lots of content and conversations. As a community grows, a problem that soon emerges is how to help individual members find the content/conversations they are most interested in. If there's only a few members in the online community, it's easy to follow every conversation. But if there's 100s or 1000s of members, it soon becomes impossible to follow or find every interesting conversation.

In researching ways that online communities help participants find the most interesting content, I've found three patterns which help.

1. First is the small-world pattern. Via Organic Groups, you split conversations up by topic. People congregate around only the groups they are interested in.
2. Second is the Twitter pattern. Using buddylist or user_relationships, you follow friends or people who's ideas you find interesting.
3. The third pattern is to read the most interesting memes as they are somehow determined by the community (This is where my module will help out).

Drupal sites normally implement the third pattern by having human editors manually promote the most interesting content to the front page. But I've found this solution to be inadequate. Human editors miss many interesting pieces of content, are prone to bias, and the solution is labor intensive. Also, promoting content to the front page doesn't group the content by conversation or meme, an important usability improvement.

A much better solution for finding the most important memes are tools called memetrackers like Techmeme, Tailrank, Megite, and others. These memetrackers automatically find the most interesting content bubbling up in the blogosphere. Using these memetrackers, blog readers can in minutes read the best content from 1000s of sources.

As the principle tech memetracker, Techmeme plays a hugely influential role in the tech community. Techmeme is the place to find conversation about the biggest memes of the day.

But at the moment there are only commercial memetrackers such as Techmeme. There are no free or open-source implementations and certainly nothing that can be easily plugged into Drupal.

I propose then to write a memetracking module that will fulfill the same role as Techmeme does to the tech community for any community website where it's installed. Using machine learning techniques, my module will intelligently filter and group content from both internal and external content sources to display to the community in real time the most interesting conversations and memes within the community as they emerge.

Other Potential Use Cases:
I think the memetracker module would find value in many areas outside of my initial community website use case. Some other ideas.

• People could use software to make techmeme-like-sites for specific topical areas (or think enhanced PlanetPlanet type aggregation).
• such as an news aggregator for Africa (Aggregate information from BBC, local media, blogs, etc.)
• or an aggregator for the edublog community
• a better personal RSS reader
• There's a huge demand for a more intelligent RSS reader that learns what types of articles that you are interested in.
• Here's a few links:
• http://ross.typepad.com/blog/2006/09/between_popular.html
• http://www.brianoberkirch.com/2007/10/18/techmeme-dreaming/
• http://www.bernzilla.com/item.php?id=661
• http://radar.oreilly.com/archives/2006/09/social-blog-aggregators-revisi...
• Plus I can personally attest to the need for better rss readers. I subscribe to 175+ blogs and I can't keep up with them most days. The memetracker module would learn what articles I'm not interested in and automatically weed them out.
• A company rolls out meme_trackers for each employee (each Drupal user could have a meme_tracker). The employees subscribe to feeds (internal and external) they are interested in. Then the best content, as determined by all the company employees, is pushed onto individual employee meme_trackers automatically.

I have many more ideas I've thought of to improve/extend the basic concept but I'll share just one now.

With the RDF module coming along nicely, some amazingly cool possibilities would open up if individual memetrackers could talk to each other and share information via RDF.

A few examples of how that might work:

• A school (like say, BYU) has a meme_tracker for each college within the university which aggregates content from blogs/og group postings within that college. A BYU-wide meme_tracker then pulls in the best content from each individual meme_tracker and finds memes/conversations underway throughout the entire school.
• A blogger writes mostly about Ubuntu news. He wants to place block on his website that displays the top Ubuntu news of the day. He writes a bit of code that queries an Ubuntu memetracker for the top five articles of the day and displays them in a block.
• My friends and I all read blogs with individual memetracker we've set up on a server. We want to share with each other articles we find especially interesting. While reading, we mark articles we want shared. Then every so often memetracker queries my friend's memetrackers and returns blog posts they've marked. These posts are automatically placed into my memetracker and tagged by the friend they came from.

I can imagine that many many more very interesting mashups between memetrackers would emerge if they all expose their information via RDF.

I'm really excited about my idea and would love to help bring it to life. I'd love now to hear feedback from the Drupal community about ways I can improve my proposal before I submit it to Google.

Thank you for your time.

NOTE:
A possible point of confusion. There are significant differences between my proposed modules and existing modules which recommend similar content (Similar by Terms, Content Recommendation Engine, Similar Entries, Conversation Pivots). These modules guide you to content similar to the node you are viewing (for the most part) irregardless of the other content's age. My module identifies "emerging" conversations and memes. It groups on a page different perspectives and thoughts on the hottest topics of the day.