RSS & Aggregation

I thought some of the folks here might be interested in a project we're working on - it's called Rootswire - http://www.rootswire.org - and is an aggregation of different political and place blogs. Our first big effort is to build a group site for all of the bloggers covering the Democratic convention. The site is up in beta right now (yeah, yeah, I know - it needs a lot of theming love) and we're currently working on the mechanical aspects of it, and getting bloggers on-board. It's all Drupal, and we're using feedapi and the calais module to import the feeds and automatically tag them.

Hi everyone,

I've been banging my head a little bit trying to get this sorted. Basically, I want members of my drupal site to have their own personal blogs and a section where they can read other members posts. This is obviously no problem if they're all using drupal blogs on my site.

However, many of the members already have their own blogs. It would be great if instead of submitting blog entries on my site, they could just add the url of their Blog feed and entries would automagically appear alongside other members entries.

I ran across a new memetracker built on Drupal/OpenCalais and custom code. It looks very cool. Check it out at Polymeme.com

They're tracking some 25,000 blogs and apparently running them all through OpenCalias as part of the process for finding "memes" along with some human editorial touch.

The creator of polymeme writes more about the site on his blog:
http://evgenymorozov.com/blog/?p=397

Have a look at PHP skeleton code for better understanding this outline.

Def:
Parser: Responsible for creating a feed data structure which is expected by the aggregator (hook_aggregator_parser). By design additional parsers could consume everything (ical, html pages, emails)
Processor: Responsible for accepting items and save them, show them to the user, etc (hook_aggregator_processor)

Planned modules:

* aggregator
* aggregator_node
* aggregator_light

I am aggregating Feeds from various site to aggregate at my site. Many of these feeds provide complete post instead of just a summary/teaser. I would like to aggregate only the teasers of these feeds (instead of the complete feed), and let the users go to the complete feed if required.
Is there a way for me to show summary only? At the moment, if the feed it aggregate is a full feed, it will show the full thing as well. I want to trim down the aggregated content, say to only 250 characters.

Thanks,
Victor

The major new feature of OpenPackage Video 5.x-3.8 is integration with Feed Element Mapper (feedapi_mapper).

The development of this functionality was sponsored by Hudson Street Media using our bounty system:
http://openpackage.biz/bounty/15
http://openpackage.biz/bounty/14

Feed Element Mapper is an add-on module for FeedAPI that maps a feed's elements such as tags or (in this case) videos to taxonomy or CCK fields on your site.

This enables your site to aggregate videos from other sites.

I have set up various video feeds on our site as a demonstration. Click on the 'Feed item' links to view the videos.

I am very excited to announce the first Daylife Developer Challenge for building applications using the Daylife News Aggregation APIs. Folks hosting Drupal based websites on http://ecproject.org and http://comminit.com have built drupal pages/blocks/modules serving syndicated content using the Daylife APIs earlier.

The prizes are: a $3500 first prize and two $750 runner up prizes. The contest ends midnight of July 25th

I'm just wondering how to import mediawiki XML and turning it into a wikitools drupal system. This should theoretically be simple, but I'm not sure where to begin.

Introduction

The Internet has empowered people to share information and ideas in unprecedented ways. It has enabled people to easily and at almost no cost to be both producers and consumers of information world-wide. Nowadays, people are able to have their own channel of information, broadcasting ideas, photos, videos, etc. This is very exciting and brings many opportunities for education, research, business and entertainment. However, it brings many challenges as well.

Problem

This project will create two complementary sites: a Local Publishing Platform paired with a Regional Aggregation Hub.

The Publishing Platform will support a range of activities, from a K12 school magazine, a community paper, a writing project (something like the National Writing Project or Youth Radio), a college paper, etc.

The Aggregation Hub will provide a way for these different organizations to work together without losing any editorial control, or diluting the mission of the participating organizations.

Some use cases

Here I summarize what you should expect from core aggregator for Drupal 7.

As maybe you heard, my application was accepted to write a new aggregator for Drupal core :) : http://code.google.com/soc/2008/drupal/appinfo.html?csaid=1222E090E875B3...
Here as a first step, I would like to collect lots of possible problems in aggregation area. Please share your opinion about these questions to help Drupal 7 to have as good aggregator as possible :) :

The items are not necessarily ordered by relevance.

Node creation

Hi,

There are many people, who ask a possibility to filter feed items based on keywords in order to exclude them from processing (or possibly based on some other criteria like on timestamp, etc.). This issue has been raised under most aggregator modules and such feature request is in the pipeline of FeedAPI project as well:

FeedAPI:
http://drupal.org/node/160692

The usual recommendation for such problem is to relay on standard syndication and use Views module afterwards for filtering.

TASKS:
WFM-IGP is a non-for-profit organization seeking a short term off-site Website Consultant to assist with the overhaul and recreation of a website, and the launching of a site under construction.

The Website Consultant will work with the WFM-IGP Website and Database Developer in ensuring and implementing a transfer of the website www.ReformtheUN.org off of its current Xaraya platform onto a new platform (Drupal, Joomla, etc.). The Consultant will be responsible for:

• ensuring the layout of the website/CMS is logical and accessible

Is anybody working/experimenting with aggregating comments for articles? I'd love to see a (FeedAPI) processor for pulling in comments for aggregated articles.

See also feature request here:

http://drupal.org/node/241935

This is my SoC proposal fro Drupal. Please feel free to post questions, suggestions or send me an email.

--------- Project proposal: RSS Overhaul (incl. Encryption) ---------

Improve over current RSS implementation:

• Richer RSS options allowing to include/exclude text contents, images, related comments, digg-like links, etc.
• Allow secure RSS and add access control settings in Admin interface accordingly.

 

 
Problems:

• Drupal's core aggregator does not satisfy the requirements of state
  of the art aggregation applications. E. g. core aggregator lacks
  pluggable configurations or the possibility of creating nodes from
  feed items.

• There is an explosion of contrib modules that try to cover the
  deficiencies of Drupal's aggregator in one or the other way. Previous
  attempts to unify aggregation functionality had limited success.

Moved to official ideas list: http://drupal.org/node/237907

I want to propose a project for building a module that pulls news content from a global news service called Daylife (http://daylife.com)

Daylife is a global news aggregation service that offers an Open API platform called DayPI (http://developer.daylife.com) for any developer, blogger, news service to pull stories and photos about any topic in the world. The API returns data in XML, JSON and PHP serialized formats. You can read the Daylife About page at http://www.daylife.com/page/about to learn more about the company. All platform documentation is on http://developer.daylife.com/docs

Status: Added to official ideas list http://drupal.org/node/234652

I'm crossposting this to the following groups, all of which have a dog in this hunt: SOC2008, Knight Foundation, Newspapers on Drupal, RSS & Aggregation.

News agencies, "legacy" newsroom management system implementors, publishers and archiving companies all support an XML standard called News Industry Text Format (NITF), developed by the International Press Telecommunications Council.

We need a robust, broadly supported common NITF feed handler that works with the Drupal FeedAPI framework, ultimately enabling loading of NITF data into CCK nodes with configurable entity mapping. This feed handler should expose its own API so that additional handlers can be added to process NewsML (primarily championed by Reuters) and Atom wrappers (used by the Associated Press in AP Exchange).

Summary:
I want to write two modules for Drupal as part of Google Summer of Code. One called meme_tracker and the other called machine_learning_api. The meme_tracker module will use the machine_learning_api to intelligently filter and group content from both internal and external content sources. The module's purpose is to find and display to a community in real time the most interesting conversations and memes within the community as they emerge.

Hello Drupliers. My name is Kyle Mathews. I'm a grad student in Information Systems at Brigham Young University working as a research assistant to several faculty members here. We are studying how and where social software can be used in education.

In the past eight months, I've built a number of classroom websites using Drupal. I've learned a considerable bit about Drupal in the process, became very involved in the community, and am writing a module (writing_assignment -- still a work in progress btw). I've loved most everything about Drupal but have found Drupal is missing an important component for building the perfect social learning website. This itches. So, in the best open-source fashion, I'm applying to Google Summer of Code to scratch my itch.

Added to official ideas list at http://drupal.org/node/237628

Overview: With Encrypted RSS/Atom feeds, buddylist-like features become possible cross-site. The project would be to develop a module which generates and consumes syndicated feeds, where reading them in only possible behind a login.

Hi,
I'm very new to drupal, and not a programmer... though I have several (without drupal experience) in my lab.

I would like to take the results of a pubmed search at NCBI, and use the feed in drupal.
I can get the RSS link to work nicely in NetNewsWire (Mac).
When I use drupal aggregator everything is displayed (abstract etc), I want to just list out the titles.
I've tried to use the NCBI controls to regulate the output, but it seems tat doesn't apply to the RSS feed which gets everything.

I made a very rough implementation of a module that just uses the nodeapi 'rss item' op to add cck fields as extra elements in the item namespace. In RSS 2.0, adding extra elements is allowed as long as they're w/in a namespace on the spec., and Drupal's nodeapi makes this pretty easy to do. My question is, is there already some functionality out there that does this?

I recently created an web site for a class at BYU that makes extensive use of FeedAPI. I was very pleased with FeedAPI as it allowed me to map each student's external blog to their Drupal blog.

Read the write-up here:
http://groups.drupal.org/node/8610

Has somebody out there played w/ SimplePie's Live service?

http://live.simplepie.org/

I would love to hear people's impressions.

Alex

The Feed Element Mapper maps now not only standard RSS tags to taxonomy, but also any RSS tag to taxonomy or CCK fields.

Check out the screencast here:

http://www.developmentseed.org/blog/2008/jan/24/pick-it-feed-stick-it-no...

and a step by step guide how to build your own mapper:

http://www.developmentseed.org/blog/2008/jan/24/build-your-own-mapper-fe...

I recently contacted spinn3r.com, and i will use their robot to spider the web, they provide a java client

I tried using the Feed URL with feedparser however the outcome was not great,

I want to try their java client

Does anyone know how to implement it
here are some links

http://spinn3r.com/documentation

http://code.google.com/p/spinn3r-client/

Has anybody bothered to take a look at the Google Ajax Feed API with the intention of using it for Drupal aggregation?

We've implemented client side aggregation for a project being released soon. This gives a similar experience to NetVibes.

I need a vary basic functionality to display story titles from an external feed as a list.
Each user have their own "profile page" which is a custom content type created with CCK. The content type has a Blog Feed URL field. I would like to parse 10 newest items from the feed and display them on the profile page without saving them into nodes. The perfect way actually would be - having aggregation of the items into nodes optional. This way users could choose if they want their content to be aggregated here, or just shown as links on their profile pages to their own content on their sites.

Hello, all,

As part of some work we're doing, I put together a proof of concept leveraging the functionality within the FeedAPI and Views.

It's a pretty simple site -- all it does is bring in some selected feeds, import any existing tags, and add on some additional tags set on each feed as it is added to the site.

But it starts to get fun when you use views to organize the content based on author, taxonomy, etc --

I was also amazed at how quickly the site came together -- like I said, it's basic, but the basic config came together in about 6 hours.

Yes, it's finally out!

http://drupal.org/project/simplefeed

More details of what has changed over the past 7 months since I introduced it:

http://tedserbinski.com/2007/11/16/simplefeed-1-0-ships

Example site processing over 300+ feeds every 15 min: http://www.momblognetwork.com/

Enjoy!

Over the last weekend I started to develop the Feed Element Mapper - a Feed API compatible module that ultimately should allow you to map any item on a feed to any field on your feed item node. Currently there is an implementation for taxonomies.

Check out the screencast on our blog here:

http://www.developmentseed.org/blog/2007/oct/30/pick-it-feed-stick-it-no...

and you can grab the module here:

http://cvs.drupal.org/viewvc.py/drupal/contributions/sandbox/alex_b/feed...

(I just upgraded it to FeedAPI 1.0 compatibility)

I am happy to announce that after a week of cleaning up and improving the API and adding some critical features, yesterday FeedAPI 1.0 beta 1 version came out. These are the features:

Embedded Media Field now ships with Embedded Media Import, for importing photosets, playlists, and the like. If you install that module, admins will have the ability to import photosets from Flickr as individual nodes. Just make sure the 'Allow import' checkbox on the content type (which must have a field defined with Embedded Media Field) is checked, proper permissions are given to editors, and look at the settings on the new tab at /admin/content/emfield/import.

Since the gods of aggregation are all here, and I've exhausted all other options, here goes my question:

In our project, we are dealing with aggregated feeds that have media enclosures in them. Photos, the occasional video, etc. How can one parse these files and use them?

My related post:
http://drupal.org/node/180385

Results:

• A release after the beta
• updated developer documentation (it changed a lot because of the concept changes)
• Screencast for users to make familiar with installing, setup and use the module

I have to admit, that i'm glad that the issue queue is full of feature requests, it seems that people want to use FeedAPI for various purposes. And yeah, their purposes are not impossible with FeedAPI (using the API, developers can write the proper modules)

The purpose of this wiki page is to provide help for developing for FeedAPI.

Again, I got lots of valuable responses from my mentors and Morbus, they really kindly follow my work and tell the problems and the suggestions.
You can install the FeedAPI modules to any Drupal 5 site and you can do the following:

I've noticed that on simplefeed and feedparser one runs out of memory if cron is set to parse a large number of feeds.

turns out this is due to a PHP bug http://bugs.php.net/bug.php?id=33595

SimplePie objects are full of circular and self references so basically no object gets destroyed until the end of the cron run, each feed you load and parse remains in memory until the end.

talking to developers @ #simplepie I don't think they'll be implementing any workarounds in the near future.

It's a question that I wanted to ask long before. Kreynen's story of how the University of Nevada could use an aggregator for covering the Tahoe fire makes me finally post: What are you using an aggregator for?

As some of you know, I work for the University of Nevada's Reynolds School of Journalism. I’ve been working with their graduate program to develop OurTahoe.org as well as individual graduate student projects looking at the new ways "Web2.0" technologies can be used by journalists. Some of the projects resulted in Drupal modules like Promise.

I discussed the further way of the Aggregation API SoC project with my mentors and the best way is to use SimpleFeed module as a base. There is a page http://aggregation.novaak.net/?q=node/224, where I tried to match the requirements of the project against what SimpleFeed does now. In addition I tried to sketch how SimpleFeed works now at http://aggregation.novaak.net/?q=node/225 . I contacted with the author of SimpleFeed, m3avrck, we'll arrange the efforts on the module. Here I would like to make the detailed plans of the planned modifications on SimpleFeed.

Previously I compared some feed parsers and aggregation-purpose modules to provide an outline of the current solutions. It will be helpful for me at the start of the summer of code project. At the project website you can find the devlog and details of the process of the work.. At this page I would like to collect the features what an aggregation modules should know and decide what's worth to include in the API.

There's a possibility for some development sponsorship for FeedParser system to allow each user to add RSS/ATOM feeds for aggregation.

So to make sure any work done is useful for others too - I wanted to gather any input on the best way to approach the implementation.

I noticed there's a CCK field available to add a new feed to Drupals aggregation process. So this could be extended to support FeedParser's FeedManager with a feed-management API for adding, deleting and forcing an update of a specific feed.

If you realize that a module or feature is missing, please edit the page!

Compared modules

• core Aggregator
• Aggregation
• SimpleFeed
• FeedParser
• Leech
• FeedAPI

I plan to compare the different XML feed parsers here in the viewpoint of functionality / speed and the interface that the API provides.
Please extend this page with any parsers / feed formats you think it's worth to take care.

This is a concept that hit me last night.

One of the issues we have with current Aggregator is that after a certain number of feeds, it starts to have problems finishing its cron cycle.

Part of this is caused by the fact that Aggregator does three actions during cron:

• Get feed
• Validate feed
• Parse feed

Really, aren't only the first two necessary during the cron run? Couldn't we save some cycles and boost performance by delaying the parsing stage until the data was requested by a user-initiated function (like an Aggregator block load)?

In working on the MySite project, I just wrote the following piece of code. It transfers aggregator feed images and writes them to a local directory.

My thought is this has security and performance benefits. But I may have missed something....

Feedback?

<?php
/**
* This function takes a Feed image and saves it locally.
* We do this for added security and speed.
* @param $fid == the feed id, taken from {aggregator_feed}
* @param $image == the image string taken from {aggregator_feed}
* @return $newfile == the filepath string pointing to the local copy of the file
*/

There are patches we can start creating and applying to core Drupal 6 aggregator today. Let's start getting them in one by in. Please flesh out requirements and notes. Link to related issues in queue. Feel free to move these items around or add extra comments. Would be great to see people pick these up and review all the others to see if we can get some great improvements in for 6 (and some are bugs that might be applied to 5.x).

See all aggregator issues.

To Do