RSS & Aggregation

This group should probably have more organizers. See documentation on this recommendation.

Working on the improvement of the RSS/ATOM aggregation provided by Drupal.

Modules for aggregation

Posted by Aron Novak on June 14, 2007 at 2:13pm
Last updated by wolfderby on Fri, 2007-12-14 05:40

If you realize that a module or feature is missing, please edit the page!

Compared modules

core Aggregator
Aggregation
SimpleFeed
FeedParser
Leech
FeedAPI

Feed parsers comparison

Posted by Aron Novak on June 12, 2007 at 12:11pm
Last updated by Skyzyx on Wed, 2007-06-20 21:29

I plan to compare the different XML feed parsers here in the viewpoint of functionality / speed and the interface that the API provides.
Please extend this page with any parsers / feed formats you think it's worth to take care.

Do not parse feeds on acquisition

Posted by agentrickard on May 31, 2007 at 1:33pm

This is a concept that hit me last night.

One of the issues we have with current Aggregator is that after a certain number of feeds, it starts to have problems finishing its cron cycle.

Part of this is caused by the fact that Aggregator does three actions during cron:

Get feed
Validate feed
Parse feed

Really, aren't only the first two necessary during the cron run? Couldn't we save some cycles and boost performance by delaying the parsing stage until the data was requested by a user-initiated function (like an Aggregator block load)?

Feed Icons

Posted by agentrickard on April 18, 2007 at 5:26pm

In working on the MySite project, I just wrote the following piece of code. It transfers aggregator feed images and writes them to a local directory.

My thought is this has security and performance benefits. But I may have missed something....

Feedback?

<?php
/**
* This function takes a Feed image and saves it locally.
* We do this for added security and speed.
* @param $fid == the feed id, taken from {aggregator_feed}
* @param $image == the image string taken from {aggregator_feed}
* @return $newfile == the filepath string pointing to the local copy of the file
*/

Patches to Core Aggregator

Posted by Boris Mann on April 6, 2007 at 10:28pm
Last updated by csevb10 on Tue, 2007-04-10 21:16

There are patches we can start creating and applying to core Drupal 6 aggregator today. Let's start getting them in one by in. Please flesh out requirements and notes. Link to related issues in queue. Feel free to move these items around or add extra comments. Would be great to see people pick these up and review all the others to see if we can get some great improvements in for 6 (and some are bugs that might be applied to 5.x).

See all aggregator issues.

To Do

Aggregation API and Upcoming Requirements

Posted by Boris Mann on April 6, 2007 at 5:54pm
Last updated by fgm@drupal.org on Fri, 2007-05-25 11:36

BorisMann: I thought I'd take Ken's post on drupal.org and stick it in a wiki format here, then we can evolve it together.

What other steps do we want to take? An aggregation_api project on d.o. to coordinate around? I can also offer SVN space if needed.

csevb10: I vote for the aggregation_api project simply so we can start moving forward. If we can decide where we're gonna start we can then get it up and running and start developing some meaningful elements.

Aggregation / Feed parsing Modules Comparison

Posted by Boris Mann on March 30, 2007 at 4:08pm
Last updated by scottrigby on Wed, 2011-09-21 16:24

A summary of feed-related modules and code.
Page needs updating

Feeds

Source: http://drupal.org/project/feeds
Description: Import or aggregate data as nodes, users, taxonomy terms or simple database records.
Status: Actively maintained

Aggregator

Description: core module that parses / aggregates feeds as non-nodes.

Current status:

needs more flexible feed handling -- any invalid XML causes it to barf
category system needs replacing with core taxonomy format
more hooks to handle different types of feeds needed

A fruit of OSCMS - aggregator API proposed

Posted by alex_b on March 26, 2007 at 4:47pm

agentrickard proposed an aggregator API here:

http://drupal.org/node/130942

Session about Aggregation in Drupal at the Drupalcon in California

Posted by alex_b on February 10, 2007 at 11:28pm

I just proposed a session on aggregation in Drupal for this year's Drupalcon on March 22nd and 23rd.

Check it out here:

http://2007.oscms-summit.org/node/100

Would be great if we could bring all aggregation heads together to get a better understanding of what's going on in Drupal's aggregation land, learn from each other's use cases and try to figure out how to sail towards more unified solutions.

Alex

Managing large numbers of feeds

Posted by budda on January 27, 2007 at 8:23pm

From the issue queue of Feedparser it came to my attention that a lot of Drupal sites are managing the parsing of hundreds of feeds. Now that I've added OPML import functionality to Feedparser I've been adding lots of feeds - which got me wondering...

What's the best way to manage the ?q=admin/aggregator page listings?
How should feeds be organised? Currently it's just a single page sorted A-Z.

What Module is used to Drupal.org feeds?

Posted by Rosamunda on January 17, 2007 at 4:04pm

Anyone´s knows what´s the module used to get these < ahref="http://drupal.org/node/63589">feeds?

ie:
http://drupal.org/security/rss.xml
http://drupal.org/taxonomy/term/14/0/feed

There are different formats of the feed...

Thanks!

Rosamunda

Slightly OT: Gregarius

Posted by bonobo on December 13, 2006 at 5:13pm

Hello, all,

Is anybody familiar with an app named Gregarius? It's a standalone app that describes itself as a"Free, Web-based feed aggregator"

From the Features List:

*  Completely web-based and runs on your web server
* Supports RDF, RSS, ATOM feeds
* Imports and exports OPML
* Simple, password-protected, web-based administration and configuration
* AJAX powered tagging of feeds and items
* Supports themes and plugins
* Search in your feeds
* Basic i18n support

Feature requests for Feedparser

Posted by budda on October 16, 2006 at 11:37pm

I'm looking for any input from interested parties on what features they like to see in the Feedparser package. I want to use the list to prioritise development, and to get an idea of what things people want out of an RSS aggregation system.

Please post your requests to the project issue tracker.

Feed parsing overview in a wiki

Posted by Boris Mann on September 27, 2006 at 4:39pm

Hi all -- I put together a short overview of current modules here: https://svn.bryght.com/dev/wiki/DrupalFeedParsing

Feel free to edit or add comments. It was meant to easily show in one place the feed aggregation / node feed creating thingies in one place. We recently wrote a very simple feed_node module that uses the core aggregator.

Budda's feedparser looks good, but basically we need more input and a concerted effort, as well as a migration plan / core integration options.

IPTC News Archetechture (NAR)

Posted by Max Bell on September 27, 2006 at 11:15am

http://www.iptc.org/NAR/

Greetings, fellow standards junkies:

I stumbled across the site above while looking to see what was being done with aggregation these days. For anyone who's ever given any thought to how much IE6 is to CSS like RSS 2.0 is to aggregation, it might make sense if I admitted I originally bookmarked the site above for future review and subconsciously repressed the memory as a hoax intended to torment me.

If it was some big deal, why wasn't anybody talking about it? Little did I know...

To be completely serious, though, I've been thinking a lot these past couple of months about what could be done to make Drupal more attractive to publishers looking to convert their operations to the web or folks looking to create web-based publications, and there's been growing interest in the subject. Given the spike in growth seen in the community following the DeanSpace/CivicSpace project and how popular Drupal has become with non-profits and the like, I've been thinking a complimentary project to the ecommerce suite aimed at publishers might both increase Drupal's popularity as a commercial platform and drive new business to contractors.

MagpieRSS library archives discussion

Posted by budda on May 28, 2006 at 3:47am

For newcomers to the RSS topic I dug out an old thread from the Drupal mailing list which discusses the use of MagpieRSS library in Drupal. Interesting read - it was before my time.

http://lists.drupal.org/archives/development/2004-07/msg00400.html

Feedparser.module

Posted by budda on May 16, 2006 at 1:22am

Drupal currently lacks any easy way for contributed modules to access the additional fields found in "enhanced" RSS feeds. Such fields include the enclosure tag which is used for podcasting and videocasting.

It has been suggested on IRC #drupal that the aggregator.module should be broken down to form a feedparser.module which does the grunt work and the aggregator.module which handles the management of the feed items.

The feedparser.module would simply accept a feed url, fetch the feed XML, parse it and normalize the data, then finally return an array of items.

An initial suggestion for the feedparser library can be found here in Ber Kessels's sandbox.

Subscribe with RSS