RSS & Aggregation

Working on the improvement of the RSS/ATOM aggregation provided by Drupal.

alaa's picture

simplepie memory usage

I've noticed that on simplefeed and feedparser one runs out of memory if cron is set to parse a large number of feeds.

turns out this is due to a PHP bug http://bugs.php.net/bug.php?id=33595

SimplePie objects are full of circular and self references so basically no object gets destroyed until the end of the cron run, each feed you load and parse remains in memory until the end.

talking to developers @ #simplepie I don't think they'll be implementing any workarounds in the near future.

alex_b's picture

What are you using your aggregator for? A call for use cases.

It's a question that I wanted to ask long before. Kreynen's story of how the University of Nevada could use an aggregator for covering the Tahoe fire makes me finally post: What are you using an aggregator for?

Read more
kreynen's picture

Providing Coverage of the Tahoe Fire Without Reporters

As some of you know, I work for the University of Nevada's Reynolds School of Journalism. I’ve been working with their graduate program to develop OurTahoe.org as well as individual graduate student projects looking at the new ways "Web2.0" technologies can be used by journalists. Some of the projects resulted in Drupal modules like Promise.

Read more

Design on SimpleFeed

I discussed the further way of the Aggregation API SoC project with my mentors and the best way is to use SimpleFeed module as a base. There is a page http://aggregation.novaak.net/?q=node/224, where I tried to match the requirements of the project against what SimpleFeed does now. In addition I tried to sketch how SimpleFeed works now at http://aggregation.novaak.net/?q=node/225 . I contacted with the author of SimpleFeed, m3avrck, we'll arrange the efforts on the module. Here I would like to make the detailed plans of the planned modifications on SimpleFeed.

Read more

Aggregation API requirements - SoC project

Previously I compared some feed parsers and aggregation-purpose modules to provide an outline of the current solutions. It will be helpful for me at the start of the summer of code project. At the project website you can find the devlog and details of the process of the work.. At this page I would like to collect the features what an aggregation modules should know and decide what's worth to include in the API.

Read more
budda's picture

Per User Feed Management

There's a possibility for some development sponsorship for FeedParser system to allow each user to add RSS/ATOM feeds for aggregation.

So to make sure any work done is useful for others too - I wanted to gather any input on the best way to approach the implementation.

I noticed there's a CCK field available to add a new feed to Drupals aggregation process. So this could be extended to support FeedParser's FeedManager with a feed-management API for adding, deleting and forcing an update of a specific feed.

Read more

Modules for aggregation

If you realize that a module or feature is missing, please edit the page!

Compared modules

  • core Aggregator
  • Aggregation
  • SimpleFeed
  • FeedParser
  • Leech
  • FeedAPI
Read more

Feed parsers comparison

I plan to compare the different XML feed parsers here in the viewpoint of functionality / speed and the interface that the API provides.
Please extend this page with any parsers / feed formats you think it's worth to take care.

Read more
agentrickard's picture

Do not parse feeds on acquisition

This is a concept that hit me last night.

One of the issues we have with current Aggregator is that after a certain number of feeds, it starts to have problems finishing its cron cycle.

Part of this is caused by the fact that Aggregator does three actions during cron:

  • Get feed
  • Validate feed
  • Parse feed

Really, aren't only the first two necessary during the cron run? Couldn't we save some cycles and boost performance by delaying the parsing stage until the data was requested by a user-initiated function (like an Aggregator block load)?

Read more
agentrickard's picture

Feed Icons

In working on the MySite project, I just wrote the following piece of code. It transfers aggregator feed images and writes them to a local directory.

My thought is this has security and performance benefits. But I may have missed something....

Feedback?

<?php
/**
* This function takes a Feed image and saves it locally.
* We do this for added security and speed.
* @param $fid == the feed id, taken from {aggregator_feed}
* @param $image == the image string taken from {aggregator_feed}
* @return $newfile == the filepath string pointing to the local copy of the file
*/

Read more

Patches to Core Aggregator

There are patches we can start creating and applying to core Drupal 6 aggregator today. Let's start getting them in one by in. Please flesh out requirements and notes. Link to related issues in queue. Feel free to move these items around or add extra comments. Would be great to see people pick these up and review all the others to see if we can get some great improvements in for 6 (and some are bugs that might be applied to 5.x).

See all aggregator issues.

To Do

Read more

Aggregation API and Upcoming Requirements

BorisMann: I thought I'd take Ken's post on drupal.org and stick it in a wiki format here, then we can evolve it together.

What other steps do we want to take? An aggregation_api project on d.o. to coordinate around? I can also offer SVN space if needed.

csevb10: I vote for the aggregation_api project simply so we can start moving forward. If we can decide where we're gonna start we can then get it up and running and start developing some meaningful elements.

Read more

Aggregation / Feed parsing Modules Comparison

A summary of feed-related modules and code.
Page needs updating

Feeds

Source: http://drupal.org/project/feeds
Description: Import or aggregate data as nodes, users, taxonomy terms or simple database records.
Status: Actively maintained

Aggregator

Description: core module that parses / aggregates feeds as non-nodes.

Current status:

  • needs more flexible feed handling -- any invalid XML causes it to barf
  • category system needs replacing with core taxonomy format
  • more hooks to handle different types of feeds needed
Read more
alex_b's picture

A fruit of OSCMS - aggregator API proposed

agentrickard proposed an aggregator API here:

http://drupal.org/node/130942

alex_b's picture

Session about Aggregation in Drupal at the Drupalcon in California

I just proposed a session on aggregation in Drupal for this year's Drupalcon on March 22nd and 23rd.

Check it out here:

http://2007.oscms-summit.org/node/100

Would be great if we could bring all aggregation heads together to get a better understanding of what's going on in Drupal's aggregation land, learn from each other's use cases and try to figure out how to sail towards more unified solutions.

Alex

budda's picture

Managing large numbers of feeds

From the issue queue of Feedparser it came to my attention that a lot of Drupal sites are managing the parsing of hundreds of feeds. Now that I've added OPML import functionality to Feedparser I've been adding lots of feeds - which got me wondering...

What's the best way to manage the ?q=admin/aggregator page listings?
How should feeds be organised? Currently it's just a single page sorted A-Z.

Rosamunda's picture

What Module is used to Drupal.org feeds?

Anyone´s knows what´s the module used to get these < ahref="http://drupal.org/node/63589">feeds?

ie:
http://drupal.org/security/rss.xml
http://drupal.org/taxonomy/term/14/0/feed

There are different formats of the feed...

Thanks!

Rosamunda

bonobo's picture

Slightly OT: Gregarius

Hello, all,

Is anybody familiar with an app named Gregarius? It's a standalone app that describes itself as a"Free, Web-based feed aggregator"

From the Features List:

*  Completely web-based and runs on your web server
* Supports RDF, RSS, ATOM feeds
* Imports and exports OPML
* Simple, password-protected, web-based administration and configuration
* AJAX powered tagging of feeds and items
* Supports themes and plugins
* Search in your feeds
* Basic i18n support
Read more
budda's picture

Feature requests for Feedparser

I'm looking for any input from interested parties on what features they like to see in the Feedparser package. I want to use the list to prioritise development, and to get an idea of what things people want out of an RSS aggregation system.

Please post your requests to the project issue tracker.

Boris Mann's picture

Feed parsing overview in a wiki

Hi all -- I put together a short overview of current modules here: https://svn.bryght.com/dev/wiki/DrupalFeedParsing

Feel free to edit or add comments. It was meant to easily show in one place the feed aggregation / node feed creating thingies in one place. We recently wrote a very simple feed_node module that uses the core aggregator.

Budda's feedparser looks good, but basically we need more input and a concerted effort, as well as a migration plan / core integration options.

Subscribe with RSS Syndicate content

RSS & Aggregation

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: