Working on the improvement of the RSS/ATOM aggregation provided by Drupal.
simplepie memory usage
I've noticed that on simplefeed and feedparser one runs out of memory if cron is set to parse a large number of feeds.
turns out this is due to a PHP bug http://bugs.php.net/bug.php?id=33595
SimplePie objects are full of circular and self references so basically no object gets destroyed until the end of the cron run, each feed you load and parse remains in memory until the end.
talking to developers @ #simplepie I don't think they'll be implementing any workarounds in the near future.
What are you using your aggregator for? A call for use cases.
It's a question that I wanted to ask long before. Kreynen's story of how the University of Nevada could use an aggregator for covering the Tahoe fire makes me finally post: What are you using an aggregator for?
Read moreProviding Coverage of the Tahoe Fire Without Reporters
As some of you know, I work for the University of Nevada's Reynolds School of Journalism. I’ve been working with their graduate program to develop OurTahoe.org as well as individual graduate student projects looking at the new ways "Web2.0" technologies can be used by journalists. Some of the projects resulted in Drupal modules like Promise.
Read moreDesign on SimpleFeed
I discussed the further way of the Aggregation API SoC project with my mentors and the best way is to use SimpleFeed module as a base. There is a page http://aggregation.novaak.net/?q=node/224, where I tried to match the requirements of the project against what SimpleFeed does now. In addition I tried to sketch how SimpleFeed works now at http://aggregation.novaak.net/?q=node/225 . I contacted with the author of SimpleFeed, m3avrck, we'll arrange the efforts on the module. Here I would like to make the detailed plans of the planned modifications on SimpleFeed.
Read moreAggregation API requirements - SoC project
Previously I compared some feed parsers and aggregation-purpose modules to provide an outline of the current solutions. It will be helpful for me at the start of the summer of code project. At the project website you can find the devlog and details of the process of the work.. At this page I would like to collect the features what an aggregation modules should know and decide what's worth to include in the API.
Read morePer User Feed Management
There's a possibility for some development sponsorship for FeedParser system to allow each user to add RSS/ATOM feeds for aggregation.
So to make sure any work done is useful for others too - I wanted to gather any input on the best way to approach the implementation.
I noticed there's a CCK field available to add a new feed to Drupals aggregation process. So this could be extended to support FeedParser's FeedManager with a feed-management API for adding, deleting and forcing an update of a specific feed.
Read moreModules for aggregation
If you realize that a module or feature is missing, please edit the page!
Compared modules
- core Aggregator
- Aggregation
- SimpleFeed
- FeedParser
- Leech
- FeedAPI
Feed parsers comparison
I plan to compare the different XML feed parsers here in the viewpoint of functionality / speed and the interface that the API provides.
Please extend this page with any parsers / feed formats you think it's worth to take care.
Do not parse feeds on acquisition
This is a concept that hit me last night.
One of the issues we have with current Aggregator is that after a certain number of feeds, it starts to have problems finishing its cron cycle.
Part of this is caused by the fact that Aggregator does three actions during cron:
- Get feed
- Validate feed
- Parse feed
Really, aren't only the first two necessary during the cron run? Couldn't we save some cycles and boost performance by delaying the parsing stage until the data was requested by a user-initiated function (like an Aggregator block load)?
Read moreFeed Icons
In working on the MySite project, I just wrote the following piece of code. It transfers aggregator feed images and writes them to a local directory.
My thought is this has security and performance benefits. But I may have missed something....
Feedback?
<?php
/**
* This function takes a Feed image and saves it locally.
* We do this for added security and speed.
* @param $fid == the feed id, taken from {aggregator_feed}
* @param $image == the image string taken from {aggregator_feed}
* @return $newfile == the filepath string pointing to the local copy of the file
*/
Patches to Core Aggregator
There are patches we can start creating and applying to core Drupal 6 aggregator today. Let's start getting them in one by in. Please flesh out requirements and notes. Link to related issues in queue. Feel free to move these items around or add extra comments. Would be great to see people pick these up and review all the others to see if we can get some great improvements in for 6 (and some are bugs that might be applied to 5.x).
To Do
Read moreAggregation API and Upcoming Requirements
Last updated by fgm@drupal.org on Fri, 2007-05-25 11:36
BorisMann: I thought I'd take Ken's post on drupal.org and stick it in a wiki format here, then we can evolve it together.
What other steps do we want to take? An aggregation_api project on d.o. to coordinate around? I can also offer SVN space if needed.
csevb10: I vote for the aggregation_api project simply so we can start moving forward. If we can decide where we're gonna start we can then get it up and running and start developing some meaningful elements.
Read moreAggregation / Feed parsing Modules Comparison
A summary of feed-related modules and code.
Page needs updating
Feeds
Source: http://drupal.org/project/feeds
Description: Import or aggregate data as nodes, users, taxonomy terms or simple database records.
Status: Actively maintained
Aggregator
Description: core module that parses / aggregates feeds as non-nodes.
Current status:
- needs more flexible feed handling -- any invalid XML causes it to barf
- category system needs replacing with core taxonomy format
- more hooks to handle different types of feeds needed
A fruit of OSCMS - aggregator API proposed
agentrickard proposed an aggregator API here:
Session about Aggregation in Drupal at the Drupalcon in California
I just proposed a session on aggregation in Drupal for this year's Drupalcon on March 22nd and 23rd.
Check it out here:
http://2007.oscms-summit.org/node/100
Would be great if we could bring all aggregation heads together to get a better understanding of what's going on in Drupal's aggregation land, learn from each other's use cases and try to figure out how to sail towards more unified solutions.
Alex
Managing large numbers of feeds
From the issue queue of Feedparser it came to my attention that a lot of Drupal sites are managing the parsing of hundreds of feeds. Now that I've added OPML import functionality to Feedparser I've been adding lots of feeds - which got me wondering...
What's the best way to manage the ?q=admin/aggregator page listings?
How should feeds be organised? Currently it's just a single page sorted A-Z.
What Module is used to Drupal.org feeds?
Anyone´s knows what´s the module used to get these < ahref="http://drupal.org/node/63589">feeds?
ie:
http://drupal.org/security/rss.xml
http://drupal.org/taxonomy/term/14/0/feed
There are different formats of the feed...
Thanks!
Rosamunda
Slightly OT: Gregarius
Hello, all,
Is anybody familiar with an app named Gregarius? It's a standalone app that describes itself as a"Free, Web-based feed aggregator"
From the Features List:
* Completely web-based and runs on your web server
* Supports RDF, RSS, ATOM feeds
* Imports and exports OPML
* Simple, password-protected, web-based administration and configuration
* AJAX powered tagging of feeds and items
* Supports themes and plugins
* Search in your feeds
* Basic i18n support
Read more Feature requests for Feedparser
I'm looking for any input from interested parties on what features they like to see in the Feedparser package. I want to use the list to prioritise development, and to get an idea of what things people want out of an RSS aggregation system.
Please post your requests to the project issue tracker.
Feed parsing overview in a wiki
Hi all -- I put together a short overview of current modules here: https://svn.bryght.com/dev/wiki/DrupalFeedParsing
Feel free to edit or add comments. It was meant to easily show in one place the feed aggregation / node feed creating thingies in one place. We recently wrote a very simple feed_node module that uses the core aggregator.
Budda's feedparser looks good, but basically we need more input and a concerted effort, as well as a migration plan / core integration options.






