Drupal currently lacks any easy way for contributed modules to access the additional fields found in "enhanced" RSS feeds. Such fields include the enclosure tag which is used for podcasting and videocasting.
It has been suggested on IRC #drupal that the aggregator.module should be broken down to form a feedparser.module which does the grunt work and the aggregator.module which handles the management of the feed items.
The feedparser.module would simply accept a feed url, fetch the feed XML, parse it and normalize the data, then finally return an array of items.
An initial suggestion for the feedparser library can be found here in Ber Kessels's sandbox.
I have investigated the use of the MagpieRSS parser, but this doesn't include any enclosure support either - so offers little in the way of additional benefits to the existing Drupal code found in aggregator.module.
Comments
Atom?
I'm new here, so don't know what work has been done previous to this, so apologies if I retread old ground or step on any toes... I'm the developer of PEAR::XML_Feed_Parser and a relative newcomer to drupal, but working with it quite extensively at the moment.
My initial concern with that particular library is the absence of any mention of atom. I don't want to ignite any format-wars, but I'm a big fan of the atom spec and as it begins to catch on (and the Atom Publishing Protocol finally stablises), it would definitely be a shame to not offer full support.
Most of my PHP XML work has been using DOM and SimpleXML in PHP5. I can see that there are definite advantages to implementing a SAX-based feed parser as it should allow much more liberal parsing, but is there any mileage in considering a system that will use a DOM-based approach based on certain criteria (environment is PHP5, feed is well-formed, etc)?
How much work has been done on the data model for managing the feed items once they've been parsed? Will that be down to the site administrators to manage or will there be a generic container that matches up the data that can currently be carried by the various feed formats?
Your very informed feedback
Your very informed feedback and further suggestions are most welcome! I'm also a big fan of Atom
A few notes:
* needs to work under PHP4
* a "native" PHP solution is preferred, rather than relying on external library dependencies; what this essentially means for core inclusion is that any module which has external dependencies is unlikely to be included
Mmmm....data model. I'd love to see mix down into Atom natively. Maybe even an Atom Store?
An Atom Store would seem a
An Atom Store would seem a great idea and my experience so far has been that the properties of other feed types can be mapped to atom without much trouble (that's how the API of XML_Feed_Parser is structured, where atom is considered normative). If the feed formats were mapped to database tables, there would need to be a clear method of handling extensions, but I think that should be quite manageable.
wrt PHP4 vs. PHP5 -- are there conventions in drupal regarding running parallel code for PHP4 and PHP5? I don't know for sure what the performance differences are for the different XML parsers, but if substantially better performance could be offered to PHP5 users would a parallel implementation be appropriate?
Parallel development
For sure...parallel development is absolutely appropriate.
Is this a matter of atom store vs. database table mapping, or do we need to do both. Could we store and manipulate the Atom Store on disk, then cache particular posts/feeds in the database table?
Anyway, thanks for the discussion, I should probably stop handwaving here and go do some more reading about Atom Store and do some real system architecture work.
Performance/Complexity
In theory everything could be done with an atom store on disk and some sort of XPath or other XML query API running against it, but I'd guess it would perform better, be less complex, and integrate with drupal more cleanly if there was at least a cache of items in the database.
When designing the architecture it might well be worth skimming the current draft of the Publishing Protocol to see how close we could get to supporting it out of the box?
PHP4 parser
For Drupal core, we avoid writing code that is specific to one version of PHP. There is no PHP5 specific code in Drupal, other than what is strictly necessary.
Atom Store = outside of the database? If so, some concerns.
(I dont know more about Atom Store than what I read in this thread...)
From the posts here it sounds like if choosing to use the Atom Store ("AS"), the feed items would be stored on the file system and not in the database. (is AS = file system only?)
For news items from other sites, which only should "roll through" and not be kept unless commented upon, etc., storing outside the database is probably fine (best) - as long as the data is not from a source that (potentially) contains sensitive information. RSS may - and increasingly will - come from any source, not only from public news sites. So my concern here is mostly related to the security of where the data is stored.
Further, some feeds receive content from own blogs and other such sites that should be kept inside the database, both for security of the data themselves and to simplify backups (some of the feeds themselves them serve as backups from other sites).
Finally, I hope any further development can provide the optional features that mainly distinguish Aggregator from Aggregator2: choosing per feed whether to make nodes of the feed items or not.
Can the Atom Store handle a bunch of such concerns? As I dont know the AS, perhaps this question is irrelevant. (I am not a programmer) Perhaps this is dealt with by the module and based on the settings for each feed, some of the content end up in an "Atom Store", and some go into the db as with the current Aggregator and Aggregator2 modules? If so, definetely great to be able to shove normal news items out of the db.
Finally: one disturbing effect of the Aggregator2 module is that when storing as nodes, the feed items clutter up the notification emails. I really hope any new developments can provide options like not include a feed item in the email notifications until it receives comment(s), for example. That example setting is relevant for a new agg.module, although the other aspect of it is an issue for the Notification module.
Somewhat related: some security concerns could also be resolved by a setting that specified that when a feed item receives a comment, it is pulled out of the AS and filed as a node in the database.
Btw - off topic:
Much missed function: flexible "blog this" feature: quote an item "into a new node" of some on-the-fly-selected type.
Feedback
If you want to aim for core (and IMO you should):
Hope that helps!
+1 on an include
Yes...an include is a way to go. I hesitate to say it, but....a feed API.
But, feed parsing is hard to do right. The universal feed parser has 3000 unit tests. Except, of course, it's written in Python...
I'd rather collaborate with other PHP projects on a base-level feed parsing implementation than have Drupal strive to maintain a feed parser that is mediocre at best. Maybe I'll talk to Zak at ezPublish about this.
MagpieRSS
We carefully investigated the use of MagpieRSS and decided not to use it; see mailing list archives for details. There was too much code duplication going on, and we would have to fork the code to make it properly interact with Drupal. This is quite a bit of work that is prone to security issues (eg. MagpieRSS's security model didn't match Drupal's). While looking at the MagpieRSS code we also identified several security issues, which didn't gave us much confidence. In general, we try to avoid the use of 3rd party code in Drupal core. If there is an aggregator.inc for core, one can probably build one on top of MagpieRSS, if they choose to.
Unit Tests
I did some work on a parser to convert the Universal Feed Parser tests to PHP for testing XML_Feed_Parser. It's not a clean codebase and is a long way from comprehensive, but the last version of the converted tests can be found at:
http://cvs.php.net/viewcvs.cgi/pear/XML_Feed_Parser/tests/convertedtests/
If anyone wants a copy of the converter code, send me an email and I'll send it across.
DOMIT parser
I agree that using a 3rd party feed parser must be the sensible way forward. However Dries comments about security issues in external developers code would make it a bit too risky to include in Drupal core ... which is where all this RSS aggregator work is heading one day hopefully.
I had a quick look at what other CMS projects are using, Joomla and SugarCRM are using DOMIT!.
I've not reviewed the code yet so not sure if it is even worth giving DOMIT the time of day. Just the fact that Joomla uses it puts me off :)
No atom?
From a quick scan of DOMIT, it doesn't look like it supports atom at all, which imho should be a showstopper.
This is a good discussion
I'm very interested in having the ability to attach feeds to nodes either through the creation of a new module or as a content type to be used with CCK. I am not a programer but I can help with documentation and development in terms of funding.
I work in the online distribution and branding and maintain a rather large network of websites. I would like to create an online directory as a community resource for people to explore the web. Every profile in the directory ideally would be a simple node type with a title, description, some other attributes, and most importantly the current feeds from a site.
I'm very interested in this group and your efforts and would like to help in some way. Pleaes feel free to contact me directly.