Aggregation API and Upcoming Requirements

BorisMann: I thought I'd take Ken's post on drupal.org and stick it in a wiki format here, then we can evolve it together.

What other steps do we want to take? An aggregation_api project on d.o. to coordinate around? I can also offer SVN space if needed.

csevb10: I vote for the aggregation_api project simply so we can start moving forward. If we can decide where we're gonna start we can then get it up and running and start developing some meaningful elements.

feed_form($settings = array('title', 'url'))
A simple form element that returns the TITLE and URL elements for adding a new feed to a Drupal site. Optionally, this function might return all the settings that an administrator has access to (including refresh rate and category settings).

feed_validate($feed)
So we can validate fields added to the feed form

feed_submit($feed)
A simple submit hook so that we can store new fields on the feed form

feed_verify($url)
A submit hook that will check a submitted URL to see if it returns a valid XML feed. A necessary stage in processing new feeds before saving.

feed_parse($feed)
Parses the feed into the elements defined by the aggregator API (see feed_store($feed), below).

feed_check($url)
Check a URL for duplication against existing feeds.

feed_update($feed)
Updates the requested feed.

feed_title($title)
A parsing routine that allows the module to grab the base element of the feed and use it instead of a user-supplied feed title.

feed_store($feed)
A standard hook for saving feed data into the Aggregator API tables. The standard data elements might be taken from the existing {aggregator_feed} table:

• fid (incremented by the API)
• title
• url
• refresh
• checked
• link (derived from feed_parse())
• description (derived)
• image (derived)
• etag (derived)
• modified (derived from feed_update())

feed_item_alter($feed, &$item)
A standard hook that gets fired after a feed has been parsed and before the feed items (as opposed to the feed description) are stored. It can be used to:

• auto-categorize items, or even...
• remove some items
• do some rewriting on items, like converting non-absolute links (think images)

Notes

• BorisMann: removed $title -- $url needs to be unique, title does not and shouldn't be
• csevb10: an area that has not been covered is caching of the feed. File-based caching is built into SimplePie, but I think we should define some way to determine/edit/override the caching for the original feed (selection between something like SimplePie's built in caching and user-defined caching). This may take some thinking to architect correctly.
• csevb10: added hooks feed_validate & feed_submit to allow contrib modules to add functionality for handling additions to the feed form. -- this is only necessary if we keep the non-node structure of current feeds (per aggregator.module vs simplefeed.module)

List of Upcoming Requirements (Contrib on top of Aggregation API)

• parse XML and map to CCK fields
• feed items as nodes (generic)
• < category > elements mapped to taxonomy terms
• to use a selected parser (e.g. choose between built in Drupal parser, simplepie, or ? -- per feed parsing?)

Requirements

• We should be able to change the parser, and to define extra fields/handling for parsers as well (feed_parse)
• We should be able to create modules that add additional functionality to the base feed forms (form_alter, feed_validate, feed_submit)
• We should be able to define storage (override or in addition) for the feed and the feed items (feed_store, feed_item_store(?))
• The new solution should optimize performance or allow to override download functionality to optimize download/node creation performance (e. g. adapt feed download frequency to update frequency of feed)

Questions that need answers to proceed

• Are we building this from scratch? Beginning from Aggregator Module? Simplefeed? Something else?