aggregator

I just entered a proposal for the session that Aron Novak and I would like to hold in Szeged: a presentation of a new extensible aggregator in D7:

Proposal:
http://szeged2008.drupalcon.org/program/sessions/new-aggregator-drupal-7

New aggregator:
http://drupal.org/project/new_aggregator

Hi everyone,

I've been banging my head a little bit trying to get this sorted. Basically, I want members of my drupal site to have their own personal blogs and a section where they can read other members posts. This is obviously no problem if they're all using drupal blogs on my site.

However, many of the members already have their own blogs. It would be great if instead of submitting blog entries on my site, they could just add the url of their Blog feed and entries would automagically appear alongside other members entries.

Have a look at PHP skeleton code for better understanding this outline.

Def:
Parser: Responsible for creating a feed data structure which is expected by the aggregator (hook_aggregator_parser). By design additional parsers could consume everything (ical, html pages, emails)
Processor: Responsible for accepting items and save them, show them to the user, etc (hook_aggregator_processor)

Planned modules:

* aggregator
* aggregator_node
* aggregator_light

Here I summarize what you should expect from core aggregator for Drupal 7.

As maybe you heard, my application was accepted to write a new aggregator for Drupal core :) : http://code.google.com/soc/2008/drupal/appinfo.html?csaid=1222E090E875B3...
Here as a first step, I would like to collect lots of possible problems in aggregation area. Please share your opinion about these questions to help Drupal 7 to have as good aggregator as possible :) :

The items are not necessarily ordered by relevance.

Node creation

 

 
Problems:

• Drupal's core aggregator does not satisfy the requirements of state
  of the art aggregation applications. E. g. core aggregator lacks
  pluggable configurations or the possibility of creating nodes from
  feed items.

• There is an explosion of contrib modules that try to cover the
  deficiencies of Drupal's aggregator in one or the other way. Previous
  attempts to unify aggregation functionality had limited success.

Moved to official ideas list: http://drupal.org/node/237907

I want to propose a project for building a module that pulls news content from a global news service called Daylife (http://daylife.com)

Daylife is a global news aggregation service that offers an Open API platform called DayPI (http://developer.daylife.com) for any developer, blogger, news service to pull stories and photos about any topic in the world. The API returns data in XML, JSON and PHP serialized formats. You can read the Daylife About page at http://www.daylife.com/page/about to learn more about the company. All platform documentation is on http://developer.daylife.com/docs

I am happy to announce that after a week of cleaning up and improving the API and adding some critical features, yesterday FeedAPI 1.0 beta 1 version came out. These are the features:

This is a concept that hit me last night.

One of the issues we have with current Aggregator is that after a certain number of feeds, it starts to have problems finishing its cron cycle.

Part of this is caused by the fact that Aggregator does three actions during cron:

• Get feed
• Validate feed
• Parse feed

Really, aren't only the first two necessary during the cron run? Couldn't we save some cycles and boost performance by delaying the parsing stage until the data was requested by a user-initiated function (like an Aggregator block load)?

In working on the MySite project, I just wrote the following piece of code. It transfers aggregator feed images and writes them to a local directory.

My thought is this has security and performance benefits. But I may have missed something....

Feedback?

<?php
/**
* This function takes a Feed image and saves it locally.
* We do this for added security and speed.
* @param $fid == the feed id, taken from {aggregator_feed}
* @param $image == the image string taken from {aggregator_feed}
* @return $newfile == the filepath string pointing to the local copy of the file
*/