Posted by cwgordon7 on March 18, 2008 at 6:24am
Moved to official ideas list at http://drupal.org/node/236135
This was originally a DROP task, but it is completely out of scope for that. So I am proposing it instead as an SoC project.
The idea is to make the functionality provided by yahoo pipes (see http://pipes.yahoo.com/) possible from within Drupal.
The module would need to:
- Have an API function that cleanly executes all the steps in the pipe.
- Have an interface for organizing the pipes' various inputs and outputs. While this does not have to be as fancy as yahoo pipes, it does need to at least avoid thousands of page reloads and millions of nested fieldsets.
- Have a hook that allows modules to register candidate steps with the pipe.
- Have implementations of the hook for useful parts of core
- Make the core module do absolutely nothing but construct the pipe and execute the pipe; doing stuff with the executed data needs to be able to be done through separate modules.
- Have two additional sub-modules, one for displaying the output of a pipe as a page, the other for displaying the output of a pipe as a block.
Why this would make a good SoC task:
- Because if we had this, Drupal would just be even more awesome.
- Because this is a fun task that any student would just love to grab.
- Because it's a bit challenging, and thus great for students who want to go above and beyond.
- Because there's a lot of room for expansion even if the base requirements are met.
- Because the base requirements are easily adjustable, so if this is deemed to difficult or too easy, they can be modified.
- Because I'm here and ready to maintain it after SoC is over, even if the student does not want to.
Mentors:
cwgordon7
...plus anyone else who wants to.
Difficulty:
Hard. Very hard. But not too hard.
Comments
Wow. Ambitious. You're
Wow. Ambitious. You're right, though, it would be fun. If this gets adopted, I may up for the challenge, provided my own proposal doesn't stick. I've only used Yahoo! Pipes moderately for a couple weeks, so I'm not sure of all its possibilities, but I'm assuming they're endless. How far along would you like to see this project in three months?
Casey
CCK + Views + GMaps + Media + Location
I made something similar to this using CCK + Views + GMaps + Media + Location.
http://fingerapps.com/locationmaps
Only thing thats missing is the aggregation of RSS feeds but thats possible.
You can use View's fields and filters to duplicate the creation of pipes in yahoo which is basically a query builder. You can also use computed fields CCK and other similar modules to fine tune your View filters.
Sounds like this might be an API/UI around CCK/Views/etc.
It's true that basically all of Yahoo! pipes functionality is do-able with these modules, but I think the idea is to make it easier to click something together than going into 50 different admin panels.
Charlie, can you confirm?
Perhaps I didn't explain myself clearly...
This would not duplicate the aggregator-location-whatever functionality of yahoo pipes. Instead it would be a framework for complex input-output systems in Drupal. This would include doing stuff with nodes, users, taxonomy terms, etc.—perhaps even the aggregator or location modules—but it would in essence, yes, be an easy way of organizing and executing a series of steps already provided by Drupal.
Does that make sense?
My biggest worry is that
My biggest worry is that this still feels a bit undefined, open-ended, and far-reaching. However it is a mind-blowingly cool idea and just the kind of thing that will attract serious passionate students to Drupal. I am also heartenede that cwgordon07 has volunteered to maintain the project when its done. A tentative +1 here.
I am interested in this idea
Hello, Charlie
I am interested in your idea. Can we discuss some details about it via email?
let's keep it public
Generally speaking (and especially in this instance) keeping discussions public is the better way to go. It means that more folks can help with ideas and also allows others to pick up with the work and keep running if either you or cwgordon7 lose interest in the project.
So, I encourage both of you to talk here so we can all benefit from it.
--
Open Prediction Markets | Drupal Dashboard
knaddison blog | Morris Animal Foundation
Agreed...
Public also means that when it comes time to evaluate your proposals, we can look back at the discussions you've had and see evidence that you've asked around about your idea, worked to further spec it out, etc. It also lets everyone provide feedback. :)
Not via email, but...
What questions do you have? :) I'm happy to clarify any questions you might have on this thread. I am also generally available in irc at #drupal.
This is just the sort of pie
This is just the sort of pie in the sky half baked proposal that leaves us with some demo code atthe end of the summer and absolutely nobody ever uses it on a live site. We have a lot of these experiences with SoC over the years (e.g. Social Network Analysis, WebDAV server, GUI form builder, etc.). Lets try to focus on small, well defined projects. Sorry to be a bummer, but I think this is an important point.
Check out existing approaches and refine
I like the idea.
We've got something in Drupal that is very close to this: FeedAPI.
FeedAPI has a concept of parsers and processors that are being configured in a content type.
A typical configuration would be e. g.:
What FeedAPI does not provide is a 'filter-like' architecture like pipes. There is no formalized passing on of processor provided data between processors.
Just today I stumbled over a problem that MIGHT be addressed by making FeedAPI's architecture more filter (or call it pipes) oriented: http://drupal.org/node/236739.
I am still not sold on the filter architecture, because it adds complexity to the API (in addition to exposing data to add on modules you have to deal with their return values, pass them on to other modules down the conveyor belt... which opens a can of problems like format negotiation, memory usage, error handling....) But I am curious to know more about its advantages.
Another drupal project that has a filter like architecture (if i got this straight from the BoF in Boston) is media mover: http://drupal.org/project/media_mover
I like your idea. But I would urge you to take a good look at existing approaches and boil your proposal down to its essence (i. e. a robust and fast filter architecture is a pretty hard cookie in itself - without the UI for it). That way you could make your proposal tighter and more specific.
Although you don't see your project that aggregation related, you should have this proposal on your radar: a more extensible aggregator for D7 as SoC project:
http://groups.drupal.org/node/9857
There might be possibilities to join efforts.
BTW, the first time I came across the yahoo pipes idea for drupal was when I was working on the feed element mapper here: http://drupal.org/node/165388
Alex
http://www.twitter.com/lxbarth
Yes!
Yes! There are unlimited possibilities for extensions! This will be an AWESOME module! FeedAPI, improved aggregator—these are great, but that is only the beginning. :)
What would you like to do with Drupal Pipes?
Could some of you provide some specific examples of how you would use Drupal Pipes? What would you like to do with it? What steps would you expect to require you to do it? What cool features would you like to see in the interface?
chx has provided some input here: http://drupal.org/node/218770#comment-722099
More examples along these lines would be great. Thanks!
pipes like UI for Views?
What about making an intuitive pipes-like drag and drop UI on top of the views platform? Views already has filters and what more. As Alex said we could then use the feedAPI to pull in the actual data.
--
Check out more of my writing on our blog and my Twitter account.