Transformations idea frenzy!

Overview: Transformations is an API for creating and executing data transformation pipelines, enabling higher-level modules to build stuff like Yahoo! Pipes or targeted import/export interfaces on top of it. The purpose of this project is to improve and build upon Transformations so that it not only provides a base framework but also satisfies a larger number of end-user needs, for example an AJAXified user interface.

Description:

Transformations was born out of the desire for a solution that can map any kind of structured data to any kind of other data, such as importing CSV or more complex XML files into Drupal nodes (with field mappings, obviously), exporting the data to the same files, or, if you will, providing an online converter that converts any RSS feed into Atom for example.

When performing such a large-scale transformation, some elements of the source data will likely need to be transformed on a smaller-scale level, e.g. you might need to convert a Unix timestamp into ISO date format at some point, or remove commas from a number string ("1,099.90") to conform to the target format, or perform string replacements on a given string field. At times, you might even need a combination of those transformations, or reference some other entity to construct the desired output data.

In order to do that kind of stuff, a simple mapping screen is not flexible enough, or would require lots of custom code to handle the more specific tasks. Transformations tries to solve that problem by splitting individual data transformation tasks into individual "operations" that can be chained into a larger transformation pipeline. (Also, whole pipelines can again be used as a single operation in more higher-level pipelines.) Given the right set of operations, you can click together your own personal transformations without the need to code, but still in a way that is flexible enough to accommodate advanced import/export needs (or whatever kind of data transformation you need).

However, the current user interface is kinda rough. Or maybe "crappy" would be an appropriate term. Like this (pure HTML, not draggable or anything):

Given a little love, we might be able to obsolete a whole class of modules (examples: 1, 2, 3, 4, 5, 6, 7, etc.) with a single transformation framework. For world domination though, the generic default user interface (Transformations UI) needs a major revamp, and maybe a few tweaks on the underlying foundations.

Here's a list of ideas that are up for grabs:

• Make the UI kick butt with JavaScript! The pipeline edit form of Transformations UI could be AJAXified so that operations are connected with nice lines instead of color codes and can be dragged around by the user... and entering options for an operation might be done in a translucent popup instead of a separate form.
• Improve schema support so that Transformations can actually figure out which operation outputs may be connected with which operation inputs.
• Enable users to create their own transformations that they can then use their pipelines like in Yahoo! Pipes - like, let a pipeline fetch an RSS feed that can be transformed to an HTML page which is then displayed by Drupal. Or stuff.
• Integrate Transformations with Rules so that any user-defined pipeline can be run on any event that happens on your site! (Now that would be awesome, wouldn't it?) Or the transformation rule might just create a Rules variable that can be used by the rest of the rule set. (Think of string transformations, array filters, and whatnot!)
• Improve support of Drupal data import/export by adding CCK field support to the Field Tool module. Also, help to centralize field support by submitting patches to various import/export modules so that they use Field Tool instead of their own module-specific hooks and callbacks.
• Write pipeline creation wizards that offer a simplified user interface for generating more complex transformation pipelines. Similar to what SimpleViews does to Views.
• Write lots of operation plugins so that people can create more powerful transformation pipelines more easily with even less code! Depending on your interests, those might be RSS importers, Views result listings, XSLT transformations, SQL SELECT queries, or whatever comes to your mind. Be creative, and think of cool new stuff!

Of course, those are only my personal ideas - I'm sure there are ways to use a "generic data transformation pipeline" that I haven't thought about yet, and the best SoC projects are of course those that the student comes up with. So bring in your own ideas! For an actual Summer of Code project, you'd probably pick some of the above items and spice it up with your own visions. Don't hold back, and also, don't copy-n-paste from the above list into your application without adding some additional thoughts.

Transformations could be to data transformations what Views is to content listings, or what Rules is to event-based execution of actions. Taking Drupal to the next level? Unmatched power and awesomeness? It's up to you!

Mentors:

• jpetso - primary mentor
• Who else? Interested mentors are welcome!

Difficulty: medium to hard (OOP skills required)