[Final] Views plugins to output node lists as XML/RDF/JSON/XHTML

Abstract

This is a proposal to extend Views with the goal of enabling Drupal semantic data sharing and interop out-of-the-box by writing views style-plugins that can spit out node lists as XML, OPML, RDF, JSON, and XHTML.

For example if you created a view with a list of events and details, you could with one click generate a page containing events in the hCalendar microformat or as items in Exhibit JSON format. Similarly lists of user profiles could be output as FOAF profiles in RDF or in the XHTML hCard format. Lists of forum posts could be output as RDF documents using the SIOC vocabulary. Or a simple list containing books from a personal book collection could be output as OPML. This idea is motivated by the accelerating activity around Semantic Web data sharing. The ultimate realization of this idea would be to enable a Drupal site operator to share datasets from her site already constructed through the Views UI in one of several open and reusable formats based on XML, XHTML or JSON, which would be consumable by other Web agents. In the screenshots below, a views list of audio tracks has been enabled to render as JSON or OPML or raw XML.

Benefits to Drupal/open-source community

This project will provide a way for Drupal site operators to open up their site to semantic data sharing and processing without requiring new tools or skills to be learned. Views is one of the most well-known and understood Drupal features and being able to use Views alone to expose site data in formats like RDF and JSON will significantly decrease the learning curve and effort involved in making a Drupal site Semantic Web-aware, and increase the uptake of Semantic Web data sharing. Given the large user-base Drupal enjoys, this will also significantly benefit the entire net-wide Semantic Web activity as a whole. This project can be thought of as a 'front-end' complement to the work going on in the Services and RDF APIs, which are targeted at developers mostly. However developers too will benefit from this project as it may lead to a common library for node serialization in different formats that projects like the Services and RDF APIs could share.

Project Details

I was really blown away by the "Video from the future" from Drupalcon 2008 showing the type of computing possible by opening up data stores using standards like RDF and microformats. It was the first time that I was really seeing what the Semantic Web was supposed to be about and it got me interested in researching the core technology base for data representation, querying and presentation in this next step for Web applications. I got to understand the core standards for representation like and "triple stores", extraction of RDF from existing documents through GRDDL, querying through languages like SPARQL, and agents for consuming data like MIT's Simile Exhibit and the Disco browser

Hanging out in the introduced me to work that had been done in Drupal with the RDF API and Web Services API and PHP toolkits for enabling data interop like ARC and Triplify. One common motif was that data should be transformed and shared from existing sources, instead of attempting to implement top-down manual markups of datasets which would be way too costly. This is the reality and focus of most Semantic Web activity today. So this lead me to thinking about how to share node data in Drupal using existing functionality.

If an axiom of Drupal content management is that data is separate from presentation, why not reuse Views and instead of generating HTML lists from node objects, generate reusable and transformable XML, JSON, and XHTML. Views already has the ability to use style-plugins that can render node data in a arbitrary way - one can create a chart using the Google Charts API and a current SoC 2008 idea is to standardize a Chart view type. The views_rss plugin can generate an RSS feed from a view and the Views Bonus Pack can export views in CSV and .DOC format. So to extend this idea - I have started to write Views plugins that render node lists as :

1. A raw XML document or an OPML document, using field-labels as attributes for each node element;
2. A structured XML that maps node fields to elements and attributes using a specified schema like Atom ;
3. A JSON data document which can either be a canonical representation of the data or use a predefined serialization format like Simile/Exhibit JSON;
4. An RDF data document that can use one of the following three vocabularies:
   1. FOAF
   2. SIOC
   3. DOAP

   Each of which requires a defined set of fields, to be present in the view.

5. An XHTML document which contains nodes marked up in a microformat like hCard or hCalendar(http://microformats.org/wiki/hcalendar)

Each XML or XHTML document could include a GRDDL profile and a pointer to a transformation fileindicating that RDF data may be extracted from the document.

Implementation

The proposal specifies building 4 modules:

1. views_xml.module
2. views_json.module
3. views_rdf.module
4. views_xhtml.module

Each module implements:

• *_views_style_plugins to provide Views types for the formats specified above:
• *_views_arguments in a like manner as views_bonus_export.module to provide links and icons for existing views to render the nodes in the format specified above:
• theme_views_*to perform the actual serialization of the node data to the format specified. Serialization can use either only the node fields defined in the view or all the node fields; this data extraction method is specified by a parameter.
• *_views_arguments to provide needed options for the serialization - for example for RDF documents specify the vocabulary to be used : FOAF, DOAP, SIOP...or for XHTML documents specify the microformat: hCard, hCalendar, Geo... to be used for rendering each node.:

I've already written working proof-of-concept code available in my CVS sandbox which implements rendering Views as raw XML, OPML and JSON. I've posted screenshots in my application post in the SoC-2008 group. Based on this experimentation I believe that all the objectives 1-5 stated above are doable in the SoC timeline. One area of major effort will be in RDF serialization as RDF conceives of data as 'graphs' and the 'flat' nature of node data will have to be processed to fit the RDF model. However several PHP toolkits exist for working with RDF including the well-known ARC. Even if I'm not able to complete all the formats I have in mind, the existing formats implemented as well as design patterns gleaned from the project will provide an excellent point to pick-up and continue development after SoC - each format is implemented around a basic design which I believe is proven to be sound.

Deliverables

1. views_xml.module
2. views_json.module
3. views_rdf.module
4. views_xhtml.module

Each module implements:

1. *_views_style_plugins to provide Views types for the formats specified above;



2. *_views_arguments in a like manner as views_bonus_export.module to provide links and icons for existing views to render the nodes in the formats specified above;





3. theme_views_* to perform the actual serialization of the node data to the format specified. Serialization can use either only the node fields defined in the view or all the node fields; this data extraction method is specified by a parameter.
4. *_views_arguments to provide needed options for the serialization - for example for RDF documents specify the vocabulary to be used : FOAF, DOAP, SIOP...or for XHTML documents specify the microformat: hCard, hCalendar, Geo... to be used for rendering each node.

Each module will implement DrupalTestCase and provide unit tests for each piece of functionality.

Timeline

I favour an iterative development process; I would like to release new versions of the 4 modules every two-three weeks which fixes the bugs from the previous releases and implement new pieces of functionality incrementally. By July 14th I expect to have working implementations of the majority of formats I want to support.

May 26th - June 8th
views_xml - OPML, and Raw XML
views_json - Exhibit JSON
views_rdf - FOAF rev. 1
views_xhtml - hCard

June 9th - June 22nd
views_xml- structured XML using Atom schema rev 1
views_json - Canonical JSON
views_rdf - FOAF rev 2
views_xhtml - hCalendar

June 23rd - July 13th
views_xml - structured XML using Atom schema rev 2
views_json - JSONP/JSON in script (http://code.google.com/apis/gdata/json.html)
views_rdf - DOAP, SIOP rev 1
views_xhtml - Geo

July 7th - July 20th
views_xml - structured XML with arbitrary node-field to schema mapping rev 1
views_rdf - SIOP rev 2
views_xhtml - GRDDL embedding rev 1

July 21st - August 3rd
views_xml - structured XML with arbitrary node-field to schema mapping rev 2
views_xhtml - GRDDL embedding rev 2

August 4th - 18th
views_XML GRDDL embedding rev 1 and 2
Complete docs and tests.

Bio

I'm currently in the 2nd year of BA program majoring in Mathematics and Linguistics. I participated in SoC 2007 with Drupal, working on the DAST project which was a lot of fun and will hopefully start gaining more traction with all the activity around testing and auto-deployment. I think I am a good choice to work on this project because I have a lot of enthusiasm for the Semantic Web and what is possible for this next generation of the WWW and enabling one of the best CMSs on the planet with this technology will be a very exciting and rewarding endeavour for me.