Coder Upgrade and Grammar Parser

Project information

Projects: http://drupal.org/project/pgp and http://drupal.org/project/coder (specifically the coder_upgrade module)
Status: Created the grammar parser module, wrote patch to use parser with API module, need to implement automated module conversion routines

Description

My project is two-fold. First, a grammar parser module will be written. This module will provide a recursive descent grammar parser to help analyze and modify a source code file. The goal is that, by organizing the source code based on the grammar of the programming language (as opposed to low-level tokens), complex changes to the code can be more readily made in a programmatic fashion (i.e. by other code using the parser).

The second part will produce a Drupal 7 version of the Deadwood module (renamed to coder_upgrade) incorporating new features such as 1) the grammar parser to enable more complicated conversions and 2) a Conversion API to enable contributed modules to write conversion routines for non-core APIs. The purpose of the module is to automate as much as possible the task of updating a contributed module for Drupal API changes, and thereby simplify the task of porting contributed modules shortly after a new Drupal release.

Project schedule

1. Design and implement the grammar parser. (June 21st)
2. Design and implement the exposed API routines. (June 28th) (a work in progress for the editor class)
3. Create conversion routines utilizing the parser. (July 7th) (a work in progress for the editor class)
4. Refine the user interface for display of conversion results. (July 12th)
5. Update the documentation for conversion routines completed. (July 31st)
6. Solicit community assistance to test the module and write conversion routines. (August 1st)
7. Test the code conversion coverage provided by the module on several contributed modules. (August 17th)

Status updates

2009-08-17

In Grammar Parser: Add PGPList class (a custom doubly linked list of PGPNode objects). Add PGPNode class to hold grammar statements, pointers to next and previous statements, and the containing list. Refactor the grammar array to a PGPList object to allow for complex edits to multiple lines.

In Coder Upgrade: Group conversion routines by processing order (beginning, middle, or end). Separate middle conversion routines into two groups (regex and parser) to utilize the PGP API. Add handler specification (regex or parser) for the middle changes applied at the file level. Revise API documentation. Implement initial batch of conversion routines using the PGP API.

2009-08-02

In Coder Upgrade: Eliminate display of conversion results online, and instead create a log file. Create patch files for each converted module or directory. Add links to log file and patch files in user interface. Revise api.php file. Revise user interface using new D7 form elements. Implement hook_upgrades for custom conversion routines.

2009-07-14

In Grammar Parser: Added PGPEditor class and custom token constants. Provided a patch for the API module to use the grammar parser for creating documentation items and be positioned to handle interfaces and classes (which it currently does not). The time required to work on this patch has resulted in a digression from the above schedule. Also added support for the 'else if' syntax (as opposed to the single token 'elseif').

2009-07-07

In Grammar Parser: Implemented heredoc tokens and function calls when the function name is a complex variable expression. Parser implements all but five of the 118 PHP named tokens (only T_USE for namespaces is of interest) and all of the known non-named tokens (e.g. '{', '+', '<', and '?').

In Coder Upgrade: Add module to coder project as coder_upgrade. Rename conversion directories to 'coder_upgrade/old' and '/new'. Change menu path to 'admin/development/coder/upgrade'. Change menu path to 'admin/settings/coder/upgrade'. Add directory called 'conversions' and move conversion .inc files there. Remove deadwood node type. Add upgrade path to .install file (require uninstall of Deadwood). Restructure process to always run all conversions. Revise help text. Add grammar parser (pgp) module as a dependency.

2009-06-23

Committed grammar parser module code to parse and rewrite PHP files. Parser implements all but seven of the 118 PHP named tokens (of the seven, only the two heredoc tokens and T_USE for namespaces are of interest) and all of the known non-named tokens (e.g. '{', '+', '<', and '?'). Still need to work on editing the parsed file before rewriting. Tested code with includes/database.inc and simpletest/drupal_web_test_case.php. API exposes routines to parse and rewrite a code snippet.

2009-06-19

Made second commit to grammar parser module with base and extended parser classes.

2009-06-01

Made initial commit to grammar parser module. Will continue to refine and test the parsing routines before further commits.

2009-05-30

Created the grammar parser module and working on initial commit. Ported the Deadwood module to Drupal 7 and integrated into Coder project.