Managing Drupal development for small distributed teams

This is a long post, grab a cup of tea... (or better still a beer)

I manage a small team (currently 3 soon 5) of grographically distributed developers developing and maintaining a few (currently 4) projects in different stages of the lifecycle. I am familiar with managing teams developing web applications but up until now my Drupal experience has been largely solo. Developing large Drupal projects and working in a team is relatively new to all the other devs so I see this as an opportunity to encourage a sound development process using the appropriate tools.

Here's an outline of my current approach (any comments gratefully received):

** In early development **

Each dev has a local copy of the site running on their machine. Code is versioned by CVS.
We have a dev server with a copy of the site which has the current stable set of code.
Code changes are always made on local machines (template files, module dev etc.) Once they have something working they commit to the repository and update on the dev server.

The Project Manager (PM) and Assistants (PA) make configuration changes and create content on the dev server. Periodically the devs take a copy of the DB with backup_migrate and refresh their local copy. Typically this would be when beginning a new task so that they have the latest set of content and config.

The general rule is Code Goes Up, Data Comes Down. (I read this somewhere else on Drupal.org and it's how we work).

This approach seems to be working well but there are issues.

1. Getting used to CVS is difficult.

The devs are finding it clunky and confusing allthough they see the point. At this early stage using version control is creating more problems than it is solving - although we all know we need it more and more.

1. Most development requires content and config changes.

For example I'm required to make a "multimedia album" which has content types for video, audio, image and show them in a shiny page. I need to make test content, install modules, configure them, as well as create templates, add to a custom module, edit css in the code. The Code Goes Up, Data Comes Down model is broken.

Codewise it's straight forward, when the new stuff works locally I check-in the code, resolve any conflicts identified by CVS and update on the dev server.

Config changes are more difficult. Content types are handled by content export/import, Views by views export/import, although one by one. Module configuration I've been doing by hand. All this takes an annoying amount of time, sometimes I miss a setting and stuff doesn't work right.

The test content has to go up to demo to the team (or the boss or the client) - this is not too much of a problem - provided that the config changes have taken place I can create the content by hand (if it's just a test node or two) which also tests that part of the system. If it's several nodes this is a real pain so (I haven't done this yet) I imagine a node import/export module should do the trick here.

** Later development **

In one of our projects we needed to provide a staging server during development where the client could see a set of functionality that the boss wants to show, while we use our dev server to make progress on features.
This adds another layer of complexity to the process where the PAs are making content, (or views or blocks or config changes) on staging to show the client - finding a display bug, the boss asks a dev to fix it so we copy that content back to dev and to their local copy, fix the bug, update the code on dev, confirm the fix, update to staging. (Are you bored yet? it gets worse). Other PAs are making content on dev which may or may not need to go to staging. Devs are making test content locally which needs to go to dev but definitely not to staging.
It's a mess.

** Live Site **

When the site goes live it gets significantly worse.

On one of the projects (which I inherited when I joined the company recently) we have a busy live site (250K unique visitors a day) although virtually no logged in users. The client is creating plenty of new content a day in various content types.
We have a staging server (which was the dev server) and we are making regular fixes, updates and new functionality which the client can review here before moving it live.
There is no time window to freeze the live DB while we copy it to staging, make config changes and then copy back to go live with new functionality so all new content and config has to moved over by hand, carefully checking that the site is still running as we go.

** Questions for the community: **

Please bear in mind all our new projects are likely to be Drupal 6 but the current 3 major projects in Drupal 5 need to keep working and may not get upgraded to a new version of Drupal for years...

1. Code versioning.
   Should we be using SVN, Bazaar, something else to make life easier for the devs?
   Whatever we use will need a windows GUI client (currently WinCVS) for the devs and should also plug in to Eclipse nicely (for me :)

2. Database versioning.
   I have read around the topic, there are various tools - does anyone reccomend one?
   I am planning a part-way solution of writing a script to dump all the individual tables into a directory on the dev server and put them into version control. At least this provides me with the ability to tag the site code, content and config. Also using cvs diff between table changes could be a useful tool for spotting why something has stopped working without browsing the settings pages one by one. Does anyone else use this or similar process?

3. Configuration & Content changes.
   There are a various of modules in this problem space - does anyone have practical experience of using them for the kind of projects I've described?

4. Docs, trackers, communication tools.
   I am thinking of setting up a Drupal for internal use and encouraging team members to use it for project discussions. First up would be a mailing list connected to a forum. Most people like mailing lists and use them really happily - I like to review them in a threaded forum format.
   We have tried out Case Tracker, which looks good for bugs. Project module at first look seems specialised for Drupal.org and not what we need.
   I'd also like us to start a development blog for internal (and community) benefit.

I think that's enough for now...
TIA
Marcus