Drupal.org Development Environment Roadmap

We'll continue to report major milestones here, but for detailed progress, follow the Drupal.org Development Environments metaissue.

Urgency

There is an outcry for very difficult and expensive work on developer tools like packaging, Git, Search, and Testbot from core developers, among others. Some of these (e.g. support for semantic versioning (8.0.1 vs. 8.1) and the ability to test core on multiple environments (e.g. PostgreSQL and SQLite) are particularly acute, and will delay the release of Drupal 8.0 if they are not dealt with in a timely manner.

The Drupal Association will not be able to staff up in time to meet these needs, so the only way forward is to give developers with an itch to scratch the tools with which to do it themselves, with the DA's support.

Finally, the community will expect that the release of Drupal.org D7 will allow them to once again participate in the development of features for Drupal.org. It’s an ideal opportunity to engage them with what’s made that difficult in the past and provide an outlet for improving our ability to make decisions and act on those decisions. We cannot afford to hold them off completely, and working toward the creation of the dev and staging environments can stimulate momentum and facilitate real progress.

Environments

Local development environments

(a.k.a. on someone’s laptop / Drupal.org in a Box)
If we want community involvement in the creation of Drupal.org tools, local development environments are necessary for the following reasons:

• It is difficult to use preferred IDEs with remote, cloud-based solutions. Developer productivity revolves around tools, many of which have high learning curves. Taking those away or making them more difficult to use is a huge barrier to entry.
• It is difficult to use your choice of debugging and performance profiling tools on remote servers, which means both a more difficult experience writing code and potentially poorer code quality. The work can be done on the server, and people with the right elevated access and their choice of tools installed sometimes prefer it that way. However, for many devs, learning new tools and getting blocked by access issues is too much to ask.
• Requiring remote dev environments throttles or eliminates contributions from developers for whom home internet is slow, expensive, etc. It's easier to save up for a laptop than to foot high-speed monthly bills. Lots of folks have access only while at work or university, but it's often not feasible to do community work there.
• Local development makes it much safer, socially, to make mistakes.
• Inactive local development environments do not occupy space that prevents more motivated contributors from working. Numerous Drupal.org improvement initiaitves have been indefinitely stalled because the resources were not ready and available at the same time volunteer energy was.

Remote environments on Drupal.org servers

We need remote space, too, in order to demo and test contributions. Depending on the contributor skills and the kind of work they're doing, these environments can provide an excellent developer experience. It's highly contextual.

In addition, the DA's ability to purchase enough space is a limiting factor. A good, sanitized local development image requires a small and finite amount of disk. It incurs bandwidth, but even that cost doesn't inherently need to be covered by the DA.

Basically, if we only have cloud environments, then increasing the number of developers will always be a budget issue, which is typically undesirable.

Staging

An environment with limited access to which work is moved and tested prior to deployment. This should match production configuration as closely as possible as it is our final opportunity to identify issues that will affect production functionality and performance.

Production

The live website - In order to build a representative staging environment, we’ll need an accurate manifest of services, configuration, and software being used in the production environment.

Roadmap for Dev and Stage Environment Setup

Puppet manifests

• Base service manifests for all required services
• Requires refactoring for use in other environments
• To embrace current best practices and reduce the maintenance burden, community modules from the Puppet Forge should be used in place of custom modules and monolithic configuration files where possible
• Production details should be segregated for prod-specific things
• Ideally this should be a set of hierra yaml files that fill in the private details, though a puppet module used to store variables could be used instead. This is how the Wikimedia Foundation does it
• Staging details should be captured for repeatable deployment to the staging environment(s)
• as close as possible to production
• ideally a less closely guarded set of hierra yaml files (still private)
• Local and Cloud environment manifests
• manifests and/or modules should be created that allow stand alone development environments to be created with all required services from the production service manifests/modules described in 1
  Review and finalize documentation and maintenance strategies

Drupal.org in a Box

• A vagrant or all-in-one project with several boxes that can be chosen for development
• A “slim” box that sets up a single local VM to run the most critical services
• shares almost everything with the setup for development vms hosted on OSL outlined above
• runs critical services on a single server
• A single “full” box that requires more ram and disk space
• has everything the slim box but more complete, featuring supporting services (solr, git daemon, packaging, etc)
• A set of deployment scripts that set up the code base and database for drupal.org’s various drupal websites
• automatically load completely sanitized and maximally trimmed copies of the database for testing
  capable of loading a small data set; targeting lower end development environments
• capable of loading a larger data set; more useful for finding edge cases and reproducing problems

Unaddressed potential blockers

Drupal.org Teams / Refined Ideation Process

Software Working Group responsibility

If our goal is wide-spread access and encouragement of community involvement, we really can’t invite a bunch of activity until we’re prepared to handle that activity. One of the most important things we’ll need in place is a process for deciding what is a “good idea” and who decides.

The good news is that forward progress on the supporting environments can be made while these processes are being put in place.

Where work is happening

IRC: #drupal-infrastructure
GDO: groups.drupal.org/drupalorg
Code: BitBucket (Puppet manifests currently contain restricted data)
IssuesInfrastructure Meta-Issue

How can I help?

There's more coming on this as we get into the project and begin issue creation. If you're already interested, contact one of us:

Contacts

• Project lead — eliza411
• Tech lead and database sanitization — halstead
• Puppet manifests — JeffSheltren Jeff_S (irc)
• Vagrant / Local Dev — TBD, talk to tizzo