Make drupal.org maintainable via more automated tests and better staging environments

What's your idea?

I've been pushing for years to get Drupal's infrastructure on a path to more automation. It would be helpful to have:

• Automated creation of new environments that clone the live environment, both local and centralized
• A battery of well-maintained tests for every level of our infrastructure, and standards to keep the quality
• A similarly comprehensive logging strategy, and accompanying standard
• A clear, transparent (i.e. inspectable by anyone at any time) process by which work moves from ideation through development and into production

My experience is that our current process of adding features makes it hard to contribute in the long run. Many features we add just add more untested complexity, which often adds more technical debt. We rarely pay down this technical debt, and the more we have, the more risky and complex it becomes to introduce new features.

In my view, the hodgepodge way in which we manage our infrastructure has caused complexity to rise to the point of outrunning our teams' abilities to really keep up with it. The classic indicators are there: projects are often unsuccessful, almost never on time, and the people involved juggle chronic problems around like a game of hot potato. We are already beyond our limit. If we can't turn the train around, moving non-brochureware aspects (g.d.o, Git, issue queues, documentation, forums) off to secondary services should probably be viewed as more of an inevitability than a choice.

What are the benefits?

• Non-professionals (aka volunteers) will be reasonably able to contribute to drupal.org again.
• The value of time invested by pseudo-professional infra team members will be amplified.
• By investing conscious effort in best practices around maintaining real sites, rather than our tradition of just caring about spewing software, we might collectively learn how to give more than lip-service recognition to skillsets other than "rockstar developer omg!" as essential to the big-picture success of Drupal.
• It will define a working model for the whole of the Drupal community as the steps one should take to have a properly maintained, tested Drupal site. No longer would that be the domain of high-end boutique shops - and in my experience, even there testing is often low quality.

I believe this change would double our throughput capacity (and if we do this well, doubling is conservative) with a year of working on this. That's just math - make the not-unreasonable assumption that at least half of our time is spent on:

• Maintaining poorly-conceived non-prod environments
• Manually testing things that could be automated
• Or worse, waiting for some busy person who knows a particular subsystem to manually run said tests
• Chasing after changes with unforeseen consequences made by someone else
• Doing all of these things in a murky environment where the state of work in the system is not readily knowable

Implementing these changes would cut out all, or almost all, of this overhead. Some of that is a reduction sheer number of required work hours; other is a reduction in the complexity of the process required to push through changes. Both amount to increased development throughput, and by a factor of two isn't unreasonable.

Assume our current capacity for 2014 is 10 features/fixes/whatever. Say that we work only on this proposal in 2014, with no new features. At the end of the 2014, we have 10 fewer features than we otherwise would have had, but a capacity of 20 features/year. At end 2015, we're tied up - 20 features. End 2016, without this system, we've produced 30 features; with it, we've produced 40. Obviously, these numbers are speculative - they're included just to be absolutely sure that the meaning of "increased throughput" clear.

What are the risks?

• We'd have to de-prioritize other areas (like marketing).
• We make huge investments in new processes, but my assessment here ends up being wrong, and productivity/volunteer-accessibility is not significantly improved.

How can we measure the impact of this idea? (metrics)

• A decreased cycle time for issues in the infrastructure, drupalorg and similar queues..
• Happiness of anyone working on the infrastructure.

Who directly benefits from / will use this improvement? (target audiences)

• Everyone, because in the medium-to-long term, everyone gets their features faster.
• Anyone wanting to help with drupal.org.
• The infra team, by reducing uncertainty and risk in all of our work.

Are additional resources available for discovery/implementation? (volunteer effort, financial backing, etc.)

There is no way to accomplish something like this without a concerted effort from all members of the infrastructure team. That means funding, and not a small amount of it. On the plus side, it's funding that ought to reduce the need for such funding in the future - the goal is to build systems that automate away many of the frustrations that we currently burn money on to deliver features.

A related anecdote: during the Git outage last Wednesday, I (Sam Boyer) was testing on production most of the day, until I finally invested the 2 hours to get the staging vm back up and working just barely well enough for what we needed to test. "How do I get a working environment together?" and "Is this environment an accurate test?", let alone "Does this change have unanticipated effects on other systems?" should not be such hard questions, especially during a crisis.