Improve and expand Drupal.org testing beyond its current borders:
- Improve testing reliability and capability
- Test on additional environments (pgsql, sqlite, Microsoft, Selenium)
- Improve documentation and understanding of our code, infrastructure, and capabilities.
- Make PIFR more deployable in environments outside D.O.
A couple months ago, at the BADCamp devops summit, we held a workshop to discuss the architecture and operation of the current Drupal.org Automated Testing infrastructure, as well as to review a proposed future state architecture for the platform.
This post is an attempt to summarize the discussion, outline the proposed architecture as presented, and discuss some next steps.Read more
Interested in participating in the 'Modernizing Testbot' initiative? Please add your name to the list below; and if there is a particular piece of the architecture that you'd like to focus on (or have previous experience with), please mention it as well!
Please edit this page to add your info below!Read more
On October 18th, 2007, Chad Phillips (hummonk) made an initial commit to the ‘Project Issue File Test’ module, with the description “initial commit of an integrated file testing platform for project issue module”. This kicked off the pursuit of automated testing integration between drupal.org and qa.d.o (then referred to as testing.drupal.org). In 2009, Jimmy Berry (boombatower) performed a significant overhaul and redesign of the system with the release of PIFT/PIFR 2.0, introducing a number of structural and architectural changes and improvements.Read more
Recently, I was asked to pull together some metrics which would help illustrate the current health of the automated testing infrastructure, and provide a baseline by which we could monitor it over time ... and I thought I'd share some of those findings here.
The first graph shows how testbot utilization has grown over time, from a steady-state count of around 1000 tests per month in the first half of 2011, doubling to 2000 tests per month in January 2012, and reaching a peak of over 6400 requests in June of 2013.Read more
Looks like you can attach to the docker container's streams and then echo them back out so Jenkins can record them. An example can be found at https://github.com/salsita/jenkins-docker-skeleton. Alternatively you can run docker logs or -- I think -- docker run -a to attach the stdout / stderr streams. Also some information on looking at the logs at http://0x74696d.com/posts/docker-logging/.
This message has been cross posted to Drupal.org Improvements, Documentation Team, and Drupal.org Testing Infrastructure.
I want to use automation to improve d.o content quality. I do not have access to do this and I have been unable to get someone with access interested enough to get this project moving.
The long version
My vision for improving the Drupal Documentation and drupal.org can be broken into three equally important ideas:Read more
This page is intended as a collection point for requirements for the next generation of Drupal.org's automated testing environment. Please feel free to update this page with your own thoughts and ideas, and help contribute to the requirements which will define the next (r)evolution of Drupal's testing infrastructure.Read more
The purpose of this document is to outline a proposed roadmap for Drupal testbot evolution, as discussed during the testbot evolution sprint at DrupalCon Denver on Friday, March 23rd, 2012. Participating in the discussion were Jeremy Thorson, Jimmy Berry, and Randy Fay. This proposal is also influenced by additional consideration and input provided by participants in the TestBot BoF held one day earlier.Read more
Note: This is a copy of a blog post original posted on March 20th, 2012; with the original located at http://jthorson.doesdrupal.com/thoughts-on-testbot-evolution ... it is being reposted here in the interest of consolidating the information on Drupal.org infrastructure, and to improve discoverability by any potentially interested parties.
Having spent the better half of the last year within the PIFT/PIFR issue queues, chasing the holy grail of sustained testbot stability, I find myself reaching a bit of a crossroads.Read more
The sprint will be mainly focused on the following areas, and the following attendees will be there:Read more
On November 1, 2011, all Drupal 8.x core files will be moved into a /core folder. This will break nearly every patch in the 8.x queue, requiring them at least to be re-rolled against the updated directory structure.
After that date, backporting 8.x patches to 7.x will no longer be trivial.Read more
The Project Dependency module is deployed and enabled on drupal.org, with initial dependencies built! Now we have to get to the next step, deploying the PIFT changes that go with it.
- Our key #1 goal for the day is to get dependencies stable and deployed, see Project Dependency.
- Fix additional project dependency issues with memory limit.
I'd love to do a code sprint on testing Friday at Drupalcon London.
The most important outcome would be deployment of Project Dependency and the related PIFT, PIFR, and Project module changes.
What we need to do to accomplish this:
- Set up a test environment (PIFT, QA.scratch + testbot) demonstrating dependencies working.
- Get the new PIFR version deployed, assuming it can be done without a PIFT change.
- Get the deploy issue RTBC'd and perhaps deployed.
I wanted to point possible contributors to a new low-level testing infrastructure doc page, http://drupal.org/automated-testing - Although this is incomplete and probably incorrect, it's a starting place for documenting our testing infrastructure. I invite you to improve it.
This wiki page has been set up to continue the 'Automated Coder' discussion initially started at http://groups.drupal.org/node/155084#comment-517764.Read more
I want to make sure that everybody in these groups knows about the fantastic simpletest + selenium integration work that ygerasimov has been doing. I just tested out his Selenium module (7.x) and it worked fine for me.
Here's what I did:
- Install Selenium module and start Selenium Server as described in the INSTALL.txt.
- Installed the Selenium Test Example sandbox for sample tests.
One of the problems we've always had that delays deployment of PIFR and PIFT code is that it has been hard to test. But now we have a workable test environment, consisting of three parts:
- A drupal.org clone, http://rfay.redesign.devdrupal.org, housed on stagingvm.drupal.org. (This is one of the new tricked out development sites that are now available to people who want to improve drupal.org)
- http://qa.scratch.drupal.org, housed on stagingvm.
- Testbots as needed can be spun up either as local VMs and configured to pair with qa.
I was trying to gauge what the best setup for a PIFR node is, and I noticed that this page (http://qa.drupal.org/node/62) seemed way out of date, so I ran some tests of my own. Here I was trying to figure out a couple of things:
- Which concurrency setting is best?
- How does performance improve with number of cores
For testing, I used two separate PIFR nodes running in KVM virtual machines with the following specs:
CPUs are Opteron 6172, 2.1 Ghz
2 GB RAM
Local disk (10K SAS)
Running Debian Squeeze 64-bit, installed PIFR using DamZ's puppet setup
My results were as follows:Read more