Posted by moshe weitzman on January 6, 2007 at 4:12am
Does anyone out there have a server which they control and is willing to let us install/maintain a simpletest-auto installation there? this will continously check developer patches to see if they break test suite. will also check core drupal itself.
Thanks,
moshe
Comments
Post to Drupal.org home page?
I just came across this post by accident when I was responding to a forum topic and wanted to reference this group.
I don't have a server, unfortunately, but I bet someone out there does, and you'll probably have a better time finding them on Drupal.org. :)
I have one
I have a server. So what do we need to do to setup this auto-testing, continuous-integrated-development enviroment contraption? :)
what we need - server config
we need the prerequisites listed at http://cvs.drupal.org/viewcvs/drupal/contributions/modules/simpletestaut.... where it says Drupal 4.7, assume we mean Drupal 6. also, here is an old email discussing more our plans for this ...
The server will initially serve a web site where developers can manually verify that their proposed patches apply properly and don't break our unit test suite. Later, we will integrate automated unit testing into our patch review process (i.e. project.module) so as to identify problem patches and problem tests as early as possible, without consuming people time.
I envision us pointing the domain test.drupal.org at your server. The server can run a few other light services of yours if you wish. Mailing lists and file sharing are OK. But I suggest not using the server for performance sensitive applications like a public web site. Unit tests can consume significant CPU if we configure many to run in parallel (if needed, we can disable this).
The next step is for you to reply back to this group on whether this is still a possibility. Then, you should work out specifics about the server with Thomas Ilsche and Rok Zlender. They have agreed to be the official maintainers of the testing service. These are our 2 highest experts on unit testing, so i am confident they can steward this service to success.
Thanks this contribution, and all your contributions to Drupal. We will recognize your gift when we announce this service and on the site's About page.
Awesome
Awesome sounds great. I guess before i can answer if this is a possibility, i need to know how integrated we're going to start with? Meaning, how many patches per-day are going to go through testing. This will help me answer some hardware/bandwidth/etc. questions. Or are we going to start as a testing/beta area, once we get the process down, start it as a semi-standard (move to a more robust server), and move on from there? You're def. right thought, unit tests can get expensive.
slow at first
there won't be many tests at first. here is my idea of a flow:
after a while, it will become habit that serious patches always include a link to their unit test report. then we can discuss how to integrate with project.module so we test pending patches automatically.
Larger issue
Ok, i see the workflow. The issue at hand is larger though. Each patch needs to be patched against a fresh co of D; thus each patch will be patched against a "provisioned" D co, tested, and reported. bascially needs to be scripts to do all this (provision a new D5 install, apply patch, enabled simpletest, run test, etc.).
I'm going to ponder on this more, but that's off the top of my head.
Simpletestauto
That is exaclty what simpletestauto is for. I'm porting it to D5 atm. Should be finished in couple of days.
Steps needed to get automation to work
There are some tasks that need to be finished before we can install test automation on any server.
Regarding how the tests will be used with patches I fully agree with Moshe.
Potential for hacking?
Is there a potential for an evil hacker to craft a patch that does random code execution and hence compromise the simple test server?
Drupal development and consulting 2bits.com
Personal Baheyeldin
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
There is a possibility.
There is a possibility of an attack with "evil" patches. This was one of my concerns last year when I was writing these scripts.
Few problems I was facing.
In order to successfully install Drupal we need PHP capable of running shell commands (cvs, mysqladmin, sed, wget,...) without these Drupal cannot be installed automatically. In order to communicate with Drupal cvs server test server must have access to the internet.
Possible solution
We could use locked down PHP. We could have one PHP that takes care of installing Drupal and can run shell programs and the other which is a bare minimum which Drupal needs and is secure as much as it can be. We would use this second PHP to actually run tests. This could be done with two PHP installations or simply with another php.ini which can then be used like this "php -c securePHP.ini runtests.php" . This way attacker can only harm his own test. This solution is currently used.
Simpletestauto for D5 ready
I have just comitted a D5 version of simpletest automation module. Currently it shouldn't be called simpletest automation as it doesn't actually run simpletest tests but just checks if patch applies. I will roll a patch that adds capabilites of installing drupal and running simpletest tests.
I tried to build a good foundation for testing. Currently all communication between server where patch is submitted and where it is actually tested is done with xmlrpc calls. No data is stored on test server side and it can be implemented in any way as long as it can communicate with patch catching server via xmlrpc.
I will be happy to answer any questions why some things are the way they are. And if someone has any suggestions please let me know.
This version of simpletestauto will soon be available here http://drupal.org/node/153814
And issue queue is here http://drupal.org/project/issues/simpletestauto