Build Systems & Change Management

What is biggest drupal project you have ever managed ? (size in man-days)

Julien Marboutin — Fri, 12 Sep 2008 23:06:09 +0000

Drupal as a Sourceforge clone?

stodge@drupal.org — Wed, 27 Aug 2008 18:33:32 +0000

I'm investigating Drupal as an internal Sourceforge alternative. I'm having trouble understanding how the Project Manager works with issues and subversion repositories. Is there any GOOD documentation that explains how to the PM module/Drupal interact with repositories?

I'm familiar with Trac or GForge, where a project is directly tied to a specific repository and any commits you make against a specific issue are added to that ticket.

I used ClearCase/ClearQuest recently, which despite its many, major flaws, is my benchmark for functionality. One prime example is linking commits to an issue/ticket so that an auditor can clearly see the software changes that were made for that issue.

I created a project on my Drupal site. I also added a subversion repository in the project administration -> vcs repositories admin page. But I don't want to have to manually create this entry every time I create a new project. Or do I have to manually create the svn repository on the disk and then add an entry in the vcs repositories admin page?

For example in GForge, I register a new project and a crontab entry automatically creates the svn repository on disk and links it to the project. Do I have to do all this manually?

I'm confused!

Any help appreciated!
Thanks

Build Systems & Change Management

Post your proposed session for DrupalCampLA 2008

Chris Charlton — Fri, 22 Aug 2008 16:27:11 +0000

This is easy and quick, nothing fancy, no voting - just post your session idea (title) and your name/contact info and show up to present that day! We have sessions open from 10am-2pm each day (Saturday and Sunday) and each session should plan for 45 minutes each.

http://groups.drupal.org/node/12528 (edit wiki page)

If you can't see or edit the wiki page, be sure to join our group which may be required for editing the wiki page.

Zen Task Force

Drupalcon Szeged hosting best practices: development, staging, deployment

Amazon — Wed, 02 Jul 2008 04:15:49 +0000

If you are interested in participating in a panel on Drupal infrastructure, development environments, deployment, and best practices please indicate what you are interested in presenting. I am hoping to get some experts in best practices to do a tutorial.

Drupal IDE

Selenium and Drupal

Souvent22@drupal.org — Sat, 03 May 2008 16:03:04 +0000

First, for those that don't want to read the full post, here's the "30-second elevator speech":
This post to to discuss using Seleinum with Drupal. Specifically using Maven to run the selenium tests and writing the Selenium tests in Java.
The example code be downloaded from the Workhabit Inc. public repository here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk
One must have the following installed to run:
1. Firefox
2. Java 1.5+
3. Maven
All 3 are easy to install and aquire. Once you download the code from the repository, you can just navigate to the directory and run:
mvn -P drupal integration-test
Maven and writing tests are beyond the scope of this blog post.
I'm writing this up for comments, thoughts, and hopefully for others to use and/or contribute.

Ok, now the long winded version:

I've been in the Java world for a bit and I'm now bringing my self back into the PHP/Drupal realm.
With that said on my last project we had to validate ALL code with unit tests, and then we did fully functional integration tests using Selenium.

The Build management tool we used was Maven ( way past the scope of discussion here ). However, what I have done is GREATLY simplified down what we used into a very short, simple, and small file which allows one to run Selenium tests out of the box. It uses Maven/Java to do the tests...but, that's neither here nor there.
It is available here: https://svn.workhabit.com/svn/public/drupal/selenium/trunk

I know that there is a Selenium module for Drupal...however when I thought about this, I also felt that this was an inherit problem. This dawned on me when I read the link that there is a patch to port it to 5.x. There in lies the problem. As we move versions/etc., we're updating the testing frame-work instep with the item we're testing. E.g. PHPUnit may be at 2.3, and we can still use PHPUnit 2.3 for Drupal 4.7, 5.1, 6.x, etc. However, by tightly integrating the testing frame work into the item we are testing, I feel one is creating a cyclic-redundancy in which the end result is chaos.

Goals

My only goals of this project is 2 fold:

Give a solid base so that anyone can download the drupal-seleinum project and write tests for their drupal site.
The community should have an entire suite of tests to test the Drupal installation as a base install ( I have currently started these in the com.workhabit.drupal.selenium.core package. Inside the package, it think they should be split into their subsystems, e.g. Authentication, Taxonomy, CCK, etc.).

Why Java/Maven?

Well, there are a number of reasons; I will not list/discuss all of them here as they are not really part of this post. But on the short list; Java allows one to set break points where Selenium is running to see why a test may be failing ( it should be noted you can also do this with PHP; if you setup your IDE correct and get xdebug installed/setup correctly also).
Also, by using Maven, one doesn't have to worry about dependencies with the unit testing suite (versions, etc.) as Maven takes care of all of that. Java/Maven will also allow us to do more in terms of continuous-build integration. Further, writing tests in Java is not that much harder/different that writing them in PHP. The Selenium IDE gives code in many languages (HTML, PHP, Java, Ruby, etc.) so anyone should be able to create tests easily. I'm hoping to write a post of how to write/add tests in Java. Most of the hard work has already been setup in the code, so all one has to do for a new test pretty much is create a new class and create an annotated java method, such as:

@Test
public void myDrupalTest() {
  BROWSER.open("/path/to/my/site");
  // ....other browser test code from Selenium IDE
  // ...Assertions
  return;
}

Also, in the coming days, I shall be adding to this suite a "MockSoapServer". This will allow you to hit a webservice in point (in this case the MockSoapServer), and it will return "canned" results. Maven will fire up both the Selenium server AND the MockSoapServer BEFORE running the test suites. It will then create a report and take down both after the tests are run.

Environment Setup

First things first, you will need to have Maven installed on your local machine in order to use the Drupal-Seleinum tests. Installing maven is very simple and straight forward. See the Apache Maven Site for more information on installing Maven.

Setup your Tests

So, lets be real; Usually when we are developing, we are changing some of the default workings of Drupal; So we will need to have a some-what custom test suite for each site we develop; or at least we will be extending the tests with more/new tests.

*EXPORT* the base Selenium maven build from the Workhabit repository.
svn export https://svn.workhabit.com/svn/public/sandbox/earnest/drupal/drupal-selenium
Update the POM (If you are unfamiliar with Maven, ignore this step)
You can update the POM if you want to better describe you project.
Modify the "setup.xml"

You will notice when you download the code that there is a file called "setup-example.xml" in the src/test/resources folder. This file is read by the test suite and contains some general settings regarding your site. The settings are pretty self explanatory. The main ones you will need to change out of the box are:
- siteRootPath
- siteDomain
- testUsername
- testPassword
I will hopefully be documenting the settings further, but for now, I apologize that the documentation is a bit sparse.
Test to see that you have everything setup correctly by running a maven integration test (Note: All tests are done with the drupal plugin)
mvn -P drupal integration-test
This should run the DrupalLoginTest which will bring up your Drupal site, and login a user.
Add the package to your own repository.
svn add drupal-selenium && svn commit -m "Adding the drupal-selenium package to my site"
You now have a customized maven/Selenium test suite for the project that you are working on.

Usability

DEADLINE FRIDAY: Include Drupal 6.x in Open Solaris

Amazon — Thu, 17 Apr 2008 15:16:10 +0000

If you are interested in seeing Drupal, and in particular a specific profile be included in OpenSolaris, Friday is the deadline for the next meeting.

Submit your comments here:

http://www.opensolaris.org/jive/thread.jspa;jsessionid=2A66929A74E346D56...

Distribution profiles

Keeping Blocks in Synch

liberatr@drupal.org — Tue, 15 Apr 2008 22:08:53 +0000

Hey folks,

I've worked on a number of Drupal sites where we needed to keep multiple sites in synch, and Drupal has been a challenge to keep that way because so many options are changed through the UI - this tends to show its weakness when we want to keep the settings across all sites (don't even mention content). Moving content types and views is easy with the import/export tools, but blocks are another story. The visibility rules, Roles, and input formats all require your time and attention to detail - time you could be spending working on other features.

Today I came across a very simple system for keeping blocks in synch: just move the database tables.

This works well, because even if block id numbers get out of synch, you are just overwriting the entire table, so no harm done. Also, if you want to create a new block, you will always be doing so at your integration point, before you push the changes to a testing/QA/beta area and then the production area, right?

Just do a mysqldump on these three tables:

blocks
blocks_roles
boxes

and import the resulting data into your production server. It saves you lots of time clicking checkboxes on those pesky Block visibility settings and trying to get all the weights and regions just right. You can also keep a copy of the SQL data in your revision control system, so you always have a snapshot of how your blocks looked at the time of the snapshot.

There was a time when I used the Insert Block module to include blocks, and that would break blocks based on IDs, but if you really need to include blocks in the content area, please look at using regions.

For panels, you might just have to move the panel data along if your site is dependent on block ids. Blocks generated by views or modules should not be so hard.

What else might prevent someone from just moving these tables over? Discuss.

Build Systems & Change Management

A Deployment Framework

heyrocker — Sat, 23 Feb 2008 20:50:57 +0000

I have been working on a framework that attempts to address some of the issues of deploying code and data in Drupal. It is outlined here:

http://heyrocker.com/drupal/content/deployment-and-change-management-fra...

I'd love to hear people's thoughts on this.

Build Systems & Change Management

Deployment and Change Management - The Problem

Amazon — Fri, 08 Feb 2008 15:37:34 +0000

Written by Heyrocker.

I had been hoping to get together with people at Drupalcon this year to discuss issues surrounding deployment and change management. Unfortunately I will not be able to go. So this will have to act as my contribution to that discussion, which will hopefully carry on well beyond Boston. There are three parts - The Problem, Things That Help Now, and Some Ideas For The Future.

One of the first things we tackled while putting together our Drupal architecture was workflow - both for our editorial team and our engineering team. Very quickly we knew we had a problem on our hands.
Our development environment is pretty typical for any large site. We have three server environments - dev, qa, and production. For us, dev is one box where every developer has a sandbox which is basically a VHost and an svn working directory. The code there connects to a local mysql database. Every developer has root on this MySQL server so we can create, trash, and share databases at will. When code is ready for testing we have a packaging script that pulls it together and pushes it to the qa server, where it is tested. This process is repeated until code is ready to go to production when it is packaged and pushed live to a series of load-balanced production web servers.
Editorially we had around 10 separate content management systems for various parts of the site that had been built up as needed over the years. This included things like Movable Type and phpBB as well as custom solutions. The editorial team passed Word documents around through the various phases of writing and editing, finally saving them as text and pasting them into whatever system was in place for final deployment. The plan was to move all this stuff into Drupal, taking advantage of revisioning and workflow.

The problem comes when you want to deploy new functionality. If there was one thing that was nice about the old way, it is that everything was isolated. Recently we upgraded our user registration system which involved some major database munging. This was not a huge deal, we just took registration and login offline for a few hours overnight (logged in users remained logged in) and did our thing. The rest of the site went on its merry way. Obviously this doesn't fly in the new world.

A great deal of Drupal's life is stored in the database. This is even more true when you're extensively using CCK, Views, and Taxonomy (and we are.) Simply deploying code is not enough to push new functionality live anymore, you need to push the database too. However you can't just replace the database every time you want to deploy new stuff - even if you refreshed QA a day or two or even a few hours before launch, you risk stomping on any new comments, logs, cache, users, etc. Drupal is constantly generating new content.

Also how does the editorial team stage their content now? They can't stage it on QA, we need to be able to have pretty much free reign on that database while testing and debugging. Even if they did how would the content move live? The problem is extremely sticky and we struggled with it for a long time before coming up with a solution nobody really liked but which seems pretty standard in Drupal-land.

The process for deploying code is unchanged, and works well. However where data is concerned the process is the opposite - Editorial does all their work on the live site. When necessary we refresh dev and qa with this database. In a nutshell, code moves "forward" and data moves "backwards." In a situation where code relies on, for instance, a new content type or view or something, code is pushed forward and the changes are made by hand through the admin.

This works but it is unweildy, and the more changes you're doing by hand the worse it is. We are working on a project now which will involve deploying 9 content types, perhaps twice as many views, a new taxonomy vocabulary, a full new set of pathauto settings (since we're upgrading to 2.0), and a wide variety of assorted admin changes in different areas. It also doesn't account for the fact that if something goes wrong while you're doing your final deployment, there is no easy way to roll back to square one. Hearing stuff like this makes business owners twitchy.

So what can be done?

http://heyrocker.com/drupal/content/deployment-and-change-management-pro...

Build Systems & Change Management

Drupalcon Tutorial: Best practices in development environments, staging, build management, and production environments

Amazon — Mon, 04 Feb 2008 15:05:46 +0000

Tutorial Panel, 90 minutes:

Presenters (Panelists)

Narayan Newton (OSUOSL)
Milind Parikh (IBM)
Sam Tresler (Advomatic)
Eric Mandel (BlackMesh)
Neil Giarratana (Lucidus Corporation)
(Note that I have only included panelists who have been involved in the conversation for this presentation. There may be others interested or with whom I have not personally interacted with; this is not meant to be a slight...just giving the list as I knew it. NG)

Track

Site Building

Session Description

Using multiple developers to build a Drupal site without stepping on each other's toes can be difficult. Marrying change management with a database-driven, content management system can be near impossible. Narayan Newton (Drupal Assocation/OSUOSL), Milind Parikh (IBM), Sam Tressler (Advomatic) and Neil Giarratana (Lucidus Corporation) will share change management best practices they have applied specifically for Drupal site implementations. You'll learn how to set up dev, test and production environments; how to simplify automated deployments; and how best to apply software tools (project management, source control, etc.) for Drupal implementations.

Agenda

Dev/Test/Prod Configurations (including Drupal specific hosting configurations)
Leveraging Source Control Management for Team Development
Automated Builds
Code vs. Content Migration Challenges
QA Best Practices
Planning and Tracking Enhancements, Upgrades and Fixes
Eliminating (or Reducing) Unscheduled Downtime (High availability, replication, load balancing)
Preparing for Disaster (Point in time recovery, backups, centralized configuration)

Goals

The goal of this session is to empower attendees with real-world knowledge of how other organizations have successfully managed change within Drupal environments.

Resources

No special resources required.

Updates

Did this tutorial session get recorded? If so, could it be provided?

Build Systems & Change Management

Drupalcon Boston brainstorming session?

heyrocker — Wed, 23 Jan 2008 15:07:42 +0000

Is anyone interesting in having a brainstorming session on deployment and change/build management at Drupalcon Boston? This could either be an officially submitted proposal with an open agenda, or an unofficial get together in a side room at the conference center. It seems like a lot of us are struggling with enterprise-level deployment issues, and getting a bunch of smart and interested people in a room talking about their own solutions and brainstorming new ones would be useful for everyone. We all have our own ways of handling it and I'd love to hear other people's ideas.

This is obviously becoming a bigger and bigger issue for the community as Drupal gets adopted at more and more large organizations and I think there's an enormous amount of interest out there in solutions.

Build Systems & Change Management

Can D7 handle enterprise-level change management features?

Senpai — Sat, 01 Dec 2007 20:46:29 +0000

Now that Dries has concocted a business model that leverages core, I wonder if this is a good time to begin talking in earnest about making D7 into a manageable, upgradeable system with built-in change management features? Dries is one of only a few people with core commit privs in CVS, and as such, he's now perfectly positioned to listen and act on revisions to core that would make everyone's lives easier, but especially his new company's options or desires.

There's a need for small changes to Drupal that allow big business to leverage it to create and manage a system while being able to upgrade that system in a live setting. We've all seen the problems. Dries has too. Now Dries has a company and wants to begin leveraging Drupal's core to provide services to businesses, but then again, I'm sure that lots of other people out here do as well.

Start listing the problems, people! Post a quick comment on this thread outlining something that you've actually run into in real life while re-publishing changes to a D5.x site. What went wrong when you last moved a set of upgraded modules from the dev server to the live server by way of the staging server? Where were the pitfalls in your last QA process? How did you get around it? If enough people start talking about this now, maybe Drupal 7 can incorporate some brilliant ideas to alleviate a few change management issues. Maybe there's some idea out there that's brilliantly simple, and just waiting to be discovered. Start talking!

Build Systems & Change Management

Why can't we just ignore the nid?

Senpai — Fri, 26 Oct 2007 21:56:34 +0000

We're all building sites that have functionality based upon the nid, right? Why can't we break out of that mold of doing nodeapi adjustments to certain nodes based upon a sequentially incrementing number, and try to use something like auto_nodetitle.module to trigger from?

I'm envisioning a scenario like the one posed by jmburnz on http://drupal.org/node/140430#comment-227293 where he says:

A fair amount of content was added to the dev site before it was branched and went live (enough to populate each site section). At times I've needed some additional test content - that is no problem as I can just create it without worrying about what is going over on the live site.
We have used vids, tids & nids as you describe, but these were all set in stone before the site went live.

Well, kidlets, what happens when your node->nid isn't the same on both the live and dev server? How do you perform actuation on content when the users are deleting old content and re-creating new nodes during the week that you've been rolling additional features onto your mirrored dev server?

We could programmatically alter nodes that had a human-readable title if that node's title was unique among all the other nodes. Sure, the uid IS unique, but its also not resettable. At all. Try this little experiment. Use Devel to create 100 random nodes for testing your newest content type. Now delete those nodes by any means necessary. Use the admin pages. Truncate the db. I don't care what you do, you'll never get those uid's back, cause the sequences table is frozen at 101, and it can't simply be reset without throwing off several other things that rely on a uid that's never been used before.

Now think about this. It wouldn't matter to a common user if they deleted a node, then recreated it with the same title, would it? A user doesn't freakin care what that node's number is. They think it's the same piece of content as the last one they had. It'd be like creating a PDF on your computer's desktop. Throw it away, and then re-create it, and when you double-click that second PDF, it will open up, rather than throwing back a <?php $node->nid != '100', 'youcrazyjackass!'; ?> What do you mean, "it's not there"? "I saved it again, and called it the exact same thing!!? It IS there!"

Having a unique element for every piece of content is a necessary evil in the CMS world. But imagine for a moment if we could merge our dev and live databases without loosing content or changing the exact node that nodeapi needed to see in order to do it's thing? How cool would that be, eh? If I want to aim my hook_nodeapi rifle at the piece of content with a <?php $title = '2008 Cherry Festival'; ?> and that title just happens to be unique across the entire site, then nodeapi would activate itself on the "2008 Cherry Festival" story. Every time. No matter what db that story existed upon.

No matter what version of which database was pushed upwards into testing. Or pulled downwards from live to dev. It wouldn't matter.

I wonder if auto_nodetitle can accomplish this for us? I'm gonna check it out. Developers, how would you handle this $node->nid situation?

Build Systems & Change Management

10 min video clip from DrupalCampLA

Senpai — Mon, 24 Sep 2007 18:50:05 +0000

http://www.bryght.com/blog/roland-tanglao/10-minutes-on-drupal-deploymen... has an excerpt from a really great discussion that ensued at DrupalCampLA during Firebright's dissertation on how hard it really is to properly upgrade the featureset on a Drupal site that's live and active.

Build Systems & Change Management

Build Systems & Change Management

What is biggest drupal project you have ever managed ? (size in man-days)

Drupal as a Sourceforge clone?

Post your proposed session for DrupalCampLA 2008

Drupalcon Szeged hosting best practices: development, staging, deployment

Selenium and Drupal

Goals

Why Java/Maven?

Environment Setup

Setup your Tests

Modify the "setup.xml"

DEADLINE FRIDAY: Include Drupal 6.x in Open Solaris

Keeping Blocks in Synch

A Deployment Framework

Deployment and Change Management - The Problem

Drupalcon Tutorial: Best practices in development environments, staging, build management, and production environments

Presenters (Panelists)

Track

Session Description

Agenda

Goals

Resources

Updates

Drupalcon Boston brainstorming session?

Can D7 handle enterprise-level change management features?

Why can't we just ignore the nid?

10 min video clip from DrupalCampLA