Building Drupal projects on Git

Somewhere between next week and next month. With any luck, next week. The two basic goals of the manifesto, as I'm writing it right now, are to a) step through the process of building a dog-managed site, and b) explore the world of possibilities that the standardized git approach opens up.

The Drush maintainer team has been talking about something like this for a while - we also really want to enable making this kind of sophisticated git workflow easy to use. We have actually be considering making the next version on the "pm" commands highly git-centric (with a much simplified wget function as fallback). From my point of view it looks like what is described here is exactly what we need - I think we would want to include in some higher level interfaces (project name parsing/validation, dependencies, pm-info, pm-updatecode etc), but this would get us a long way.

I considered writing this as patches to drush core (albeit quietly) for a couple weeks before deciding to roll it out as a separate package. Lemme quickly be clear about what, in my mind, having it as a separate package does/does not mean:

I ultimately arrived at dog as the right approach after considering a number of different levels of drush integration and considering the implications that each level would have on ease-of-use for users, consistency of the overall experience, and the ability for the system to 'right' itself after user interference, etc..
I do not see dog as a challenge to or replacement for the existing pm functions, or even necessarily the git_drupalorg package handler. They work well for the patch-together-your-workflow case, and however cool dog might get, we shouldn't ever take away the swiss army knife. Folks can be luddites if they want :)
Also, I have no problem with eventually rolling dog directly into drush once it's more complete, if we all think that's the best way to go. I think it'd be great, actually - would certainly make provisioning with dog easier. But for now, I'd like to keep it separate until it matures, at least into alpha code.

That's where I'm at on it. Sorry to have been a bit quiet about it and just handwave about having a manifesto (which is, yes, still in progress); I kinda went dark on big, wide discussion when I realized that this needed a foundation. Now that this bit is out at least, I'd really welcome some wider discussions with drush peoples - though waiting for the manifesto could really give the fullest possible context.

The only thing I am not sure I understand is the manifest file - doesn't this just replicate the contents/purpose of config and .gitmodules? The URL for the collab repository should already be sufficient to rebuild the exact tree using "git clone --recursive git://host/repo.git", so I am not sure that distributing a file to do the same thing adds much. Of course, it would be really useful and important to be able to list out a manifest to summarize "what is in the site and where does it come from/go to", since collecting that info with submodules is really not that fun to do by hand. This feels more as a human readable command output though, not something that needs to live in a file.

Buncha reasons for the manifest. Here's a super-simple one to start: pretty much every dog repository is going to have two remotes - upstream and collab. There's no way to record that directly using submodules - it needs to be kept somewhere else.

There's also the case where you want to be able to make a pseudo-submodule that tracks a particular collab/ in a particular subrepository. You can't really take advantage of submodules very effectively at all in that case (nor will a recursive clone get you anything). That case is not on my immediate critical path with this, but it still shouldn't be impossible.

There's also the case where you might want to use a working directory for a repo that's outside of the webroot (helps address http://drupal.org/node/1119802). No way to set that up natively with submodules, but it's easy if you have a meta-system like dog retaining config setttings for core.worktree that it can roll out in every new instance.

That last case highlights where I expect a lot of additions to be useful for the manifest file - custom repo config (as in .git/config) that is impossible to transmit using any built-in git functionality. And not even just config - this system will seriously leave GUI users behind unless we can roll at least some of the functionality into git hooks. And ensuring that the right sets of git hooks are attached to a repo can't be done without local action - managed by something like dog, reading from its manifest.

So while I most definitely agree that part of dog's responsibility is to automate complex-multi-step processes into unified concepts with single commands (because, let's be honest - submodules are really frustrating and annoying), it'll definitely need its own manifest data to do it properly.

Yeah, we talked about this a fair bit last week. There's no problem with doing that at all. It means you maybe get a slightly mucky history in your core repo, but oh well, we're all used to that. Best part is, if later on you decide to take that custom module and make it its own real repo that you contribute back, you just run a quick git filter-branch and separate out the commits on the subtree for the theme/module and make it its own separate repo. Ipso facto magico, push it up to d.o.

Also, if repo proliferation is an issue, you can also just use a d.o sandbox for that code (if the code is appropriate to put in public).

Sam: I'm looking forward to discussing all of this at our call next week. :-)

Me too :) But in advance of that, let me clarify one thing, re: this point -

Those last examples imply another desirable property, if not a requirement. It should be possible, and in fact the most common setup, to have the primary repo for a site hosted at the hosting environment. When a new server needs to be launched, it is much more likely that the hosting environment can reach a repo on its own servers than that it can reach github or whatever, especially when the site is hosted in Singapore or Australia or on the moon.

I think there's a key misconception here that threads throughout, so let me at least clear that one up. While 'upstream' repos are at some canonical location (e.g., drupal.org), 'collab' repos can be located anywhere. And 'collab' is the only repo from which a new site instance is ever rolled out. So a hosting provider invested in dog can (and should, to smooth the process) provide collab repos for everything - so all the repos are indeed locally available within the hosting provider's network. Which also means writeability. And notification hooks that are readily available. My thought all along has been that hosting providers like Dev Cloud would do all repo hosting locally, and maybe possibly in the future allow hosting from an external provider.

One other point to address, though:

Also, if the hosting environment needs to know anything about the methodology the repo owner is using in order to find the code (e.g. "this is a DOG repo"), that is likely to lead to complexity and unreliability, because there will always be edge cases, people who need to do things slightly differently, whatever. So saying, "there are remote repos you need to contact, but you can find them in the file foobar/.manifest", while seemingly implementable, is going to be painful long term.

The goal of dog is to create a portable Drupal package that is both versioned and deployable, both human & machine read/writable, and easily encapsulated within wrapping deployment workflows, config management, and/or build systems. In the current implementation/spec, it carries a pretty minimal amount of information required to do the internal assembly - and most of that is just git settings to pass around. To that end, I should note that building workflow-specific applications on top of Git is not something we should be circumspect about - because the Git developers aren't. Read the git dev list for a bit - you'll see that the express intention is for people to build systems around Git.

That said, I entirely agree that one of the primary requirements for dog is that it is internally robust - that it is nigh-impossible to get projects managed by it into unrecoverable, or even unpredictable, states.

Liked how this was written; great documentation!

Saw a few things I stumbled on while learning git fairly well explained.

Would really like to see something about subtree instead of submodules... I remember that there was a nice thing written about those in the git pro book just after the chapter on sub-modules (and how not to use them).

Now that you’ve seen the difficulties of the submodule system, let’s look at an alternate way to solve the same problem.

http://progit.org/book/ch6-7.html

Need to read more of the thread to see if this is discussed down there, but I'd like to see if some though have been put into using subtree merges instead of submodules in this proposed workflow.

As long as that alternative core is a good steward and is itself based on a clone of the original Drupal core repository, then it's not too tough. It'd be more or less the same as a standard update from upstream, actually - except that you change the git URI for your upstream source first.

Ah, I actually misunderstood this a bit at first because I didn't read the link. So:

• As I said, $Id$ tags should be gone. They no longer have any place in Git repositories...though I'm a bit ashamed to admit that, after spending an inordinately long time getting all the logic in place to make that happen, it turns out there's the gitattribute ident that would allow us to continue simulating the feature. Oh well. Anyway, point is that the tags should pretty much be gone from repositories.
• LICENSE.txt won't make it in to repositories anytime soon. It's been suggested to me by legal-y people that this is a potential issue, though that conversation's gone dead in the water, and I'm operating under the assumption that it's a small enough problem that we can deal with it.
• The information in the .info file is for the update manager. That's rendered totally unnecessary by git_deploy.

Updating existing projects to use dog is not something I'm looking to deal with right away. It's important, obviously, but the basic functionality needs to be working for new sites before we can think about scooting existing ones into this format.

And to be clear, it's not just "especially" relevant for existing projects. It's ONLY relevant for existing projects. If you're using dog, you should never ever ever EVER have a single tarball from d.o in your system. All git repos, all the time, period. Hybridizing makes things unnecessarily complicated.

DOG is similar, but somewhat more ambitious than Drush Make. You should review what we're talking about in terms of using git submodules.

The Aegir+Drush Make creates "one big repo" for each platform. That's fine, but you're stuck doing all upgrades by hand. A big part of the value of the DOG process is having the ability to pull updates/upgrades to Core, Contrib and your own Custom code from their actual sources. This lets you really leverage git much more.

Whether or not you then use Aegir to deploy is a totally other question.

As others explained above already, DOG could work as a perfect tool to manage your code, while Aegir and Drush Make are designed to work on different level, since they help to manage your sites and their environment (the platform) but they don't help in managing the code at the low level at all.

It helps to think that in Aegir context the app is not a site, it is an install profile with corresponding makefile, used by Drush Make to create an environment (the platform) where the app lives, and the site is just a deployed and managed (with the help of Aegir) instance of the app.

But that instance (the site) and its environment (the platform in Aegir) still has the code you may want to maintain both on the platform level and on the site level (for site specific stuff), and you don't want to maintain separate platforms per (every) site, plus, Aegir will not help at all with the code in the site specific space (it just moves it between platforms as-is w/o any comparison checks etc), so you still need something to manage/track the changes at the code level, and this is where the DOG can be a perfect match for Aegir and Drush Make, not a replacement for anything, imo.

I see the default is the latest stable branch. Stupid question anyway...

Got the reply from sdboyer at #drush
eventually the plan is to make it work so that someone else can prep an upstream dog repo for you and then you use dog-init to grab it down, but it's gonna be a little while before we get all that in order

Lot of great points here, thanks for the thoughtful response. Took me a while to respond because I was thinking a lot about it :)

Coming back to the important missing feature I mentioned early. With the approach dob takes, it not possible to reuse the contrib Drupal repo across multiple projects. You could probably have different branches for different project, each branch having different submodules at different version. But that's a mess when switching branches because Git at the moment is not able to remove unused submodules when switching branches. Then, depending on your workflow you might have different branches per project (master/staging/production) so that you would have 3*(number of projects) branches plus all the branches coming from upstream, plus the features branches you use for development. This is worse than ugly!

First and quickly, on the annoyances of submodules (such as them sticking around across branch switches): yep, and if you read over some of the other comments in this thread, you may see that while I was initially inclined towards always using submodules everywhere, problems such as that, as well as the spammy history it can result in, led me to only use them in certain cases. Where submodules are used and a branch switch occurs, the post-checkout hook is where we do the cleanup. Getting the right hook into place in an automated fashion is one of those things dog is good at doing.
Feasibility of sharing a single contrib repo (e.g., Views) across multiple site projects: when I was first thinking about dog, the case where a shop would want a base install to start with was actually near the front of my mind. And yes, if you use a remote configuration that's unmodified from the base set up by git clone, e.g., this:

then you'll have a proliferating mess of branches. But do just a little namespacing magic, say for a project called "projectfuntime":

And your local master branch is linked to a namespaced branch in the collab repo called "projectfuntime/master". (There are a number of other tricks, but that's a start). It is true that there's no git-native way to just fetch a namespaced subset of branches with a glob (fetching a single branch is quite easy, but we also need to be able to discover feature/hotfix/etc. branches), which means the remote listings will get cluttered - unless you do some cleanup in some custom porcelain. Like, say, dog.

So, I strongly disagree that it's impossible to reuse the collab repos on multiple projects. In fact, I don't actually see how, looking at your description, your pattern makes anything more reusable than the above outline for dog does - it's still just submodules pegged to a specific ref. Dog can do that, or it can attach to the repository more loosely. Where the subrepo is ultimately placed within a particuarl dog-managed project is irrelevant to reusability - you still have all the same basic problems of many-branches-to-contend-with, unless you make the assumption that all your projects will be in lockstep on their shared repos.

Also, a quick note - I'm hoping to roll Phing in to dog somewhere, also probably pretty transparently to the top-level interface, as the tool simply won't be that useful unless you have a way of managing target-specific variations between instances (e.g., differing mysql connection strings on your local dev vs. staging boxes). Build targets ftw.

(On side note: I think it's a terrible idea to call that remote contrib, Git best practice is to have an origin and an upstream remote!)

It's collab, not contrib - contrib wouldn't make sense at all. I disagree that it is a "best practice" to use origin, especially in a case (like this one) where you have two very important remotes that need to be interacted with regularly. Using "origin" works best when there is just one remote repository you interact with regularly; I came up with this convention in the first place specifically because I wanted to avoid those connotations, and the names reflect the purpose.

The solution we came up we like to call deployment-platforms and it's basically a very small Git repo with a lot of submodules. It looks like this:

I considered a repository layout like this, with an outer super-repo that contains the core clone. . Truth is that there are advantages to having the base dog repo != to the webroot - a lot of potential advantages, and it's my preferred solution for the long term. Many sites, especially bigger sites, want to put stuff into the repository that really shouldn't be under the webroot, for example. I initially rejected it, though, because it hadn't occurred to me that we could place all repos in the thin, outer super-repo, thus avoiding the hellish commit noise of submodule commits from a module echoing first up into the core repo, then again up into the super-parent dog repo.

However, having reflected for a few days on this layout you've suggested, I'm now about 60% sure that I'm going to take a note from it. We'd adopt a strategy where we have a base repo that contains ALL other repos, including core (there's a nice symmetry to that). The biggest drawback is an ease-of-use consideration - before, you only needed to set up one collab repo per project. Making this switch would mean at least two are necessary - the dog super-repo, and the core repo. I worry that that could be just enough of a hurdle right at the outset to keep people from adopting dog.

Now, Git is NOT a software deployment tool! So, what we're trying to do, is to have Jenkins/Hudson use the deployment-platforms to run tests and build Debian packages for Drupal, modules, themes, libraries. Those packages are automatically injected to our package repositories, which later are used by APT and Puppet for deployment to the target systems.

We're pretty well in agreement there. Neither Git, nor even dog, are deployment systems in and of themselves. I think people get confused because they're mistaking necessity for sufficiency: you need good versioning & collaboration tools to build your project in the first place. Add in the fact that git can do all kinds of transport, and it's easy to make the mistake of thinking that deployment is just a hopskipjump away from your normal dev process. But it isn't - or if it is, you probably lucked out. Dog is really about building that portable package that a true deployment-focused tool can easily roll out elsewhere. The set of tools you've described are a shining example of a way of how these tools are package-builders that slot nicely into tools which really do deployment and provisioning. Actually, I'm very interested by what you guys have set up and would love a tour, if that's possible? :)