Support Composer on testbot and packaging

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

What's your idea?

I can't believe I didn't think to mention this one already...

Drupal 8 is making extensive use of 3rd party components, pulled in via Composer from Packagist.org. This is good.

Drupal 8 is checking all 3rd party components into our main repository, including the generated Composer files. This is bad. It's directly contrary to all Composer best practices, and makes any attempt to update our dependencies consist of 500 KB patches at least. This is not how it should work.

Why are we still doing that? Well, because
1) Testbot can't handle downloading extra code first.
2) The tarball generators can't handle downloading extra code with Composer first. (We can for some other things with distributions, but not core itself.)
3) We're worried about making it harder for developers to work on core because they'd need to run "composer.phar install" periodically.

The solution:

1) Make testbot run composer.phar install on a checkout after applying a patch but before running tests.

2) Make the tarball generator run composer.phar install on a checkout before making the tar.gz file.

3) Accept that running one extra non-Drupal specific command that is quite fast once cached is not a high burden, and is way way way way less of a burden than "learn how to use Git".

Naturally only points 1 and 2 are relevant for d.o in particular.

Composer also has local caching now so after the initial download it should be quite fast, and there is no/less issue around "switching branches on a plane". (Really, we change our dependencies so rarely at this point that is a non-issue even without offline caching.)

What are the benefits?

1) Less code in the repository.
2) Way easier to update dependencies.
3) Development workflow closer to all other Composer-using projects. (Good for bringing in new developers, and teaching developers best practices, and not being a special snowflake.)
4) Easier for site-specific customizations of and experimentation with the composer.json file (which should NOT be viewed as hacking core any more than modifying .htaccess is, as that is also necessary in many cases). That includes using an optimized classloader in production.
5) We can discuss allowing Drupal to be installed directly from Packagist via Packagist's installer. (Optional; wouldn't really require any additional d.o work I think.)

What are the risks?

It's one more step for developers checking out a fresh copy of core. But it's a step we shouldn't be hiding from them anyway.

Advanced developers who checkout from Git directly to start a new site (eg, Pantheon) would have an extra step; but if they're already taking that advanced of an approach then they probably want that level of control anyway.

How can we measure the impact of this idea? (metrics)

I'm not sure.

Who directly benefits from / will use this improvement? (target audiences)

Developers who actually pay attention to our dependencies, or who want to add new libraries without resorting to hacks like composer_autoload or composer_manager. (Or at the very least it can help make those easier to implement in Drupal 8.)

Are additional resources available for discovery/implementation? (volunteer effort, financial backing, etc.)

This should be a very simple change. The Composer documentation is available, though.

Comments

Drush Composer

RobLoach's picture

As far as I know, the Testbot has Drush installed. If downloading composer.phar is not an option, it could in theory, use the Drush Composer wrapper. Drush Composer is simply a Drush component, allowing you to run any Composer commands through Drush.

The following commands accomplish the same thing:

composer.phar install
drush composer install

Either method is an option for the testbot!

This fits with or desire to

larowlan's picture

This fits with or desire to look at using behat for functional tests
Also phing is worth looking at for testbot builds, so then you can replicate the build locally

+1 for both phing and behat

attiks's picture

+1 for both phing and behat

Changing the way Drupal core

webchick's picture

Changing the way Drupal core is packaged is not something that the DA can or should be willing to touch with a 100-foot pole unless there was consensus reached, or a directive from Dries, on the "parent" core discussion at https://drupal.org/node/1475510.

However, making testbot more flexible sounds like a great task for core developers interested in at least opening the doors for this flexibility.

Define consensus

Crell's picture

I freely admit a bias here, but I don't recall any strong voices against doing this other than you and sun. sun's concerns (the airplane switch) have been addressed upstream.

The big blocker isn't consensus, it's the ability of the infrastructure to even do it. That's what this thread is asking be done (which does fall to the DA or a working group therein). Once there's no technical barrier to "doing it right", I think consensus around that will be a lot easier to achieve.

DrupalCI project is moving

moshe weitzman's picture

DrupalCI project is moving along slowly and will eventually let us run composer install during test runs. Just wanted to leave a breadcrumb for anyone who lands here from the future.

Drupal.org 2014 roadmap brainstorming

Group organizers

Group categories

Difficulty to implement

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: