Disclaimer: I haven't been living and breathing Composer for the last few years. I took some much needed time off from Drupal to get an album out and to reestablish some balance in my life. But I'm back in it again, and discovering all the joys and sorrows of our evolving Composer support. I'm not philosophically opposed, and I'm all in favor of the Get Off The Island(tm) movement.
However, either I'm missing something (entirely possible), or the folks who want us to believe Composer solves more problems than it creates love manually fixing merge conflicts in composer.lock and vendor/composer/installed.json.
Assuming I'm not sufficiently Composer-compliant and that I'm doing something stupid, I've just spent the better part of two "work" days reading through nearly every word of each of these posts and issues to understand why things are the way they are and how we're expected to use these tools:
- Using Composer with Drupal (handbook)
- Using Composer to manage Drupal site dependencies (handbook)
- #1975220: Allow a Composer user to manage Drupal, modules, and PHP dependencies with a custom root composer.json
- #2380389: Use a single vendor directory in the root
- #2551607: [meta] Path forward for Composer support
- Using Drupal 8 and Composer (Pantheon.io docs)
- And others...
No where that I can find has anyone brought up a fundamental pain point of trying to use Composer for Drupal sites. While we now have a "root" composer.json that we're supposed to be welcome to modify without being branded as "kitten-killing core hackers", and now that the only way to install certain contribs I need to use is: composer require "drupal/address ~1.0" (which modifies composer.lock for me, as it should) I'm apparently hereby doomed to a lifetime of merge conflicts in composer.lock.
For example, here's exactly what happened to me...
- Create a new Drupal 8 Pantheon dev site back when 8.2.6 was the latest stable core release
- git clone the site repo to my local dev environment
composer require "drupal/address ~1.0"to get address.module and its dependencies (required)- git add composer.* vendor modules/address; git commit -m "address (8.x-1.0-rc3)" (I've automated this part with a simple script to parse modules/x/x.info.yml to get the version string). Pantheon requires me to commit everything (including vendor) to git since that's how the platform deploys the code.
composer require "drupal/inline_entity_form ~1.0"(cuz hey, this is the "best practice" now, why not?), run my git commit script, etc.- ... repeat for the other contribs I need. My composer.json has diverged from core in small and obvious ways. However, composer.lock has diverged in huge ways./li>
- Watch Drupal 8.3.0 come out
- Try to merge the new version of core into my Pantheon git repo
Boom, the last step leaves me with ~60 impossible-to-untangle merge conflicts in composer.lock, since core has touched composer.json, regenerated composer.lock, and pushed the commit as part of the 8.3.0 release. There appears to be no way to recover from this conflict.
Possible (failed) approaches:
A) use git pull/merge -ours when trying to do the merge. Sure, I ignored core's changes to composer.lock and composer.json, but that's obviously not right and will break any code in core that is relying on the latest versions of the dependencies.
B) Use git pull/merge -theirs when trying to do the merge. Sure, I've got core's changes, but I ignored all of mine, and now I have to manually re-run all of those composer require commands (each of which takes 2-6 minutes to run on my totally decked out new Macbook Pro with 16 gigs of RAM, SSD, and the fastest CPU I could buy), and watch composer re-do a metric crap-ton of work. Then I have to re-commit all that crap.
C) According to various blogs that claim to know, "It's okay, just remove composer.lock and run composer update", but that totally defeats the purpose of having composer.lock in the first place! Now I've lost all the data about the specific versions of dependencies that 8.3.0 was released with, and (after 10 minutes pass) I watch various core dependencies get updates that I definitely don't want to have to review and verify myself (changes to zendframework, twig, symfony, etc).
D) Painstakingly and manually try to resolve the merge conflicts in an auto-generated file. The diff is around 2500 lines on my project. I refuse to have to do this kind of BS manual conflict resolution every time a new version of core comes out.
WTF? :/
I understand that dependency management is hard, and that we want to re-use libraries instead of stuffing duplicate code directly into our modules. But surely there must be a real solution here that I'm missing and hasn't been documented anywhere.
I've been a computer scientist for 20+ years, have worked with Drupal for over 11, and I've been intimately involved with Drupal packaging and release management for most of that time. If I can't figure out how to get a relatively simple D8 site deployed to Pantheon and be able to update the code, it seems to me that Drupal is doomed. :/
Help?
Thanks!
-Derek (dww)
Comments
wow, yes, you really need to
wow, yes, you really need to think of managing your modules differently than before.
I have not deciphered all of your post, but I can safely say that you should not be committing the vendor directory and not the module/address directory. Those are both pulled in by composer.
the only thing I keep in git is the composer.json, composer.lock, but I also do need to build with other root files and sites directories, so you may want these in git also.
All of the other directories are built with composer including core, modules, themes, libraries, vendor
Honestly, if you get composer.lock conflicts then you are probably doing something wrong.
That's not how deployment via git works
You misunderstand. Pantheon deploys the code via git. Period. I know the Composer kool-aid expects everyone to have access to "composer install" on their production servers, and likes letting a 15 minute job run to resolve dependencies every time you want to deploy something, but that's a) problematic in all sorts of ways and b) simply not how Pantheon works.
So no, I can't avoid committing vendor (and modules/address) to Git. That's the only way to deploy it.
Regardless, that's not even the problem. The actual problem I'm having is all the merge conflicts in composer.lock, which everyone (for good reason) states should be committed to your VCS.
You shouldn't run any build
You shouldn't run any build operations on production, that is just causing unnecessary load and downtime on the production environment. The best practice is:
This is the quickest possible way to switch a live website to a new codebase. Of course you also need to toggle maintenance mode, perform database updates etc.
If your host only allows you to deploy using git, I would suggest not to use your development repository as a source for the production builds. Instead use a separate "production builds" repository, and commit the entire build in that (including vendor folders and everything).
Then you can keep your development repository nice and lightweight.
This is the way that Acquia
This is the way that Acquia BLT does it—you have your 'source repository' (usually GitHub, but could be any other Git repo), and from there, you either manually build the 'deploy artifact' (using a couple commands on the command line), or you let Travis CI, Jenkins, Circle CI, etc. do it for you.
That process does take at least 5-10 min for most sites (and might also be a time when automated tests are run).
Then, the Acquia Cloud, Pantheon, etc. git repository is only used for these 'production deploy artifacts'.
This is a lot more heavyweight than most people are used to from Drupal < 8 days (it's a lot more like many Java, Rails, etc. apps)... but then again, you can always stick to the typical "download zips/tarballs of everything and commit it separately, and only use composer for specific modules which require it".
I actually use this (more frowned upon?) method for most of my smaller side projects, and it serves me well. I don't want to maintain a separate 'artifact generation pipeline' for these projects which usually only get security and maintenance updates, and I don't want to spend more than a few minutes ever maintaining their codebases!
What am I doing wrong? ;)
All I'm doing is:
composer require drupal/address ~1.0I'm using:
Composer version 1.4.1 2017-03-10 09:29:45Honest question: Where am I going wrong?
Thanks,
-Derek
https://github.com/webflo/dru
https://github.com/webflo/drupal-core-strict is an attempt to solve this problem, by having explicit dependencies on the versions that core has in its lockfile. However, it doesn't really work that well because it seems to hit weird composer dependency resolving bugs: https://github.com/composer/composer/issues/6131.
.
Combine that with using https://github.com/drupal-composer/drupal-project and drupal/core instead of drupal/drupal, then you don't get a composer.lock anymore and drupal core is just a dependency like anything else.
I've been using drupal-core-strict a bit, but it is currently pretty annoying due to the mentioned bug. However, anything that composer update would install within the constraints in drupal core's composer.json that would result in bugs is a bug in core or in those dependencies. Maybe we could have a special test bot run that runs regularly with a drupal update to catch any regressions quickly.
PS: platform.sh works beautifully with composer and doesn't require everything to be committed :)
Thanks for the feedback!
Apparently Using Composer with a Relocated Document Root on Pantheon could let me use https://github.com/drupal-composer/drupal-project and the drupal/core dependency instead of the drupal/drupal one and still host the thing on Pantheon.
But I don't really understand how a dependency on drupal/core is any better than one on drupal/drupal. You're saying that drupal/core doesn't provide the top-level composer.lock. But then aren't we back to the problem where each time I try to update anything, I'm potentially slurping in a crap-ton of updates that I don't want? All roads point to me wanting a composer.lock. But I can't wrap my head around the conflict/tension between needing composer.lock to specify an exactly reproducible set of code and dependencies vs. the need to change composer.lock every time I add another module/theme to the site. If core and I are both changing composer.lock (for the right reasons) aren't we doomed to merge conflict hell?
Still trying to understand the fundamentals here. But I suppose another half day dorking around with https://github.com/drupal-composer/drupal-project and seeing if I can get it to work is my next step...
Ahh, the joys of "progress"... ;)
-Derek
As I said, drupal-core-strict
As I said, drupal-core-strict is what would allow you to force the exact same versions that core uses, if it weren't for the annoying bugs there.
The point of the lock file is to ensure that you get the same dependencies when you run composer install. It doesn't really matter when you commit the dependencies, because then you have them anyway in git already. So it doesn't really make a big difference in your workflow whether you have one or not. But it's a huge difference when you are using composer install and need guarantee to get the same versions.
But it still allows you to add and update specific modules/dependencies, without using the latest version of everything else too at the same time.
composer update still respects the constraints in composer.json in drupal/core, it won't update you to symfony 3 or something like that unless core updates its version constraints. You're just not locked down on every single patch release in core's dependency tree.
You're bumping into a very real, very not solved problem
There's really a couple of issues here that are thornyish with regards to composer, and the way core expects to use it, vs how composer expects to be used.
A composer.lock file is intended to be a manifest of your project, such that you do not need to carry around your dependencies, or keep your dependencies in git.
Core is trying to use it as a lock file for drupal, as if drupal itself is the project. The conflict that results is due to the fact that there is really only ever supposed to be one lock file - yours. Drupal core should really only ship with a composer.json, and not a composer.lock, since drupal is expected to change once users download it, and its becoming increasingly difficult to build a site without needing it. I think either core needs to adjust how they build and ship, and we need to adjust how we build and test core itself (test on max and min dependencies), or we need to make it clear that starting with a tarball of drupal is generally a very poor way to build a drupal site if you need to also use composer.
So, composer does not have any sort of concept whatsoever of cascading locks. There's multitudes of reasons why this is, namely that it would likely be impossible to actually resolve conflicts if it did honor this.
As a result your lock file is no different than if drupal core shipped with a drush make file and you had to change it for your project, then resolve conflicts once newer versions of core came out.
So, the current suggested best practice is what you found: https://github.com/drupal-composer/drupal-project
And like Berdir mentioned, if you want to use exactly what core has been tested with then you should use the drupal-core-strict package as well to lock things down to an exact version.
In reality core should take the following steps:
Then one would be able to start a project with either the tarball, or 'composer create-project drupal/drupal' and everything would work.
Re Point #3: Please make an
Re Point #3:
Please make an issue so we can get this stuff settled.
Also root-core should have a lock file, but it should also require drupal/core instead of replace it. Which is my way of asking: When's the subtree split happening? :-)
I must be missing something
I must be missing something then, when does core change composer.lock?
ahh, when you update. Lol,
ahh, when you update. Lol, yes that would be a problem.
drupal-composer helps separate this issue a bit.
But what I do is only download the git repo from drupal the first time,
Then for subsequent drupal and module updates I let my root composer manage them.
Here is my composer require for drupal 8.3.x-dev
"require": {
"drupal/core": "8.3.x",
"composer/installers": "^1.0.21",
"cweagans/composer-patches": "^1.5.0",
"wikimedia/composer-merge-plugin": "~1.3",
"drupal/address": "~1.0",
I will change 8.3.x to ~8.3.0 when it is released
For what do you use the
For what do you use the wikimedia/composer-merge-plugin package exactly?
Merge compser.json files
So say you have multiple composer.json files (some of which come from packages not downloaded via composer). This plugin will allow you to add thier location and their contents are merged into the requirements, repositories, etc... as if they were all one composer.json file. It's essentially a work around for when you have more than one composer file in items not downloaded by composer.
Why do we need wikimedia/composer-merge-plugin?
Normally, composer-based projects only have one composer.json file. In Drupal 8's case, however, the root project (drupal/drupal) needs to inherit the dependencies of drupal/core, which are specified in core/composer.json.
wikimedia/composer-merge-plugin allows drupal/drupal to merge in the dependencies of drupal/core. You'll see in the
extra:merge-plugin:section of the root composer.json file that drupal/drupal merges in core/composer.json.Eventually, drupal/core will be a 'subtree split,' meaning it will just be a regular dependency of drupal/drupal, and the merge plugin (probably) won't be needed. See: https://www.drupal.org/node/2352091
At that point, you'll (ideally) just be able to say
composer require drupal/core 8.3.0without messing around with all the Drupal-flavored complications in this thread.thanks @generalredneck and
thanks @generalredneck and @Mile23 for the explanation. I've used the drupal-project/drupal-composer project which doesn't have the need for this ATM (doesn't use drupal/drupal), but I can see it's usefulness :)
ooh, 8.3.0 is available I
ooh, 8.3.0 is available
I made the change to my root composer.json and run update to update drupal core
inside composer.json:
"require": {
"drupal/core": "8.3.0",
and from site root:
composer update --prefer-dist --optimize-autoloader --no-progress --no-dev
Yay! updated from 8.3.x-rc2 to 8.3.0
don't forget:
drush updb
drush entup
Composer.lock represents a
Composer.lock represents a set of latest, resolved dependencies from the time composer update was run. If you're not running composer update after merging, how can you be sure that your dependencies introduced in your branch do not conflict? Short answer is you need to run the tool that resolves your "intentions" in composer.json to a specific state.
I'm not going to get into the part about "why doesn't it just merge cleanly when my changes are only additions which don't affect existing dependencies?". No idea... but I'll happily let composer just deal with it by rm'ing my composer.lock and running composer update (again).
There is a Pantheon version of drupal-composer/drupal-project
If you're planning on deploying on Pantheon, use:
https://github.com/pantheon-systems/example-drops-8-composer
This is drupal-composer/drupal-project, altered for Pantheon. This project is referenced in the blog you linked to above, https://pantheon.io/blog/using-composer-relocated-document-root-pantheon. That same blog post explains how you can use the Terminus composer plugin to do Composer update operations directly on Pantheon.
With a relocated document root, you now have a separate composer.json and composer.lock that will not be interfered with when updating Drupal core.
Conflicts in composer.lock can be common, even with a relocated document root -- for example, in a multi-developer project, your change to composer.json / composer.lock may conflict with a change some other dev merged. Most folks resolve these conflicts via:
$ rm composer.lock
$ composer update
Some people are more rigorous about their lockfile, though; see: http://blog.doh.ms/2016/11/28/solving-conflicts-in-composer-lock/
Finally, although you do have to commit your vendor directory when deploying on Pantheon, if you want to work with a repository that does not have the vendor directory committed, you can create a "canonical" repository on GitHub, and use CircleCI to update the vendor directory and push your changes to Pantheon. The Terminus Build Tools plugin can help to set this up with the
terminus build-env:create-projectcommand. See: https://github.com/pantheon-systems/terminus-build-tools-pluginThe Composer Way(tm)
First off, since I have a very obvious bias here I'll just state it explicitly: I work for Platform.sh, and we do full composer-based deployments of Drupal 8 "the composer way", and it works fine. That's how I maintain my own website, currently.
As Mixologic notes above, the problem here is, basically, that Drupal core didn't switch to Composer. It half-heartedly nodded in Composer's direction without actually doing it right, because we were afraid of alienating non-CLI-using users but also didn't want to entirely ignore the obvious Composer-centrism of modern PHP. So we ended up in a half-hearted middle, which is just no good for anyone. Basically, the official tarball is useless for Composer-based sites.
Really, I never use the tarball from Drupal.org for Drupal 8. I only use the Drupal Composer project. At Platform.sh our Drupal 8 project template is based on that specifically, because it does things the "Composer way", fully. Specifically, core becomes a dependency of your project, and the only lock file is yours. That means that, yes, the .z of a dependent package could be different on my site than on the Drupal 8 tarball. This is normal Composer Way behavior. To date that hasn't caused me a problem.
If you really want to fix your dependencies at a precise version (not Drupal's dependencies), then you can modify the top-level composer.json file to specify an x.y.z release explicitly. Then it will never change unless you modify composer.json yourself. (I do not endorse this, but you can certainly do it.)
I've done several point release updates of my site now, and no issue. I even just updated to Drupal 8.3 this morning with a simple "composer update", no issue or conflicts. (Granted, my site is not exactly complex; it does have a few contribs in it, though.)
As far as not checking vendor et al into the repository, that's a limitation of hosts that still do "raw git" deployments. That worked for a long time, and was a huge step up from FTP-based deployments, certainly. But modern web apps need a compilation step. There's Sass to process. There's composer to install. There grunt or gulp or whatever the cool thing is to run. There's running your make-modern-JS-look-like-its-2004 script. There's generated PHP code. (Drupal manages that in a weird dynamic way, but almost no other projects do.) That means deployment pipelines that have no "artifact generation" step and just dump raw git commits into production are now as out-of-date as FTP-based deployments were 5-10 years ago.
Again, there's no way for me to say this without showing my bias so I'll make it explicit: That's why Platform.sh does what it does. :-) See: https://platform.sh/2016/12/production-artifact (with major hat tip to Michelle Krejci, who now works at Pantheon).
The intermediary alternative is to do that build step yourself, and only commit the generated output to your "hosting Git" repository. That is, I believe, what Acquia recommends for instance. But for a modern web app you really do need a build step of some kind between your repository and the production server. There's no way around that. You're just hitting it on Composer first/hardest.
Long story short: You need to either double-down on The Old Drupal Way of managing the site or The New Composer Way of managing a site, and go all-in. Trying to mix and match is going to leave you with a headache, and conflicts in composer.lock. :-) Obviously I strongly favor doubling-down on The Composer Way.
Unfortunately going all in
Unfortunately going all in with "The Old Drupal Way" is no longer an option for most sites.
As soon as you decide you want to include an address on your website that module requires you to switch to a composer workflow. https://www.drupal.org/project/address "Address must be installed via Composer, in order to get the required libraries. The tarballs are provided for informative purposes only."
I am starting to see this requirement more and more from other php-library based contrib as well, and even trickling back into Drupal 7 through a shudder composer manager requirement.
This leaves anyone on traditional hosting platforms with the equally unpalatable options of building locally and hoping for the best or building (and maintaining) their own continuous integration workflow to handle this for them. This leaves larger organizations with the budget able to adapt and smaller ones stuck in that half-hearted middle with only bad options.
old comment, but still ran into the problem with Address module
Most are probably using composer already, but just in case you are not, and don't want to and only need it for the Address module, look to the Ludwig module to help you get those dependencies for the Address module. https://www.drupal.org/project/ludwig
Works like a charm! But only for the Address libraries.
Related to all of this is we
Related to all of this is we need to improve our instructions. See https://www.drupal.org/node/2867757 - the current instructions are a recipe for breaking your site.
Missing the point
Derek's question isn't about deployment or if the vendor folder should be in git or not.
The question is about the .lock file and how git mangles the ever living heck out of it.
You see the lock file is pretty printed json. Git doesn't know how to merge entire arrays so what ends up happening is that git tries to merge multiple dependency arrays together creating an unmanageable mess.
The problems presented using "composer update" and/or pinning your modules to specific releases both range from distasteful to disastrous.
What we need if for the lock file to be written in a smart way for git to handle conflicts.
If each required module/dependency was able to write a single line array of it's config, instead of pretty printed, then git would be able to manage conflicts in an easy to understand way.
Composer does have an input parameter for JSON_PRETTY_PRINT in it's jsonFile->write method. I do not think that any of the code that calls the write method includes this param, so it always defaults to JSON_PRETTY_PRINT on.