Use GitLab for all projects

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

What's your idea?

Instead of moving all projects to github, we might consider using a self hosted GitLab, see also Move Git repositories to Github

What are the benefits?

Since GitLab is very similar to github adoption will be straightforward, but it also allows us to completely control the environment (community edition is open source w/300+ contributors).

Build-in features:
- hosted on our own servers
- better git viewer
- integrated dreditor (comment on line level)
- inline editing of all files
- linking issues
- pull requests
- protected branches
- integrated CI server (a bit limited)
- ability to use something else for issue management
- private repos possible
- extensive rest api

Comparison with github

Github is better because

  • no hosting cost for public repositories
  • slick interface
  • well known
  • supports Prose.io out of the box
  • supports Travis CI

GitLab is better because

  • full control (open source and self hosted)
  • supports external authentication
  • 'free' private repositories (security.d.o. amongst others)
  • might support prose.io for doc team.
  • uses a unique issue number cross all projects, similar as d.o. is doing.

D.o. is better because

Spreadsheet

https://docs.google.com/spreadsheet/ccc?key=0AusehVccVSq2dEtuWm1JQ1ZSV1Z...

What are the risks?

How can we measure the impact of this idea? (metrics)

By the average time it takes to write a patch and get it commited. By the number of people contributing to Drupal core and Drupal projects.

Who directly benefits from / will use this improvement? (target audiences)

Developers, Doc team

Are additional resources available for discovery/implementation? (volunteer effort, financial backing, etc.)

Since this is a self hosted solution, we will need hardware (funding) and sys admins to manage the new environment.

Screen Shot 2013-09-02 at 10.48.49 AM.png

Screen Shot 2013-09-02 at 10.48.59 AM.png

Screen Shot 2013-09-02 at 10.49.29 AM.png

Screen Shot 2013-09-02 at 10.49.40 AM.png

 gitlabhq | GitLab.jpeg

Comments

If I'm reading this properly,

webchick's picture

If I'm reading this properly, I guess the main reason to do this would be to not be reliant on an external, commercial entity for such a critical piece of our project's future. Or is there anything in Gitlab's feature list that Github doesn't have? It might be easier to start from there, so we understand what we'd gain by incurring the ongoing costs you note down below (hardware, sysadmins...), which of course wouldn't be present with the Github option.

I wonder if Gitlab could also

Liam McDermott's picture

I wonder if Gitlab could also be themed to at least use the Drupal logo, since the cohesive nature of everything being on d.o is very important to the sense of community. I noticed drush have moved and having to go to github to get Drush honestly feels like it's no longer a Drupal project.

Hopefully using gitlab is a decent compromise, I'm really scared a github move is going to destroy the sense of community on d.o, something most developers who're jumping on the bandwagon probably haven't considered.

Gitlab

attiks's picture

I send an email to gitlab, asking to chime in.

I'm here

sytse's picture

Hi attiks, thanks for the invitation. I'm a GitLab.com co-founder and we would love to have Drupal use GitLab. I'm wondering about one thing, would you need public repo's?

Hi, sytse! Nice to meet you!

webchick's picture

Hi, sytse! Nice to meet you! :D

Yes, Drupal.org currently hosts around 30,000 public repos and 28,000 developers. Our needs for private repos are fairly small... I can actually only think of one or two we would need, all for "actually running the website" type of stuff.

Nice to meet you too! Public

sytse's picture

Nice to meet you too! Public repo's are currently not in GitLab but we might add them in the near future. Drupal planning to use GitLab would certainly speed this up :-)

Public mode

attiks's picture

My bad, I thought that was what public mode was for, but I tested it and it seems you still need an account first.

Currently public mode only

sytse's picture

Currently public mode only gives you a clone url and some basic info, not everything that people expect. For an example see https://gitlab.com/public

Will be fixed on October 22

sytse's picture

Public mode will be fully browsable in GitLab 6.2, released on October 22.

Will be fixed on October 22

sytse's picture

Public mode will be fully browsable in GitLab 6.2, released on October 22.

This feels like the right direction to me

rgristroph's picture

This feels like the right direction to me. Another plausible alternative is continue with the current drupal / git based infrastructure and enhance it to have pull requests and other desired features.

It feels right enough to me that I would contribute some resources to it, in terms of money and whatever time I have available (not much). Maybe there's something that could be done to prove the concept and move it down the road quickly.

What I have in mind is some sort of plan to:

1) Set up a gitlab that mirrors the d.o repos, or maybe just a few of the more popular modules
2) Have it pull some issues from the feeds so that it is actually useful for trying things
3) People won't actually try it in any meaningful way if work they do gets stranded there (with out manual effort). So we'd need something, maybe an extension of Dreditor or another greasemonkey script like it, that allow one-click generation of a patch & comment back to the home d.o issue.

It seems like a lot of work in glue code that will be eventually thrown away, but that's they way these types of integration projects go, I guess. Also, instead of the gitlab, I'd be willing to entertain an alternative of a Drupal Project with a lot of enhancements, but I fear that would not get us a good PoC very quickly.

When I say I'm willing to put in resources, I'm thinking from $1,000 to $5,000 and recruiting other money. Maybe one of the crowdsourcing systems out there could be used for it. I'm willing to put in that much, because I see Drupal as worth much more than that to my career, I see that it needs better infrastructure so that I can make money off of all those other contributors shared work, and I think the github solution won't cut it in the long term.

Wow, this is great!! Please

webchick's picture

Wow, this is great!!

Please make your intention to provide resources around this known by editing the wiki page. I think your approach sounds very sensible; not sure how long it would take to put something like that together.

Test site

attiks's picture

I installed it on my local machine, and will try to install it on a public server sometime this week, so it can be tested.

Great, please email

sytse's picture

Great, please email support@gitlab.com if you need any help.

Github issue tracking is awful

jhodgdon's picture

I am using GitHub for a group project developing a Drupal site, and I personally think that their issue tracker is terrible, compared to what we have now on drupal.org. Using the Github tracker would be a big step back in issue queue usability for my purposes. I can't even begin to elaborate on all of the features we take for granted in issue tracking that it doesn't have, because it's so basic that it has hardly any features.

Did you checked the gitlab

attiks's picture

Did you check the gitlab one as well? It's similar to github, but the good thing is that it can be changed.

This is actually a good

webchick's picture

This is actually a good point... improvements made to Gitlab would help benefit the larger open source ecosystem using Gitlab as a whole, as opposed to improvements made to Project* module which only generally benefit Drupal.org.

Though I'm somewhat dubious about us being a major contributor there, given that most people don't contribute to our current set of tools, which are written in PHP and not Ruby. Still, there are a lot of tinkerers out there and we might be able to help. :)

Yeah, I think the interest of

sytse's picture

Yeah, I think the interest of Drupal users will make GitLab better. There is also interest of Fedora in using GitLab, and we hope for more open source projects, see http://blog.gitlab.org/packaging-gitlab-for-fedora-a-gsoc-2013-project/

There's no easy way to use

danillonunes's picture

There's no easy way to use Project and other Drupal.org tools for private purposes. It's so hard to install a replica of d.o. project management tools and it's so specifically designed for Drupal project that nobody bothers to try it unless they are really seeking to contribute to d.o.

GitLab can be configured to

sytse's picture

GitLab can be configured to point to external issue trackers, see https://github.com/gitlabhq/gitlabhq/blob/master/config/gitlab.yml.examp...

External issue tracker would be good

jhodgdon's picture

I posted on the Github thread about the specific problems with Github's issue tracker (as did others). Basically it lacks most of the features of most other open-source and commercial issue trackers, such as dedicated meta-data fields for status, severity, category, etc. (which IMO are absolutely essential. the ad-hoc tags that Github's tracker support are not a replacement), and the ability to upload files.

Rather than spending time developing another issue tracker system or contributing to this one, the ability to use an outside tracker would be better.

Assuming that we no longer want to retain the Drupal-based Project Issue, Bugzilla would be the logical choice. Some people don't like it, but it at least is adequate and actively maintained, and many many other large and complicated open source projects use it (Mozilla, Linux kernel, etc.).

Bugzilla?

Crell's picture

My experiences with Bugzilla have been rather uniformly awful. I once tried to install it at a previous employer and couldn't because it required so many system dependencies, eg, it required an LDAP server to even install. That was admittedly 8 years ago and I'm sure it's improved since then, but I've not had a pleasant experience with it.

Project* and Redmine are the issue trackers I hate the least out of the ones I've used. (Redmine wouldn't be bad, actually.)

Redmine wouldn't be bad,

Gaelan's picture

Redmine wouldn't be bad, actually.

Except that nobody but me (in the Drupal world) would touch the code with a 10-foot pole. :)

I've used Redmine for smaller

haydeniv's picture

I've used Redmine for smaller projects and quite like it as well. It'd be nice to integrate some of the features but I don't know how well it would scale for something as large as d.o. It does have some really nice REST APIs though and if you are sly with your commit messages it will automatically update issue statuses for you. For those who are wondering, Redmine is written in Ruby. :)

Oh and I like how you can respond to an email notification and it automatically posts the response to the issue queue.

duplicate

attiks's picture

There's a new - duplicate - proposal created at https://groups.drupal.org/node/314193, which should be merged into this one.

GitLab has a free hosted

lpalgarvio's picture

GitLab has a free hosted service since a bunch of months ago - GitLab Cloud.
http://www.gitlab.com/cloud/

Unlimited private and public repos, organizations/groups, etc

So we can use it self-hosted or hosted at gitlab servers.

Luís Pedro Algarvio
Drupal and DevOps Developer, Evangelist & Trainer
lp.algarvio.org

Gitlab podcast Nov 4th

Elijah Lynn's picture

I am listening to this now and recommend having a listen. Gitlab has over 300 contributers so far and works very hard on growing their community.

http://pcasts.in/Ni32

FWIW, I added a feature

aboutblank's picture

FWIW, I added a feature suggestion to enhance GitLab issues with similar parameters provided on the d.o. issue tracker: http://feedback.gitlab.com/forums/176466-general/suggestions/5083989-mor...

We're using self-hosted GitLab on a project I'm currently working on, and I'm happy with it so far, besides for pretty crumby issue queue. However, the GitLab core development team is very responsive, so I'm sure they'd be open to enhancements.

I've been familiarizing myself with the GitLab codebase, so I may just start working on this and see what they think.

// Jason

Gitlab Webhooks and API

minorOffense's picture

We're working on a series of modules to integrate the API layer in Gitlab with Drupal using Services and WSData.

You can find the first of these here.

https://drupal.org/project/gitlab_webhooks

These could potentially help quite a bit in the process of switching to Gitlab. As we complete further API modules I'll post back.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

Gitlab feedback

attiks's picture

I had a great talk with the Sytse from Gitlab

We would love to see Drupal using GitLab to enable better collaboration!

As discussed the free GitLab CE also supports LDAP user login.

If you do need to use GitLab EE we are prepared to give Drupal.org a free license for this.

We can also do a free call with your operations people to set up a highly available system.

The PHP API library I mentioned was: https://github.com/m4tthumphrey/php-gitlab-api

Where did this end up?

gabesullice's picture

I think gitlab would be a great solution, has this gone anywhere?

Edit: This is also something I would be very interested in working on.

Looking at

cjoy's picture

Looking at https://drupal.org/project/gitlab_webhooks it would appear that @minorOffense is still working on something.

Big fan of GitLab as well and I feel it would be preferable to direct migration efforts towards another major open source platform, rather than go with a proprietary Git solution.

Even if few d.o contributors would venture into the Ruby codebase (not sure the language is the main issue anyway - how many people contribute to d.o tools now?), the scale and exposure of d.o alone would probably help improve GitLab.

We have other modules to

minorOffense's picture

We have other modules to publish as well. Just haven't gotten to it.

I'll see about getting those up for next week.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

Drupal GitLab module

minorOffense's picture

Here's the API module. It's not 100% API complete just yet but it works (we use it for Dropfort.com)

https://www.drupal.org/project/gitlab

It doesn't do anything on it's own, just if you want to play with the API.

We didn't use the PHP library mentioned in the API docs for GitLab since we wanted to build it in a "Drupaly" way. We wanted to make it possible to link GitLab data to fields or entities or just use the webhooks. Hence the dependency on WSData.

WSData (https://drupal.org/project/wsdata) and RESTClient (https://drupal.org/project/restclient) which both take care of caching, the actual HTTP requests, adds the ability to link data to entities and other configuration.

Anyways, if anyone has questions on how to use it or want a demo, just post to the issue queue. We haven't gotten any example code published yet but when we do I'll update here and on the module's project page.

Thanks.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

The question that kills github

chx's picture

The current drupal.org workflow is such that if A posts a patch then B can post the next patch. This is simply impossible with github unless A gives permission to his/her repo to B which of course is not viable. Does gitlab allow for such "swarming"?

A Forking Model

gabesullice's picture

Could we not make all repos public (Drupal requires all modules posted to it to be open-source anyway) and simply follow a forking model? I imagine it would look something like this: A forks the module and creates a topic branch (this could probably be automated), does some work, then submits a pull request (akin to a patch now) and the PR is posted and linked in the issue. Now B can review the PR. If it needs work, B can fork the module repo themselves, add A as another remote and then merge the branch of A into a topic branch in B's repo. Finally, B submits a new PR and closes A's.

This is superior to the patch model in that commit histories can better be fully maintained, while taking advantage of proper commit message standards. Plus, PRs can quickly be updated to apply to the latest module code by rebasing them onto the latest work. Currently, patches quickly become stale and in a large fast moving module, patches need to be rerolled many times.

This also falls nicely in line with Dries' Amsterdam proposal because we can come up with a really clean and concise means of attribution if we have all the space that a full commit message offers over the one liners that are standard today.

I asked the same thing to

minorOffense's picture

I asked the same thing to webchick when she was here for DrupalCamp Ottawa. The problem with that is if the owner of the fork disappears all the work in the repo could be lost, stuff like that. Instead of having it all in the current repo where the module maintainer can keep track.

But yeah it would be nice to use.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

Seems like an easy problem to

gabesullice's picture

Seems like an easy problem to fix. Sort of like user deletion option in Drupal, "Remove user, but keep all content." WE could simply tie user accounts to d.o and when an account is deleted, simply change the namespace of their fork to anonymous or some random hash to avoid name conflicts.

Out of the box

attiks's picture

Out of the box, no, but since it is open source we can easily use the API (or custom code) to create a fork and give everybody (or everybody in that issue) access to it, or his/her own branch.

We can setup the 8.0.x branch to track Drupal and only allow merge requests to it, so once the issue is finished we have a clean patch.

AFAIK this is a just technical problem that can be solved

Gitlab also has the concept

minorOffense's picture

Gitlab also has the concept of protected branches. Where only the repo master or owner can commit to them. But keep the release branches locked.

Maybe use issues branches, let people push/pull to them and when the code is good the project owner just has to merge it in.

Just a thought.

Here's a breakdown of the permissions.

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/permissions/perm...

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

Proof of Concept

gabesullice's picture

Is there a way that we could demo this or take some concrete action in this direction?

Gitlab.com is free and you

minorOffense's picture

Gitlab.com is free and you can try out all the features.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

I should have been more

gabesullice's picture

Sorry, my bad, I should have been more clear, I'm really all for this and have already given GitLab a try. Now I'm just wondering what steps we could take that would move this more toward taking a decision. Be it demoing that we can tie GitLab into a Drupal site, or actually working with the GitLab guys to discuss some of the features we need etc.

gitlab guys

attiks's picture

I spoke to the gitlab guys in the beginning of this year and they are open to help us in any way they can, but I guess we can already solve most thing with what is already available

Well I know we can tie it to

minorOffense's picture

Well I know we can tie it to drupal for some of the workflow required. We built a tool for people to package and release private modules against both github and Gitlab.

http://dropfort.com

Not done yet but core functionality works.

The issue queue API we haven't played with yet but it's on the roadmap.

I know you can set the protected branch stuff through the API and create branches. The initial repo config you can't get through the api but I'm sure that's a feature can get made. Also making commits through the api doesn't work yet. But they recently added a lot of nice api calls fire dealing with tags and labels for issues.

Works pretty well actually.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

As has been repeated many

Jaypan's picture

As has been repeated many times in this thread, moving Drupal GIT to a 3rd party service is a bad idea, and it's very doubtful you will ever get a consensus agreeing to do so. Time will be much better spent focusing on improving the Drupal infrastructure to implement the positive functions from Github into Drupal's infrastructure, rather than talking about moving to Github which will almost definitely never happen.

Only local images are allowed.

Gitlab is self hosted just

minorOffense's picture

Gitlab is self hosted just like the cgit or whatever drupal is currently using. They have a hosted version too but you can run your own local copy.

Using a standard or existing solution follows with d8's motto of using what's already available rather than having to build and maintain it all yourself.

Gitlab is a perfect example of a tool we could use. It's supported, feature rich and open to customization as required.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

My bad. I was misreading

Jaypan's picture

My bad. I was misreading Gitlab as Github. I agree that integrating a complete system within Drupal.org would be a good move, particularly as it's almost impossible to do development for Drupal.org unless you are in the inner-circle.

Only local images are allowed.

Drupalcon BoF

gabesullice's picture

Would anyone in this thread be interested in attending a DrupalCon BoF session?

I would love to get something more concrete started as well as enlist a group of people able to bring something like this to fruition.

I'll be there. If there's a

minorOffense's picture

I'll be there. If there's a BoF I'll attend.

--
Mathew Winstone
CEO/Co-Founder - Coldfront Labs Inc.
http://coldfrontlabs.ca
@mathewwinstone

git workspaces

YesCT's picture

I would recommend attending https://events.drupal.org/losangeles2015/sessions/issue-workspaces-bette... which should have an update for how the DA staff plans to tackle this.

Cathy Theys

Those interested in this

davidhernandez's picture

Those interested in this subject, who are attending DrupalCon LA, may want to attend this session.

https://events.drupal.org/losangeles2015/sessions/issue-workspaces-bette...

"In the past few years there were numerous conversations in the community about the need to improve issue queues and Git workflow on Drupal.org. Suggestions ranged from implementing pull requests to migrating to a different self-hosted solution, such as GitLab, or 3rd-party site, such as GitHub. Meanwhile, new technical solutions have emerged for some of the challenges we are facing. One of them is issue workspaces."

It may be a done deal but....

metzlerd's picture

Gitlab 8.0 added continuous integration support that would allow for distributed test bots etc. I'm starting a pilot to use it at my work place. If this is something we're thinking about doing, I'd loved to stay involved.

Still on the table.

heather's picture

I just started working for GitLab, and as a long-term Drupal user, I'd love to see this.

I'm not sure if anyone is still considering a move to use a repository management system for the Drupal project? I think there are wide-reaching benefits in terms of making it easier to track, monitor and collaborate for all users.

Anywho! If it's still on the table, I'd love to know more/talk about it :)

+1 for GitLab

jimafisk's picture

I've also just started looking into GitLab and enjoyed my experience so far. It seems to fit with the "get off the island" theme. It'd be great to see these two open source projects work together to make each other better!

I think this sounds like a

alexdmccabe's picture

I think this sounds like a great idea.

Related article

ultimike's picture

A good primer on why this stuff is so difficult: https://www.drupal.org/drupalorg/blog/the-drupalorg-complexity

-mike

Not only git and gitlab

DevElCuy's picture

There are other areas that need change, I did a summary and proposal in this post:

https://steemit.com/drupal/@develcuy/is-drupal-the-right-tool-for-drupal...

Feedback is welcome!

--
[develCuy](http://steemit.com/@develcuy) on steemit

Finally

colan's picture

Looks like this idea has gained momentum: Technical Advisory Committee Update.