Thanks to everyone who voted on Drupal.org ideas! Based on that, as well as a few conversations with some key Drupal.org infrastructure maintainers, here's a draft of the proposal for the board meeting next week, for your review!
(Original at https://docs.google.com/a/acquia.com/document/d/1_kTZrxPHArdDeyAZvV3-mj_...)
How the Drupal Association is going to make Drupal.org rock for developers and site builders
Background
In December 2011, the Drupal Association board held a two-day retreat to decide its strategic priorities for 2012. Two of the six priorities identified by the board were “Improve the collaboration tools on drupal.org and make it rock for developers” and “Make drupal.org awesome for site builders.” (Of course, this also means an un-specified third priority of “Make drupal.org awesome for businesses” so they can help us pay for this. :))
Board members Angela Byron (webchick) and Donna Benjamin (KatteKrab) embarked on some initial brainstorming and research:
- Interviews were conducted with members of Drupal.org’s infrastructure and webmasters teams to determine their priorities (since that’s what keeps the plumbing running).
- Drupal Association Executive Director Jacob Redding (jredding) was consulted regarding sponsors’ needs (since that’s what *pays* for the plumbing :)).
- And finally, a public voting site was put up at http://drupal-association.ideascale.com/ to get community feedback on the Drupal.org roadmap. (Huge thanks to tvn for ENORMOUS help in gathering up and organizing an initial list of ideas to populate the site).
Through this process, we’ve developed the following plan for improving Drupal.org in 2012 that helps fulfill the needs of these distinct groups. And we will need your help to make it happen!
Quick Summary For Busy People
The Drupal Association wants to hire a development team to focus full-time on Drupal.org, supplemented by subject matter expert consultants where appropriate. We’ll start small (just reallocating existing staff to focusing 100% on Drupal.org) and work our way up from there, based on fundraising success.
Here’s the short version of what we plan for that team to work on:
Site builder improvements
- Add ratings and reviews to projects, as well as other metrics, in order to help determine module quality and aid in comparing modules to one another.
- Create a dedicated support section on Drupal.org in “question and answer” format, to help users get useful help faster.
- Improve the case studies page
- Revamped events section on groups.drupal.org to find user groups and camps in your area.
Developer improvements
- Back-link commit messages on issues, to show a full history of code changes within the context of a discussion about the change, as well as a better way to catch bugs.
- Improvements to the “Follow” functionality on Drupal.org to make it more usable, and provide additional details such as who is following an issue.
- Turn project pages into groups to do per-project documentation, events, announcements, etc.
- Drupal 7-based development tools, leveraging modern modules/practices that folks are already using on their “normal” Drupal projects (Entity API, Rules, Organic Groups, etc.), which are much easier to extend.
Business improvements
- Improved hosting page
- Revamped marketplace
- Revamped jobs section
- Drupal.org running Drupal 7, with responsive, mobile, HTML5 theme.
The Team
In the past, the Drupal Association has made improvements to Drupal.org “big bang” style: pick a near-impossible sounding goal, spend a bunch of money on hiring the community members who know the most about those things, and get ‘er done. We’ve had some huge successes going this route—the Drupal.org redesign and the CVS to Git migration—but this approach also has some major flaws:
- These are short-term gigs, which means the Drupal Association is paying contractor rates for the work
- When the money runs out (with a few very notable exceptions), the leaders of these initiatives tend to prioritize other things and their contributions to Drupal.org are reduced.
- The Drupal Association is left with little institutional knowledge of how various pieces of Drupal.org work and fit together. The knowledge is instead scattered among 15+ very busy people.
Going forward, we want to change how we spend our money on Drupal.org, by building an internal team within the Drupal Association to work on Drupal.org full-time, supplemented by subject matter experts where appropriate. They will attack both big problems and quick wins, as well as support community volunteers who wish to scratch their own itches.
The team, as we envision it, breaks down like this:
Q1 2012:
- Volunteer Coordinator / Project Manager
- Senior Architect
- Junior Developer
Q2-Q3 2012:
- Add:
- Additional Junior Developer
- Allocate funds for 1-2 contractors
Q4 2012:
- Add:
- Design/UX lead?
- DevOps Engineer?
- Additional development/contractor resources?
As you can see, this is not a huge team, and it’s going to take time (and funding) to get it up to speed. This means two things:
- The more we can fundraise and the earlier/faster we can fundraise, the faster we can hire folks and get rolling on the big things we want to tackle.
- Regardless of how fast we can fundraise and hire, we will need the community’s help in resolving these critical issues. Not even the smartest crew of 2-6 people can cover all of the bases on a site as diverse as Drupal.org.
The Timeline
Key
- #xx = Placement in top 100 in community voting; a lower number is more important.
- SB = Site Builder
- D = Developer
- B = Business
Q1 2012
- Bootstrapping: Hiring interns, winding down Neil Drumm’s work on DA priorities so he can focus 100% on community stuff, setting up basic project management processes (issue tags, documentation, reporting, etc.).
- Low-hanging fruit: Analyze list of community priorities, identify items that are fairly straightforward and/or already have significant community momentum, and get them finished/deployed as fast as possible. Possible candidates:
- #5 - Better showcase of Drupal websites on Drupal.org [B, SB]
- #7 - Check for duplicates when posting nodes [D, SB]
- #9 - Show user pictures on Drupal.org [B, D]
- #21 - Move groups.drupal.org to new design [B]
- Fix broken usage graphs [B, D, SB]
- (Sort of #1) Download counts on projects [B, D, SB]
- #18 - Improve usability of Follow button
- Office hours: Establish “Drupal.org office hours”: a weekly time when community members can get updated on what’s happening, and those scratching their own Drupal.org itches can get dedicated help from DA resources for reviews, deployment scheduling, etc.
Q2 2012
- #2 - Drupal.org to D7 upgrade [D]: Set up dev/staging infrastructure, hold sprint in April with key infrastructure team members to develop crowd-sourceable roadmap and get as much done as possible, bring on key contractor resources and continue working throughout quarter at least 50% of the time on this.
- Other things that we could (don’t necessarily have to) couple with this upgrade include:
- #4 - Responsive, mobile-friendly HTML5 theme [B, SB, D]
- #20 - Enable group functionality for projects [D, SB]
- #29 - Full staging environment for Drupal.org [D]
- Project* revamp on modern tools [D, SB]
- Completely redesigned and improved user experience for collaborating in the issue queues:
- better workflow and UI for participating in individual issues
- activity stream for individual issues interweaving comments, commits, edits to the summary, etc.
- tracking of related issues
- tools for organizing larger initiatives
- Separate content types for the different types of projects (modules, themes, distributions, apps) that are optimized (with custom fields, UI, etc) for each sort of contribution. I.e. the “projectness” is a field that you can apply to different node types.
- More long-term flexibilty and simplicity by using the Field API for all of our collaboration tools, making them easier to extend and expand in the future.
- Plan for Testbot??
- TODO after BoF at DrupalCon
- Medium-hanging fruit: Pick the next round of improvements once the easier ones are taken care of. Possible candidates:
- #1 - Project ratings and reviews [SB, D, B]
- #3 - Back-link commits in related issues [D]
- #6 - Improved support tools on Drupal.org [SB, D]
- #19 - Make events on groups.drupal.org amazing [B, SB, D]
- #36 - Better Drupal.org marketplace for companies [B]
- #16 - Dedicated jobs board on Drupal.org [B, SB, D]
- Continue incorporating community-driven improvements.
Q3 2012
- #2 - Drupal.org to D7 upgrade [D]: Complete the port by DrupalCon Munich. Almost 80% of time gets allocated to this.
- Continue work on the “bigger” medium-hanging fruit as needed:
- #1 - Project ratings and reviews [SB, D, B]
- #3 - Back-link commits in related issues [D]
- #6 - Improved support tools on Drupal.org [SB, D]
- Continue incorporating community-driven improvements.
Q4 2012
- Re-assess community priorities again to see where things are at, and develop plan/staffing accordingly
- Finish up other “top 10” stuff that’s still hanging out (e.g. “advanced” things outside of staff’s current skillset, things that are better off waiting until after D7 upgrade, and/or particularly contentious community issues). Possible candidates (I can’t stress enough that these things are not necessarily going to happen [through DA-driven initiatives, at least]):
- #8 - Automated Performance testing [D]
- #10 - Donations for project maintainers [B, D]
- #11 - Reputation/Badges system [D]
- #14 / #27 - Better Git collaboration tools [D]
- #39 - Topic pages [D, SB]
- Continue incorporating community-driven improvements.
Aw. But the thing I want to see on Drupal.org isn’t on your list. :( (Or it is, but way later than I want to see it!) What now?
It’s only natural that with 80+ ideas submitted (not to mention all of those floating around that haven’t been written down or even thought of yet!) there are going to be some folks disappointed that their pet project isn’t on this list (yet!), or that other itches are being scratched before it. Since there’s so much out there, we really tried to focus staff time on things only the DA could do.
However, fear not! You can make changes to Drupal.org yourself, too! In fact, thanks to the voting site, you have a list of people who also want to see your thing happen. Get in touch with them, form up a team, get set up with a sandbox, and start crankin’!
The Drupal Association will be holding “Drupal.org office hours” every Monday at 11:00 - 12:00 PT to support volunteers working on their own community initiatives. This is guaranteed, set-aside time for volunteers to get ahold of someone who can answer any questions you have about Drupal.org development, perform reviews of your work, and get RTBC improvements signed-off on and scheduled for deployment. We’ll use that feedback, balancing with the other priorities of the moment, to populate the list of weekly Drupal.org priorities that we’re going to work on.
Hope to see you there! We can’t do this without your help! :)
Comments
Drupal 7 upgrade
The Drupal 7 upgrade should open the possibility for so many cool things. Sebastien Corbin is working hard on the Drupal 7 port of localize.drupal.org already at http://drupal.org/node/1424984.
No Docs support?
Cross-posting - I see no specific funding for docs projects.
Under "Developer Improvements" it says: "Turn project pages into groups to do per-project documentation, events, announcements, etc."
But this doesn't have any relation to the existing long term plans from Docs Team to build the Better Help System and Curated Docs, though it's probably something that can be worked with/around, of course.
I refuse to believe that there is not enough need for better Documentation Infrastructure to warrant some major support from the community leadership and DA...
The Docs Team may have dissipated due to the difficulty working with our existing structure, but I am 100% confident, they're waiting in the wings and will be ready to do what they love when there's an infrastructure to properly support it!
Again, better docs is the single most requested improvement for Drupal.org that has been communicated by the user base at large. These plans are a clear path to helping everyone be able to maintain better project docs, and therefore help everyone learn and understand (and then keep using Drupal!)
One of the problems with
One of the problems with "better documentation" is that if you ask 10 people what that means, you will get 30 different answers back. I think the Documentation Team's plans laid out at http://drupal.org/node/1095012 make a ton of sense, and I was really hoping to see the "Curated docs" idea rank higher in the voting tool than it did, but unfortunately it didn't. Now, to a large extent, these numbers can be taken with a grain of salt, since they're primarily from community "insiders" who largely don't need docs. However, the fact remains that this doesn't seem to have strong community will to get it done.
The DA has gotten the best "bang for its buck" by resolving long-standing collaboration pain points in the community that there is near-universal agreement have to happen, and that only the DA can do because it's clear that they're not going to happen any other way. Git migration, Drupal.org redesign, +1 subscribe, distro packaging tools are all examples of really successful investments in our community tools that all meet this criteria.
And while it's possible that putting funding behind curated docs could help "engineer" community will to see this prioritized and fixed, that's generally not how it works, in my experience. People either are motivated/passionate about something, or they're not. And throwing funding behind unmotivated people/initiatives is generally a terrible idea, because it means either the work on that thing dies off entirely when funding stops, or else you need to keep paying for those initiatives indefinitely, which means we get less done on the grand scheme of things, which is not good for our community.
So if you contrast something like curated docs (definitely a pain point, but no universal agreement that it's the right solution, efforts could largely be crowd-sourced) with something like "Port Drupal.org to Drupal 7" (absolutely, 100% not going to happen without DA involvement) and "Get ratings/reviews on project pages" (a most-wanted feature that's been hanging out there for over 7 years, so obviously isn't going to come from grassroots efforts either), it unfortunately doesn't rank in terms of what the DA staff should be prioritizing. At least, not from what I can see.
However, one area where the DA has definitely fallen down in the past is when teams like the docs team have had a bunch of energy and momentum around their grassroots, crowd-sourced initiative, their improvements have sat there waiting forever. and ever. and ever. for someone to take a look at them. Months go by, people start to go find better things to do with their time, and by the time it gets looked at folks are demoralized or have even forgotten what the heck it was they were working on to begin with. THAT is a huge tragedy, and this again is an area that only DA staff can rectify. So here is where we're planning to make three big changes:
1) Put our senior developer (Neil Drumm) 100% on d.o stuff, so there's someone reliable there who understands how all the parts fit together, and can do regular deployments.
2) Hire a project coordinator to keep on top of various *.d.o happenings and help prioritize/schedule when they can get deployed on an ongoing basis.
3) Start holding "Drupal.org office hours" to give community members to escalate things they're working on to DA staff attention.
I think those three things are going to help TREMENDOUSLY to empower the community to get things they want to see done, including a curated docs section if that's what comes to be.
And in case it wasn't clear from the above timeline, the basic plan is this:
Step 1: Get a basic team/community processes in place, and start knocking out little things so the team gets its sea legs.
Step 2: Port Drupal.org to Drupal 7 and get project ratings/reviews done (our top 2), while continuing to bang out other small wins to keep momentum happening in the meantime.
Step 3: Re-evaluate community priorities after the D7 port is done (hopefully by DrupalCon Munich) and see where we're at. I think at that point, our list will look a lot more interesting because "OMG Flamingly Critical Stupid Plumbing Crap We Absolutely Just Need To Just Fix™" will be dealt with, and we can start looking to more exciting things like new features/capabilities for the site.
Hope that helps explain where I'm coming from, and why I plan to present this to the board without docs (or many other features I'd like to see) in there.
Priorities by voting
Priorities for 2012 in general are at: http://groups.drupal.org/drupal-association ... Probably the voting system (which is only/mostly getting the "insider" perspective I think?) was good for capturing the sense of the community on 2012 priority #1 (make it an awesome place for developers/contributors), but I don't think it probably gets at #6 (make it an awesome place for site builders) -- those people aren't well connected and probably didn't vote in large numbers on Ideascale.
So is there some way (like using previous surveys) to capture their desires, other than basing it on a voting system that mostly fairly well-connected people saw, or just guessing what they would want? When the re-eval happens, maybe a different mechanism could be used rather than community/insider voting, to capture the desires of the vast majority of not-so-connected users who make up the "site builder" population.
Drupal programmer - http://poplarware.com
Drupal author - http://shop.oreilly.com/product/0636920034612.do
Drupal contributor - https://www.drupal.org/u/jhodgdon
In terms of site builder
In terms of site builder improvements, project ratings/reviews is an absolutely killer site builder feature, so I'm not too worried about that. Figuring out which of the 16K modules you should use on your site remains a deathly difficult problem only solved today by either lots and LOTS of experimentation or making lots and LOTS of mistakes and learning from them. So I am not too worried about that in terms of fulfilling our mission. We also already have pretty detailed spec on how this should work at https://infrastructure.drupal.org/drupal.org-style-guide/prototype/modul... so this should actually be achievable.
And while the D7 upgrade doesn't directly translate to developer features, it will end up doing so in the port of Project* to modern standards (entities/fields, Rules, OG, etc.) and will be a huge enabler for other things, so I'm pretty comfortable with banging out some awesome developer improvements this year too.
In terms of the voting tool, I'm totally fine with whoever does the next round of vetting using some other means than a voting system for it. The reason I went that way is because it somehow fell on my shoulders to make "THE Plan For Drupal.org™" and I didn't feel like I had a good sense /at all/ of what the community actually wanted to see, nor what they were working on... including insiders and outsiders. It's impossible to overstate how totally fragmented this information is right now and how bad Drupal.org's "bus factor" truly is.
Dries's survey was nice in that it was to a broad audience, but the items on the list were way too vague to be actionable. Same with the Prairie survey. So I needed a tool that would allow more fleshing out behind the bullet points of what people actually mean when they vote for "Badges/Reputation system", including arguments pro/con, and including who was pro and who was con to get a sense of audience segment.
So the voting tool definitely had its flaws. But in terms of what I was using it for—essentially a "gut assumption checker"—it worked really well. For example, my gut thought was that "Git Phase 3" improvements would rank a lot higher than they did, meaning we'd need to raise another several tens of thousands of dollars this year. But really, it was mostly low-hanging fruit stuff developers are asking for, and this is awesome because we can actually do something about those! :) But I highly encourage the next person to do this to use whatever tool makes the most sense for them.
IMO That's just not good enough
Thanks for the response webchick, but I have to disagree with the choice not to include funding for docs. And I agree wholeheartedly with Jennifer that the "voting process" for this funding was hugely skewed.
I blatantly disagree that things like the project ratings, the case studies page, reputation/badges, job boards, or support sections for example are more appropriate to get DA funding for (a couple may be equally important, but they're not more in need/more appropriate), and also i think that the DA and community leadership have really not supported documentation, which is not helping raise any support community wide. If the people everyone listens to say "this doesn't matter enough" it's going to continue to be that way.
It's not the first time the DA has explicitly excluded Docs from their mandate, and I don't understand why there isn't a wider understanding and admission that this is a true problem.
The lack of docs infra and resulting lack of clear, easy to follow, and easy to maintain docs is a huge detriment to everyone in the community, as well as any potential Drupal users.
This is not because of lack of efforts by the TINY Docs Team, it's because we simply can't make this stuff happen on our own with the resources we have. The Docs Team has a number of people who can't wait to write and maintain docs, but has hardly any resources to actually build the infrastructure. I don't understand how this can be said not to be the sort of project the DA should support.
So "[it should be a project]...that only the DA can do because it's clear that they're not going to happen any other way." Yes, this is exactly one of those.
By stating "People either are motivated/passionate about something, or they're not. And throwing funding behind unmotivated people/initiatives is generally a terrible idea, because it means either the work on that thing dies off entirely when funding stops, or else you need to keep paying for those initiatives indefinitely..." you're not even giving docs a chance to improve. Whatever hope and energy there is in the docs realm, it sure won't improve when it's assumed that docs initiatives will fail. There weren't many people energized about doing the git migration, but it happened (albeit difficultly) with the support that was provided. And I don't think these Docs projects are even on that scale, so are lower risk.
I don't understand why these Docs initiatives couldn't be put on the 2012 priority list and worked on in parallel to the D.o upgrade - all that's needed is a test site to work on the tools in D7 alongside the upgrade, and then potentially even launch them together.
And as far as Dries' survey results not being actionable, what? Of course the docs part is actionable, because we've spent a year doing infra planning - the action is already 80% defined! We just never had the resources to carry it out, and yes this has led to major demotivation of both the team and the lead(s). That doesn't mean these people aren't ready and waiting to act if/when this had some support and a stamp of "yes, this is important" from the community leadership.
The reason those other things
The reason those other things you mention are on the list is they're either:
1) Very easy/straight-forward changes so we can get some quick wins under our belt to get the team bootstrapped, and also to act as little nuggets of usefulness to keep community momentum up while our efforts are primarily focused on the Mega Project Of Unholy Death™ that is the Drupal 7 upgrade.
2) Changes that have near 100% community buy-in (within their respective sub-communties they affect) that we need.
3) Things with pre-existing community momentum behind them that we just need to give a little push to get them done.
Curated docs, unfortunately, is none of those things. The docs team has 100% buy-in, but the community doesn't, it is not a "bang this out in 2 days" project, and there are no existing community members working on it, which means we have to boot it from a cold start. Until the D7 upgrade is done, I just don't see how we can do that, short of an huge infusion of cash to the DA from interested corporate sponsor(s) who wanted to improve docs, which would allow us to increase our headcount faster.
So for now, it seems like the best way for the DA to support the documentation team (as well as the translation team, usability team, core developer team, testbot team, git team, etc.) is via community initiative support. Put processes in place so when you work on Drupal.org-related things they get seen by someone who has the power to make them happen.
However, get me a proposal and a budget, and I can certainly look at what can be done with any remaining funds after the D7 upgrade (which, once again, is only going to happen if the DA does it, and must happen this year). I asked Jennifer one for last night but she said no, because the work would almost assuredly start happening after her tour of duty as docs lead ends. :(
Or, if you have something else specific in mind with how "DA and community leadership" can better support the docs team, let's talk!
Let's calm down and step back
So... Let's take a step back for a moment, and calm down:
Angie: You did a great job pulling this together on short notice, and I agree some kind of voting on specific projects was needed. Sorry for criticizing the process without offering a viable alternative.
Ariane/Jennifer (yes, I'm talking to myself): Yes, the docs team has been frustrated and felt abandoned for years. But let's let bygones by bygones and the past be the past.
Anyway... After reading through all of this and several IRC conversations yesterday and today, and doing some thinking this morning, I actually have a plan for how to revive things in the Docs team and make it worth the DA's while to fund some very specific plans we might have:
a) I have several small items in the d.o infra realm that I'm going to use as tests to see how the deployment process is improving, with the new funding for Neil Drumm, and new processes. We'll see how that goes in the weeks immediately after DrupalCon Denver.
b) The Monday after DrupalCon, I have plans to post to the Docs Team g.d.o group to get the Docs Team (hopefully) energized again on working on some updates to the Community Docs infrastructure. (I even have the post written.) This will be another test of how the infrastructure deploy pipeline is working, and also of whether the Docs Team can get energized like we did last year for the Community Docs redesign.
c) Assuming all that works out well, which I should know in a month or two, I will get back in touch with cweagans and see if the two of us (and anyone else we can recruit) can build the modules we need for the fabled Curated Docs, using Drupal 7. Hopefully this will somewhat coincide with d.o updating to Drupal 7, and at that point, we can either build a separate docs.d.o, help.d.o, or whatever, or we can deploy this on Drupal.org itself -- whatever seems the most sensible at that time.
d) And assuming that works out well, I plan to apply to the DA for funding to get about 6-10 people together to write a really good user manual for site building in Drupal, using the new Curated Docs site. And using the new curated docs site, it will be topic-based, modular, reusable, maintainable documentation, because we'll have this awesome tool for writing good technical documentation. And the docs will be navigable and searchable for users, because of this awesome tool. And everyone will be happy.
How does that sound for at least the outlines of a plan? I am proceeding cautiously, because of our experiences as a Docs Team over the past many years, and specifically last year with the battle to get the Community Docs redesign stuff deployed. But I am cautiously optimistic, and hopefully this is a productive plan rather than more antagonism.
Drupal programmer - http://poplarware.com
Drupal author - http://shop.oreilly.com/product/0636920034612.do
Drupal contributor - https://www.drupal.org/u/jhodgdon
I think that plan sounds
I think that plan sounds excellent! +100! :) Please let me know how I/the DA can help support you, and touch base a month or so before it feels like a good time to get a bunch of folks together. We have space in the Portland office that we can use.
Yay :)
Yay, go jhodgdon! I'm glad that all these discussions last week resulted in a renewed plan of action (regardless of DA funding status).
I had a really good chat with Cameron (cweagans) today, and gave him a big braindump about all the infra plans and how they intertwine - we should put him in touch with Lin (linclark) and Johan (itangalo) if he doesn't manage to meet them this week too, just so he can get an idea of the code they were working with last year - maybe some can still be useful.
Guess we ought to stop hijacking this DA thread, follow up next week on g.d.o/documentation-team, and forge ahead slowly but surely...
There's an em tag exposed in
There's an em tag exposed in (weird since pasting it here rendered it correctly):
Video + slides from DrupalCon Denver
Here are the video and slides from my talk at DrupalCon Denver about this: http://denver2012.drupal.org/content/whats-next-drupalorg I'd encourage anyone interested in this discussion to watch that, because it (hopefully) lays out the vision here a lot better than this document did.
To the board, I presented slides ~19 - 37 (the background, the team, what the team's going to work on, and timeline), and have a $200K budget (including staff headcount) to get this stuff done. WOOHOO. :)
Quick Start Guide
What would go a long way to getting closer to new users is making something like Quick Start Guide (check http://www.applicationcraft.com/tut-qsg). I spent like 8 hours watching those videos and in the end I felt like I knew enough to start working with Application Craft.
So Drupal Association is
So Drupal Association is turned off :(, Come on! I was having fun over there lol!~ I need votes!~, can't Drupal Association be active for votes???
Drupal Association:
and When will it be active again????????????????????????
I've just to say: Whoa -
I've just to say: Whoa - Awesome idea!