To all reviewers ... I'd like to send everyone a not-so-subtle reminder:
The mission of reviewing Full Project Applications is to ensure that new contributors have a basic understanding of the Drupal community's core values concerning contributing code to Drupal.org, AND to promote lasting, long-term contributing to the project.
Over time, the project apps process has strayed away from it's original mandate. It was intended as a means to help on-board new contributors into the Drupal community, and provide a level of mentoring and guidance before setting them loose in contrib ... but has since transformed into a manned security gate that only those who can master the secret handshake can ever break through. We have gone from ensuring 'that new contributors have a basic understanding of the Drupal commumity's core values' to ensuring that 'only the perfect and persistent' are ever vetted. In the meantime, the hurdles and delays do nothing to promote 'lasting, long-term contributing' for the vast majority of applicants.
So please try to remember ... we as reviewers are not an enforcement body. It is not our job to go out looking for applicants who have done something wrong, and then punish them for their crimes. The role of reviewer is to help the applicants ... and every action we take should be in the applicants' best interests.
Case in point ... I recently reviewed an application posted at the end of November, which was subsequently looked at by three different reviewers, and marked RTBC within a week. It then sat for an additional 12 weeks before I came along for a final review. Sure, it was short, and still had a couple of issues ... but putting myself in that applicant's shoes, I knew how frustrating seeing another 'needs work' status would be, especially after three months of RTBC status while waiting for final approval. So I flagged the issues, glanced through the user's other sandboxes, and made a judgement call to push them through.
Well, it appears that the 'enforcement' role has taken another evolutionary leap forward, in that even a git admin's 'final approval' is now subject to peer review ... today, I was surprised to discover that another git admin had unilaterally decided to reverse my decision to promote the user, and had even gone so far as to remove their 'git vetted user' status.
Now we can go back and forth all day debating about whether that decision was justified, but that isn't the point of my story. Instead, consider the crushing effect it must have had on the applicant ... someone who initially made the leap and decided to give back to the project, only to find themself blocked for three months because "we're too busy" despite their application having been accepted. Then, once those "in power" finally spare a minute to look down and click the magical checkbox that accepts them into the contrib community ... ten days later the applicant finds themselves back outside, their application marked 'wont fix', and the waiting game starting over once again. Just what is this supposed to do to that person's morale, or their chances of contributing anything else?
Drupal's motto is "Come for the code, stay for the community!". As an on-boarding process for new code contributors, however, the motto for today's projectapps process could be "Give us your code, and get in line ... the community's too busy to look right now, but if you stick around long enough, we might let you in."
It's important to note that this is not a criticism of those who are still active in the queue ... the reviewers do what they can with what they've got. They put in a ton of time for the sake of helping others, and receive almost no credit for their effort. But the sad fact remains that the process itself is broken, and simply not sustainable on the backs of a group of volunteer reviewers. EDIT: while we've spent about 18 months trying to improve the situation, applicants continue to face exceptionally long delays and growing frustration within the queue.
To this end, I will be bringing forward a proposal for discussion amongst the larger Drupal community in the next week or so, suggesting changes to the process of publishing code on Drupal.org and the role that the Project Applications process plays within it. Please keep an eye out for the post, and add your voice to the discussion when it occurs.
Comments
This is really important, and
This is really important, and adresses a lot of the itches I've had about how the process has been applied.
But since most of the reviews seems to be done by people chasing review bonuses (as opposed to "regular" reviewers), I'm not sure how likely they'll be to read posts in this group before reviewing. I wonder if this should be more emphasized in the review guidelines.
Not directly solving the specific situation you mentioned, but could it be an idea to relate the "severity" of the reviews to the issue priority recommendations, for instance by saying that minor issues should never be enough to hold an application back?
Thanks very much for posting
Thanks very much for posting this. I think it's worth looking both at the incident and the larger picture.
One quibble:
Depending on the standard you hold, there are a lot of parts of Drupal that are "broken and not sustainable." But...lo these many years...Drupal is still here and unsustainable "things" are still marching into the future. You can say I'm wearing rose-colored glasses to state it this way, but I think instead of claiming things are "broken and not sustainable" we should instead say "things are not as good as we would like them to be" and simply work to improve process and tools to make them better. Maybe a call for constant improvement doesn't seem as motivating as despair and death of the project but it is at least more accurate.
knaddison blog | Morris Animal Foundation
I couldn't agree more with
I couldn't agree more with greggles post and quibble ++
Ditto ... adjusted the post
Ditto ... adjusted the post accordingly.
Thanks, it feels a bit
Thanks, it feels a bit offensive to me every time somebody states that the project application process is broken beyond repair and hope.
OK - You are wearing Rose Colored Glasses to state it that way.
Let's play the fiddle while Rome burns.
Why does sound like the Chief of Police promising to look into the latest complaint of police brutality?
RTBC is the highest form of review. All questions have been answsered, all objections overcome. The odds are high that reviewers have installed and tested it and found that it works as advertised.
Why the hell is an RTBC waiting 3 months for approval from some final authority? That is not only an insult to the applicant but an insult to those who RTBC a project. Now they have to be peer reviewed on whether their review is "good 'nuff'?
Long post. I'll get to the point.
jthorson has worked so hard on this project. He knows his stuff. He made a judgement call and it was usurped. This is so wrong.
The only thing that is so
The only thing that is so wrong is that you spend so much time complaining and none contributing to improving the process in a constructive way.
knaddison blog | Morris Animal Foundation
Good answer!
Perhaps you could address the other questions too.
Sure!
Sure!
It doesn't. It looks to me like a reasonable and thought-provoking post, which identified an instance where I allowed emotion to influence my words and exaggerate the true situation. Doing so actually detracts from one's credibility and ability to influence their audience ... a lesson you might consider in your own posts.
Because that is the Drupal process, and it is the same in any issue queue. Anyone in the community can mark a patch as 'RTBC', but it is the project maintainer (or core committers, etc) who have final say as to whether that patch actually makes it into the code base. In this case, the project maintainers are the 'git admins', and they've either been too busy with other priorities (babies, work, D8 core development, etc), or have abandoned the role because they no longer get any joy out of the process ... in which case angry diatribes are not likely to bring them back.
You appear to have missed a sentence in my post ... Now we can go back and forth all day debating about whether that decision was justified, but that isn't the point of my story. Please don't try to make it so.
Thanks for the lesson
Thanks for that free lesson in How to Win Friends and Influence People.
BTW, I voted for your OP because I thought you made some very good points. I am very curious about the reasons for reversal. Some transparency here would be nice.
There was no reversal ...
There was no reversal ... while I'm frustrated with the inefficiencies in the current process (as per my post), that's easily trumped by my frustration with posts that complain about the process but provide no new insights, alternatives, or suggestions for how to improve it.
As you said, I've been involved in this process for a while. We've all seen the complaints, and know the issues ... repeating them does nothing but take attention away from discussion of the solutions, which is where I'd much rather spend our combined energy.
That's not what you said earlier
If this can be debated all day, as you have said, then obviously it's not cut and dried, is it? What happened? What are the reasons?
You are right. 12 weeks in RTBC is bad. You made a judgement call and from my take, it was a good one.
But, as you can tell, I don't know WTF is going on anymore.
While the application process
While the application process can be wearisome & verbose it's still totally up to the applicant on how proactive they are within the community. Drupal contribution has high standards and I hope that never changes. However like everything peer reviewed its overly subjective.
Is this an example of Cognitive Dissonance?
I read the first two sentences and my head nearly exploded. On second reading, the last sentence is worth a +1.
There are DO members that have been around for 4 years and work for big Drupal shops that complain about this process.
The standards are high now, for one member class. Too high.
But this was not always the case. Three of the last 4 contributed modules I have tried had problems. None of these modules would pass today's code review process.
This process can work, but it has to be applied to everyone.
Reviews for everyone!
The idea of reviewing all projects has been suggested a few times, and I've never bothered giving it much thought before, mostly because no one ever bothers describing how it would work. And at face value, it wouldn't work at all. We have a backlog of projects needing review, so adding more projects to the backlog would, without any mitigating factors, simply make the problem much worse. Without any explanation of what those mitigating factors might be, this suggestion just comes off as a non-serious "modest proposal."
But because it's been suggested before, I thought it through a little more this time, and I realized we already have reviews for every project, in the issue queues of those projects. And when projects have serious problems that go unaddressed, project owners can and do lose access to the projects, through the security team or the abandoned project process.
I see this happen all the time to some of the most prominent members of the community, so I don't think it's really true that we have two different classes of contributors. We obviously do have two different review processes, but that's just a logistical difference, e.g. we can't prevent bad project releases after they happen, lacking a time machine. The same standards apply to all projects. If you see projects not following those standards, you can and should open issues and push them just as much as reviewers push applicants to improve code in this process.
You can also, if you like, apply the issue queue review process to applications. I've done this in the past, just like I would on a full project. That proved to be time-consuming, with the need to cross-post updates in the application issue, but it had the advantage of using the same feedback process for applicant and non-applicant projects alike.
The Issue Queue?
Ha ha, Very clever reference to that Jonathan Swift satire. Apropros too, because the current PA review process comes off as a perfect example of a society rationalizing eating their young.
The issue queue is in no way the same as this PA process. Problems in the issue queue often get closed or dismissed or just go down that memory hole.
Perhaps it's time to think about the "same rules for everyone" idea one more time because it's really not that complicated. There is obviously a a lot of modules that would be approved quickly even if they required 1000 votes for an unconditional pass.
The 'all-in' model is NOT impossible. It would be healthy and, as fuzzy has asserted, it scales nicely. Drupal should be sponsoring and mentoring new contributers. It should not resemble a silly court drama that takes months to resolve.
Details
Please explain it, and I'll vow to think about it. It's hard to think about without any details on what is being proposed. In this hypothetical scenario, what exactly happens to all the existing full projects? Does every one have a review issue? Do they lose releases if they don't make it through in X days? Is the review process the same as we have now, just with 11,000 issues needing review rather than 70? Or does it significantly change?
Something I've had in my mind
Something I've had in my mind for months:
This should actually get us a lot of the way on itself. It would not need manual reviews at all, and it would probably shame projects with bad code into actually getting their act together. Or atleast prevent people from using them.
If desirable, it can also be expanded with manual reviews, but I'm not actually sure it would be necessary.
We cannot let everyone create
We cannot let everyone create full projects because of spammers and name space squatting.
Also,
Based on recent feedback I've collected, I suspect the spam/namespace squatting concern is secondary (and falling) related to the concern regarding the security team's corresponding workload and ability to keep up with SA's on full releases if everyone and their dog was allowed to generate releases; along with maintaining Drupal's reputation for having modules that are 'generally' functional and secure by the time they make it to a stable release.
That could be fixed by saying
That could be fixed by saying that projects with a score of 80% or higher get a Gold badge, and are the only ones that gets "full" attention by the security team. Over time, most people would probably stick with Gold badge projects only.
There's a proposal from Heine
There's a proposal from Heine from 2009 on the topic of how the project decides which projects get a Security Advisory or not. It's interesting to consider because he talks about a lot of changes that have come to happen that are good and some that are generally perceived as bad (e.g. holding back applications because of code style issues even when the code style issue is not a hindrance to reviewing the module in general).
I think that the current system of "stable release gets security advisory coverage" works out relatively OK. It communicates the idea in a vague enough way that people aren't likely to confuse "gets a security advisory" with "reviewed and approved by the security team" (to be clear, there's no such "reviewed and approved" certification of modules on d.o and it is unlikely there ever would be). It might be nice to add an additional step or two before a 1.0 can be created (though easy to overcome). I wouldn't want to make it so people have to complete 100% code style issues since there are false positives and room for deviation if it makes a module better.
knaddison blog | Morris Animal Foundation
That should be easy to fix
That should be easy to fix with a combination of manual reporting of such projects (with a prominent button somewhere) and some kind of automatic tests as well, such as remove modules based on low usage numbers, lack of commits/releases and extremely low "overall score" as calculated above.
Actually, having to pass the automated tests with fever than a fixed number of errors and warnings would probably prove better than most CAPTCHA's.
I'm betting you could even make a Mollom-like check on the projects.
Relevant discussions elsewhere
http://groups.drupal.org/node/114264 has a long thread explaining why the community decided not to do #1. You're welcome to restart that discussion, but I think it's important to recognize that the decision wasn't made by the project review group alone (or at all, for that matter) and can't be reversed by the project review group alone. So this maybe isn't the best place to have the discussion.
http://drupal.org/node/1637534 covers most of #2-5, with real implementation plans, so that's probably a more productive place to discuss those.
I don't have a complete plan
Just a few ideas.
Split the current DO into two sites. One is for contributed modules only, just like api.drupal.org is for just for the API.
The new site is in D7 or maybe even D8. It has voting (up and down), polls, comments, accountibility and good metrics. I like the StackOverflow (SO) model. I thought much of this was possible in D6.
The current DO gets archived and then a serious spring cleaning takes place. The cleaned out DO has all the great stuff that is in there now and none of the cruft.
This can be transitioned over time. It does not have to hurt that much. But we have the same standards for everyone, the same exposure for everyone. I see nothing wrong with reputation ratings either. People put much more effort and thought into posts when they can earn reputation from it.
Needs code
Okay, so that's at least a 6 month plan. I suggest you start by taking over http://drupal.org/project/arrayshift which is the closest thing we have right now to code for that kind of site. We can get into the details once that has a stable D7 release. Or D8 if you prefer.
Until that's somewhere near ready to use, let's focus on ideas we could act on in the short term.
Give project/Git access but block on having a release?
One of my concerns is that the reviewers are trying to ensure quality, but as a potential contributor, one of the main reasons why I would be motivated to contribute one of my custom modules is to get feed back and improve quality.
Would it be possible to give people Git access and the ability to create projects, but have a separate application to get their first project non-dev release? This would allow much of the clean-up and final review to happen on the project's issue queue.
An interesting test case is @sylus and The Government of Canada's attempts to add a distribution to Drupal.org: http://drupal.org/node/1494484
The distribution has been on Github for months with multiple releases, over 1000 commits, dozens of users and followers and almost 700 closed issues: https://github.com/wet-boew/wet-boew-drupal/issues; yet it took nearly a year to get though the review process. There were a number of legitimate issues, but by keeping the project off of Drupal.org for so long, Drupal has missed the opportunity to bring a very big, active and well funded community onto Drupal.org.
In the WET distribution's case, the project/git access could have passed the initial "close enough" hurdle, and then the activity could have moved onto Drupal.org as the project was cleaned up and made ready for an official release.
Exactly ...
That's exactly what I'm suggesting in the proposal I mentioned in the original post (which I expect to release this weekend) ... essentially opening up the packaging scripts to all users, and pushing the project review process back to make it a gate for publishing a 'stable' release (i.e. one that would be covered under the security team's 'Security Advisory' policies).
I wanted to first run the concept past a few key stakeholders before publishing it to Drupal Planet. The feedback thus far has been positive, and I haven't encountered any heavy opposition.
I'd also like to suggest
I'd also like to suggest letting "all" projects be sandbox projects until their first stable release. This would mean that people can't claim project namespace (think url) for years without releasing anything (which I've seen quite a few places on d.o).
Current process for this
Without knowing the details, this sounds like a likely violation of the policies at http://drupal.org/node/709790 and/or http://drupal.org/node/251466. Those policies commonly result in removing project maintainers. Prior to a policy change, you can and should report these in the webmasters queue when you see them.
If you're talking about the
If you're talking about the WET example, I think it's an excellent illustration of what happens when our own internal processes don't work ... and it doesn't seem right to challenge it further, now that a community has risen around the GitHub code while they waited to establish the true home here. (In any case, the drupal.org repo does host a copy of the code, so it's not in violation.)
EDIT: Never mind ... I mistook the context of the post. I've got it now. :(
Thanks for posting Dustin@Pl
An excellent example of why change is needed. In the year it has taken for get approval, another ecosystem was used to advance and develop the module by a large and engaged community. The DO approval process became irrelevant and so did DO.
So what can be done to insure something like this does not happen again?
Bit off-topic..
With all the due respect to the drupal.org project application review processes and the various concepts/suggested improvement about it published/voiced by the various drupal geeks, I believe because of it tediousness and volunteer based persuasion drupal looses many willing contributors ( I would be happy to be wrong).
In d.o there is a link http://drupal.org/project/usage by which more or less any body can see that users are well aware of the quality of the modules/themes/distributions over the time and things does not work goes down (read rejected by the users) by the usage stats.
So whats wrong if we can automate the basic quality testing of the contributed modules/themes/distros and let users choose.
I believe by adopting this d.o will be having a win win situation.
Post duplicated .. deleted.
Post duplicated .. deleted.
The Review Bonus program has
The Review Bonus program has been doing more bad than good, tempting newbies to post reviews, which are mostly namesake, with a primary motto of qualifying for a bonus review. This has eventually brought down the average quality of each review.
The Review Bonus program has also not helped in anyway to fasten the approval process or to speed up the queue. Because even if the newbies posted their reviews in hope of receiving a bonus review, the reviewed projects would stay in RTBC forever.
Taking off the Review Bonus Program will definitely increase the quality if reviews. At the same time it would not make the approval process any slower.
--
Tanay Sai
Bangalore
skype: tanay.co.in
Just to clarify: I personally
Just to clarify: I personally make sure that applications that had a review bonus get pushed through eventually when the get to RTBC. So don't worry: once you are on the review bonus train you will get approved in a timely fashion. The only requirement there is that you react to the feedback and clear up remaining application blockers.
Also: could you post an
Also: could you post an example where the review bonus has failed an applicant? Maybe I have overlooked something and we can clear that up right away.
It works!
I recently went through the PA review process. It didn't take long to make it through the process. I submitted my application on February 1st and the project went into RTBC by the 8th and I was granted full project rights on the 9th.
There are several reasons I can think it moved so quickly.
1) I did the review bonus process. In total I reviewed nine other projects and learned a lot about Drupal coding standards in the process.
2) The project itself was fairly simple to understand for reviewers. Complex projects take longer to review and understand. If you have thousands of lines of complex code it simply takes a real long time to review the code. What if Views was my project that I wanted to get vetted on? It would take really long. Instead it was a simple plugin for Views that had 200ish lines of code.
3) I've done enough Drupal coding that it really didn't need a lot of changes. It only went back into "Needs Work" twice so I only had to review another set of projects a couple of times. And when it did go into "Needs Work", I made the changes quickly, reviewed another round of projects and got the PA Review Bonus again within a day or two. That makes it easier for reviewers because they probably still have your module installed somewhere and don't have to remember back weeks ago as to what the project was all about.
In summary: Do the bonus program and learn a bunch. Don't pick a super complex module if you want the review to go quickly. Fix bugs quickly and go do some more project application reviews so you can get back onto the fast track again.
I totally agree with heddn.
I totally agree with heddn. If you are actively on your issue and fix bugs and do manual review of some other project quickly, you will found this process fast but make sure that you are following the PA review process guidelins accordingy.
.
I got RTBC in 14 days for my first project and I don't think 14 days is too much time where lots of projects are in queue for review.
Same feeling here: I had a
Same feeling here: I had a not-so-small module I was working on. I decided to create a small module just for the application process. And I definitly believe, it is the best way to go. You might be able to publish quicker a bigger module by writing a 200 lines module easy to understand by other developers...
Alexandre
http://w3w.fr
Well summed up
I fully agree as well with heddn and by experience this is exactly what happened for me and several other colleagues or friends on D.O. as well:
Based on my own experience as well, going through the process, this is exactly the type of recommendation I have given to any of my colleagues, quick tips during meetups/presentations or users I help mentoring.
Thanks for summing this up so clearly.
Cheers!
How good are these Namesake reviews!
To qualify for a Pareview reivew bonus, you have to do a manual review. Simply repeating what the bot found when sniffing through the code is not enough.
The reason why i find these name sake reviews interesting.
It gives the applicant a feeling of not abandoned. I had seen people getting really happy with a Pareview review and thanking the bot. So a comment from some real person might mean something.
In order to review you must know what you need to review, this is a kind of self learning process in which these "namesake-reviewers-who-are-applicants-themselves" get to understand the process. (I did)
The knowledge gained through step 2. would naturally be applied to improve your own process. (I did, it just felt unfair to complain about it on somebody else's page and not following the procedure in mine :P )
It gives a chance for the new-applicants to read other reviewer's comments. Again this is getting to know more about Drupal and its environment. (I did, came to understand the hook_uninstal and naming conventions better. Which helped me in implementing the same in my projects and ALSO recommend other projects to do the same)
After few namesake-reviewers-review when its time for a "so-called-real" reviewer to review the project, the code is much cleaner and it could save 10-15 mins of "so-called-real" reviewer. I am not a "so-called-real" reviewer, so somebody can correct me if they believe/find it doesn't save any of their time.
I spend few hours in irc #drupal-support and not a single day has gone without a complaint that some module "xyz" is not supported anymore, please help!. If somebody is not willing to heed a request from "namesake-reviewers-who-are-applicants-themselves", i suspect if he/she could be a good maintainer as mentioned in Dries' post on Responsible maintainers.
Not to forget, these "namsake-reviewers" might become "so-called-real" reviewer one day.
I agree having more number of reviews doesn't mean a better quality per review. But hey, if we look at it quality of review per project... i'd say its much better than having no reviews.
Now i would like to say something which is more like a personal issue with reviewers rather than a Pareview Bonus issues.
The noob reviewers are not liked by some applicants. ( Ego issues ). I had experience of such an ego clash in one of the reviews. Surprisingly and oddly enough, the noob had pointed out a application release blocker and the applicant didn't consider his word to be "good" enough to be considered.
I wish the above be made clear to each reviewer, not just the namesake ones. The project application reviewing environment is quite tight and tensed and people seems to be shouting at each other (at least i felt so). I think we need some kute puppies and kittens images around the application process to remind the reviewers about the same. ( I am not joking here )
End Note:
I was able to get my project application approved in about 10 days. I did some nine reviews meanwhile. All reviews included me downloading the project through git and inspecting the code and also trying the module out.
Almost two year's back i had an application for review for almost more than a year, after which i decided to not pursue it. So yes, as you have guessed i am great Fan of Pareview review bonus.
This all reminds me of Debian
This sounds just like Debian before the fork.
Back in the day you had several choices:
Compile from source
"Optimize" your machine with Gentoo
Enter into RPM hell
Or Use apt-get with repos that were woefully behind, but the best package management
Then Ubuntu came along and it put a fire under everyone's ass.
Maybe we need an alternate project application que. You can go stable or multiverse. Developers choice.
Many folks agree with your
Many folks agree with your conclusion, which is why we have sandbox projects. Is there something in sandbox projects that doesn't feel like multiverse?
knaddison blog | Morris Animal Foundation
Change status of sandbox
I use sandbox as my development repo and launching point for review.
Maybe you can have a new status for projects awaiting approval into stable.
Sandbox -> Multiverse -> Stable
I do not trust anything in Sandbox to be complete, and would not download it. I would download something I knew was in Sandbox but actively being persued in the que. Especially if it can point to its position in the que.
It would be nice to actually show up in module search results.
How about some drush love.
Some modules may become so popular this way it will excite other developers to become part of the project.