I don't know if this has really been discussed yet, so I apologize if I missed it somewhere else.
First, it is clear to all of us that the review process needs some major changes and a lot can be done to improve the workflow. I am seeing a lot of talk about many great ideas (better use of statuses, establishing tags for gates or security issues, managing gates in other ways, use the sandbox issue queue, don't use the sandbox issue queue, use multiple reviewers for each application, etc) but how are we going to reach a consensus? Everybody has their own opinions (yes, part of collaboration), and this just seems to be resulting in a lot of opposing suggestions with some more insistent about their method than others. It's likely that we can find a happy medium and develop an efficient system with some collaboration and official decisions. So, why not get to it and set up some sort of official meeting every few months? (I think I saw this mentioned before) Has this been discussed more extensively?
I guess I'm with ccardea (?) on desiring more structure in some ways (but maybe that's just the military man in me). And I suppose it just looks to me like the queue is growing with no end in sight, but at some point we all know that needs to turn around and the sooner these efficiency solutions are implemented the shorter the backlog will get ultimately. I'd hate to see it pass two or even three months (I think a possibility) before any short term solutions are put in place.
Comments
Never final
It would be nice in many cases to have a defined decision-making process, but that's hard to do without ... a defined decision-making process. Few initiatives in Drupal have a defined leadership role, but everything else tends toward polite anarchy, an unclear, conversational consensus-building process. That's not entirely a bad thing. It has flaws, of course, but generally works pretty well around Drupal.org.
Data points: the overall queue isn't growing especially quickly, and the review list isn't actually growing at all.
I'm not at all suggesting we shouldn't make improvements. I don't think the current process is sustainable in the long run. But this is not an emergency. We have time to try things, see how they work, and iterate. I mention this primarily because I think the sense of urgency around these issues is ironically slowing down our consensus-building by making everyone unnecessarily entrenched in their opinions. If I don't think something is a good idea and we do it anyway, that's okay. We'll see it doesn't work, and we can undo it. When we start talking about "a final solution" (unfortunate choice of phrase, by the way), any concerns about something not working become much more serious, because now we're talking about it not working forever and ever.
Back to the data points. Over the last few months, the overall issue queue has grown from around 280 issues, to the current 323 issues. That's a pretty slow increase. But more relevant than that, the increase is entirely in the "needs work" status. The "needs review" section of the queue, the only section we're dealing with, has changed over the last few months from under 150 issues (3 pages) to ... under 150 issues. It has gone up and down a bit, but overall it hasn't changed at all.
One thing that has changed significantly is the longest wait on reviewers (the top "last updated" here). That has gone from about 4 weeks to about 6 weeks over the past few months. But it's currently heading down, after being up about 7 weeks.
You're right!
You're right! I think I meant "can we finally have some solutions?" maybe. I'm just seeing a lot of talk and disagreement with no ultimate conclusions being reached.
I look forward to being able to see when issues were started as a field in the queue. Today I found an application that was started in the first half of April and hasn't been touched aside from the author updating it. Similarly, my application has been in the queue since May 1st. I have been posting updates which probably makes it look active when in fact I have been talking to myself for a month now. So, determining application lengths by last update dates is obviously flawed.
I understand and appreciate that an open-source community can be a somehow efficient cluster**** of disagreement. Maybe I'm too new to try to understand decision making in a group without much of a structure or leadership (probably the military man in me again, thats getting annoying). But I do think I'm starting to get the hang of it now (just try it and see if it works!). And I like it.
Post Date
The patch to add the 'Age' date to the Project Applications queue has been posted ... you can lend some support for it at http://drupal.org/node/1179586.
I provide this not only because you referenced it, but also as an illustration that, despite the outward appearance, solutions and improvements are occuring. :)
Love it!
I love it! This is badly needed! Should make things much easier. I originally wrote that some issues were being resolved and referenced this news, it just didn't make my final cut. +1 on http://drupal.org/node/1179586
DONE!
DONE!
Unfortunate, indeed
Ahh! I didn't click the "unfortunate choice" link until after I replied. Wow! Quite an unfortunate choice indeed!
Longest wait
Speaking of waiting times, we just discovered this project application today. http://drupal.org/node/1094750 It has been in review since March 16th. Unfortunate, but if the application continues I think he deserves attention. Is this because it was not under the "module" component? Anyways, I look forward to the new view! (hopefully)
Not exactly "longest wait"
The current issue queue view allows us to order applications by the last time they were updated, which isn't quite the same as when the status last changed. While that issue had been in "needs review" status since March 16, it was updated with a simple "is this thing on?" kind of comment at the end of April, resetting the last updated time. That's a pretty common scenario, unfortunately. Being able to sort by creation date will hopefully make this a non-issue.
I wasn't really saying that
I wasn't really saying that issue was the longest wait, because who knows what is? Doesn't matter to me. I'm only pointing out that it fell through the cracks, and it is indeed a long wait. But maybe I'm not understanding how the fact that the author updated the issue has anything to do with how long he has been waiting. I do understand the mistake of not catching it because of how the queue is set up, and that's exactly why I was already talking about that problem earlier.
Which was the point I made earlier in this thread. Determining the length of the queue by looking at last updated dates is unreliable, but that was the statistic I got in response to my concerns about the queue length.
Anyways, I just meant to point this out because I felt bad for him and thought he deserved attention because of the wait time, so I'm not really sure what you're disagreeing with or even if you're disagreeing, but it seems like you are. Then again, I haven't slept all night...
Not disagreeing
I wasn't really disagreeing. I was just trying to clarify why exactly that happened, partially because I didn't understand it myself at first, but also so we can avoid it happening more in the future. Expediting it now is great, but ideally it would never happen in the first place.
Indeed!
Indeed! Obviously the new view will be a HUGE help in this area. The simple lack of accurate data is a clear problem. Thanks for the clarification!
Priorities
Considering that we don't yet have the "Created date" patch for the issue queue, wouldn't it make sense to make use of the priority field in this case? We were discussing using priority for applicants who help with reviews, but I think there's a solid argument to be made for doing so on applications that somehow fell through the cracks as well, just as this one did. I'm ready to implement this now if you agree.
Priority guidelines
I think priorities could be useful here as stopgap. I don't believe we're using them for anything else, and prioritizing older applications is an accurate description of what we're doing.
It would be important to document what we mean by the priorities (which would be a little different from the standard priority definitions) to avoid the appearance of arbitrary or unfair preference being given to certain applications, and also give applicants some guidelines for changing priorities on their own issues if reviewers miss something. That could be really simple, e.g.:
Major: 2+ weeks awaiting review.
Critical: 4+ weeks awaiting review.
Done
I've documented this at http://drupal.org/node/539608 and http://drupal.org/node/894256
Ralt has updated most of the
Ralt has updated most of the applications according to the priority guidelines. This definitely helps in the absence of created dates.
Hi, I've just updated the +4
Hi,
I've just updated the +4 weeks applications to Critical. I might've missed some, since I used the "Last updated" column, so the less than 4 weeks applications that were put to "Needs review" more than 4 weeks ago weren't updated (I hope I made myself clear :)).
Sounds great!
Sounds great!
Consider demoting too!
I've reviewed some projects where I've pointed out problems / made suggestions, then never heard back from the applicant. What happens when a "critical" project finally gets reviewed but is set back to "needs work" or "needs more info"? Does the issue remain in the queue as "critical" while we're waiting for the applicant to address the problems? Does the priority clock get reset when the applicant marks it as "needs review" once more?
I propose we have a corresponding guideline that if an issue is "needs work" or "needs more info" for more than 4 weeks the issue should be closed. If an applicant doesn't respond within 4 weeks, will the project really be maintained if approved? I don't want to discourage applicants, but I don't want to discourage reviewers either. If priority is to be equated with time, then surely lack of timely response should be construed as abandonment of an application.
Escalating based on time without having a guideline for de-escalation or removal from the queue will eventually result in the queue being clogged with high-priority needs-work issues, and priority will then no longer be a useful concept for project applications.
Data point: A 4-week limit for applicants to respond to "needs work" or "needs more info" would close off about 100 of the 350 applications that are currently open.
10 weeks + 2 weeks?
I think we should set priority to "normal" when we set status to "needs work." Those applications are an increased priority for reviewers, but not necessarily for the applicants.
Before closing applications, I think we should wait far longer than 4 weeks, as we can't reasonably demand applicants respond more quickly than we do, and we still have a few that have been "needs review" more than 4 weeks. I also suggest we follow the process for dealing with abandoned projects (since that's a very similar situation) by contacting the applicant directly and giving them another 2 weeks to respond.
We shouldn't get distracted by that 350 number; the relevant metric is the "needs review" count. Those 100 applications have already been partially reviewed. By closing them, we waste that effort, risk losing new contributors, and spend more time. They're not really causing problems staying open as "needs work," so I don't think we need to rush to close them.
I suggest we let them sit for 10 weeks in "needs work" status, contact the applicant, document that contact in the issue thread, wait 2 more weeks, then close the issue.
I agree with setting the
I agree with setting the priority to normal if a reviewer has responded. And I mean a legitimate response, outlining actual concerns that need to be addresses, not "Hey, this looks cool. I'll review it." The critical nature of the review lies in an applicant waiting for a response from someone, which is something out of their control. If they've received a response, regardless of how long ago they started the process, I don't consider the current state critical. Some reviews just take longer.
I also agree that we should wait more than 4 weeks before considering something abandoned. A month is a relatively short period of time when you are busy.
I agree as well! The Purpose
I agree as well! The Purpose of the priorities is to get a reviewer on the application, so once that is done it need not remain that way. And we don't need or want 80% of the applications to be critical. Besides, keeping those applications at critical will only muddy the waters for applications that haven't gotten any reviewers in along time. Priority is most important for applications that might seem like they're being overlooked by reviewers. I can update the documentation again to reflect this as "two weeks since the last review" or something like that.
Coincidentally, I was also wondering about all the applications that have been in Needs work for four or five weeks at least. Certainly those are not critical, as it's not even a reviewers responsibility at that point. It's also a good point that we should be in no hurry to close applications, and ten weeks sounds reasonable before trying to figure out if an application is abandoned. But they should be closed at some point. Ten plus two weeks is certainly reasonable. A message should be posted if the application is closed notifying the applicant they can reopen the issue and correct whatever problems existed that caused it to be in "Needs work" maybe.
Documentation
I've updated the documentation at How to review Full Project applications to reflect new priority guidelines as well as abandoned application proceedings. The priority guidelines basically apply only to applications with a status of needs review and the guidelines say to return the priority to normal once a reviewer has continued the application process according to the project application workflow (meaning not just a simple response, as davidhernandez mentioned). Additionally, I updated the priority guidelines at Application review process for Full Projects; what to expect. Abandoned application guidelines are the same as suggested by sreynen - ten + two weeks - with documentation in the application issue.
RTBC
By the way, what had happened to the git administrators? Applications marked RTBC were being fixed within a couple days until recently, and unfortunately that's what I've told some applicants to normally expect, but we have applications marked RTBC for more than a week now. (admittedly including my own, so yes there's a bias, but I didn't apply for nothing!)
Interesting conversation in the peer review group too.
Busy
I can't speak for everyone, but I think git admins are just busy. I moved your application from RTBC to fixed and will try to get to others when I have more time.
My problem is that I try to
My problem is that I try to fix these and then do a quick review and find that I wouldn't consider them RTBC.
I'll try to work on these a bit more...
knaddison blog | Morris Animal Foundation
busy I believe it! But
OK, let me be the first to
OK, let me be the first to try it out. Here's an application that has no code and has not responded to comments for 11 weeks: http://drupal.org/node/1098004
I contacted the applicant, documented the contact in the issue, now I'll wait two more weeks.
I think after a long time (4
I think after a long time (4 weeks seems ok) we can just move them to a closed status. The important thing is that we are clear with the text in the comment that it can be re-opened.
Some proposed text:
knaddison blog | Morris Animal Foundation
forgot about re-opening
I was thinking of closed as a permanent status, but it's not really. As long as we make that clear (I like greggles' proposed text), there's not much risk in closing applications. So I take back what I said before about 4 weeks being too short.
Postponed?
Is there any particular reason we wouldn't handle these the same was as the CVS applications during the Git migration? (ie. marking them as 'postponed' as opposed to 'closed'?)
Priority guidelines for RTBC applications
There are currently 6 project applications that have been RTBC for more than 2 weeks.
Should the guidelines for Major/Critical also apply here?