Drupal.org Process: Bringing idea to production.

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

The draft policy, procedures and requirements for bringing *.drupal.org initiatives onto the live sites are published here:

Process for getting changes deployed on drupal.org
Drupal.org development guidelines

Hopefully as ideas are generated and refined through the Prairie Initiative and people start implementing solutions, implementors can be proactive rather than reactive to the process and requirements necessary for improving Drupal.org.

Comments

Heavy Weight Process - who will particpate?

pcher1bw's picture

I fully agree with the concept/process you are proposing, but one of the comments to my early response in the “Module Reviews will kill Drupal” discussion was that I can’t expect volunteers (this is open source) put the proper amount of time into reviewing other peoples code. What hacker is going to follow THIS heavy weight process?

What you are describing here is a full blown software engineering process, which is truly heavy weight. I’m not sure a hacker will even know what a Data Flow Diagram is. Which notation should we be using? Just to start this discussion, why use data flow diagrams, start with use cases? Try to use an up to date design paradigm like Object Oriented Design, UML can definitely be applied, the objects designed are the modules.

This process will need document templates, and what tool should be used to design diagrams, I use Visio myself, but that probably won’t work here.

Paul Chernick
CEO
Chernick Consulting
(310) 569-2517

You cant do it now.

MGParisi's picture

First, a hacker does not follow traditional project design, documentation and project management. They do not follow quality assurance practices, standards etc. That's what makes them a HACKER!

Let me ask you pcher1bw... You have a handful of posts to your name on d.o. you have 0 commits, never seen in IRC, I dont see a solution, though you have thrown around some dollars on a few things. Granted I would definitely like to hire me a private teacher for Drupal. What initiatives have you started? You clearly don't know what a Data Flow Diagram is, since yes you use Visio, and it would probably take me if 2-3 hours to map out the common Prairie initiative IF I was also providing support on a very active day at #drupal.

Want me to point you to a million issues that get buried by people like you. You don't offer solutions, all you do is offer criticize. You bury projects by weighting them so heavily down with comments that you A) no one reads them and/or B) after 1 week they loose their initiative. I'm calling you (pcher1bw) and all the other people who decide to be a negative manipulative unhealthy cancer to all of Drupal out. Infact your the exact reason why we need a very heavily documented approach to doing things.

But currently we have NO way of accomplishing anything to Drupal.org, unless however the change you want A) does not require any significant modification B) no larger then adding a button, drop down or maybe a title on a table C) unanimous approval. And even when you have a great idea that's simple like moving the #drupal channel to the #drupal-contribute channel it takes years an act of god for it to happen. (I am still curious, I know I, and about a dozen other people had made that suggestion, and from what I remembered there was nobody saying holding it up, what actually got people to move from #drupal to #drupal-contribute? For those that where around, it took at least 2 years for it to happen, and EVERYONE seemed to agree that the Channels needed to be moved around.)

Now look at http://groups.drupal.org/prairie-initiative here people are seeking real solutions to real problem, yet even the good ideas wont make it to the website. There are flow charts, and mock ups, and many other tools and methodology to do this type of developer. Quite frankly if a DFD had to be drawn, we would do it in a heart beat. We have people who will program the changes, we have really good improvements on design, UX, IX, yet I have seen this a thousand of these ideas die because no one has the authority to make change. If somehow you got Michelle, Merllin, Chx and Webchick to agree to the change (which I have gotten) you would still need an act of God and a Mythical holy Drupal grail to get it done.


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

steady on...

leisareichelt's picture

@MGParisi - I don't know if you two have history, but this seems like a pretty personal attack to a relatively innocuous reply. I really appreciate your passion for this topic, and it is a very important topic (if we don't resolve it, I'm wasting a lot of time here!) but can we pls keep in mind that it is exactly the kind of person you're describing ('You have a handful of posts to your name on d.o. you have 0 commits, never seen in IRC,') that we are trying to attract to participate in Prairie Initative, and this kind of response is not conducive to encouraging people to participating, is it?

Also, I tend to agree. I think that suggesting a TQM based methodology for an open source community is pretty optimistic... for me (as someone who has a vested interest in some of the design of the Drupal.org site and who has seen a LOT of talk go on recently about making changes to it), the main thing that I'm interested in is how we can make sure that all the required perspectives are considered before changes are made - this goes back to the old chestnut we're tackling in Prairie with making sure the right people are around the table at the right time, knowing who they are and at least giving them the option to be notified and be involved in the process.

It's too easy to go - 'here, this fixes that problem' - without understanding the full, rich complexity of the problem.
Conversely, there does seem to be an absence of process for demonstrating that you have done this and that your work should be put up for testing/put live etc.

So, summary - I applaud the work you're trying to do here. I'd be interested in trying to define a lighter weight methodology which is more focussed on trying to make sure that you've gone to the trouble to understand how your solution will affect the full range of audience/use cases and consulted the people who should be consulted rather than going through a more formal development methodology. I hope that's useful feedback!

leisa reichelt - disambiguity.com
@leisa

TQM

MGParisi's picture

I have heard allot of resistance to Quality procedures. Implication of them are always met with skepticism and hesitation. Yet these same procedures are later seen as invaluable resources and tools later on. Am I suggesting to do follow all of the System Analyst methodology to every proposal. Absolutely not. This is a HUGE field with so much to learn, and tons of methodology's for all sorts of situations. Its the job of all of the parties involved to dedicate themselves too providing quality, and to create a system that works for the individual environment. Not a one size fits all.

Does this mean applying the entire practice of TQM to Drupal. Absolutely not. I left this open so that we can come up with a process that would fit creating changes to Drupal.org! This process will be unique to Drupal and can be reviewed, tested, tried and modified as we go along. You can add more too it, or if it becomes too cumbersome reduce it. Thats the beauty of it:)

I have seen Mock Ups, Flow Charts, DFD's, and TONS of other documents and methodologies being implemented with Drupal Projects, specifically when dealing with the website. The shear number of parties involved makes documentation and process creation a much. For example, you come up with a proposal. Then users come in and say X wont work because of Y. So you fix X. Then Drupal Management say X wont work without Y. So you come up with a X solution with parts of Y. Then the website performance team speculates your solution is resource intensive! Your project gets floored because there is no process for performance testing to see if your solution will work!

The problem is much deeper. People fill a proposal with statements and contradiction's, yet because there is no way to test if what they are saying is true your project is shelved. Processes create opportunities. As a person who is proposing changes a Performance Process would allow you to look at what they require and be able to play within their guidelines. Likewise the UX and IX people also require certain documentation, but their is no documented requirements for this either. So the UX and IX come to your project and say it wont work, yet they don't provide you with the tools necessary to make it work.

I know from first hand that when I want to make changes, its better to get a wink and a nod, and then start making changes then to open a discussion. IF you can not generate enough political capital, or get in charge with one of the few people who has the authority to give you a blank check, then you must goto the community. Unfortunately, since there is such a lack of processes, you need to pick your battles, and on the few things you decide you want to fight, you better come incredibly prepared for a battle. Ultimately little change is made that way.

So no, I see processes as an empowering force. They allow people too know what needs to be done to satisfy the needs of each party involved. When resistance is met, the process provides the tools you need to adjust your proposal to go ahead with the project. Essentially it stops the fighting, bickering and back room dealing that is currently needed to be done to get the job done.


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

Re: You cant do it now

pcher1bw's picture

@MGParisi
Feel free to point me to initiatives like the Prairie Initiative any time, thank you for the pointer.

A good TQM based software process would adopt your proposal, I think it would good to have a design document prior to creating a sandbox, and that a process gate could be no sandbox without a design document.
However a lot of people would scream at that.

Let me address a few of your issues:
1) I made a public apology for my comments in “groups.drupal.org Peer Review: Module Approval Process will KILL Drupal (http://groups.drupal.org/node/139754)”.
2) I've been into drupal for 25 weeks 1 day, so it's not surprising that I have 0 commits and I never help on IRC. I do what I can being only 1/3 of the way up the steep Drupal learning curve.
3) If I need private teachers for Drupal, I have them, Christefano and Lee of Exaltation of Larks, BTMash (Ashok Modi), MikeLikesIt (Michael Thorne, Open Chimp), otseld (Oliver Seldman), stevenator (Steve Rifkin), mike stewart, highermath (Cary Gordon), Chris Charlton, and several other senior members of LADRUPAL all of whom I know personally.
4) I attend 3 to 4 Drupal meetups a month to learn as much as I can.
5) I'm familiar with Yourdon and Demarco, and have used Dataflow diagrams on some projects I've developed on in C / C++, I don't find them as useful (http://www.keystonesandrivets.com/kar/2007/12/understanding-a.html) as UML, Flowcharts and other software design tools, and I've been a software engineer long enough to have practiced Top Down design as well. As of Visio 2007 Vision supports Dataflow diagrams as well.
6) In the year 2000 I was certified by SEI at CMU (http://www.sei.cmu.edu) to teach CMM so I have quite a wealth of knowledge on Software Process and Software Process improvement, as well as heavy weight versus light weight software processes.
7) I'm well trained in TQM, since the company I worked for as a software engineer required everyone to be involved, and they started me down the path that got me certified in training the CMM.
8) I'm not into name dropping, if I was I would have mentioned earlier that I've discussed issues and Drupal 7 in person with Webchick and Amazon, and that Webchick personally introduced me to Dries at DrupalCon Chicago.
9) I'm all for heavy weight software process, I think it might do Drupal good, but I don't think it will be adopted by the Drupal Association because most contributors would be against it.

Paul Chernick

Paul Chernick
CEO
Chernick Consulting
(310) 569-2517

Heavy Weight verse Light Weight

MGParisi's picture

All I am purposing is that we know the rules when Designing for DRUPAL.ORG (not Drupal). In specific projects that make major changes to the site as seen in the Prairie Initiative and other groups. Drupal Core and Drupal Modules will continue on their light weight path, and their path and requirements are quite different. I am also not planning a heavy weight process, just a bit more information so that as we go through this we can be pro-active to the concerns of the people who actually run the many aspects of the website. In fact the current development for Drupal.org process is relatively VERY HEAVY. I am purposing a more light weight process.

I am not purposing a specific documentation standard because I want to provide flexibility especially in getting the first process revisions up. Drupal does not need a specific quality standard, nor does it need to be certified. Systems Analyst's have their own language, and I can't teach it. For me it has been a life long learning process, but if you have worked with TQM principles you will know the knee jerk reaction you get when you mention it. TQM is seen widely as a more restrictive system, not an empowering one. I truly believe in TQM and I know that it is not only flexibility but liberating. I also know not to make TQM into an over burdening monster. TQM is a case by case situation. What works for project/company A wont work for project/company B. I am interested in GTD, not TQM, but I will be foolish to ignore my systems design and analyst experience as it has always lead ultimately lead to a GTD approach.

The thing is that these processes already exist they are just not written down. I have been working in Drupal for years, and have spent a large amount of time in Drupal's community. It is not unusual for Me to spend 8 hours a day in Drupal. As far as Drupal.org is concerned it is amazingly difficult to get anything done. Currently Drupal.org development is not just heavy weight, but is also almost completely undocumented (though we are discovering some material scattered around). Writing these processes down has the opportunity to actually make the system more light weight.

As I stated, we are not even developing new processes, we are just writing down what currently exists.


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

Leadership, Policy and Procedures

MGParisi's picture

I would love a world where we could come up with an idea, everyone agrees with that idea, and then we simply (magically) develop and deploy the idea. But for those that have written proposals, gone through the discussions, and then been ignored or got no "go" or "no go" response it becomes exhausting. So let me make my suggestion.

We need a D.O. Benevolent Dictator. I see this role as being a leadership role. One with resources, individuals and power. We may disagree with this person, we may have our suggestion die at their hands, but its better then our proposals slowly being pushed off the main page and into the Drupal Idea bin.

Part of the problem is a lack of procedure. How do we move a project that has been discussed into development. How do we deal with security, performance testing, security testing and implementation. When we are faced with the Response "That solution is too process intensive" what procedure do we follow to show that it wont crash the database (or make modifications)? How can we come up with a proof of concept? Can we get a Drupal.org test bed? When people disagree with specifics, who can break the stale mate.

I understand the fear of procedures. They are a pain in the butt. I don't want to spend 24 hours in documentation for a Drupal.org change. But honestly, I am more then willing to deal with all of these things if I know I am doing more then writing about something that will eventually float off the page and into the Drupal Ideas bin.

Unfortunately procedures are a must.

I have seen procedures being implemented on Drupal.org and felt/feel strongly they will lead us down a wrong path. We need defined goals, to setup measurements of the success of our projects, and to understand and overcome the restrictions placed upon us. Policies will empower us because we will know what we need to do! However if people are not reviewing and changing policies then they can become impenetrable walls. Thus the procedures need to accept feedback, statistics and other forms of data and be Dynamic. They are not LAWS but a continual state of improvement. Procedures may become so cumbersome that the shear drag of them stops all progress, thus they need to be cut back. Other times; procedures need to address valid concerns and issues. When they fail, they need to be reviewed and improved so that we can learn from our mistakes.

This is not the work of an impatient person. Drupal.org receives a TON of traffic. Website development for sites of this scale require a large amount of work to move foreword. This is inevitable. We need code to be secure and fast. User interfaces intuitive, beautiful and powerful. We need to set goals and measurements of success. Collect data, run surveys, draw diagrams, draw mark ups, program proof of concepts, etc.

I am not afraid of this work, I enjoy it. But right now we have no tools and no methodology to go foreword on some of the best initiatives the community of Drupal has developed. There is no one to direct us, no documents to follow, no way to progress. I am NOT here to talk about Pipe dreams, I am here to see results. I am willing to work, want to work, willing to jump through flying flaming hoops, but I (Like most of you) are trapped in an endless cycle of talk.

Finally Drupal.Org needs a dedicated Benevolent Dictator. I find Core Developers have Dries to guide the team, yet Dries is largely absent and/or silent when it comes to D.O. I don't blame him, D.O. and G.D.O. produces a large amount of content, the issues we face are complex, the amount of feedback we get is large. It really is a task onto itself, and we need a leader who can dedicate themselves to it. I am not purposing a populous uprising against Dries. I am proposing someone who can work with on these issues and who can get the ear of Dries when they need too.


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

An Example of a Process in Progress

MGParisi's picture

We use processes all the time, most often they are not written but are learned via many methods... an example of a written process is
http://groups.drupal.org/node/139754#comment-467184

This was done by ZZolo. I am not sure his knowledge of TQM, but this is not rocket science, its a natural process:)


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

Start

MGParisi's picture

Drupal.org development document. DRAFT version .0001 - http://drupal.org/node/1127550


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

Older document already spells out the criteria

dww's picture

Actually, someone just pointed me to Drupal.org development guidelines which is an older page covering similar stuff. I just merged my content into there, removed http://drupal.org/node/1127550, and added a redirect to the older page -- it's bad to have this list in two places.

Anyway, point is, some of this stuff is already documented and well-defined, and we're making an effort to make all of this more explicit. The Infrastructure Team is already working on a better framework for dev/staging/live, since this is something that's going to impact not only the Prairie Initiative, but also Git Phase 3 and a lot of other things. I don't think there's a big a crisis as MGParisi is making this out to be. And I definitely agree with Leisa that MGParisi's response to pcher1bw was an overreaction that doesn't help anyone want to contribute or get things done.

Let's try to keep our efforts around here positive and constructive...

Cheers,
-Derek

More documentation on the process

dww's picture

I also just added a first draft of this new page: Process for getting changes deployed on drupal.org.

So, between that and the now fairly detailed Drupal.org development guidelines I think we're in pretty good shape for at least documenting the current process and criteria. Certainly the deployment process needs a lot of help, and as I mentioned above, we're working on it. But, at least the current reality is now explicitly defined...

Cheers,
-Derek

Overwelming Success

MGParisi's picture

I have heard a number of very positive comments regarding the success of this process on D.O. and how it is enabling growth. I would like to improve site wide process documentation. The processes are written, they are just scattered around and difficult to find. Please help me find them all so that I may make a proposal on a new navigation structure for improving Drupal. [#1307688] http://drupal.org/node/1307688


--Sig--
Owner of Proper Programming, LLC a software and website development firm.

Peer Review

Group organizers

Group categories

Project Type

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: