COD ships with the ability for "crowd-sourced" rating of sessions. The idea is to the the wider community have a say in DrupalCon session selection. I think this is a wonderful goal.
The practical reality, however, is this:
a) There are some 350 proposed sessions for DrupalCon London. Let's be generous and say that it's only going to take me one minute each to read the session description and presenter list, consider what score I'd like to allocate to that session, and cast it. I would need to find nearly 6 hours of time in order to give each session its proper consideration.
b) As a result, I'm totally not going to be able do that. Instead, what I'm going to end up doing is hunting around the list for people I know give good talks, and rating all of their things up, without really reading the session descriptions overly carefully other than the titles, because I'm pretty confident they'll do a good job. This turns DrupalCon into an insider's club that is unfairly biasing my votes against newcomers in the community who might have a lot of really cool, insightful things to say.
c) Further, people with louder voices in the community, such as a large, well-known Drupal company or people with several thousand Twitter followers, are going to have a much easier time peddling for votes than smaller shops and independents/freelancers. This turns voting into a popularity contest, not anything remotely resembling a merit-based selection of sessions. There are a lot of brilliant people in our community who don't have 25+ employees, who don't have 500+ Twitter subscribers, and who don't have a mailing list. This shouldn't affect their perspectives brought into DrupalCon.
As a result, the DrupalCon session presenter list tends to look pretty stale, with the same old same old people in it all the time. There's a big perception of unfairness among the "little guys" who are not as well-known or don't have as big of a "base" from with which to draw votes. And, because of these factors, some track chairs ignore the results of voting altogether, making it a colossal waste of time for all involved.
So I would love to open up a discussion on how we take what we want to do (give the community a voice in DrupalCon session selection) and merge it with the practicalities that such popular voting entails.
Comments
Blind peer review
I just want to throw the concept out there of blind peer review. At the organization I work for we hold an annual academic conference with 450-600 submissions every year. Our abstract review process strips author names out of each submission, and routes them to volunteer reviewers based on the submission categories (spreading the workload out) to score them. Those scores are then brought back to the conference planning committee to choose the highest-scoring submissions and schedule them into slots in the conference program.
Excellant topic & Excellant point.
Thanks so much Angie for bringing up an oh so relevant topic!
I've helped organized 3 camps and was a co-chair of one of the Drupalcon-Chicago tracks. I've been to almost 20 camps and 3 cons and what Angie says is often true. The final session selection often misses out on new undiscovered talent.
A few weeks ago I wrapped up the schedule for DrupalCampSC.com we had voting open and it served a single purpose; determining which TOPICS were of interest. I made it a point to give everyone at least one of their proposed sessions. I don't know most of the presenters, but their descriptions alone proved to me that they know their topic very well and that I would learn a lot by sitting in front of them.
It used to be true that beginner-friendly sessions were not very common on the proposed session lists for camps and cons. This lead to my 2010 world-tour which included 4 sessions in DrupalCon SanFran and 12 camps across America. But these days I am seeing more and more people step up to the beginner's needs. Surely they saw me doing this and thought, "MAN! I can do better than that!" and so they have. ;-)
My whole point is this... If you are a camp organizer you have a powerful position and can use this to build the community and the project by welcoming new speakers.
The obvious caveat applies. If no one your team has ever heard of the person then engage the individual and have a conversation with them. I have been to one event where an unqualified speaker delivered some pretty erroneous stuff. In the after camp debrief we discussed how to avoid that ever happening again. The conclusion was to ENGAGE the unknown and possibly team them up if need be. I'm sure the individual was intending to be awesome.
I am so thankful that Angie has brought this up. We NEVER want any small shops or individuals to feel squeezed out or unwelcome. I will say that the small shops and individuals should really jump on board [even more than they already are] and engage the community as much as possible. I gave a session one time on Community Involvement where I laid out the Three Vs; Be VISIBLE, be VALUABLE, and VOLUNTEER. See the session description and slides here: http://colorado2010.camps.drupal.org/drupalcampcolorado.org/sessions/get... I believe they will help anyone [individual or shop] move along in the community.
And yet topics go uncovered...
Crowdsourcing content always will introduce speakers that are unqualified, either as speakers or, on occasion, as subject matter experts. Short of requiring references and/or "demo tapes/videos," like many large conferences do before hiring speakers, I'm not sure there is a great solution.
And yet the current process of asking people to submit session proposals on topics they are interested in presenting, then letting the audience vote on those proposals, misses out on one whole area--topics that are interesting to the audience but not thought of by the speakers.
In the ideal case, there would be a time when a DrupalCamp or DrupalCon is first announced, to let the audience suggest topics, sort of a reverse session proposal, along the lines "I would love to hear about xxxx." Then let the other potential attendees vote (I don't think I would have the requirement that you be registered to attend at this point in the voting).
Then, as session proposals are coming in, and a topic with a lot of interest isn't getting proposed, the organizing committee can do some recruiting to make sure the needs of the attendees are being met as much as possible.
Thank you for posting this.
Thank you for posting this. Very well said.
I love the followup idea of having the community vote for topics rather than for individual presenters. It encourages speakers who work in similar areas to collaborate, rather than compete for votes.
It would also probably help to have more transparency around exactly how the votes are used. Looking at the past couple DrupalCons (for example, http://chicago2011.drupal.org/news/drupalcon-chicago-session-selection-p... and http://london2011.drupal.org/conference/drupalcon-london-2011-speaker-fa...), it's been clearly stated that votes are only one factor out of many, but if it were explicitly stated that they were only a tiny factor, then the hype over voting would go down and (paradoxically) the session voting might actually then become more meaningful.
On voting
I'm not sure there's much we can do about the number of submissions or the time it takes to read through them all. But there are things we can do with the voting process to more accurately get a feel for the pulse of the community.
Gather meaningful data. Something like vote up down (as used by DrupalCon San Francisco) gives organizers much better data than fivestar. Up means 'yes' and down means 'no' and the tally for a session reflects how the community actually feels about it. With vote up down: sessions that aren't voted on get a zero, sessions that are liked get +1, and sessions that are disliked get -1. This means that it's okay if you don't get to vote on all the sessions because a non-vote is the same value as neutral feeling. With fivestar: sessions that aren't voted on get zero, where sessions that are disliked get +1. (How does that make sense?)
Make votes mean something. We also toyed with the idea that each person attending the event should maybe only be allowed to cast 10 votes. This would prevent the "I'm voting because these people are my friends (or co-workers)" phenomenon, and people might be more careful with where they placed those votes. If people only had a few votes to spread around, then they'd certainly vote on what they cared about most.
Respect the community. I believe in the need to curate the sessions so that the quality of conference remains high. Most good conferences pick a number of pre-approved sessions, recruit great speakers, propose talks in popular subjects, seed tracks, or otherwise pre-determine the outcome of some or most of the sessions. I'm flexible on what that percentage should be, but the higher it gets, the more important it becomes to let the rest of the sessions be a reflection of what the community said they wanted to see.
There is a lot of complexity to session scheduling, and getting a good process in place for voting is just a start. As our conferences continue grow this problem is only going to get harder to solve. I'm happy to see us talking about it now :) Thanks Angie!
http://www.jenlampton.com | http://twitter.com/jenlampton | http://jenerationweb.com
DrupalCampLA uses flag module
DrupalCampLA uses flag module -- a simple choose only the sessions you'll actually attend. It works well for crowd sourcing, IMO. People talk. Each person gets up to 14 selections (7 per day). My background in IT in manufacturing before I got into Drupal made me aware of capacity & scheduling. The benefits for organizers are a true idea of attendance and how to schedule rooms,,, including bofs -- which helps us keep costs down.
by the way, we haven't yet announced, (so, you know, don't tell anyone) but DrupalCampLA 2011 will be either the last weekend of July 30/31 or the first weekend of August 6/7 (most likely)
--
mike stewart { twitter: @MediaDoneRight | IRC nick: mike stewart }
flagging sessions
Like DrupalcampLA now, Drupalcon Szeged 2008 back then (http://drupal.org/node/358129) did voteupdown flagging for sessions as well. The sessions you've "voted" for showed up in your personal session list/schedule. By treating votes as promises to attend, we could also use the data to place sessions into their physical rooms based on a "heatmap" of votes. Those sessions that got more votes got the bigger rooms. Because voting for more sessions littered your personal schedule, our experience was that people actually voted reasonably. Also, the conference was much smaller, and we might have only opened voting for attendees, that part I don't remember anymore.
Overpromising risk?
Gábor, did you guys already have the sessions filtered into time slots prior to deciding the rooms? My concern from an attendee point of view is that I'd say OK, I want to attend X number of sessions...but then be out of luck when the sessions are scheduled at conflicting times. Did you force people to pick a particular track to avoid this? I think it's pretty common to mix-and-match a bit from different tracks (e.g. freelance developers might attend a couple business sessions and otherwise mostly code/development sessions).
WizOne Solutions - https://wizone.solutions - Drupal module development, theme implementation, and more
FillPDF Service - https://fillpdf.io - Hosted solution for FillPDF
timing trick
I think the trick is to have the sessions selected reasonably earlier before your schedule needs to go to print (if it needs to that is). So that you still have room to fiddle with your program and move things around as you see the interests. But the initial placement of sessions in slots can already be done based on the popularity if you can reasonably assume a vote means someone really wanted to attend. I think its more important to have good sized rooms for sessions compared to all sessions of a track being at the same place. For those mixing and matching sessions, having separate rooms might even be good to stay around for a different session. For those looking for one specific track, walking around, looking at the sponsors and actually talking to people does not hurt :)
I believe San Francisco used
I believe San Francisco used flag in a "+1" manner for voting and then changed it to mean "I will attend this session" after the voting was over. The only votes in the votingapi_vote database table are for the session evaluation system. I did some analysis of when sessions were flagged in this earlier post analyzing copenhagen and sf voting users compared to registered users.
It is true that with fivestar sessions that are disliked get a +1 (really a 20 and the other stars are 40, 60, 80, 100), but I think people who want to use votes to compare sessions can combine the average rating with the number of votes to get the same conclusion you mention.
Some quick queries about chicago session voting where fivestar was used show that people want to use all 5 of those values. Often systems based on a 1-5 or 1-10 scale end up with clustering at the 1 or N value which indicates that rating system can be reduced to a simple up/down. But when the item being rated is complex enough users of a 1-N system will show that they need all those levels and in Chicago they did. (Pages 61 and 62 of Web Reputation Systems discuss that idea in detail).
I'm waiting until voting is over to post more thoughts on this...
knaddison blog | Morris Animal Foundation
Disagree
I disagree with this point. I don't think it's fair to say that fivestar is worse; I think each approach has its merits. I think it's not out of the question to say every session will receive at least one vote, and thus the problem of sessions without votes getting 0 is not a problem. The difference, then, lies in the information provided: up/down provides information about how many people want to see a session, whereas fivestar provides information on what people predict the quality of the session to be. Personally, I disagree with the up/down approach, because, with 360 sessions, people are unlikely to find every session they would want to attend, thus turning it into the popularity/elite club problem: only well-known presenters will get voted on at all, so if a lesser-known presenter had 10 people look at the session page, and 9 of them give +1s, that is looked at as not as good as a well-known presenter who had 100 people look at the page, but only got 50 +1s. However, that's not to say that fivestar has many of the same issues as well: fivestar is also used as a popularity contest, only giving high ratings to well-known presenters. I do, though, think that the problem is mitigated somewhat by the fact that the scores are averaged out.
That's just my 2cents. I haven't done a very thorough analysis of the problem, so don't take this as an expert opinion.
Thanks Angie for opening up
Thanks Angie for opening up this discussion.
I have actually thought back and forth of how to do the session voting for this year in London, and the best I thought that was done so far, is what Chicago did. The 5-star rating gives an indicator about the quality of the sessions as well as number of votes, whereas the former should be considered more important. (a side note: we also put an effort in communicating that people should "rate" or "evaluate" sessions and not "vote", as to avoid the perception of the process being a popularity contest, in the hopes that people will look at not so known speakers.)
Having said that, I was also thinking of abandoning voting all-together and let the track chairs, representatives of the community, do the selection. I know that this would take away the "community-factor" but in the end, it is often as many say: track chairs are considering votes as one of many factors.
I also second the voting on topics before session submissions open instead of voting on sessions. I proposed this a while back but it was then deemed to complicated to execute. Maybe this is the direction we are moving into, having more resources and support.
There will always be an issue regarding popularity votes (even if limited, likelihood is that you vote for the top anyhow) or pre-selecting what you would attend (people can always un-flag after the selection has been done).
Voting on topics before submissions
Voting on topics before submissions seems a reasonable step toward making sense of crowd-sourced input; maybe if the concept were developed in more detail, it's execution wouldn't seem as complex.
At DrupalCamp Western Mass
At DrupalCamp Western Mass there was a feature for "Requested Sessions" that people submitted. This helped to guide session submission and acceptance. It was actually really widely used (which surprised me, I thought it was a bad idea). ;)
It was a great event
I was fortunate enough to speak there and really liked the mix of sessions. It was a great event. I think it's a pretty good model to explore.
Jason Pamental
[ @jpamental ]
Rotating Panel of Selectors
In other conference settings, I've seen good use of a rotating panels of heavy-weight experts used to help in the decision making process. Some form of public voting can help inform decisions, but the final say comes down to these 6-8 people. The make-up of the panel shifts yearly, so there is no long term entrenchment and so that various points of view are represented. This helps avoid some of the problems with blind peer review (ie - blind peer review is never really blind - most folks can tell who is presenting what based on topic, writing style, etc) while ensuring that the value of peer review is maintained.
It also means that the folks on the panel actually get those many many hours to properly pay attention to the submissions.
The Problem With Requested Sessions
This is a community largely of developers. We have an intrinsic myopia because of that that creates a "tyranny of the majority" where I could envision nearly every session being about such-and-such code or such-and-such code initiative. This ignores topics related to business and project management that get significantly less votes than your typical coding session, but make DrupalCon valuable to folks like me. I think there are ways to combat this, but it needs to be a part of any new system.
what about requested topics?
What do you think about suggesting topics and vote on topics suggested rather than suggesting specific sessions?
Similar Problems...
Really, any time suggestions are crowd sourced you end up with the same problem - "conference by committee". It's the same issue with designing websites ironically: if you let everyone dictate the requirements for a site you end up with a hodge-podge of stuff that doesn't make for a particularly interesting or innovative site. I think having the community vote on core themes for a conference shouldn't all-together be thrown out, but the best conferences I've attended have always had strong curation by a small group of representatives who are tasked with creating an interesting, diverse, and meaningful conference. Unfortunately, that flies in the face of how we do things in the Drupal community.
If I were king, I'd say we should have the DA members decide on a conference theme (or some small set of themes... think like prom in High School if you grew up where they have such things). Then you have a conference team that is tasked with ensuring that the core themes are true within the sessions that are selected and come from a diverse set of people.
the problem is complexity..
...the solution is mitigation complexity. It makes sense that getting input on the strength of interest on general topics and desired target level, (beginner, intermediate, expert) before getting input on specific presenters on topics, would be a useful step in bringing focus.
Probably the bigger problem is eliminating well written proposals for desirable topics at the right skill level submitted by individuals who end up as not able or not prepared to make the most of the valuable venue opened up to them.
I think that would depend on
I think that would depend on how the topics are suggested. If they were picked by a select group of individuals, then you would be right back where we started, though possibly worse because new ideas would be less likely to enter the arena.
The most prevalent concern (IMHO) should be that of stagnation. Keeping the sessions and ideation in DC sessions fresh is paramount. A restriction on presenting in subsequent DCs would make sense to me.
Restricting diversity
The problem with this idea is that it would narrow the rich diversity which is available at the conference. Drupalcon is the perfect platform to introduce new ideas which fall outside of peoples regular palette. I want to be challenged with the new and exciting.
Paul Johnson
http://www.twitter.com/pdjohnson
Global Social Media Lead for DrupalCon
What about "insider's club"?
Great comments/discussion so far; yay!
Something I don't think the comments so far address, though, is the "insider's club" issue.
I don't know how we find that unheard of person with a lot of great things to say when their session proposal is pitted against well-known community rockstars and people who work for companies with 25+ employees with a guaranteed base of votes. It's a completely unfair contest.
Since another big goal of DrupalCon is to draw people in from the "outsiders" pool to grow the Drupal community, I also don't know how we factor in the votes of people who aren't following @drupalcon on Twitter/the Drupal.org front page to even know that DrupalCon sessions are open for voting. Votes are therefore always going to be skewed towards what "insiders" want to know, which is generally going to be more advanced, technical topics that are utterly meaningless and frustrating to someone new to the community who's evaluating Drupal.
I throw up a bit in my mouth at the very mention of this word, but... should we consider "quotas" for DrupalCon sessions? Shoot for X% of speakers (maybe 10-15% or something) who have never spoken at a DrupalCon before, but whose session ideas were vetted by the track chairs? Shoot for X% of sessions (10-15% or something) aimed directly at new users (an entire "Newbie" track?) I am also fully open to other, less vomit-inducing ideas. But the question is, how do we stop people like me from only voting for their friends and on only highly technical session topics, and thus skewing the results?
Leveraging Drupal Camps
I think we need to start leveraging drupal camps somehow to help determine what presentations/presenters are good. More newcomers could also be found that way. If possible, a simple framework for drupal camp organizers to follow to rate presenters/presentations that drupalcon organizers could reference would be nice. It would also be nice if we as a community could encourage people to present their presentation at a drupal camp first before doing it at a drupalcon. I think this would allow for more solid presentations also.
This could be helped by
This could be helped by adding in a non-public field for use by organizers "where I have presented before". Lowers the possibility of someone who will completely bomb and provides data that won't be present in all d.org profiles or a quick google search. Opens the pocess up more to newcomers who only have presented to smaller regional groups.
We did do that in Chicago,
We did do that in Chicago, I'm not sure it is there for London or not.
Yeah, realized that it's been
Yeah, realized that it's been done before after getting to the bottom of the thread.
Nothing like this was there when I submitted a session for London. Having never presented at anything larger than badcamp I would have loved to be able to offer a little more context to the reviewers. Sometimes "I promise I'll do a great job" isn't good enough.
DrupalCamp Twin Cities post-camp data
Just wanted to note that the Twin Cities group has been doing some follow up work after last week's camp. Session head counts are posted, there's a wiki page for organizers to post thoughts on what worked and what needs improvement, and an attendee survey is in progress.
DrupalCamps as previews
I agree with this 100%. I would only add the hope that focus is on good presentations, not good presenters. Except for people who present for a living, most people will have some great presentations and some iffy presentations. What makes a good presentation?
If the presenter tries to phone it in, or doesn't bother preparing, or is just posing, it makes a huge difference. Even rock stars can hit a wrong note. And that doesn't mean that next time he or she doesn't rock the house again.
DrupalCamps (& summits & meetups) can be kind of sneak previews of potentially hot sessions for DrupalCons. Great idea. +1.
PINGV | Strategy • Design • Drupal Development
Will it Play in Peoria
I've said for a while that DrupalCamps serve two purposes:
1) Reruns. (Didn't make it to DrupalCon? Lots of people recycle DrupalCon sessions at camps over the course of the next year.)
2) Peoria. :-)
Quotas, or rather guidelines
Future of Web Design started having 2 tracks of speakers - the 'big names' and a second track of newcomers, and it has brought to light some really great presenters. Looking at DrupalCamps for new presenters is a great idea too.
Having the tracks for business, design, etc is helpful - it keeps things a bit more balanced. It would be nice see more variety within the tracks, but it's certainly a start.
Jason Pamental
[ @jpamental ]
Insiders? counter with a stream for new voices
Having a stream/track dedicated to new voices is a good idea. It's promoting another facet of diversity.
Voting is one part of the mix but shouldn't be the only deciding factor.
I like the way DrupalCon London has assigned people to evaluate each track. That's a great idea to counter against the bias of tech people only voting for tech sessions. If each stream has a champion - not only helping to evaluate the sessions, but even encouraging new people to submit sessions, then that's a good way to grow the knowledge base.
As for reviewing hundreds of sessions... rather than asking the whole session committee to review every session, set a quota - Number of sessions divided by number of reviewers... if everyone is expected to review a fair percentage of the whole, then it's likely those reviews will be better. This is what Zookeepr does for linux.conf.au - it also randomly assigns you sessions to review - helping you avoid the natural bias of reviewing your friends sessions.
You could also suggest that you shouldn't review the sessions of your friends, - which is precisely why blind peer review is used in Academia.
We have written Drupal modules to do blind peer review.
https://github.com/creative-contingencies/converge
Donna Benjamin
Former Board Member Drupal Association (2012-2018)
@kattekrab
I agree
Having chair tracks is a way of 'fudging' noteworthy proposals which are in the interest of the community.
Paul Johnson
http://www.twitter.com/pdjohnson
Global Social Media Lead for DrupalCon
+1 on the quotas. That's
+1 on the quotas. That's something I did anyway when organizing DrupalCamp W. MA I think DrupalCon organizers are aware of it, maybe not officially though.
I also agree about using camps as "minor leagues".
I'm in the big or at least medium name camp and can promote myself and have gotten a session for several camps in a row. But I do hate the nerdpotism. Still, the fact is, it's taken me 5 years to get here and a lot of hard work. If I wasn't rewarded with professional recognition, I'd be a little bummed about that. Plus, many companies have a "publish or perish" model w.r.t. conferences. I like going to D.C. and meeting old friends, and my sessions get me there.
So I don't see anything wrong with giving slots to established names. I think CHI did it right though that you can only have one talk. That helps spread the floor a bit. I suppose if I support a quota I might get bumped... but hey, I guess it's bound to happen eventually anyway with all these smart youngins...
Insider information
I cannot speak for other track chairs, but for the Coder track for DrupalCon Chicago here's what Robin Barre and I did:
First, we got a dump of all sessions into a spreadsheet, complete with descriptions, presenters, average score from five star, and the number of ratings. All of that is useful information. Contrary to what Jen says above, "no rating" does not imply 0, or less-than-one. The fivestar average is only of votes cast, and combining that value with the number of votes is valuable. A session with a rating of 4 after 50 ratings clearly has more interest than one that has 5 after only 2 ratings. That's information that track chairs need to take into account.
Once we had our list, we threw out most of those sessions with a low overall average; I believe anything less than a 3 average we concluded people weren't generally interested in. That still left us far more sessions than we had slots for, as we knew going in that we had only a limited number of slots for Coder.
We then clustered the remaining sessions by their general topic. For instance, there were sessions relating to "introduction to stuff that is relevant to the coder track", "scalability, performance, and other similar stuff", "servers and administration", "some specific hot new core API", "some sweet contrib module you should know about", etc. That sort of clustering allowed us to determine what our potential breadth was. We wanted to have a good range of session topics covered, so picking 4 "here's a cool new module" sessions and nothing on performance, even if all of the module sessions were more highly rated, would not have resulted in a good track.
Going through each topic area, we tried to identify common threads. Were two sessions essentially the same idea, or different spins on the same idea? Was there a lot of apparent interest in a given topic area, regardless of how many sessions were submitted in it or how each one was individually rated? In some cases there was clearly only one session with community interest. In others, there were different sessions that were rated similarly.
We also looked at the presenters. Certain topics, like intro to module development, we knew we had to include no matter what the vote count and it needed to be covered by someone known, experienced, and trusted to nail it. We wanted an "insider" with lots of speaking experience. Others, we could be more experimental. That's one of the key reasons that the session submission form for Chicago asked about a presenter's previous experience. Even if we didn't know them personally knowing that they had presented at several DrupalCamps before, or even non-Drupal conferences, helped to boost our confidence in them.
Another factor we included was feedback from selected "bonus reviewers" that we reached out to for their experience, specifically Gabor and Jeff Eaton. Gabor had previously been a track chair and Eaton we all know has a good eye for sessions that have potential to be interesting. Their input we treated as a sort of "extra special bonus community input".
That ended up as an interesting balancing act: Broad topic coverage, ensuring we touched on areas we felt were important to include at a DrupalCon, speaker experience (even if we didn't know them personally), speaker background (if we didn't know someone but they maintained several modules, we checked up to see if their modules were any good), and general level of community interest (as determined by both rating average and vote count) were all weighed against each other, topic by topic. Eventually in each topic we managed to whittle the list down to one session eventually, which was not always the highest-rated. All of the above factors played a part in our decisions. One by one we selected sessions that balanced those criteria well. Sometimes that meant picking a session that didn't have as high a community rating but fared very well in the other criteria.
The end result didn't quite cover every topic area we wanted to, as we had only limited session slots. (Chicago started off by reserving an equal number of slots for every track, so we had far fewer coder slots available than past conferences; overall I think that was good thing.) However, the sessions we did have we felt were all strong and covered a broad base of subjects rather than simply "lots of popular sessions about cool modules". We also selected a number of alternate, backup sessions that we wish we'd had room for, and one or two of them did get added to the schedule later.
It wasn't a perfect process, but it did involve far, far more data than simply five star ratings. That was only one piece of the puzzle.
If there was anything we could have done differently, it would have been to reach out to more experts in the general PHP world to try and actively recruit, especially during the pre-seed period. As it was I reached out to two, one of whom declined and the other already had a speaking engagement elsewhere at the same time. In hindsight I wish I'd started that process earlier and tried to bring in more outsiders, because I think the Drupal community desperately needs to be further exposed to non-Drupal PHP expertise, something we are sorely lacking. (Note to Denver: Let me know if you want suggestions on who to try and bring in.)
Why one overall vote per session submission?
If one flag is good (or up/down), then wouldn't more be better? Some would cringe at the thought, but let take a look at what this could mean.
If a person takes the time to submit a session proposal for review, it should be given due consideration on several levels including quality, relevance, and uniqueness, etc. Can't each of these measures be voted on by all reviewers rather than a single up/down or flag vote for the whole submission and some kind of weighted total score produced? Determine all the qualities you want in a session and get an overal score from evaluating each criteria.
Determine how many sessions will be allowed. Let session organizers go through the top scoring sessions to look for abnormalities in the voting. If one can use flags to calculate a session proposal's average score, one should be able to average the reviewers given scores as well. A vote from someone who blindly and consistently marks everything with all fives (outside one or two standard deviations of the norm) should not have their votes count with the same weight as a more cautious and deliberate voter. Some people are just uniformly harsh or easy. You need a way to account for that.
Picking the first 10 of 20 session from the top 40 scored of 200 submission should be a trivial matter. The next five a bit harder and the last five harder still, but not so hard that anyone would lose sleep over it, especially when backed up with data.
I think the hardest part of session selection for conference organizers would be having to choose between two highly scored, but somewhat identical session proposals. I mean really, how many sessions on the Views module can one endure, highly rated or not?
I would hope the goal is to have wide range of session topics from a wide range of presenters that appeal to a wide population of attendees, from Drupal newbie (like me) to Dries.
Ultimately, it is up to conference organizers to choose the sessions, as a poor session reflects as much on the organizers as the presenter, in my opinion. But hey, the reviewers just successfully whittled down 200 submissions to the top 40. It's time for the organizers to makes some calls and check up on the potential presenters. Tell the unknown new blood you like thier submission, but would like them to present it to a local group and tape a few minutes for review. Do some due diligence. Have some fun.
James Sinkiewicz
Drupal Site Builder and Generalist
http://MyDrupalJourney.com
Why I did my own ranking tool
This is an interesting debate. Some of the reasons given by webchick led me to the development of this ranking tool for Drupalcon London: http://www.happer.at/drupalcon/
Backround: I tried first to rank "honestly" - give stars from 2-5. But then I realised that - given the dynamics of the voting - the only thing that makes sense to do is to rank those that I want 5 (and then maybe even rank those that I don't want 0). And finally I thought it would actually be best to know where sessions stand currently. To see who is already at the top and who is not.
So, while not addressing all questions in the article, here are a few suggestions:
- have only "likes" (Facebook showed us that 5-star ranking is past)
- publish the rankings for transparency
- mention how sessions are chosen (rankings + chairs + ??)
- maybe have part of the places reserved for "popularity" and others on purpose chosen by the panel (to keep outsiders in)
- why not propose to the wordpress/joomla/sharepoint/enterprise community that they each chose one session that would interest them
-- www.stefanhapper.com
More Insider Information
I was a track chair and the co-chair of programming (with Matt Farina) at DrupalCon Chicago, and prior to session submissions I gave these topics an enormous amount of thought. Here is some of that background and how I felt it worked out.
In the last two DrupalCons before Chicago I did an experiment - after sessions closed but before voting started, I made a prediction as to which sessions would get accepted. My success rate hovered around 90% correct. It is, in general, pretty obvious who the most popular and well-known speakers are these days. Note that this is not just insider-ism, the Drupal community has some very very good speakers, and they are always highly voted so we reward quality too. Lets also keep in mind that over and over again there has been one piece of attendee feedback that has risen above all the others in regard to speakers - the quality was too low and inconsistent, and too many people were unprepared or didn't give a talk that matched the audience's expected skill level. This is a huge problem because you want new blood, but you want high quality, well-prepared, knowledgeable speakers as well.
All that said, I definitely wanted the ability to get some fresh blood into the mix for Chicago. I had a couple of solutions for this. First was the controversial pre-selection of sessions. Why not just give a session to webchick and eaton and morten and etc? We all know they're getting picked anyways. Get them out of the voting pool and maybe those who are left can attract more attention. The track chairs were given the opportunity to recruit for these sessions as they saw fit.
The other controversial rule was the speaker limit. At several past DrupalCons, both crell and webchick have spoken three or more times. I don't have an inherent problem with this, they are both smart and engaging and have great big ideas to spread, however there is no doubt that it also limits the ability for new blood to get in. I actually wanted to go further than we did, limiting to one session of any type at all but I was outvoted.
In practice I don't think either of these efforts was successful, and I believe the reason is that at this point we have more hot topics and domain experts than we do DrupalCon speaking slots. I don't think there's anything we can do about that outside expanding the conference which many feel has already grown far too large.
On to other topics, it is also no coincidence that the companies with the loudest voice in the community often get well voted as well. A company with ten employees who each have 1,0000 Twitter followers is just going to attract more votes when everyone pimps each other's sessions. Whereas the independent dev with a great idea just doesn't have that platform.
Given all this, after DrupalCon Chicago, I came to believe that the utility of voting has come to an end. There is nothing to be served by it anymore, and the reality is that it hasn't meant much at all in a long time.
What I would like to see is careful selection of track chairs that are far more empowered to recruit for and curate their tracks. These track chairs should be longtime community leaders who know their subject areas deeply and have some sense of the trends in the community at large. They should also be people who spend their time researching things like blog posts and DrupalCamp videos, keeping an eye out for people who are not necessarily well known but who have good ideas and can articulate them well. This does not rule out session submission, but it does leave the final call to the judgement of the track chairs and their committees.
Obviously this opens up whole other cans of worms. How do we stop the NodeOne folks from selecting all their co-workers? How do we make sure everyone's not just choosing the co-maintainers of their modules? I think this can be mitigated by making sure the selection committees are reasonably diverse, with a spread of three to four people in each topic area. However I also think it can be mitigated by the fact that we all love Drupal and we should trust each to not be assholes and serve the greater good. I have a hard time coming up with high level contributors I wouldn't trust with this task. I have also been a proponent of a rule that states that a track chair can not speak at any DrupalCon in which they are helping select sessions. This seems like a fairly obvious rule and I'm a little shocked it has never been implemented before.
I'd also like to bring up core conversations as a place where new contributors probably have a better chance than normal to make their voices heard and get their ideas out there. In my mind, core conversations is specifically for the purpose of presenting ideas that are a little more "out-there" or conceptual, and I think we will encourage a lot more thought-experimentation there. Success presenting to such an intimidating audience could easily raise someone's profile in an incredibly meaningful way.
Finally there is one more thing to consider which I haven't seen talked about but which is definitely an issue. There is an ENORMOUS economic incentive to getting a DrupalCon speaking slot. If you are a company launching a new product, there is nothing more valuable than getting an hour of undivided attention from the most engaged and interested members of the community, and if you're a Drupal shop there is no better publicity for recruiting and publicizing your services. There were some instances in DrupalCon Chicago where track chairs came under varying levels of pressure and lobbying from people in an attempt to get their sessions selected. This is very bad, and it makes the task of being a track chair become extremely uncomfortable because it adds a whole new level of crap to have to deal with in what is already somewhat of a political minefield. This is a huge amount to ask from volunteers who are trying to focus on implementing a high quality program. I praise the DrupalCon Chicago leadership and the DA in helping to tamp down these instances, and I would hope that in the future that this effort can continue so that we can keep the session selection as much about the community and the future of Drupal as humanly possible.
In heyrocker camp
I whole heartedly agree with heyrocker here. Couldn't have said it better. This should come as no surprise after he and I spent so much time working together on DrupalCon programming.
I just wanted to officially point out my agreement. Being on the organization side of this and seeing all sorts of different factors that come into play has been an eye opening experience.
agree very much with heyrocker
This being the third DrupalCon I (help) organise and being involved in session selection, I definitely second what Greg said. Voting has become a factor that track chairs struggle with increasingly, as it is hard to leverage and there are a lot of other factors to consider.
I do believe that well selected track chairs and track committees can solve the issues regarding a diverse session selection. Criteria such as usual suspects, experience, topic, description, trends within the community and Drupal project play a far more important role than the voting results we have at the moment.
I do like the idea of pre-selecting the sessions that would get in anyhow, limiting that to 30% of sessions. Then have one round of session submissions open for 2 weeks, selecting another 20%-30% and then get another round of session submissions to fill the final slots. This may help the track chairs and committees to not be too overwhelmed and give them time to get in touch with speakers, do their research etc, and resolve the issue around getting in new "blood". If we stress that this is one factor (and I don't think that quota is a negative thing if it makes sense), then track chairs will consider it. Having said that, it always comes down to common sense and there is no clear recipe. Just as we shouldn't force 12 sessions in a track if 5 of those aren't of high quality, whereas another track needs more slots, cause it has too many quality sessions.
Apples, Oranges, Juicy Tangerines and other assorted fruit
I think it is probably impossible to have the perfect selection process. But the current one can definitely be improved. There are a lot of great ideas above and perhaps a way to organise them is to think about the underlying goals for Drupalcon itself. Here are some of my thoughts on that:
These are probably beginner sessions and you want the best speakers here - I don't care if they have spoken in every Drupalcon since the first one or have never spoken in a Drupalcon before. You simply want a great, engaging speakers that know the subject and will excite the attendants and make them love Drupal. Whoever chooses these speakers needs to know them, have heard them before, etc. Think of it as small scale keynotes. Pre-selection / invitation is probably the way to go here. The point is to guarantee a good conference for new people to Drupal.
Drupalcon can also be a chance to really pull in people who are engaged to the cause but not fully into it yet - that have interesting ideas (not necessarily earth-shattering) but if you give them a slot and a chance to present you've hooked them for life. If you continuously ignore them they might simply come to the conclusion that there is no way in this event and they need to find some other place to contribute. Voting on topics rather than sessions, followed by session proposals with voting + committee selection + chairs is probably a good solution here.
Session proposals should probably also be a bit lengthier - since the person is not that well known a half-page abstract might help to better understand where they are coming from.
Here the academic process is probably the best - blind reviewing from a program committee with chairs making the final choice. We don't need to get to stuck up on the blind review process - if you get an abstract about Workbench you know there are X number of people that can do that
This also sounds a lot like a Core Discussions type situation although that is not always the case. You can have novel applications of Drupal, etc that are not about Drupal core.
To mix and match all of the above quotas might be a solution. Guarantee that 15% of sessions will be from brand new speakers - the beginner sessions will all be invitation only and the rest a mixture.
Also don't understimate the flexibility of shorter presentations. A lot of academic conferences have 15-20 min slots. The point is not a mini-tutorial of 50mins but simply to say "hey - here I am - this is what I am working on - here are the key points - if you are interested let us talk afterwards".
Finally, perhaps the simplest most important thing is transparency. Right now we have some vague idea about votes mattering or maybe not but not much else.
Interesting stuff
Idea #1
Start with very few number of slots really open( no insider business please) for fresh bloods and monitor it hard, voting or no voting get this proposal reviewed by the drupal rockstars panels.
Say for particular con start with 2 open slots for new speaker, ask proposers to propose with a pre-decided topics.
Idea #2
At BOF sessions keep one room alloted to the fresh bloods only and have a panel of reviewer ready to choose one or two for your next con.
How many more Dries's are out there not getting a chance?
Imagine if a guy called Dries approached the current voting system with an idea called Drupal, that's how he started out, on his own. How many more Dries's are out there not getting a chance?
Many small Drupal shops / individuals have fantastic innovations or success stories they want to share. Not all brilliant ideas come from charismatic speakers.
One problem with the current semi democratic process is that crowd pleasing topics and presenters always win. Popularity does not always measure the benefit a session represents to the Drupal community. Security will never be as sexy as jQuery, features we all use will get more votes than emerging revolutionary new ideas.
It's fabulous that Drupal has achieved wide adoption and the numbers of Drupalcon attendees is soaring. The downside of this is its hard to establish your place in this crowded space.
Paul Johnson
http://www.twitter.com/pdjohnson
Global Social Media Lead for DrupalCon
Drupalcon Session Selection needs a new incentive structure
Right now it's a sort of no mans land trying to please greeks and romans. I know that track chairs work with a lot of constraints and try to do their best while sailing the shallow waters of sponsor and insider pressure. But let's face it right now I don't think there's much in terms of new things happening.
I know that there's space for core stuff in the core conversations initiative, but in my case whose main interests reside in performance and security, there's very little space for such. Those are, IMHO, very important aspects. Any project that has the ambitions that Dries puts forth in his keynotes cannot overlook such aspects. Yes there are more specific events, like the developer days and government days. But we're loosing the opportunity to expose the largest possible number of persons together in a face to face setting when we commit something deemed "topical" to events where the exposure will be much less.
Drupalcon should be about celebrating innovation, new ideas, new approaches. You cannot decide before hand who's going to be a good presenter. You have to take risks, make decisions and bear the consequences. I think that playing it safe and letting the usual suspects take the lion share of the slots is a bad move. I'm not criticizing those usual suspects capacity or presenting skills. They're great presenters and make important contributions to the drupal project. What I'm saying is that the risk of inbreeding and working in tight loop is there. And unless we do something explicitly to diminish our exposure to it we're closing ourself as a community to new approaches, perspectives and people championing them.
We need to work this out and see how we can make Drupalcon be more provocative and a source of new ideas and directions. Perhaps that means that the business part should be more clearly separated from the technical. Remember that to keep business flowing and increasing we need to keep pushing ahead with new ideas and approaches.
Not being a fan of quotas anywhere I find that it's a start. Thanks webchick for getting the ball rolling. Let's keep moving the ball forward.
For just a moment I read that
For just a moment I read that as "geeks and romans"..
Voting has to be clear to have meaning (and it doesn't here)
In fivestar voting, what does a 1-star vote mean?
What does a 5-star vote mean?
What does no vote mean?
I feel the big problem with fivestar is that it has no clear meaning. We can't count on some legend to give meaning. People don't read legends. Compare this with ratings on travel sites or CNet or shopping sites, where the vote is clearly about a narrow parameter. (Cleanliness, Staff, Accommodations, etc.) Those have more meaning -- although even those have only limited value without comments/reviews.
The benefit of a simple +1 widget is that it's clearly an endorsement. You're either for it or you're not. If you combine it with a kind of "shopping cart" of sessions, sort of a "pick sessions you plan to attend" kind of functionality, then it's even clearer: Give this a vote only if you're going to go. Don't vote it up if you think it's good but you have no intention of going.
In the end, though, these votes can reflect only one small subset of the crowd attending the conference itself. How does voting help the majority of people who don't spending hours voting on sessions?
What I don't like about the sessions we've been getting lately is that they seem to be picked based on people's popularity and/or topic knowledge, with little appreciation or consideration of not just ability to present well but desire to really rock a presentation. Some sadly typical disappointments in recent 'Cons are:
What I miss from the earlier cons (my first was OSCMS in 2007):
Picking great sessions is hard, and I don't know what the answer is, but popular voting several months before the event seems to be of limited value, and perhaps quite misleading.
So how does a DrupalCon team pick winners? Perhaps one answer to the "same old people all the time" thing is for track chairs not to be able to pick themselves. Perhaps it's focusing more on sessions that give and less on sessions that boast, pitch, pose.
In the end, though, I think the problem is upstream: What are we pitching? Slideshows with blah blah designed to fit into one of these 4 or 5 predefined boxes. I submit that DrupalCons are perhaps asking for the wrong things, and trying to second-guess what Joe Decision-maker wants to see, rather than just going for the best, most interesting, most provocative-sounding ideas people might propose to spend an hour on at DrupalCon.
SxSW doesn't have this problem, it seems. They have voting, but they always seem to have an abundance of provocative, rockin sessions. What do they do that's different?
PINGV | Strategy • Design • Drupal Development
Totally!
Laura has made some candid and important points here. What she is really saying is Drupal should grow but not forget its roots, what made it the great success story it has become.
In the end, with so many bright people attracted to Drupal, there is never enough slots for the talent available, and this is a good sign.
Paul Johnson
http://www.twitter.com/pdjohnson
Global Social Media Lead for DrupalCon
There are a lot of
There are a lot of comparisons here that I don't think are at all relevant. DrupalCon to SXSW is a total apples:oranges comparison, they are immensely different animals. Remember that before SXSWi came into being, there was a long-standing music and film festival that they were able to hook their boat to. This gave them a lot more freedom to develop what they needed to and find their own business model. It also means the risk is spread out a lot more across these individual pieces. SXSW has the advantage of always being in Austin, so they have a base of people there to run it year after year. This is their job. It is also an industry con, not a product-centric conference like DrupalCon is. SXSW has more talks listed under the letter 'A' than DrupalCon has for their ENTIRE CONFERENCE. I refuse to believe that all of these hundreds of sessions are so provocative and rocking, again the size of the con allows them to do all sorts of stuff we really can't. The needs and goals of these two conferences are wildly divergent and comparing them is only damaging to our goals..
Likewise comparing a current DrupalCon back to Sunnyvale... the differences are so large that it is difficult to know where to begin. You can't even compare a modern DrupalCAMP to Sunnyvale. BADCamp has reserved a 1,000 capacity venue this year!!
I think it may be time to look at some truths about the modern DrupalCon (note this is all mostly centered on North America, where I have attended three DrupalCons, one of which I helped organize. I have yet to attend an EU DrupalCon.) The fact is that DrupalCon is rapidly approaching the point where it has as much in common with a trade show as it does a technical conference. I honestly do not think this is a bad thing, but it does mean that people need to change their expectations around it. It is going to continue to have a much broader spectrum of attendees, which is going to mean that sessions need to become more generalized to support that. It is the main funding arm of the DA, which means it needs to somewhat prioritize business and sponsors for funding opportunities. The fact that the Drupal community has matured to the point where it can support such a thing is actually really impressive. We should not be disgusted but proud.
If you want something other than that, opportunities abound. In Chicago, I had a blast attending Core Conversations, which contained some incredibly passionate and forward-thinking sessions from a really broad range of contributors. DrupalCamps are sprouting up EVERYWHERE and offer an enormous opportunity for newcomers to bounce ideas off of veteran core devs and longtime community members.
None of this is to say that we shouldn't be spending more time picking quality sessions, or that we shouldn't be focusing on bringing in new blood, or making things more exciting or interesting. It is just to say that we should focus our efforts on the crowd and goals of the conference, and that there are TONS of opportunities out there for a different kind of experience if that isn't your thing.
It is going to continue to
This is the assumption I question. Sure, there are changes inherent in larger events with larger crowds, but this is still Drupal, still open source, still about community leadership, and all the best sessions I recall from the early days would be perfectly delightful at DrupalCon today.
PINGV | Strategy • Design • Drupal Development
New blood go to drupalCamp
I could not completely agree with heyrocker. But I also agreed with heyrocker about the sense of business drupalcon has able to inject within it and I truly believe that it helps drupal getting the momentum it required.
But I also belive that along with business side we need to nurture our community which is very very hard at this point of time and specially in DrupalCon. I am believer of that let drupalCon be the showcase conference of entire drupal and it's community and let start/help/support/encourage regional(across the continents including Asia) camps/workshops to get the fresh ideas and new bloods in the community, and for that there need to be a published process( you can't really just expect local/regional camps could able to get sponsors always).
really off topic, but...
Luckily there is a whole group for that: http://groups.drupal.org/drupal-event-organization
It has multiple wikis on how to grow a group and run camps.
I have personally never heard of a camp failing to run b/c of lack of sponsors. The main philosophy is to balance the level of spending to match sponsors, but you can always reach out to international sponsors if the local sponsor community isn't strong enough.
knaddison blog | Morris Animal Foundation
Drupal shops need some branding
As a digital agency, we actually proposed a session to help Drupal shops to better brand themselves, both within the Drupal community as well as towards clients. Better branding results in better chances of having your session proposal accepted.
We're marketing guys and we had the intention to share our marketing knowledge. Unfortunately, our session was not selected either, so we might have to work on our own brand.
The Pickwick Project - Digital agency
COD Ships with Plus1
To clarify a technical detail of this discussion as it relates to COD: COD ships with Plus1 for session voting, whereas the Drupal Association has done voting with Fivestar for DrupalCons since ~ Chicago 2011 as a standard practice and an extension of COD.
1 session proposal or session per person
Hi all,
got forwarded to this discussion from a post by webchick on Morten's recent block about voting re Drupalcon Denver.
My thoughts are:
+1 to limiting either the number of session proposals a person can make, or just limiting the number of sessions they can be chosen for.
In either case, the effect is to increase the number of voices at Drupalcon's, but the former has the advantage of making voting slightly easier because there are fewer proposals to read through. The latter has the advantage of allowing feedback on what exactly our great speakers talk about to ensure we get the best speakers AND the best topics.
Of these two options, my preference would be for limiting the number of sessions a person can be chosen to present to just 1 but allowing multiple proposals from an individual.
My other thought concerns the overall level of sessions/feedback about the conference in general. This information could be used to guide track chairs in future conferences and supplement any voting process.
While I know we collect micro data on sessions (what did you think about session x), does anyone know if we have tried to collect macro data (ie. at the conference level - e.g. 'Overall, did you find the sessions too advanced or simple'. Were there enough sessions for your interests?' etc. etc.).
Personally, I suspect the sessions have not been challenging enough on average in the last few 'Cons but aside from personal experience (at London) I have no real evidence for this. It would be great to know if we can collect macro feedback, or whether we have already done so - and if so, what it has said.
As an update: DrupalCon Munich
The DrupalCon Munich team made the choice to abandon the idea of voting this time around and instead use the comments on sessions as a means of gauging session quality: http://munich2012.drupal.org/news/improved-selection-process-drupalcon-m...
However, my totally unscientific reading of various rants on Twitter, IRC, etc. leads me to believe that this pisses off at least as many people as voting ever did. What should we try for São Paulo?
Different read
I probably have a completely different totally unscientific pool than you did, but I didn't see anyone complaining about comments per se. I saw people disappointed that they or someone else didn't get selected to speak, not about the selection process itself per se.
I don't think that will ever be resolved. When you have 400 session submissions and 80 slots (or whatever the extremely disparate numbers are), a majority of people are going to be disappointed. No amount of process is going to change that math.
Multiple data points
Why not use multiple data points and create an algorithm to calculate quality. Votes count for a certain amount and comments for another. You could also rank social interaction on social site by way of hash tags or such. Gather as much data as possible to determine what will be well received. I'd be happy to collaborate further on this if you like it.
Apathetic electorate?
When I looked around the Munich session proposals (and did actually comment on a tiny handful) it struck me how few comments there were (insert unscientific method, confirmation bias, and other methodologic faults here - @todo: check data!)
The issue may not be that comments aren't useful for judging proposals, but rather that commenting requires more time/thought commitment than most browsing users will commit.
If the issue is lack of comments (or indeed other crowd-sourced ratings), then we could consider a couple of potential solutions to generate more comments.
We could also consider automated metrics as a factor: for example, number of visits, or time-on-page. This could be subject to gaming, but excluding outliers and calculating a median could mitigate that somewhat.
--
Marcus Deglos
Founder / Technical Architect @ Techito.
The luxury of too much good stuff
Actually I think voting and comments are both good ways to surface community opinion - and I'm afraid that I, for one, don't think there is a better way.
The problem we have is actually a rather wonderful one - we have too many interesting people, with too many good ideas for talks, across too many disparate and fascinating specialist topics in our area of work - and simply not enough slots to fit them all in.
Having been a track chair at a previous Drupalcon I remember the long evenings of agonising over the details of 100+ proposals in a track, of which 60% or more were suitable, but with only 12 slots available.
Even as the track chair I was frustrated and disappointed at having to leave some sessions out - so I can easily imagine the frustration of those who submitted them, or really really wanted to see them.
Sessions generally weren't being left out because of any fault of the presenter. The reasons why sessions were not favoured could simply be something like "Well there were some good sessions on [Subject x] at the last two cons, so this time let's give some time to [Subject Y] which doesn't get enough attention but is growing in popularity/interest".
So I think the limitations we have are always going to leave a majority of people frustrated at this time before a Drupalcon. People in the community are very passionate about and focused on their 'thing', and it's frustrating if other people's 'thing' seems to get prioritised at any point.
There's simply no way to make everybody happy all the time in this session selection process. One thing that does exacerbate the problem though is that some big Drupal companies say that in general staff can't go unless they are presenting - so for some people rejection isn't just about not getting to present, it's about not getting to go at all. I think they are the ones who complain loudest (that certainly turned out to be the case when I was a track chair). I hope companies can learn to see the value of staff going to Drupalcon to listen, participate and learn - rather than just promote!
Finally, I also think that 90% of the same people who might complain about session selection now will then leave Munich saying "Wow, that was the best Drupalcon, like, ever!"
Some sessions I wanted to see didn't get in - but the schedule looks packed full of interesting stuff, and I'm still really looking forward to it.
Thanks to the Munich track chairs for their hard work.
Steve
=======
Steve Parks
WunderRoot
http://www.wunderroot.com
Steve nails it
As a track chair for several past DrupalCons and one of three people who had final say over the DrupalCon Chicago session program, I agree wholeheartedly with Steve's assessment. There are always far more good sessions to choose from than slots available, and the choice often comes down to factors other than just session quality (e.g., diversity of content).
While I was also disappointed to see that some really good session proposals and some amazing speakers didn't make the final cut, I also know that's not an indictment of them or the quality of their session proposals.
At DrupalCon Chicago, we had both comments and five-star ratings to use as tools as part of the session selection process, but those were never used as deciding factors. I think that allowing people to rate session proposals has some value and should be part of the content team's toolkit, but at the end of the day it comes down to the team picking the sessions that make the most sense as part of an overall curated program.
That's what the Munich team has done, and while I might have made some different decisions in their shoes, I can also understand, respect, and appreciate the choices they've made and the effort that went into it. This is a very well thought-out program, and I'm personally looking forward to it!
That's what the Munich team
Do we know that for sure?
I'd like to hear someone from the Munich team weigh in (after they get some well deserved rest) about their process.
I'm pretty surprised by some of the people who are not presenting this year who had top 10 or top 5 presentations at previous cons and some people who are presenting (or featured) who got negative reviews at past cons.
knaddison blog | Morris Animal Foundation
Speaker reviews from past
Speaker reviews from past DrupalCons are the most valuable tool that session selection teams have available, and I certainly hope that the Munich team made full use of them. I can understand why some well-reviewed speakers from past DrupalCons might not have been selected for Munich, but certainly poorly-reviewed speakers should not have been chosen, unless the team is planning to provide those folks with individualized, hands-on mentorship from more experienced speakers.
I personally haven't had access to any speaker reviews since Chicago, but my understanding is that the goal was to start building a library of quality speakers who we could recruit for future conferences.
Another example of the feedback loop
People usually improve over time and everyone can have an off day. Please don't rule speakers out because they had bad reviews at a previous event and a mentor scheme isn't viable for some reason (not enough volunteers, no mentor near them, or whatever).
Not meaning to hijack, but reacting to one part here:
It's a shame that the voting was deemed unhelpful for selection, but retained nonetheless in the UX of the submitters and attendees. After seeing all the gaming and downrating of sessions, the entire process struck me as largely arbitrary process rating popularity over quality. It left such a sour taste in my mouth, I gave up the idea of submitting DrupalCon sessions. DrupalCon Chicago was the last time I submitted a session for DrupalCon.
I don't know what actually happened in the process, but what seemed to be the process was weighting votes over prior experience, prior presentation ratings, etc. If the votes don't really matter, they shouldn't be offered up.
I realize how hard it is to make effective selections. Past performance is not an indication of future results. I've been too many frankly slacker presentations by people who were capable of doing better and had done better in the past but this time thought they'd be cute or be able to skate by with precious little preparation, as well as too many sincere presentations by people who really did not have the skills to present information clearly and thus left everyone bored and/or baffled. (Being able to lecture to scores of people comes naturally to only a very few people.) There is no easy recipe for curation.
But the popularity voting over the years has been the suck, imho, especially when configured with downrating. It's one thing to have a +1/like/thumbsup kind of widget (and even then, what does that mean?), and something altogether to have a system that includes downrating, but no context for what these votes are supposed to mean. If voting were to never ever make an appearance again, I wouldn't miss it.
Due to personal and family matters I've had to attend to this past year, I've not paid close attention to the process as it has evolved, so I'll refrain from further comment.
PINGV | Strategy • Design • Drupal Development
HI Laura - As one of the
HI Laura -
As one of the DrupalCon Chicago co-chairs and one of the people in charge of overall content, I'd love to respond to this:
What I said was not that voting was deemed unhelpful, but that it was never a deciding factor (at least in the tracks that I was aware of) for whether or not an individual session got picked. We wanted to give people as many opportunities as possible to provide feedback on session proposals, but voting was just one of many factors considered.
I'm sorry to hear that the session proposal rating system at DrupalCon Chicago left a bad taste in your mouth, and I'm really sorry to hear that you haven't proposed any sessions for any subsequent DrupalCons as a result. There was certainly some gaming of the process (e.g., sessions proposed by people from larger firms tended to have higher ratings, no doubt because they got their colleagues to vote for them), but we were aware of this and weighed it accordingly in our deliberations.
That was not our intention at all, and in fact our track chairs and committees were expressly told to look at past speaking experience and presentations when deciding who should speak. I believe Chicago was the first DrupalCon to ask those making session proposals to list their prior speaking experience.
The +1 system had been used at several previous DrupalCons and had proven to be an ineffective tool for determining interest in a session. For example, my session proposal for DrupalCon Szeged got the most votes of any apart from Dries' keynote and was scheduled in the large auditorium. Fewer than half a dozen people actually came to the session and I had a largely empty room, while one of the other sessions happening at the same time in a smaller room (one on Panels, I believe) was overflowing.
My take was that while my proposal was for a session that many people thought should be presented at a DrupalCon, it wasn't actually a session a lot of the people at the conference were likely to attend.
When we created the five-star system for Chicago, we assigned the following labels to each of the stars:
One star - I have no interest in this session
Two stars - I would probably not attend this session
Three stars - I might attend this session
Four stars - I would probably attend this session
Five stars - I totally want to see this session
Note that the ratings are framed in terms of one's desire to see a particular session, not as a judgment of the quality of the speaker or the session proposal itself. Our goal was to get people people to tell us what sessions they wanted to go to as opposed what sessions they thought should be presented at DrupalCon. Was that completely clear to everyone? Possibly not. If we had to do it over again, we would definitely tweak some things, and we passed those recommendations on to be included in the handbook used by future DrupalCon teams.
I do however, believe that five-star ratings can be helpful, when used in combination with comments, past conference evaluations, speaking experience, and other tools.
Another positive impact of voting is that by encouraging people to vote on their sessions, we get more eyeballs on session proposals, and some of those people might leave more detailed feedback in comments. It's also a pretty low-cost way to help generate publicity and enthusiasm for the conference.
I want to see as many people as possible submitting session proposals for DrupalCon, and I'd love to talk with you more to see what future organizers and staff could do to help you and others who feel like you do to feel more comfortable proposing sessions for future conferences.
Part of the problem in the past has been that there's been very little consistency or knowledge transfer from conference to conference; with the introduction of staff and global content teams, we have more of an opportunity now to better manage expectations across conferences.
Conflicting scheduling, perhaps?
I think that one problem is that many of the same people who "would probably attend your session" or "totally want to see it" found themselves having to choose between that session and the one on panels that they really, really "totally wanted to see". Care needs to be taken when it comes to scheduling the most popular sessions so that they don't take place at the same time. Even if they are considered to be on different "tracks", most of us won't stick just to the "coder", "core conversations" or "site builder" tracks (whatever best describes our "role" in the community) and will attend a mix of the sessions that look most interesting, so "coders" may well attend "community" or "Design/UX" -track sessions if the description looks particularly promising or if they know that they definitely want to see webchick or eaton.
As I see it, that is one major downside to the "no votes" system used for the Munich DrupalCon. There is much less indication of which are the most popular sessions since so few in the community commented and since, even then, it’s probably harder to get a good indication of whether the comments reflected a "definitely would attend" or simply "this looks interesting for others, so I'm going to say something" (or, worse, "my buddy is proposing this session, so I need to promote it").
From the look of things, I think the track chairs did a great job selecting some interesting sessions for Munich, but it might be wise to add Fivestar voting now, after all, and get community feedback to help with the "how big a room will this session need" and "which sessions should we try not to schedule at the same time" kind of questions. IMHO, this voting should be limited only to those who have purchased a ticket. And our answers should be stored to automatically populate our personal schedules with the sessions we indicated we wanted to attend (even if some sessions need to conflict, time-wise, for many of us, this could help minimize that).
About me:
I agree wholeheartedly, but
I agree wholeheartedly, but the problem is that for various reasons, the schedule needs to be finalized weeks in advance of the time when most people will be thinking about what sessions they'll be attending or putting together their personal schedules.
The experience of submitting should be positive
Thanks, George, for the thoughtful reply. In hindsight, my latest comment above was rather self-pitying.
My main observation above is that the experience of submitting was like throwing oneself into the meat grinder. The voting was rather abusive in the Chicago instance. Having a legend is inadequate, imho -- I commented in full about that last year, elsewhere in this thread: http://groups.drupal.org/node/151174#comment-511489
What I'm trying to get at here is the experience of submitting (as opposed to the tools useful for selection).
My main interest here is that great potential presenters -- not just the insiders who always seem to have multiple sessions -- have a positive experience submitting their proposals, and don't feel like they were beaten down for the effort. I think very few people actually expect to be selected every time they submit. I certainly don't. Despite my own track record of successful presentations at DrupalCons of yore, I have no expectation of being chosen. I do, however, hope for and expect to be seriously considered. And I think I'm pretty typical in that regard. It's a big community, and lots of people have great things to contribute! That's one of the things that makes the Drupal community so awesome, and DrupalCons so fun, for me.
In the end, selected or not, we want the submitter to feel good for actually making the effort. Anyone would feel disappointment at not being selected, but it's all the worse when one feels completely ignored or, worse, is kicked virtually in the experience, especially when that experience provided little benefit to the actual selection committee. One will brave the initiation of a thousand knives once or twice, but it's harder to feel inspired to take on the effort next time.
PINGV | Strategy • Design • Drupal Development
Thanks Laura - These are
Thanks Laura - These are really good thoughts. I wanted to comment on one in particular:
The ability to customize session acceptance and rejection letters would be an awesome tool for DrupalCon content wranglers. For Chicago, I ended up having to write and send all of the acceptance e-mails by hand on New Year's Eve so that folks knew which of their session proposals had been accepted; after that, we had no choice but go with a generic message for rejection letters. This is still an issue: for Munich, I got both a generic acceptance and a generic rejection letter because I had submitted multiple proposals.
I think telling people why their session got rejected would be dangerous, however. In my experience, you're always going to be end up with a few people who are really upset that their session didn't get picked and will not accept any explanation offered. It seems to me that this would just open the door for arguments with the track chairs and/or content committees and more hurt feelings.
I think telling people why
Yes, I can see that. It would be difficult to get right, and could easily be misinterpreted.
Thanks again.
PINGV | Strategy • Design • Drupal Development
customizing accept/reject
It would seem this would be easy when using COD to send emails directly from the site?
My impression has been that the generic nature was due to sending via a separate service?
I can't speak for where COD
I can't speak for where COD is at now, but that functionality didn't exist at the time of DrupalCon Chicago.
Added in June 2011
Contacting people who proposed sessions was added to COD in June 2011 at http://drupal.org/node/1023600.
A Case For Restraint?
I'd like to see more people consider sitting out on some cons and camps as well.
I'm organizing a few more camps this year and I'm not sure if I'll present at any of them. I've been to 25 camps and somewhere in the 20's I didn't know I was going to a camp until after the schedule was printed thus I had my 1st camp where I wasn't speaking.
IT ROCKED! ((just picture me not talking... and just listening!)) that wasn't hard was it?!?! ;-)
Now I'm trying a new thing. Many times I'll ask the camp organizer, hey... if you really want me to, I'll submit a session on blablabla, or the topic-of-your-choosing and understand that I'm not broken hearted if you make room for some fresh blood or a more engaging topic. Honestly, I'm scared to death that my presence will tip the scales in my favor and some one else will miss their chance to shine.... So I sometimes keep my name out of the mix entirely.
I think more people should consider sitting it out every now and again. And what do I know? Maybe many more do than I realize...
How many times has some one given a con session that mimics a recorded session from a DrupalCamp from 90days prior to the con? OR giving a con session then turned right around and redelivered it at a camp where it was also recorded.
I'm not suggesting that we should reject ALL sessions if they DO have a recording of that session already published, but I think that is a great reason to disqualify them.
I was a co-chair for one of the tracks at Chicago DrupalCon and I can tell you, there were occasions when I cast my vote for the little guy if I felt that it would result in a great experience. While casting that vote I KNEW that I was not voting for one of the "usual suspects" and THAT in itself was a key factor.
Pick 10 names you always see giving sessions at every con for the past 3 years and ask yourself, would it still be a con without one or more of them? The answer is certainly yes. We shouldn't feel the least bit awkward that one year so-and-so doesn't actually have any sessions at all.
I liked the Chicago plan where there was a quota. I can't recall what that quota was, but I loved it. It may have been two sessions and one panel. I guess my idea is to take that one step further and actually skip over some of the oh-so-familiar faces of the community and to do so with the express purpose of making room for less familiar faces.
I applaud the efforts of any selection committee who has, as its core intention, to provide fresh people with fresh topics, and well delivered common topics. If Munich's approach delivers the results then we know it should be duplicated.
...
Just my random thoughts.. :-)
Every con since Chicago has
Every con since Chicago has enforced the quota system and in Denver we also had a target for each track to include more presenters who had never presented at Drupalcon (I think 20% was the goal). I was a track chair in Denver for the commerce track and have heard some feedback about the event but don't think I heard much of "wow, lots of great new ideas, this is so valuable!"
I can see some merit to this, but can think of two notable exceptions:
1. I will always want to hear, for example, Nate Haug talk about webform. He's a great presenter and there's always new stuff in Webform worth hearing about.
2. The 2007 to 2011 Drupalcon report shows that very few people go to more than 1 conference (part of this is explained by the fast growth, but still. So even if a person has presented on a topic 5 times and even if the "regulars" are bored of the session the majority of Drupalcon attendees have never seen that before.
Look at the people in this debate - we've all been to a large number of camps and cons and yet "insiders" are not the majority demographic in attendance at the event.
knaddison blog | Morris Animal Foundation
1. I will always want to
Great idea! Then why don't we reserve 80% of the spots to those same presenters that we already know to be the best and open up the rest for vote? This will make the "selection" of sessions so much easier (80% easier in fact)!
You came to a conclusion I
You came to a conclusion I didn't (and don't support) and then attack that conclusion. I guess you "win." ?
knaddison blog | Morris Animal Foundation
I did not attack anything
I did not attack anything (unless I was being ironic, which you can't really tell).
But I did support your conclusion based exactly on what you said without distorting it. You do support setting aside a certain percentage of slots for established speakers, right? This is how DrupalCon works already (see gdemet comment below).
I think Greg is objecting to
I think Greg is objecting to your facetious assertion that he would support setting aside 80% of speaking slots for pre-selected speakers for the sole purpose of making session selection easier.
My feeling is that there is value to setting aside some slots, but not to make session selection easier (if anything, it makes the process more complicated and involved). The number of slots set aside should also be a small minority of the overall session total, not four out of five as you suggested.
80% + 20% = 100%
The reason I mentioned 80% was because I read 20% was set aside for new speakers! :)
Well, I guess i'm confused. I
Well, I guess i'm confused. I thought you were being sarcastic and had an emotional/frustrated reaction to that, but maybe you weren't.
To state it less emotionally, I don't think we should set aside any specific percent as a mandate of people who did well at previous conferences. If there are known good presenters who have done great presentations either at Drupalcamps or outside of the Drupal community then they should be considered even if that pushes us over some limit. It's interesting that our keynotes, some of the most important/well attended/well liked presentations, are usually from outside the community.
knaddison blog | Morris Animal Foundation
If there are known good
I can totally agree with that! :)
I would suggest making this a prerequisite for proposing...
I know it's not a majority of session proposals, but for those who might try to read through the list, it's frustrating to even spend any time on looking at proposals that clearly never should have been added. If someone does not have any real history as a speaker (previous DrupalCamps, Cons, or "outside" events) which they can point to as proof they are prepared to present in front of hundreds of people, then no matter how interesting their topic is, they should build their presentation experience before throwing their session proposals into the DrupalCon list.
About me:
Feedback loop
I think it's a bit dangerous to close drupal events down so that only existing speakers can speak. Communities need new ideas and fresh looks, else they become stale, dull and cliquey. Of course, you don't want every slot taken by a nervous new speaker, but we all had to start somewhere.
Right... but start at a smaller DrupalCamp
Unless you are already a “known commodity” you just aren't going to be selected or fill a room at a big DrupalCon. It's easy to get a session at a smaller Drupal event where you can build your skills (and better still to start building speaking skills in Toastmasters, of course), while also building a "track record" that you can point to when you propose your "big idea" to present at a DrupalCon.
Even if you had a great idea and it wasn't selected as a session, you can still present it in a BoF or find others interested in working with you on it in the coder lounge, so unless getting your session approved is the only way you are getting to a DrupalCon, there are still avenues open for you to share your ideas. And of course you can blog or otherwise discuss your ideas online, as most have done many times before they are accepted as a speaker at a larger event.
Of course we all need to "start somewhere", but that "somewhere" usually isn't on a stage in front of hundreds of people. Business school graduates aren't normally recruited to be the next CEO of a Fortune 500 company. People with a good idea, but no history in politics don't usually get to join a major party as the frontrunner for a national office. But if one starts “locally” (that could be online, in this case) and build up one’s résumé over time, one can follow a course that could lead to the "big stage".
If I were a track chair and had almost ten times as many proposals as there was room for accepting them, I most certainly would not entertain the notion of selecting anyone I'd never heard of with no history of public speaking, regardless of how interesting their proposal was.
About me:
Point b of the introductory post
Point b of the introductory post was that DrupalCon should not be an insider's club, so making it a requirement that you are already an insider with a speaking record seems like a bad move, doesn't it?
OK, I agree, don't put new-to-all-speaking speakers in a 100s-capacity stage, but surely there should be smaller spaces for them within the event? If rejected speakers settle for a BoF or lounge group, would that help them get accepted in future?
Also, there should be spaces for new-to-drupalcon experienced speakers if they're on-topic and interesting. There doesn't seem to be space in the DrupalCon London pages for someone's general speaking experience, so I'm not sure how you would evaluate it fairly.
(It's on public record that I'm a local councillor and that I dislike groups like Toastmasters because I'd prefer people to learn communication skills as part of participating in their own productive communities, like Drupal, so I'm not sure if you're trying to troll me with comments about politics and that organisation.)
Not trolling ;-)
In the final selection, there are a number of sessions given by people who are not well known within the Drupal community, but within other communities of interest to Drupalistas (e.g. other CMS communities and the Symphony community, etc). So, yes, I agree that it shouldn't be an "insiders club", but the speakers should also attract registrations, so it's good for them to be a "known commodity" to at least some of the potential attendees, whether that's because they wrote an awesome module that loads of people use, or write a popular blog or books, or have spoken at other Drupal events... not "no track record, but have a good idea" (especially if a similar "good idea" for a session is also proposed by someone who we know will do a good job and attract attendees.
I suspect that being on record for having led a BOF is a is worthy of mention when applying for a future DrupalCon. It probably is more helpful to give a presentation at a DrupalCamp and be able to provide a link which includes video of the talk... or at a similar event (non-Drupal) if the presenter comes from outside the community, but wants to share something we might be interested in or if their previous speaking experience was not specific to Drupal.
I think that, at least with DC Munich, you could write up your speaking experience outside of DrupalCons/Camps, etc.
I'm a Toastmasters member and I think it's a great way to be aware of communication issues that you might not pay attention to if you weren't performing such activities. I only mention it as a way to build public speaking confidence (and eliminate some bad habits) in a group that's specific to the needs of public speakers (within a local community), while also practicing giving talks at Meetups (Drupal or whatever), camps, etc. My mention of politics was purely an analogy. Nobody is likely to vote an "unknown" to be the next president, prime minister... whatever. Likewise, it helps to build ones influence locally (or in smaller circles) and progressively gain enough "followers" to attract the international attention that is necessary to stand out when there are 10 applicants for every speaking slot (or whatever it is).
Finally: I generally agree with you. But I also think that the application process should be more restrictive in general, so that there are not so many "unlikely" applications. i.e., if you haven't got a track record (somewhere, if not in the Drupal community) for public speaking and/or IT-related / community-related / business-related prowess of some significance, save your proposal for a more local event than a DrupalCon. If you want to preview all proposals, it's currently an ENORMOUS task and it would be better if more of the people knew from the start that they would not "make the cut" and we only had 50-60% of the proposals (or less, perhaps), but they were all "worthy" proposals (or close enough that it would really be the track chairs' call, not a judgment that almost anyone could guess). The way it is now, I suspect VERY few people, indeed, look at all the proposals. It shouldn't have to be that way.
About me:
We do set aside a certain
We do set aside a certain percentage of slots (~25%) for pre-selected speakers who we know will help draw an audience for the conference.
I disagree strongly that
I disagree strongly that people who have given similar sessions at past DrupalCamps should be disqualified from consideration. Speaking from my own experience, speaking at camps and other events are great opportunities to develop and hone a presentation before a smaller audience before presenting it on the big stage.
For example, my presentation on Hollywood storytelling and the Drupal development process was first presented at Drupal Design Camp Boston in May 2011 before a room of ~40 people. While it was a successful presentation, I very quickly identified the fact that I was trying to cram in too much content and made adjustments accordingly, so that when I presented it in August at DrupalCon London before a couple hundred people it was a much more polished and professional presentation. I'll be presenting it again at OSCON this July, and will again be making some adjustments and tweaks to the content so that it's even better before that audience.
I do hear the concern that "the usual suspects" present at just about every DrupalCon, but there's a reason for that: people want to see them speak. No matter how many times they appear, folks like Crell, Morten and Eaton will always fill a room because they're great speakers who are established experts on the subjects they talk about. Think about it, would you rather hear someone you've never heard of speak about a subject, or someone who you know is an expert?
That said, there is tremendous value in bringing new blood into the process, and DrupalCon selection committees should always be on the lookout for folks who can bring something new to DrupalCon. I would argue that this should be less about "people you've never heard of speaking about familiar subjects" and more about "rising stars talking about new and emerging topics", but in any case the point is that we should strive for a good balance.
One of the ways you can achieve this balance is by limiting the number of sessions that any one person can give. This not only helps ensure that more people will get to speak, but it also reduces the burden on individual speakers and gives them more of an opportunity to focus on creating a quality presentation. Chicago introduced the "one session per speaker, unless you're also on a panel" limit and every DrupalCon since has largely followed it (though Munich has at least speaker who's presenting two solo sessions).
Doug, what session track did you help out with at Chicago? I don't have you on any of our old track committee lists.
clarifying
George,
I looked it up and I was a co-chair at the request of Allie and Jer for the Site Building / Implementation track.
Your disagreeing with something that I didn't exactly say. That's more my fault for not being clear, so I'll clear that up.
I certainly don't want every one disqualified for con sessions just because they gave the same session in the days leading up to the con. However, imagine if the chairs or selection team was looking to whittle down the list between Peter, Paul, and Mary. Peter just gave the same talk 30days ago at a camp and it was recorded. Paul has a 2month old d.o user, has never blogged, and it's not entirely clear what kind of speaker he would be. Mary is not a camp junky [like I am] but she blogs well and is very involved. In that scenario I see a tremendous opportunity for Paul to take a back seat [or actually be GIVEN the back seat] while Mary gets to shine.
As for your session George, you're a fantastic speaker and you could never be considered a usual suspect. Fine tuning your session at camps is a staple in the session workflow.
I want to be clear that I'm not advocating for the immediate rejection of any one's con session simply because it was recorded at a camp the months prior to the con. I do, however, maintain that the occurrence of that fact should weigh in somewhat in the decision making process.
I purposely avoided naming names, but you mentioned three. I enjoy the usual suspects as much as any one else does. I accept that there are going to be some fixtures in this mix. I'm stating that we need to be careful and not have the "fixture-list" get too big. I did 4 sessions in SanFran, and one panel in Chicago. I sat out Denver for the exact reason that I'm stating here... I do not want to take a spot that could go to some one who doesn't have the visibility I already enjoy.
Caveat... If you see me propose a session in Portland, don't think me sanctimonious or hypocritical LOL I just may be cooking something nice up!
;-)
Good to know. It sounds like
Good to know. It sounds like you may have helped out as part of the Site Building track committee, but Allie Micka was the sole chair for that track. It's a minor point, but the reason I want to make sure the distinction is clear that is that for Chicago, track chairs (as opposed to track committee members) had defined roles that included regular meetings with the other track chairs, reports to the content committee, etc. That certainly doesn't diminish your efforts though, and thank you for helping put together a great program!
In your hypothetical Peter, Paul, and Mary, scenario, I would look at it a little differently, assuming each of them has submitted a similar session proposal:
What kind of session is this, and what is the target audience? If it's aimed at an audience that's more likely to be people new to Drupal or business evaluators, I'm definitely going to favor the more experienced speaker. If it's a very technical session aimed at core developers, then speaking ability is probably less important (though established knowledge of the subject matter is).
Have any of them spoken at past DrupalCons? If so, how well were their sessions attended and how were the session reviews? If someone has previously spoken at DrupalCon, but did a poor job of it or was unprepared, I'm unlikely to want to pick them again.
If this is a topic that's new to DrupalCon and/or this person is an established expert in their field outside the community, then I'm more likely to pick someone with less DrupalCon experience.
If none of them have spoken at DrupalCon before, I would review all available information (videos of past presentations, blog posts, etc.) to try to get a better understanding of what their session might be like. I might even talk to other folks in the community who know them to get their thoughts.
If these three speakers want to talk about the same thing and it's a topic that a lot of people appear to be interested in, I might also reach out and see if they're interested in joining forces on a panel, particularly if they have different viewpoints/backgrounds. Being part of a panel is a great way for a less experienced speaker to get exposure, but everyone needs to be on board and coordinate in advance.
If I do, for whatever reason, end up picking a speaker who has no established track record, then I might ask a community member with more speaking experience to help provide individualized guidance and mentorship. At the very least here should be regular check-ins from the track chair to make sure that speakers have everything they need and that their session presentations are on track. The #1 cause of bad presentations is lack of advance preparation.
Does one of the speakers bring something special to the table beyond just their speaking experience and knowledge of the subject? If so, I should also consider that.
The point is that session selection is a complex process that can't easily be broken down into easily solvable algorithms; it requires careful consideration, research, and strategic thinking.
With regard to my own speaking abilities, I actually don't think I'm always a great speaker, and sometimes I do get bad reviews from my audience. What I do know is that my best received sessions are usually the ones I've had the most time to prepare for, and that's a big part of what track committees bring to the table. Session selection should just be the beginning of a process that goes all the way to the day of the presentation itself.
Lost in the weeds
Yah.. We're really getting lost in the weeds here. Allie asked me to be [what she referred to as] co-chair and I agreed. That's all I know. I don't need titles or anything. I was just happy to help. :-)
As for your expanded scenario... If I'm ever a helper or a co-chair you can count on me to take into account many factors. The scenario you proposed seems fair and balanced to me.
Let me be perfectly clear and agree with you when you said:
"The point is that session selection is a complex process that can't easily be broken down into easily solvable algorithms; it requires careful consideration, research, and strategic thinking."
:-)
well... I do wonder what..
First of all, I'am one of the sessions that didn't got picked for Munich so my opinion might be a bit over-biased here. But i do wonder what the selection criteria's where for the Frontend category. If I look at the sessions that actually made it to the finals I can't help thinking that the final verdict on those sessions was made with some target audience in mind, and to be brutally European style honest with my thoughts here.
I doubt that that target audience was actually Joe the themer, and not Bob the designer. As I spot a overload in sessions about the designing process and only a few sessions about actually doing frontend coding. This disrupts the balance between theory and practice in my eyes is way. My session was the only session about the new theming layer that most likely is going to end up in D8. So giving the frontend coders a head start in introducing this new layer sound right in place for a drupalcon to me, yet it got killed. So yeah, i openly wonder about the selection process here, and wonder even more if the right people where in the committee.
Yes i'am aware that a drupalcon should be interesting for everybody, but I do think that in the frontend department a lot can and must be gained here. Frontend is way more then unicorns, rainbows and pretty stuff in photoshop.
To do or don't, there is no try!
First of all deep breath - if
First of all deep breath - if you wasn't pissed of and full or rightious anger, you shouldnt have posted the session in the first place!
Nothing is worse than getting a session rejected that you know deep down is what the Drupalcommunity needs to move forward & that fills that whole in the program, that nobody else wanna cover
That happend for me in London last year, and im still pissed off about it (...)
There can be many different reasons why a session is chosen & rejected. To make it clear i have had nothing to do with the session process in Munich, but i do have been vocal about what i think would be good, and i have a hard time seeing how a twig session at this point wasnt chosen.
Anyways there can be a million reason. There was 333 session and only 80 or something was chosen & all groups needs their spots, so something needs to get cut. and that sucks ass.
BUT instead of doing the usual answer "ooh just do a bof" - which usually ends up in a session in a corner that noone knows about & makes it impossible to get you self hyped up about...
well Do a goddamn BOF, but do it right - Plan it now, and build up for it.
Theres the core conversations, where i hope you can push in more about the twig, so theres more that can enlighten people like me who just wanna have someting to work with
Theres room for a lot of things when we have 2000 people in one location.
Theres already movement now for putting together a kickass Typography bof, but atm we dont have an idea about rooms or anything else. But belive me we will dominate that fucker ;)
So in Short put on the armor, up on the warhorse - get you ass to munich, pack you session & be ready to rock!
/morten.dk king of rock
morten.dk | geek Royale
core conversation!
Ongoing / future work on core is better suited at core conversations as for how Drupalcons are structured. There is no mainstream sessions about the new Dependency Injection layer or routing system either :) Core conversations submissions just opened up: http://munich2012.drupal.org/program/core-conversations - these are highlighted on the same level as regular sessions but are naturally focused on the crazy new stuff and things that core and contrib developers need to learn / should participate in.
(I was and am not part of the Drupalcon Munich session selection process).
I understand that there are a
I understand that there are a limited number of session slots available, and that it may not have been feasible for Munich, but I personally would have liked to see "Frontend" split into two tracks, like we did in Chicago: "Theming" and "Design/UX". While we definitely want (and need) to foster Drupal's design community, a very large number of attendees at past DrupalCons identify themselves as "themers", and we need to make sure that audience is getting access to quality content, especially when there are so many new emerging fronted technologies (HTML5, CSS3, Sass, etc.)
I definitely agree - but sometimes hard to distinguish
I think that the past few DrupalCons have had separate Design/UX and Theming tracks and I wish they had followed that as well - though there are extremely valuable tracks in place. I think that's the challenge: expanding the offerings and tracks to provide needed content for new market segments (like having dedicated 'Business / Strategy') and serving the 'core audience' with more technical content.
I think there's enough to learn about Design/UX and all the new theming technologies & techniques that it bears consideration.
Jason Pamental
[ @jpamental ]
Fact is
that drupalcon is not the starry eyed community enabled event that celebrates the drupal community. It cannot be. There are too many interests and too many people for that. Core conversations and sprints notwithstanding, the con is mostly a trade show.
I for myself after London, decided that the cost/benefit of a con is not very interesting. You're much better investing in more specific events, like the devdays and some camps. Those are usually real community events without the the towering influence of sponsors and where there's space for new things and new people.
There's little incentive of the organizers to do it differently. In scientific conferences, for example, organizers know that without a good number of accepted sessions, the conference won't be viable, because no one will go there. There's an incentive for the organizers to try to please as much as possible. The question is that Drupalcon is not a scientific conference or a free software conference: it's a trade show.
Accepting that will avoid the general malaise that ensues each drupalcon session selection. Specially on this side of the Atlantic (Europe).
I think that São Paulo will be different because there's not much interests from the usual suspects. You have to know the culture and speak the languages of latin america to thrive there commercially.
I have plans to go there. I hope that the local organizers are aware of the pitfalls and avoid killing the newborn by aligning with the trade show modus operandi.
I call BS Fact is that the
I call BS
Fact is that the Drupalcon is a big Festival for geeks celebrating the Drupal succes.
No matter if they are in a suit or in a tshirt & funny hats - as with music festivals sometime the program isnt as good as you hoped for. But guess what here you can create you own program & session if you plan it out ahead.
its pretty obvious that you think the Drupalcon is a corporate monster thats the powers that be only want a Tradeshow & dont want the "real" community...
Last time i checked the Drupal Community was anyone from the big scary corporate monsters to the freelancers & casual users, that are using a Drupalcon to peek into this world, where else can you meet & talk with all parts of the Drupal Community (from suits to geeks)
Going back to a Drupalcon where its in a venue thats borrowed because somebody had the right contacts, isnt an option. When we passed 1000+ attendees, its not even possible to put together without a paid staff. With 333 sessions its not even possible to have em all if the conference is open 24 hours a day. Let alone make sure the quality is getting high enough.
Its fair enough that some dont wanna do Drupalcon's because they are big, and it can be hard to meet the people that you used to meet. thats just what happens when youre 2-3000 attendees. It can never be the same as a camp. and thanks god for that.
Claiming the drupalcon as a tradeshow...That is pretty much tells me that you missing out on a lot of what 2000+ Drupal enthusiast can bring when they are all in the same location, and that is sad.
The Drupalcon is what youre gonna put into it -You can try to take some control & push push it towards a platform for the community... or jump up on a high horse and complain over the "tradeshow", and its not for the community.
The real pitfall is when things change & people begin to whine because it wasn't as it used to be, instead of looking for the oppertunities that we get with having that many people all loving drupal in the same spot.
/mdk
Disclaimer: I was only very short in London (last day + codesprint) I did hear alot of ppl being not that happy about the program, that the level was way to low & to much marketing bs.
/morten.dk king of rock
morten.dk | geek Royale
I don't even think Drupalcon
I don't even think Drupalcon is about the sessions. I have been to been to four Drupalcons and actually been to very few sessions.
I feel the real reason to attend is for the community and networking.
Come for the community, stay for the code.
evolution
Absolutely agree - as a community grows the selected sessions need to have more wide appeal, costs increase so more sponsorship helps, and new events crop up to cater for differing needs.
DrupalCon will never be the same as it was the last time, that's what makes it special!
backwards slogan?
The slogan is Come for the software, stay for the community - did you reverse them on purpose or just a mistake? ;)
knaddison blog | Morris Animal Foundation
+1 for the old switch-a-roo
+1 for "Come for the community, stay for the code."
+1 for having been to 5 Drupalcons and seeing less and less sessions each year.
Fact is ad hominen
Don't jump to conclusions judging me.
You're free to have your own opinion. Don't pigeonhole mine as simple "moralizing high horse". It's not. It's just the statement of what I see as a fact.
If you have your starry-eyed-kumbayala-lets-hold-hands-together-and-make-a-fence view of the con, then enjoy it :)
As a wise man once said:
First of all, I cannot find
First of all, I cannot find anything in Morten's post that indicates that he is judging you or committing ad hominem. Morten is a bit, umm verbal, as we all know :), but he's not a mean person.
As a long-term member of this community, as well as a business owner, I am well aware of the fight to commercialize it vs keeping its spirit. The charm of DrupalCon is exactly what Morten described so well. That description doesn't fit the term "trade show". Conclusion: DrupalCon is not a trade show, and will hopefully never be.
Jakob Persson – Leancept – Results-only digital and marketing consultants – Personal blog
It's a trade show
we trade ideas, knowledge, code - sometimes for monetary benefit, sometimes not.
One thing I noticed at Denver was the talk of "community" over one side and "business" over the other - I see the two as inter-twined, more-so in Drupal than any other Free Software community I'm a member of as simply more people make money out of Drupal because they use it to build websites for people.
Even those who do it for free need services such as hosting, etc. so in the end yes, it's a trade show and indeed a "Trade" show, and we should be thankful for the wide appeal Drupal has, makes it much more interesting than just a gaggle of geeks or a showroom full of sales people. I can't wait... and I get to visit another wonderful place in the world!
oooh well sorry i dont see
oooh well sorry i dont see your statements as pure facts (in bold .. seriously?)
Its little hard to not seeing as being up on a high horse moralizing.
sorry man but back it up with facts, that the Drupalcon is actually a Tradeshow & I wont call you out on it
I will in fact be the first in line ready to attack anything that moves.
btw no idea what ad hominen means & really dont care, guess its an clever insult in some way ;)
/morten.dk king of rock
morten.dk | geek Royale
Definition of Ad Hominem
Ad Hominem arguments are where you say something is wrong because of the character of person making the argument.
It is a particularly unpleasant logical fallacy that is far too common in politics.
I completely missed this
I completely missed this important change to the session selection procedure. Mainly due to being incredible busy with other things.
If i had known about it I would have done much more work lobbying for comments about the proposed Media session (that didn't get selected), that's for sure... In a way, same as the old way of pitching for votes...
Sure, it pleases me that the organizers are trying to listen to those who plan to go to DrupalCon about what sessions they want there. But now when the Drupal project/community is so big it is impossible to really understand what sessions etc that will give me the most if they are selected.
Also, for me personally, most important is the (unoffical) Media Initiative. It is also where I have most knowledge about what is needed for it to really change things for Drupal.
I don't have the same insight in the other initiatives. I do know a fair bit and try to keep myself informed, especially when it comes to how they will affect the Media Initiative.
So, at least for me trying to find a method for the community to vote/talk-up-in-comments or another way is not something I believe will ever work. It will in one form or another just turn into a popularity contest where those with the most "friends" will get voted in.
No, there needs to be a selection team, that includes track chairs, Dries and others that are involved in defining our future. Those are the best to be able to select those sessions that best fits the general plan and gives me as the visitor the best possible information about how to plan my Drupal life.
The official sessions aren't the only thing happening at DrupalCon. Maybe we are making them more important than they really are....
I don't want to spend tons of time learning about all the proposed sessions. I just want to focus on my things in the community, then go to DrupalCon and get blown away about all the other awesome stuff happening...
--
/thomas
T: @tsvenson | S: tsvenson.com