Expertise/Reputation system: initial draft

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Current proposal for reputation system: http://groups.drupal.org/node/225824

Code for this sub-project is published as Reputation module.

For current state check out demo site.

Overview

The goal of this sub-project is to build an expertise/reputation system which will:
a. expose all the different types of contributions (including non-code ones, which can't be measured by simply counting data in database)
b. expose field of expertise of individuals, to make it easier to understand "who is who", who is active in which field, whose opinion is backed up by expertise in this specific field
c. encourage more contributions, especially non-code ones, by recognizing them, showing that community values them, giving feedback to the new contributors when their contribution is useful and motivating them to continue
-------------
Main wiki for reputation system sub-project: http://groups.drupal.org/node/144594
Previous discussion: http://groups.drupal.org/node/142559

Principles:

- Quality of participation is more important than quantity
therefore action itself (comment, patch, bug report) adds no reputation points,
other people evaluating action as useful add points
- If there is no action, there should be no evaluation
therefore no direct voting on profiles is available, voting is available only on actual actions - issues, comments etc.
- The goal of the system is to recognize expertise and encourage contribution, but not to
regulate how people interact with the site
therefore there are no additional privileges for people with higher amounts of reputation, system does not affect users' permissions on d.o
- Collaboration, not competition
therefore there should be no "leader boards" or other lists directly ranking people and displaying their score and "place".
- System has no goal of moderating d.o content or punishing people for "bad contributions"
therefore there are no negative votes, reputation is always positive and can never go below 0.

Implementation

There is a list of "reputation score types" or "fields of expertise":
- Development (patches, reviews, testing etc.)
- Design & UX
- Support
- Documentation
- Event organization/Marketing
- D.o - something for all webmasters, infrastructure and other people working on *.d.o
maintanenace and improvements.
- Security
- ?

This list should not be too broad to make it easier to capture score points.

Capturing:

In various places on d.o there is a way to vote for contribution as good/helpful in specific
"field of expertise".

Places:
- issue comments
- g.d.o discussions and comments
- forum posts
- documentation pages comments
- module maintainers pages?
- release nodes?

Tricky places:
- documentation pages
- issue nodes
- g.d.o wiki pages
this places can be edited by multiple people, how do we define for whom we vote?

A way to vote yet to be defined, as an example it can be:
- drop-down list, where you can choose "field of expertise" and vote button/link
- line of icons for each "field of expertise" clicking on which performs vote in that specific filed.

While in some places, such as documentation, we probably could limit available vote types, in the issue queue it can be anything.

Measuring

Each time someone votes for contribution, its author gets reputation point of specific score type.

To make the system more representative, we will also consider who votes. If person who votes has a lot of reputation score in specific field, author of contribution will get more points.

Q: Should there be some limit on votes received/cast per day to prevent gaming?

Display

Reputation will be displayed in following places:

  1. Right where voting occurred.
    So right at the place of voting will be shown information - how many votes piece of content has, who voted. (initial information will be short and on expand we can show full list of people who voted and which types of vote they casted)
  2. At user's profile page
    Most voted at: Documentation (1+ votes), Development (5+ votes), Support (100+ votes)
    Voted by: X, Y, Z and 7 others.
    Number of votes can be shown or not.
  3. Aggregated information
    To make it less competitive the title will be something like: "Most voted in Documentation", followed by the list of users, sorted by their score but score count itself won't be displayed.

    The place for this aggregated information yet to be defined. Suggestions are next:
    a. if the list of "rep score types" will match list of "top level" topic pages - "Most voted in" can be displayed at those pages, instead of "Most active" as we have in mockups for topic pages now.
    b. some special page with description of the field of expertise, ways to get involved, and list of "Most voted in: N".


Badges - as additional tool to encourage and recognize contributions

Certain badges can be made available on specific events or to recognize specific contribution.

1. Team badges - we create team pages for "officially" defined teams (ux, docs, d.o, security etc.), adding people as a member of such team automatically adds badge of this team to his profile. The purpose of this is to recognize participation and let people, especially new people, know "who is who", so that questions like "is greggles on security team or not" from recent issue did not appear.

2. Badges encouraging first contributions of people. So lets say badge when user was first time voted in some field of expertise etc. Stackexchange does this really well. It should not be too many badges, only for first time contributions. They will be displayed only on user profile pages and we can hide them with time or when person gets some amount of reputation, when showing badge for his first contribution will not be relevant.

3. Badges for specific levels of contribution / dates, for example 1000 documentation edits, or 1000 commits, or 1000 reputation points, or 10 years member of the site etc.

4. Badges for specific 1-time events, for example for module maintainers who took D7CX
pledges and actually had D7 versions of modules ready on D7 release day.

5. Badges for attendance of DrupalCons and Camps to encourage wider participation, participating in code sprints etc.

Comments

Great start!

Graeme Blackwood's picture

On Capturing – issue nodes, it makes the most sense to me to reward for those fixing the issue, not the person that flagged it. I agree documentation is a little harder. It could easily be abused just by making tons of edits – though could be offset by points only being applied after a period of time in which revisions aren't reverted, but that gets pretty complex.

On Badges #2, I don't think we would need to hide the first contribution badges – it's fun to collect them and show the history.

In certain cases it may make sense to upgrade them – Level 1 for first contribution / positive rating, Level 2 when you hit 50, Level 3 when you hit 500 and so on.

I suggest we start small on

greggles's picture

I suggest we start small on this and expand based on feedback.

The biggest thing, IMO, is being sure we don't create a feeling of competition or leaderboard.

I made one small edit to remove "n votes" and turn it into a banded system (1+ votes, 5+ votes) similar to the way that the number of handbook edits on project pages works.

I've decided to start indeed

tvn's picture

I've decided to start indeed from the basic system, which will allow to

  • define the list of "types of reputation"
  • vote for comments, choosing one specific type of reputation to cast a vote to
  • accumulate votes for specific comment and display vote count on comment
  • accumulate votes for specific user and display vote count on user profile page

We had short discussion today on IRC regarding the input method for votes.
Proposed variants are:
- icons as vote buttons for each type of reputation (http://dl.dropbox.com/u/33894147/voting.png)
- drop-down list with vote button (sort of http://twitter.github.com/bootstrap/components.html#buttonDropdowns)
- vote button with list of reputation types appearing either on mouse click or mouseover (method used on Google+, sort of: https://skitch.com/royscholten/8dpus/untitled)
If anyone has better ideas for input method - please share!

As the developer of the

Morbus Iff's picture

As the developer of the Achievements module, I'd like to be
involved in this process. It could satisfy the Badge requirement quite easily.

That is great! As reputation

tvn's picture

That is great! As reputation system is quite sensitive subject, the decision was to go slow, step by step. I am currently working on the basic system, described in my previous comment. So the plan is to first implement it, get some feedback and then expand it with badges etc. I will be updating this post with progress, and plan to setup demo site as soon as I have something to show.

This is a great idea. I know

cwithout's picture

This is a great idea. I know you mentioned in your outline recognition for things other than code contribution (bug reporting, patch testing), but I noticed some comments kind of steer away from that direction.

I think it's important to reward people for things like legitimate bug reports. Testing for bugs is a valuable service and encourages people who can't write code to contribute via testing and reporting. (Of course, you don't want to reward people for every bug report or people would abuse the system and make false reports to get points, but you've already outlined a good solution to solve issues like that.)

It'd also help maintainers see when a person has a good reputation for not only coding but good bug reporting. Being able to see when a person is a good bug tester could assist maintainers when they're trying to filter legitimate bug reports from mistakes made by newbies.

I'd also like to see recognition for testing patches. There's not really a motivation to test pending patches right now unless you've had the issue yourself -- and even then people still often don't test. I'm not sure how often other people encounter this, but it's kind of frustrating to find a issue in a contributed module, write a patch to fix it, and have it just sit without anyone ever testing it.

Capturing the participation of non-developers is also important to encouraging wide-spread use of Drupal.

Yes, we definitely want to

tvn's picture

Yes, we definitely want to recognize other than code contributions. But one of the problems is that there are just so many of them, so we need to find right balance between:
1. recognizing as many different types of contributions as possible
and
2. not having too many options to click when voting so people would actually use the system and vote.

I currently think that maximum number of types of reputation we can use in the system is 6. It is already big one, but imagine we have more then 6 buttons (or options in drop-down list) on each comment on the page. I have great doubts that people will bother to vote in such a case. :)

So what 6 types would you like to see?

Integration with BadgeFarm.com - help needed!

jeremyprz's picture

Hi - I'm CTO of badgefarm & ideascale. We'd like to create a drupal plugin for turn-key gamification in drupal hosted sites. BadgeFarm is already built into your ideascale community -- http://drupal-association.ideascale.com . BadgeFarm is designed to have portable profiles, so the reputation being built up in here could be shared the ideascale community. I'd love some help with this if a drupal expert is out there who can help out. BadgeFarm has a full, undocumented REST API that would be simple to integrate with.

Related discussion:
http://drupal-association.ideascale.com/a/dtd/Add-a-reputation-badges-sy...

You might instead consider a

Morbus Iff's picture

You might instead consider a module that integrates with my Achievements module - regardless of what remote badge farm you use (and there are many, of course, with Mozilla's OpenBadges gaining some mindshare a lot lately), you're still going to need logic on the Drupal side for badge creation, unlocking, and tracking, and that's what Achievements does nicely enough.

BadgeFarm is more than 'achievements'

jeremyprz's picture

It is a full gamification platform including the detection of when achievements should be unlocked, notifications, link sharing & social influence tracking, feedback widgets, etc. Mozilla OpenBadges is basically a badge issuing system not really a gamification platform.

BadgeFarm issues badges by receiving events from remote systems through web service calls. The events are collected for users, algorithms on our end detect when badges should be issued, then we issue the badges.

We think this is the way to go because SaaS platforms should focus on their business model, not a game layer. We are essentially the equivalent to the unity game engine + xbox achievements but for SaaS providers. Most new game developers don't create their own rendering engine, they leverage existing game engines & build content upon it. That's what we are doing.

I'm very interested to talk to the people in charge of any efforts in Drupal. Are you one of those people?

I think I am one of those

Morbus Iff's picture

I think I am one of those people, yes. But, to be pointlessly blunt (and necessarily confused): if BadgeFarm has been out since early 2011, how come the only people using it all seem to be... yourselves? No insult intended, but where are your users?

Only a few months into development right now...

jeremyprz's picture

A perfectly valid point.

IdeaScale, BadgeFarm's parent company, is a privately funded company that I helped grow in the last 2 years. The first year (2010) was product stabilization from its spin-off from the parent company. The 2nd year (2011) has been about growth & features.

The BadgeFarm concept was thought up early 2010, domain registered late 2010, and we only around September of 2011 got any real horsepower around it.

Everything in starting a new company is about lead generation. No customers, no product - this is especially true when you are bootstrapping. BadgeFarm has a baked in customer base - IdeaScale's customers & partner companies. We decided to focus on these customers first & branch out to new types of customers second.

The website itself even shows this - we are only now starting to get functionality in place to expose the features we have built for ideascale. I'd be happy to discuss product roadmap, features & plans.

Hit me on skype or twitter jeremyprz or @jeremyprz

JP

So, I've published what I've

tvn's picture

So, I've published what I've been working on as Reputation module. It is work in progress and has only basic features so far. I've also set up a demo site for it.

Next I am planning to:
- figure a way to allow translating reputation types names (probably using http://drupal.org/project/variable)
- show only first 3 votes on comment and expand the rest by clicking "more" link or similar
- implement different ways to show voting results on profile pages e.g. show vote count, show range (over 5, over 10), don't show vote count at all
- fix all the things I did wrong

looks good tvn - great job

larowlan's picture

looks good tvn - great job

Thank you!

tvn's picture

Thank you!

Prairie Initiative

Group organizers

Group categories

Prairie tags

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: