Designing modules for fast, formative feedback

Hi all,

Like to get some ideas for a set of modules I'm going to (re)implement.

The science

Learning science tells us that fast, frequent, formative feedback is a Good Thing. Formative means the goal is not evaluation of the student, but helping the student learn. So:

• Students get details about what they did right and wrong.
• Students are able to resubmit until they get it right.
• Each exercise is chance for a conversation between student and reviewer.

"Fast" means that submitted solutions are assessed within a day or so. "Frequent" means there are, say, 40 exercises per semester that are evaluated. This work load means that the evaluation system has to be fast and easy for the instructor/reviewer.

Current system

CoreDogs has a system like this. http://coredogs.com/article/feedback-system is a quick description of it. In sum:

• There are many exercises embedded in a virtual textbook.
• Student enter solutions into the textbook.
• Reviewer(s) evaluate the solutions, using an interface that removes every possible click, keystroke, and cognitive context switch. E.g., clickable rubrics for each exercise.
• Students get feedback, and can resubmit. A conversation builds up, as students resubmit, and reviewers reevaluate.

I wrote several modules to implement the system. They're not great code, but they work. To prove that, I've attached a number of partial screen shots from CoreDogs to this post.

There are other aspects to this, e.g., student progress is evaluated constantly, making possible interesting data analyses. The exercise system makes a natural eportfolio possible. CoreDogs implements that through a "make public" flag students can add to solutions. CoreDogs gives students URLs they can email to other humans (e.g., recruiters), who can see solutions students have made public.

The Grand Plan

Imagine a Drupal distro with a bunch o' modules that helped individuals or small teams of authors write online textbooks based on learning science principles. The distro would include the feedback system, support for deep learning, outcome-based learning (think Learning By Design), and other things.

As part of this, I plan to redo the feedback modules in D7. Changes might include:

• Improve the reviewer workflow even more.
• Improve the exercise creator workflow, to make the creating and editing of content and rubrics easier.
• Improve the feedback reporting system.
• Add a way for instructors to enter a timeline for their expectations of student progress. Students can compare themselves to the timeline. Instructors could see who is falling behind.
• Let students compare themselves to each other, as well as the timeline.
• Add a badge system for students. There is a nascent one already - you can see it in one of the attached screen shots.
• Integrate with a group system of some sort, so that reports are over the right comparison groups. Use OG? Maybe, maybe not.

Other changes will happen, too, like improving the lesson creation workflow. But the feedback system is central to everything.

Current goal

There are two:

What features should be in the new feedback system?

The above, yes. But the current system has no grading part to it. Other instructors would want that.

What else should be part of the system? What reports?

Architecture

Like to figure out a good architecture for this. For example, there might be separate modules for:

• Managing groups (classes, sections, whatever)
• Creating exercises
• Allowing students to submit solutions in various ways. Text, photos from a cell phone,...
• Evaluation
• Reporting
• ?

They would need interfaces to each other (user interfaces is a separate issue - I'm talking about code interfaces). For example, there might be a standard way of representing a solution:

• Ref to exercise entity
• Ref to student entity
• State - unsubmitted, waiting for review, reviewed
• Solution history
• ...

Thoughts?

Does anyone care?

I've done a lot of work by myself on this over the last few years, and will continue. But I don't know if the broader Drupal community cares about it.

I'm interested in everything from general reactions, to ideas for features ("I would like it to...."), to architecture thoughts. E.g., if there are already standards for representing exercises (e.g., SCORMy things), let me know.

Or anything else you'd like to say.

I'm not looking for commitments to help, just reactions.

Thoughts?

Kieran