Research Repository

Aggregating partial rankings with applications to peer grading in massive online open courses

Caragiannis, Ioannis and Krimpas, George A and Voudouris, Alexandros A (2015) Aggregating partial rankings with applications to peer grading in massive online open courses. In: 14th International Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2015-05 - 2015-05, Istanbul.

1411.4619v1.pdf - Accepted Version

Download (404kB) | Preview


We investigate the potential of using ordinal peer grading for the evaluation of students in massive online open courses (MOOCs). According to such grading schemes, each student receives a few assignments (by other students) which she has to rank. Then, a global ranking (possibly translated into numerical scores) is produced by combining the individual ones. This is a novel application area for social choice concepts and methods where the important problem to be solved is as follows: how should the assignments be distributed so that the collected individual rankings can be easily merged into a global one that is as close as possible to the ranking that represents the relative performance of the students in the assignment? Our main theoretical result suggests that using very simple ways to distribute the assignments so that each student has to rank only k of them, a Borda-like aggregation method can recover a 1 - O(1/k) fraction of the true ranking when each student correctly ranks the assignments she receives. Experimental results strengthen our analysis further and also demonstrate that the same method is extremely robust even when students have imperfect capabilities as graders. Our results provide strong evidence that ordinal peer grading cam be a highly effective and scalable solution for evaluation in MOOCs.

Item Type: Conference or Workshop Item (Paper)
Additional Information: Notes: 16 pages, 3 figures, 2 tables
Uncontrolled Keywords: Social choice; MOOCs; peer grading; Borda count
Divisions: Faculty of Science and Health
Faculty of Science and Health > Computer Science and Electronic Engineering, School of
SWORD Depositor: Elements
Depositing User: Elements
Date Deposited: 20 May 2020 13:50
Last Modified: 23 Sep 2022 19:39

Actions (login required)

View Item View Item