Leaderboards: The Problem Is the Neighbour, Not the Ranking

In autumn 2012 the game researcher Scott Nicholson had the students of his online course, which was called, of all things, “Meaningful Gamification”, climb a mountain with characters they had invented themselves, a made-up Mount Gamification on which every discussion post earned a few metres of elevation and a leaderboard showed, week by week, whose character stood how high. After six weeks he put a choice to the class that he borrowed from a well-known science fiction film: the red pill stood for a new adventure the students would help design, the blue pill for the existing system, leaderboard included.

Everyone voted for the red pill except one person, and that person was at the top of the leaderboard, while several others from the upper group also voted to scrap it, even though, as Nicholson records in his account of the course, they admitted that the old system had probably suited them better. Someone from the middle of the table described their position in a sentence familiar to anyone who has ever stood seventh on a sales ranking: “I had an okay position on the leaderboard and very little chance of upwards mobility.”

The obvious lesson of this scene is that leaderboards are bad, and I held exactly that view for a long time, so stubbornly that I contradicted clients who asked for a leaderboard almost by reflex. The lesson has only one flaw, which is that it does not survive the evidence. In a German wholesale warehouse the productivity of around 65 order pickers working on piece rates rose by 4.0 percent after the firm told each of them where they stood relative to their colleagues, and it stayed up without a single extra cent of pay, as the economists Blanes i Vidal and Nossol report in Management Science.

At a North American furniture maker the exact opposite happened over three years, because when the Wharton economist Iwan Barankay stopped telling some of the salespeople their rank, they sold eleven percent more. Both studies are carefully done and both examine the same instrument, so the effect of a leaderboard evidently depends not on whether it exists but on how it is built. A leaderboard is a device for social comparison, and social comparison is as old as the species, since as early as 1954 the psychologist Leon Festinger described how people read their own abilities mainly off their neighbours whenever no objective yardstick is at hand.

Why you give your own friend the worse clues

In 1980 the psychologist Abraham Tesser published, together with Smith, a small and slightly wicked experiment in which pairs of friends and strangers played a word-guessing game modelled on the television show “Password”, where you help someone else guess a word by giving clues. Some groups were told that the game measured verbal intelligence and leadership ability, and as soon as the result seemed to say something about one’s own worth, the stranger received the more helpful clues than one’s own friend in ten of thirteen sessions, while in a supposedly meaningless game it was the other way round.

Tesser built a model out of this which he called self-evaluation maintenance and which can be stated without jargon in one sentence: another person’s performance threatens our self-image all the more, the closer that person is to us and the more the field in which they shine matters to us. A stranger’s success in a distant discipline can even make us proud, because we bask a little in the glow, whereas the success of the person at the next desk in exactly our job is a verdict on us that is handed down afresh every morning at eight.

An office leaderboard, if you take Tesser’s model seriously, is the most precise machine one could possibly build for producing that verdict, because it places side by side exactly the people who are closest to each other and measures them on exactly the field that matters most to them. Stephen Garcia, Avishalom Tor and Richard Gonzalez showed in eight studies that rivals compete all the harder the closer they stand to a salient threshold, at the top, at the bottom or at a cut-off such as tenth place, and that near such a threshold they even turn down gains from which both sides would have profited.

The flip side of the model explains why two other forms of comparison work remarkably well, namely the distant role model against whom you measure yourself without ever meeting them at the coffee machine, and your own earlier self. Anyone who has played Mario Kart in time trial mode knows the translucent ghost that replays your own best lap and against which you race lap after lap without ever feeling humiliated, because the only person who can lose is the same one who wins. The educational psychologist Andrew Martin studied this ghost in 1,016 students at Australian high schools, and goals oriented to one’s own personal best predicted persistence, class participation, educational aspirations and even enjoyment of school.

Who writes the rules

After the red pill Nicholson’s class did not get a better leaderboard but an assignment: in groups the students drafted new syllabi and new game systems, the whole class voted on the individual parts, each student set a personal goal grade and chose a matching set of tasks, and in the end everyone but one reached the goal they had set for themselves. The same course, the same professor and the same people produced an entirely different result as soon as one single thing had shifted, namely who wrote the rules.

In 1978 the philosopher Bernard Suits wrote a definition of play that ought to be pinned to the door of every gamification project: “playing a game is the voluntary attempt to overcome unnecessary obstacles”. All the weight rests on the word voluntary, because an obstacle you choose yourself is a game, and the same obstacle handed down from above is work with an audience. The motivation researchers Edward Deci and Richard Ryan call the need behind this autonomy, the experience of being the author of one’s own actions, and how quickly a reward imposed from outside eats away at that experience is something we described in a separate piece on reward systems and motivation. In a university course in which Michael Hanus and Jesse Fox introduced a leaderboard with mandatory badges, the very things they had hoped to raise went down in 2015, namely intrinsic motivation, satisfaction with the course and, in the end, even the grades in the final exam.

The metric and the corner

Barankay’s furniture salespeople actually had no reason to react to their rank at all, because their pay depended solely on their own commission and not on their position, and yet they sold more without rank feedback. The explanation lies in a detail of the study: only sales of the manufacturer’s own brand were ranked, while almost all of the salespeople also sold other brands in their stores. Whoever found their rank worse than expected apparently shifted their effort, on Barankay’s reading, to exactly the place the leaderboard was not looking, and the data fit this account, since the effect tended to be larger where switching was easier. The order pickers in the wholesale warehouse, by contrast, were measured on exactly the work they were paid for and learned their position in private instead of reading it off a wall, and so I read the difference between the two studies as one metric hitting the work itself and the other only catching a corner of it.

How expensive a corner can become was demonstrated by Meta this spring, when an internal leaderboard called Claudeonomics honoured the 250 most eager users of AI tools with titles such as “Token Legend” and the workforce consumed 73.7 trillion tokens in thirty days, partly with agents that spent hours working through tasks without any goal for the outcome. Once the press had reported on it, the leaderboard was scrapped within days, and we have told the story behind it at length under the term Behavioral Capital. Consumption was counted while usefulness was meant, and the anthropologist Marilyn Strathern found the wording for this confusion in 1997 under which Goodhart’s law is usually quoted today: “When a measure becomes a target, it ceases to be a good measure.”

Someone has to be at the bottom

Every leaderboard has a last place for purely arithmetical reasons, and people fear that place in a way that money can hardly explain. Ilyana Kuziemko, Ryan Buell, Taly Reich and Michael Norton observed in the lab that people in last place accept gambles they would reject in any other position, merely to get away from the bottom, and that those in second-to-last place would rather give money to the person directly above them than to the person directly below, who might otherwise overtake them. The researchers called this last-place aversion, and the term describes with surprising accuracy what happens in a team whose leaderboard crowns a new candidate for the bottom every Monday.

What it looks like outside the lab was shown by an accident in California, when the Sacramento Bee newspaper put a database online in 2008 in which anyone could look up the salary of every state employee, the University of California included. David Card, Alexandre Mas, Enrico Moretti and Emmanuel Saez pointed a randomly chosen part of the university’s staff to the site and then surveyed everyone, and those who earned less than the median of their own unit were less satisfied with their jobs and more often thought about leaving, while the better paid did not become one bit more satisfied through the same knowledge. Visible comparison thus costs something at the bottom and earns nothing at the top, and this asymmetry is the quiet reason why Nicholson’s class voted its leaderboard out: “The more opportunities for earning points I missed, the more I felt disengaged and resentful.”

One sales team, built twice

How the test grid works in practice is shown by a case from inside sales, assembled from patterns that are typical there. A sales manager with twelve people in an open-plan office wants to put the number of calls per day on a screen on the wall, and the grid triggers on all three points of this design at once. The neighbours sit two metres apart and see each other on the list, the rule comes from management’s office, and the metric counts calls, although the company really wants good conversations with the right customers, so that the shortest way to the top runs through many short calls to the wrong people.

The same manager can build the same wish for visibility differently, by having each person first see their own curve against their own best week, by letting the team decide once a quarter which metric it considers worth playing for, and by making the only public comparison one of the team against its own previous year or against a distant industry benchmark at whose end no colleague is standing. None of this abolishes comparison, it only moves it to where it generates energy instead of consuming energy for self-protection.

Eighteen years of contradiction

For eighteen years I contradicted clients who wanted a leaderboard, and in all that time I rarely convinced one, because a contradiction is a stance and a stance stands against another stance, while the client sees competition working in his own sales team with his own eyes. The test grid replaces that stance with a prediction that can be checked against any design before it goes up on the wall, and a prediction has the priceless advantage over a stance that the client can see for himself whether it holds.

In ski racing a course setter plants the gates in the slope in the morning, and his work decides whether the same piste produces an exciting race in the afternoon or a series of falls, even though the snow, the skiers and the timing stay the same. Whoever introduces a leaderboard is also setting a course, and the slope in this case is their own team. The three questions of the grid can be answered for any existing leaderboard in a quarter of an hour, and the most honest one begins with a look at the current list: who is standing right next to whom today, and who actually decided that this is what gets counted?