This article was originally published on Lullabot.com
Whoever said "comparison is the death of joy" was onto something. Comparing ourselves to others can create all kinds of problems, whether we think we are worse, better, or equal. Most of us probably know to avoid comparisons, and yet we can't seem to help ourselves. We do it in our personal lives and in professional settings.
This article focuses on three attempts to compare people (and organizations) in the Drupal community: Certified to Rock, DrupalCores, and the Drupal Marketplace page. It explores how these methods were useful, where they might have been lacking, and what we can learn from related academic research into leaderboards.
Rockstars Are Cool Because They Have Lots of Fans
CertifiedToRock.com, which launched on April 19, 2010, rated Drupal.org users on a 0-11 scale. This site used a secret algorithm to assign value to individuals in the Drupal community. People could speculate what activities would affect a person's score, but for the most part, the algorithm was secret. Finger-picking skills on the guitar almost certainly did not factor in, but contributing a Drupal module probably did.
The name might suggest this was a joke, but it was not. Quite the contrary, this site claimed to be "serious about certification."
Indeed, many people in the Drupal community took these scores seriously. For instance, a decade ago, Lullabot's application process asked developers to provide their Certified to Rock score. Seth Brown, who is now the CEO of Lullabot, explained why in a 2011 blog post:
"Our application form asks for a developer's Certified to Rock score; it's a web application that attempts to track each Drupal.org user's activity in issue queues, their activity with local user groups, how long they've been participating on Drupal.org, and so on: the very things we look at when we're examining how engaged with Drupal an applicant is. Although it obviously can't tell us everything about an applicant's involvement with Drupal, a reasonable CTR score immediately tells us they are involved. In my experience, anything above a 2 or 3 is a great sign."
The site claimed to offer an "answer to the question of certification for the Drupal community," perhaps a bit like how Acquia certification appeals to people that desire to "validate and promote" their Drupal skills. Certified to Rock served a role for some people involved in hiring decisions as well as developers who liked to think of themselves as rock stars.
Nevertheless, the site did not last. By 2013 the data had become stale, and the owners were looking to sell.
People over Profit
If Certified to Rock served a purpose, it likely helped businesses more than it helped people. People working in Drupal agencies might have occasionally uttered phrases like, "well, I'm not sure about this applicant, but they have a formidable CTR score." Less often said were phrases like, "this gal I know is really mean to other people, but I still consider her a friend because she has a Certified to Rock score of ELEVEN!" In other words, someone's score should not factor into their human relationships.
Proponents of secret evaluative systems believe these tools make more information public (anyone can look up another person's score), that they motivate people to get more involved, and that they provide important unbiased feedback. However, academic researchers who have looked into such systems have found they more likely become "engines of anxiety," eventually hurting both individuals and organizations.
The influential meditation teacher, Luang Por Sumedho, often tells his students that the kindest thing to do for the people they care about is to not create them. Meaning, do not create versions of people in your mind with which you compare the actual, real-life people. By letting go of preconceived ideas about people, we can interact with them in a more kind and human way. The same advice holds true for free software communities. Treating people --- both yourself and others --- according to a ranking is not especially friendly. Perhaps Certified to Rock did not persist because the Drupal community agreed.
DrupalCores, Leaderboards, the Leftovers
The DrupalCores project offered a slightly different method to feed people's desire to compare themselves to others by providing a ranked list of contributors. Hosted on GitHub and available under an open-source license, DrupalCores.com offered a seemingly unbiased list --- "a very basic table of all contributors to Drupal 9 Core." The list could be filtered to a list of contributors, companies, or countries. DrupalCores was essentially a "leaderboard." DrupalCores.com, like CertifiedToRock.com, no longer exists on the web.
Extensive research into leaderboards has revealed their pros and cons. Kraut and Resnick, for instance, demonstrated how leaderboards discourage contribution when "leaderboards elevate the top ten or twenty-five participants in populations of tens of thousands" and the "vast majority of members ... perceive that they have no chance of making the list" (50). DrupalCores seemingly avoids this pitfall by listing everyone, except that it omits just about everyone who doesn't contribute code.
It may seem like lists such as DrupalCores are harmless, but many academic researchers have been compiling evidence disputing this view. For his Princeton University Press book, The Tyranny of Metrics, Jerry Muller examined various ranking systems and found "that while they are a potentially valuable tool, the virtues of accountability metrics have been oversold, and their costs are often underappreciated."
Communities tend to measure what is easiest to measure, which often leads to problems. For instance, some people begin to fixate on the so-called unbiased systems and question their experience-based personal judgments. Others spend the majority of their time trying to game the system rather than contributing something of value. While leaderboards can benefit some people, they should be used with caution.
The Storied History of the Drupal Marketplace Page
The final category of comparison for discussion is the type of leaderboards that rank organizations --- rather than individuals --- such as Drupal's Marketplace page (sometimes called the "Drupal Services" page and available at drupal.org/services), which lists/ranks organizations that provide Drupal services. This page has been well-used for over 15 years, and the Drupal community has generally accepted it as a useful tool.
The "Drupal Services" page was not always a leaderboard. In 2005 it listed less than a dozen "individuals and companies that have contributed to Drupal," presumably with people who contributed the most at the top of the list.
By the 2010s, the "Drupal Services" section of the "Marketplace" page on Drupal.org listed Drupal Services providers alphabetically. It stated so near the top of the page: "Please note that the directory is sorted alphabetically, and not in an order that would imply preference or relative importance of one listing over others." Yet another claim of non-bias.
By the later half of 2012, the "Drupal Services" page had added a separate category for "Featured providers," though still offered an alphabetical listing.
Drupal's issue credit system launched in July 2015, and by the fall of 2015, the Drupal Services page officially became a leaderboard. From 2006 to 2015, the Canadian Drupal consulting company 2bits.com had enjoyed the top spot on the Drupal Services page because it always came first alphabetically. In August 2015, Zyxware Technologies was on page 12, but by 2016 they were on page 1.
The credit system made the Drupal Services page seem more equitable because counting commit credits allowed for the sorting of organizations by contributions rather than alphabetically. As Dries Buytaert and I wrote in 2017:
"Credits are a powerful motivator for both individuals and organizations. Accumulating credits provides individuals with a way to showcase their expertise. Organizations can utilize credits to help recruit developers or to increase their visibility in the Drupal.org marketplace."
We used data from the credit system to produce a wide variety of lists and graphs, including lists such as "the top 30 contributors" (and Dries has updated the list annually). We felt that such lists provided a way to highlight the work of these prolific individuals as well as offer a lens through which to investigate the role of sponsorship in the Drupal community. However, the Drupal Association has intentionally avoided creating lists of the top individual contributors to the Drupal project, and there is wisdom in the choice to keep the focus on organizations.
The credit system marked a significant improvement, but it was not perfect. According to Campbell's Law, "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Lots of people and companies found ways to game the system, and the contribution system has needed a few "tune-ups."
And then there are the sad stories about people in the Drupal community negatively affected by changes to the algorithm that determines the order on the Drupal Services page. Nobody in the Drupal community wants contributors to the project to feel worse about their contributions to the Drupal project as a result of a leaderboard, but that was exactly the experience for some people. For instance, one change resulted in a company dropping from page 2 to page 6 in the list, and a long-time Drupal contributor who had worked hard to get his company to page 2 found that drop to be "incredibly demoralizing." Another well-known community member-owned a company that dropped on the leaderboard, which led him to feel that his "contributions are now devalued." Unfortunately, these are the guaranteed results of any comparison system.
In reviewing the (unpublished) findings of Drupal's Contribution Recognition Committee --- which interviewed the leaders from various companies and offered a survey about how the Drupal community recognizes contributions to the Drupal project --- we found similar stories. People who filled out the survey and supplied their Drupal.org username received contribution credit (seems kind of meta, doesn't it?), so there were a lot of responses. Some respondents specifically called out "leaderboards," saying they "are not the way to go." Others clearly appreciated the attempts in the Drupal community to give credit appropriately. In fact, many people in other free software are envious of Drupal's contribution recognition system, and we are currently trying to port it to GitLab so that others can benefit from it.
We know that such systems can be valuable. Individuals and organizations that evaluate free software and open-source projects might like to know which organizations are involved in a project and the extent of their contributions. Drupal's Marketplace page can provide detailed information about the companies most involved in the project. In fact, the lessons we've learned in the Drupal community have directly influenced a new "Contribution Attribution" candidate metric in the Community Health Analytics Open Source Software, an organization focused on creating analytics and metrics to help define community health. In other words, leaderboards can be used wisely to help understand and grow a project in a healthy way.
When we look beyond the world of open-source and free software, ample evidence suggests that something like Drupal's credit system must be used with caution. For instance, Wendy Nelson Espeland and Michael Sauder's research into the effects of the U.S. News and World Report rankings on law schools revealed some problems with leaderboards. Espeland and Sauder found law school administrators who consistently felt pressure to choose between what is good for the law school and what is good for rankings. They encountered administrators who lost their job after a small decline in rankings. Schools offered scholarships to certain populations, altered their definitions of what it means to be "employed," and engaged in other activities to "game" the system. While some of these activities may sound foreign to the Drupal community, there are a wide variety of people who have noticed attempts to game Drupal's credit system, and the Drupal Association has had to monitor these attempts to game the algorithm.
To Err Is Human, to Blame Someone Else Shows Management Potential
We should not conclude that something was inherently wrong about these attempts to compare people in the Drupal community. They all appear to have been rooted in a genuine desire to help. Let's assume positive motivation. We also must be mindful that comparison, though it may be a natural human tendency, must be undertaken with care.
Thich Nhat Hanh, the peace activist Martin Luther King Jr. nominated for the Nobel Peace Prize, believes, "Where there is perception, there is deception." All attempts to single out people for their hard work in the Drupal community necessarily divide the community and risk alienating others. This isn't a radical proposal for the Drupal community to abandon all tools that single out individuals who contribute to the project, such as MAINTAINERS.txt or the Aaron Winborn Award.
Rather, we should continue to proceed carefully in our stewardship of the Drupal Marketplace page but also that we exercise a healthy level of skepticism the next time someone creates a leaderboard or related system that claims to provide a non-biased measure of individuals in the Drupal community.
If you are interested in leaderboards, Matthew Tift will be speaking about them on September 29 at an Open Source Summit panel entitled "Contributor Leaderboards to Incentivize Good Community Citizenship."
Comments