This article was co-authored with Tim Lehnen, CTO of the Drupal Association, and cross-posted on Drupal.org.
Sustaining open-source projects is the challenge of this decade. Understanding how contributions are made—by volunteers, sponsored by an organization, or both—can create incentives for ongoing support. The Drupal project has measured contributions in Drupal.org issues since 2015 using an issue credit system that captures contributions—including code, documentation, speaking at events, security review, etc.—from both individuals and organizations. This insight is used to shape incentives for the business entities that participate in the Drupal ecosystem.
Issue trackers can explain what changes have occurred and why they are important. Version control systems such as Git track changes to a code repository, who changed what, and when. Drupal's issue credit system dramatically expands the kinds of questions that we can ask about an open source project to include questions such as, "How much do volunteers contribute to the project?" or "How many of the contributions to this project are sponsored?"
In this article, we will describe how Drupal's issue credit system works and why we would like to bring it to GitLab and other code collaboration platforms. We hope that other free/libre and open-source projects and organizations that want to understand their return on investment in open source can model their approach on this issue credit system and benefit from the insights we have learned in the Drupal community.
How It Works
Drupal's issue credit system provides an interface that allows a person to describe the circumstances that facilitated their contributions for each issue on Drupal.org. Each issue on Drupal.org can credit multiple people, and may pertain to a code contribution, Drupal initiative meetings, Drupal event planning, podcast, and more. For instance, for each meeting for the group of people working on Drupal's new front-end theme, Olivero, Matthew creates an issue that is tagged as "meetings," the meeting notes are posted in the issue, and each attendee for that meeting receives credit.
Use of the issue credit system is optional, and there are three potential aspects of each issue credit.
First, a Drupal.org user can optionally attribute their participation in a specific issue to a company that allotted time for them to work on the issue. This is typically an employer that pays that person's salary or wage. This organization must be directly tied to the contributor's user profile on Drupal.org as a current or past employer and the organization must have an organization profile on Drupal.org. For instance, in Drupal issues, Matthew can credit his employer, Lullabot, whereas Tim can credit his employer, the Drupal Association.
Second, in addition to attributing a contribution to an organization, the credit system also allows someone to attribute their work to a "customer" (client), which also must have an organization profile on Drupal.org. Thus, in addition to crediting Lullabot, Matthew could also give credit to Georgia Public Broadcasting (GPB) when he is doing work on their behalf, such as porting the NPR and PBS Media Manager modules to Drupal 8. Tim, who works for a nonprofit, does not usually do work on behalf of a "customer" and would not typically use this field.
Third, for each issue, someone can check a box that indicates they volunteered their own time to work on the issue. For instance, Matthew has a long history of working in public media, feels a deep connection to the mission of public media organizations, and often added features to the public media modules on his own time after work. He added features that were not necessarily useful for GPB, but that would allow other public media organizations to use the module. Therefore some of his work was on behalf of his employer, for a customer, and volunteer. These are the sort of contributions that are difficult to measure without Drupal's credit system.
Finally, because we don't want to overwhelm our contributors with "paperwork," the credit system remembers a user's attributions as the default for their next issue.
This shows how the UI works:
The project maintainer can then select which issue comments truly contributed to the resolution of the issue, and all of the individuals and organizations attributed in those comments will receive credit:
Why and how do we use this data?
We have been using the issue credit system in the Drupal project since 2015. We created the first analysis of the data in 2016 in a report titled Who Sponsors Drupal Development? that has been updated each year since.
Those reports contain answers to question such as:
- What is the Drupal community working on?
- Who is working on Drupal?
- How much of the work is sponsored?
- Who is sponsoring the work?
- How diverse is Drupal?
The data are also used to determine the order of the organizations listed on Drupal's Marketplace page. In contrast to a simple alphabetical ranking, or even paid placement for marketplace ranking, using the credit system in combination with other factors allows us to promote organizations that are good open source citizens first — and thus incentivize that behavior from other organizations. In addition to considering data from issue credits, those lists use an algorithm that also considers factors about an organization such as the number of case studies, whether they financially support the Drupal Association, and the number of projects supported.
Why We Want to Share
Every free/libre and open source project adopts slightly different values. Some projects consist entirely of contributions from people who are all doing work for one employer and Drupal's issue credit system might not seem to be particularly useful to them. However, most organizations know that an open-source project with the support of just one company is a risk. For example, many of the open-source projects used in the financial sector were originally created by people working at a single company. However, the Fintech Open Source Foundation (FINOS), which works with companies in financial services, created a rubric to help organizations evaluate the health of open source projects, and one of their definitions of a healthy project is one that "is composed of individuals from 3+ organizations, ideally of 2+ org types, and including 1 bank." A tool like Drupal's issue credit system can help outsiders get a much better sense about the diversity of the contributors to a project.
Interestingly, FINOS also views it as a positive sign that "No requirement exists for developers to create a separate 'work' /corporate Github ID." They encourage organizations to pay their employees to work on open source projects. They want people to contribute as part of their jobs, with contributing written into their job descriptions and NOT something they do in their volunteer time.
In addition to the FINOS rubric, there are other metrics that specifically refer to volunteer and organizational support. Here are a few examples:
- Internews created the BASICS program (Building Analytical and Support Infrastructure for Critical Security tools) that includes a rubric to help reporters and organizations determine which open source tools they can trust. Among the questions they ask include "Are the maintainers primarily volunteers, or are they employed by an organization or company specifically to work on the tool?"
- The Apache Maturity Model considers if "contributors act as themselves as opposed to representatives of a corporation or organization."
- The Community Health Analytics Open Source Software (CHAOSS) community, a Linux Foundation project focused on creating analytics and metrics to help define community health, suggests a wide variety of questions and considerations regarding "organizational diversity" such as, "What is the number of contributing organizations?" and a consideration of "ratio of contributors from a single company over all contributors." This issue credit system also could help outsiders understand the potential "elephant factor," which measures how dependent a project is on a small set of corporate contributors.
We've also seen end-user organizations begin to use Drupal's contribution data in their formal RFPs when seeking a vendor for work. Because this contribution recognition system acts as a proxy for good open source citizenship, both private and public sector organizations have found value in asking companies to provide their contribution history as part of their bids. This feeds a virtuous cycle of continuous contribution.
We would love for other organizations and projects to benefit from the kind of insight we've been able to gather in the Drupal community. But for Drupal's issue credit system to be useful to other organizations, it cannot only exist on Drupal.org, which is why we want to port our system to other development platforms, including GitLab, GitHub, and GNU Savannah. Drupal has already moved its code repository to GitLab and we have an open issue on GitLab to port Drupal's issue credit system.
Limitations
Drupal's issue credit system is clearly an improvement on previous systems that the Drupal community tried and later found insufficient, such as Certified to Rock (too opaque) or DrupalCores (only measures code contributions). However, the issue credit system is not without limitations.
When Dries Buytaert first proposed that the Drupal community adopt a method for giving credit to organizations that contribute code to open source he did so in order to "encourage more organizations to contribute to Drupal and hire core developers." Many of us who care about Drupal hoped this new system could be used to better recognize all aspects of contribution to the Drupal community. We saw people making a wide variety of useful contributions to the Drupal project and we wanted to give credit where it was due.
While there's a common belief that we can use metrics to increase the number of organizational and individual contributors, a great deal of research suggests that trying to use metrics to shape communities can lead to undesirable results. In fact, Goodhart's Law, named after British economist Charles Goodhart, states any measure used for control is unreliable. Indeed, we have found that Drupal's issue credit system, like any public metric, has encouraged some people to shift their focus from making useful contributions to "gaming the system." While some people have found ways to do the minimum amount of work in hopes of raising their profile or that of their employer, we encourage all contributors to break issues down into small parts so a large contribution doesn't count the same as a small one.
Drupal's credit system seems to avoid one of the pitfalls that Jerry Muller mentions in his book The Tyranny of Metrics: "measure output, not input." Rather than giving people credit simply for trying, Drupal's credit system captures committed changes to the Drupal codebase --- the output. However, this system has grown to include many other kinds of contributions, including conference presentations, initiative meeting attendance, accessibility reviews, and much more. Consequently, it could be argued that Drupal's issue credit system has grown to measure all sorts of inputs and the Drupal community has shifted focus toward one of the outputs of the issue credit system: the "leaderboard" on Drupal's Marketplace page.
In some instances, leaderboards positively motivate community members. But leaderboards can also demoralize people when a change to the algorithm lowers their company's rank on the Marketplace. For instance, a long-time Drupal contributor found a change to the algorithm that caused his company to drop down dozens of places on the Marketplace listing "incredibly demoralizing" and another well-known community member felt that his "contributions are now devalued."
For everyone who feels proud to see their company placed highly on the Marketplace page there are many others who are left out. This is consistent with research on leaderboards in other communities that found how such rankings discourage contribution when "leaderboards elevate the top ten or twenty-five participants in populations of tens of thousands" and the "vast majority of members ... perceive that they have no chance of making the list" (50).
This is an ongoing area of concern that requires careful attention for any community looking to implement such a system. For the Drupal community, we are careful to ensure that this contribution credit system is used to provide recognition for individuals, but not rank. We believe ranking individual contributors is an unhealthy model that doesn't account for the privilege of free time.
On the other hand, because presenting an ecosystem of Drupal service providers creates a defacto ranking system—it's in our best interest to choose one based on contribution behavior that promotes good open source citizenship, rather than something more capricious or arbitrary like a simple alphabetical or randomized list.
Start with a Goal, Not a Metric
The CHAOSS community recommends that free/libre and open-source communities adopt a Goal-Question-Metric format. In this format, a community first establishes goals. Then they need to establish questions that characterize the assessment or achievement of a specific goal. If the answer to the question does not matter, the community does not need information (metrics) and therefore does not need to measure.
Consequently, we view that Drupal's issue credit system is most well suited to organizations that have specific goals and that have questions pertaining to sponsorship, volunteerism, diversity, and related topics. In the Drupal community, many people who had been participating for a decade or more saw a clear shift in the community from being that of primarily hobbyists to that of professional developers. If nothing else, Drupal's issue credit system has shown us that organizations have played a significant, and growing, role in the development of Drupal. We can now state rather confidently that more and more contributions are sponsored, but volunteer contributions remain important to Drupal's success. We know that of the people using the issue credit system, 5% were "purely volunteer" compared to 69% that were "purely sponsored." We know that Drupal's contributors have become more diverse, but that we would still like to increase diversity.
Despite its limitations, we feel that Drupal's issue credit system offers significant value and we would like to see it adopted more widely. We are in the process of making individual and organizational credit an official metric of the CHAOSS project. We would like to see it adopted on GitLab, GitHub, GNU Savannah, and other platforms. Furthermore, if we standardize across these platforms, then the project health analytics researchers can study these attributes across all the projects and tooling ecosystems that support this data, and we can share the effort.
Would you like to know at a glance whether an open source project is primarily sustained by individuals or organizations?
Would you like to know who they are?
Would you like a window into good citizenship in your open source project?
Please join us! You can show your support by adding your own feedback on the issue to add the feature to GitLab.
Comments