Project Health Measurement

How to measure your project’s health.

While popularity measurements, like stars / forks, might be interesting, they don’t provide much real insight into project health. We encourage projects to focus on metrics that help us understand which specific aspects of a project are healthy so that we can make improvements in any areas that are not.

What makes this challenging is that every project is a little different, and interpretation of the metrics will vary based on the project. For example, the thresholds of what is “healthy” for a very large project, like Kubernetes, might not be the same as what you would use for a smaller project. The measurements in this guide are not one size fits all measurements, but instead are designed to help you think about how to apply some metrics as a way to improve the health of your project. Also keep in mind that data and metrics dashboards (including DevStats) aren’t going to be perfect; you will find inconsistencies and inaccuracies, so you’ll always want to apply a bit of common sense and a reality filter based on your knowledge of the project.

CNCF does not specify a one size fits all measurement for project health. Instead, CNCF requires that a project specify explicitly how they measure project health, and they provide DevStats dashboards with some metrics that can be used to help measure project health.

Measurements of Project Health

The DevStats dashboards can be overwhelming, especially for people who aren’t familiar with them, so we recommend picking a small number of metrics as your focus and providing those in an easy to consume way that any member of your project can understand.

Here are a few examples of what you might want to measure to determine project health. These do not include everything that you might want to measure or every option for each metric, so think of this as a starting point. In many cases, we’ve included links to DevStats dashboards for various projects to serve as an example of the type of metric you might use, but you will want to go to the DevStats dashboard (or other data sources) for your specific project to see how well your project is performing for any given metric.

Responsiveness

Projects that are responsive are more likely to retain both new and existing contributors. 

Contributor Activity

Stable or increasing numbers of contributors indicates that people want to continue participating in your project. A decline in contributors can sometimes indicate that there are deeper issues that should be investigated (e.g., inclusivity, toxic contributors).

  • Overall contributors: Are the number of contributors increasing or decreasing for your project?
  • Pull request creators: is the number of people writing code and documentation going up, down, or staying about the same?  A slow but steady increase here is the best result, but staying about the same for mature projects can be OK too.
    • PR Authors In Repository Groups
    • Note: if you see a sudden decline in PR Authors, look for organizations leaving the project, or project changes that make it hard to contribute.
  • New and Casual Contributors: Are you regularly getting new contributors into your project? Are these new contributors sticking around to become regular contributors?  Are you regularly getting “drive-through” contributors, usually users who are fixing their own bug?
    • New and Episodic Contributors in DevStats.
    • Note: having excellent onboarding to help new contributors get started quickly and easily while feeling welcome and included can make a big difference in your ability to attract and retain new contributors.
    • Note: having easy and clear PR submission and review procedures makes it possible for “casual” contributors to just fix one bug.  This lets end-users participate in your project, and is an eventual source of more regular contributors.

Contributor Risk

Your project will look less risky and more appealing to potential contributors and users if you have contributions from a variety of people and organizations.

  • Individual Risk: Are the contributions spread across enough people that the project wouldn’t be completely disrupted if a key person left (sometimes called bus factor, pony factor)?
    • For this one, it can help to look at contributor data for PRs / Commits by repository group to see where you may need to recruit new contributors to reduce your risk.
  • Organizational Risk: Are people from a variety of organizations participating so that the project could continue with minimal disruption if one company divested from the project (also called the Elephant factor)?
    • “ Companies table” in DevStats can shed some light on this metric.
    • Note: If your project is dominated by contributions from a single organization, it might help to understand whether there are barriers to contribution (e.g., planning / roadmapping conducted in private, governance with no clear path for others to move into leadership roles, new contributors not feeling welcomed / included).

Project Velocity

Active projects will have a steady or increasing flow of contributions. A decrease in project velocity could indicate that a project is mature and requires less work, or it could indicate that a project is no longer interesting or useful and might eventually need to be sunsetted / archived.

  • Commits, PRs, and Issues: Are commits, PRs, and issues remaining steady or increasing over time?

Release Activity

Regular releases provide peace of mind for users that they will be able to get new features, bug fixes, and security updates in a timely manner.

  • Major Release Cadence: Does your project make regular releases to make it easy for users to get new features and bug fixes?
  • Security Releases: Does your project have a process for quickly making point releases and patches available to fix security issues? How quickly do you respond to security vulnerabilities of various severity thresholds?
    • This information is not in DevStats, but can be tracked separately by your security response team.

Inclusivity

Being inclusive and welcoming helps projects attract and retain a more diverse set of contributors.

  • Code of Conduct Enforcement: Do people feel safe when reporting code of conduct violations? Do you address code of conduct violations in a timely and satisfactory manner? 
    • This information is not in DevStats, but it can be tracked by your code of conduct committee.
  • Difficult Contributors: Do you address issues with difficult or toxic contributors quickly to minimize the damage to your project?
    • While this can’t be measured directly, decreases in the number of contributors, especially new contributors, can indicate that you might have difficult or toxic contributors. See Contributor Activity section above. 
  • Diversity of Leadership: How many people from underrepresented groups in tech do you have in positions of leadership, especially at upper levels (e.g., Technical Oversight Committee, Steering, top level owners / maintainers)?
    • This usually needs to be tracked manually, but it is worth the effort to understand and address any lack of diversity in your leadership ranks.

Best Practices for Measuring Project Health

Timeframe 

You should pick a window of time to use as your measurements. If your project has regularly scheduled releases, running metrics for the period of each release is a great option. For projects with sporadic releases, it might be better to run them every quarter.

Whole project or Repository Group

In some cases, especially for larger projects, you might want to look at some of these metrics by repository group or sub-project to better understand whether the project is healthy across all areas. If some areas of your project are experiencing issues, those might be masked by other areas doing very well. This can help you identify specific areas that need improvement.

Interpretation

Every project is a little different, so the way that these metrics should be interpreted will vary by project based on the size of the project, its maturity, and other factors. Each project will need to determine what metrics work best for them, and verify the integrity of the metrics and data.

Last modified November 6, 2020: Create website for maintainers and contributors (c5bba84)