How do you measure the impact of a community champions program?

This post was co-authored by Yanina Bellini Saibene and CSCCE Staff, and can also be found on the rOpenSci blog.

How do you measure the impact of a community champions program? This was the central question of a working session at CZI’s Accelerating Open Science in Latin America workshop, convened by rOpenSci’s Community Manager Yani Bellini Saibene and attended by CSCCE’s Founder and Director, Lou Woodley. 

Measuring the impact of any kind of community program presents a series of challenges : 

  • What is the impact that you’re hoping your program will have? 
  • Is the impact you hope the program will have something that can be measured?
  • What types of instruments can be used to measure impact? (e.g., surveys, focus groups, etc.)
  • How many times can you reasonably ask your participants to give feedback?
  • How do you (or can you?) reliably follow up with participants months or even years after a program has concluded? 

The session took the form of a facilitated discussion of these points that all of the 15 –  20 participants were invited to contribute to – with the goal of learning more about one another’s experiences and creating a list of useful resources. In this blog post, we summarize these conversations with a view to continuing them in the rOpenSci Slack, the CZI Slack, and CSCCE’s Slack workspace for STEM community managers. 

What is a champions program? 

Community champions are individuals who take on more responsibility for the success, sustainability, and/or running of a community (see the image below, and read more in CCSCE’s guidebook). Champions programs formalize these roles, and may offer champions incentives, opportunities to connect and access to training and other support. They also offer community managers a means of maintaining, growing, or evolving a community by partnering with champions to further the community’s mission:

A figure describing the various roles of community champions. Maintain: Champions support the day-to-day running of the community (e.g., by welcoming new community members or serving on the code of conduct committee). Grow: Champions support the growth and extend the impact of the community and its outputs (e.g., by referring new members or sharing resources outside the community) Evolve: Champions contribute to steering the direction of the community (e.g., by serving on a steering committee/task force or participating in focus groups)
Image taken from: Center for Scientific Collaboration and Community Engagement. (2021) The CSCCE Community Participation Model – Exploring the Champion mode. Woodley and Pratt doi: 10.5281/zenodo.5275270

Champions programs have a variety of names in STEM, including fellowships and ambassador programs. You can find out more about champions programs, including a collection of tip sheets for those interested in creating a program for their own community, on the CSCCE website. 

What is “impact,” and how can you measure it? 

Community managers running champions programs are invested in their success for a variety of reasons. But deciding how to evaluate a program, and define the data that will support that evaluation, isn’t particularly straightforward. 

During the session, participants considered a range of nuances around evaluation, starting with understanding the meaning of “impact.” Champions programs ideally impact both the champions themselves and the community writ large. Asking participants to complete surveys or exit interviews is a great way to collect initial feedback on the former, but measuring longer-term community impacts is less straightforward. 

A complicating factor for many community managers is assigning “value” to different types of activities, which is especially complicated when measuring outcomes that rely on “invisible labor.” For example, in open-source communities, impact is often measured by tracking code contributions, but the work that champions may do to organize a monthly community call or translate documentation is neither so obvious nor so quantifiable. 

Some champions programs include a training and mentoring aspect, which adds roles such as mentors and instructors in addition to champions. Measuring the impact of the program on these individuals is useful, too.

Combining qualitative and quantitative metrics

Some solutions to these challenges included defining clear goals for the program and then designing evaluation tools to match; benchmarking champions’ expectations at the beginning of the program during an initial intake meeting; and using regular check-in meetings to ask about how the program is going as a way to learn more about less visible work. By combining “annecdata” with quantitative metrics (e.g., number of commits, attendance at training sessions, participation in a community forum, etc.), community managers can gain a more complete picture of the overall success of their program. 

As you design your survey, only include questions that matter to your evaluation, and clearly communicate to your champions and other participants what you’ll be doing with any data you collect. It is also  important to think through the metrics you’re collecting so that you aren’t always valuing the same type of contribution. Program participants can help identify and collect non-traditional contributions, especially in open software projects. And champions programs can be very good places to start valuing these contributions by using and displaying the metrics associated with them.

Resources