Background

CEF is CSCCE’s longest-running training offering, launched in 2020. As of April 2024, we have trained more than 300 community managers at 150+ STEM organizations, associations, and institutions since 2020. At the end of each cohort, we ask our learners to share their feedback on the course via an exit survey, and we explore how they plan to implement what they’ve learned in more detail via our graduation activities. With this study, we wanted to follow up months-to-years later and find out if and how participants’ experience in the CSCCE online classroom has impacted their work life. 

CZI has hosted multiple private cohorts of CEF, and so was particularly interested in funding a medium-term evaluation of the course and to explore not only learning outcomes, but additional, less easy-to-capture impacts such as socioemotional outcomes and the value of connections formed. It is our hope that the findings presented in this report will help demonstrate to prospective learners and their organizations the value of participating in CEF, while also informing the continued iteration and improvement that is intrinsic to our work here. 

Goals of the project

Our research questions were focused on exploring the impact of the CEF on three levels – the individual, their community/organization, and the wider STEM ecosystem. 

Specifically, we wanted to know: 

1. In what ways have CEF participants applied concepts, frameworks, and activities from the course in their community management work?
2. What impact has CEF had on participants as individuals, particularly on their confidence levels and career-related outcomes?
3. What impact has CEF had on participants’ organizations and communities, particularly on member participation?
4. What impact has CEF had on their organizations’ and communities’ relationship to the broader STEM ecosystem? 
5. What other resources, activities, or training have participants found useful in their community management work?

Our assessment suggests that CEF is having a lasting socioemotional impact on the individuals who take part in the course, as well as long-term, positive impacts on the engagement within communities in STEM and even the connections between some of those communities.

How the project was carried out

We distributed a multi-part survey to 285 CEF alumni, and received 86 responses between November 2023 and January 2024. Responses were anonymous, with only limited CSCCE staff accessing raw data, and survey participants were informed that the anonymized data would be shared in a report, on the CSCCE website, and during a community call (April 2024 – read the recap of that call). Anyone who completed the survey had the option to be entered into a raffle to win a free place at a CSCCE Mini-workshop. 

Once the responses were received, we coded and analyzed the data to prepare a final report, which you can download from Zenodo. 

Further information

• Link to CEF homepage
• Related blog posts:
  • Evaluation project underway to assess the impact of CSCCE’s foundational training, Scientific Community Engagement Fundamentals
  • How does CSCCE online training impact community managers and their organizations? Read our report to find out!

Summary of findings 

The report goes into much more detail, but in brief we found that:

• 95% of respondents have applied multiple CEF concepts, frameworks, and activities in their roles to improve overall community strategy (68.6%), improve existing content and/or programming (67.4%), develop new content and/or programming (64%), and build alignment with team members (55.8%) – among other things.
• On an individual level, the vast majority of respondents reported positive socioemotional outcomes such as increased confidence (87%) and connection to peers (90%).
• On a community / organizational level, 74% of respondents reported at least one improvement in member participation within their communities, and 51% reported multiple improvements – as described by deeper engagement across the modes of CSCCE’s Community Participation Model. An even greater percentage (79.1%) selected multiple additional improvements beyond member engagement, such as such as the development of scaffolding and community engagement strategy.
• On an ecosystem-level, 38% of respondents reported multiple improvements in their organizations’ and communities’ relationship to the broader STEM ecosystem, such as improved communication with external stakeholders or communities, greater clarity about their community’s place within the broader STEM ecosystem, and new collaborations with external organizations or communities.

We conclude that CEF is serving its function well as CSCCE’s foundational training, and supporting our mission to professionalize and institutionalize the role of the community engagement manager in STEM.

Download the report

Acknowledgements

The CEF course was designed by Lou Woodley and Camille Santistevan, building upon some of the curriculum Lou designed for the Community Engagement Fellows Program and adapted for an online setting. Over the years it has been taught by Lou, Camille, Alice Martinic, and Maya Sanghvi.

We are grateful to the Alfred P. Sloan Foundation for financially supporting the development of both CEFP and the CEF course, and subsequent support to establish CSCCE.

The research in this report was funded by the Chan Zuckerberg Initiative (CZI), and we’re grateful to Priscila Martinez at CZI for championing the wider application of this work.

Thank you to all our CEF learners and especially to everyone who took the time to complete the survey. Many of you were very generous with your time by completing the open-ended questions and providing specific examples. 

CSCCE consultation 

CSCCE can help you apply similar research methods to better understand the impacts of your community or project. We’ve evaluated multiple champions programs in a range of settings from open source software projects to open data science initiatives. To learn more about how to work with us, please email training@cscce.org.