CLI's Impact Study: Measuring participant transformation
Introduction
The Congo Leadership Initiative is proud to present the results of its 2019 study that measured participant transformation over the course of the nine-month CLI program. CLI has long tracked metrics and monitored results related to the impact of our program on the communities in which we work. We have done so by diligently monitoring the number of individuals whose lives were affected and the profits generated via the businesses that our young leaders found and lead. However, the results reported here are from a study where, for the first time, we focused specifically on the transformation experienced by individual participants themselves.
Further, to our knowledge, this study represents the first serious evaluation of psycho-social change experienced by youth in the DRC over the course of participation in a leadership skill-building course. It represents a promising start that we hope will lead to more evaluations of this nature in the DRC and similar countries. As an organization, we are pleased to have quasi-experimental evidence of the transformation that we have long anecdotally monitored among the youth who are part of our community.
Framework of the Study
The design of CLI’s study was created by an expert team of international development professionals at the Johns Hopkins University SAIS program. They proposed a quasi-experimental design in which youth who were wrapping up their participation in CLI’s nine-month leadership training program were tested and then compared against a similar group of youth who were just beginning the program as part of the next cohort. This allowed CLI to understand the shift in mentality between two similar groups whose only discernible difference was their participation in CLI’s program.
Both iterations of the study took place in both Beni (in the North Kivu province) and Kinshasa, the capital. These diverse settings lent more validity to the survey results. There were 57 metrics in each survey, each of them measuring one of seven different dimensions of leadership efficacy and self-sufficiency:
Collaboration
Common Purpose
Self-efficacy
Citizenship
Commitment
Optimism
Self Esteem
Each scale was measured on a five-point Likert scale (strongly agree, agree, neutral, disagree, strongly disagree) and some statements were deliberately posted in an inverse manner so as to disguise the survey design.
Sixty-two youth were part of the sample/test group and forty youth were part of the control group, though not every participant in the survey responded to every metric.
Results
Overall, youth in the test group improved along the dimensions measured by 41 out of the 57 metrics (72%). The improvement for youth who completed the program versus youth who did not was largest for the following metrics:
“It is easy for me to stick to my aims and accomplish my goals”
“Collaboration produces better results”
“Others would describe me as a cooperative group member”
“If someone opposes me, I can find the means and ways to get what I want”
More broadly, youth who had completed CLI’s program showed improvement on five out of seven themes or dimensions described in the prior section. Specifically, they improved on collaboration, common purpose, self-efficacy, citizenship, and commitment. The highest area of improvement was on dimensions measuring collaboration, a key focus of the CLI curriculum and our leadership training methodology.
In two of the seven dimensions measured, youth who completed the program did not improve and, in fact, scores were worse for the youth who had completed the program than for those who had not yet completed the program. These dimensions were optimism and self-esteem. Ultimately, the lack of improvement in optimism and self-esteem is logical given the fact that CLI’s curriculum does not focus on these areas. While these results do suggest a need to add content with an emphasis on these dimensions, we are also interested in better fitting the survey design to our specific lesson goals in the future.
Further, attendance during the program period measured was lower than normal due to unusual election-related violence in the DRC. In future years, as these attendance numbers improve, we expect that the survey results demonstrating efficacy of the program will improve as well.
The most important note, however, is that the study this year is simply a baseline for future work. The most relevant data have to do with changes in the metrics described above over time. As indicated below, CLI will continue to monitor data of this type in future years and, over time, we expect to develop and cultivate a rich set of relevant statistical analysis.
Next Steps
CLI intends to continue program evaluations of this nature both for our own edification and to contribute to the overall knowledge base of groups doing similar work to ours. As a baseline, we will conduct the same study (with some regular modifications) each year with a similar or larger number of participants.
Further, as is briefly discussed in these results, CLI plans to use this evaluation to shape curriculum innovation and improvement. In this way, best practices suggested by the survey will influence the future direction of the CLI program. At the same time, future surveys will be better matched to the specific lesson goals that CLI has in order to better evaluate our impact against the goals we choose.
This evaluation represents the next step of a continuous process of improvement and evaluation for CLI as an organization. We are proud of these initial results and excited to grow into the future.