SAIE has always been committed to evaluating its work, independent of stakeholder accountability issues, because of its interest in learning and improving practice. A summary of the main reasons and goals for evaluation for SAIE are:
To be accountable to all stakeholders
To improve practice and the impact of interventions
To understand the impact SAIE is having
To understand what triggers or programme elements are most effective for helping participants
Generous SAIE donors have supported a range of evaluations for the organisation, including a pilot schools’ evaluation using rigorous quasi-experimental methods as well as a wider roll out evaluation using this methodology that was reported on in the 2005 Global Entrepreneurship Monitor (GEM Report) done by the UCT Graduate School of Business’ Centre for Entrepreneurship and Innovation
SAIE’s overall evaluation approaches, and especially approaches for impact evaluation, were outlined in last year’s annual report. While the two evaluations mentioned above showed positive impact results, they also raised important questions about appropriate evaluation methodologies for SAIE’s purposes. Reflecting on this process led us on a journey towards more comprehensive evaluation processes that include both quantitative and qualitative methods, and more specifically our efforts to develop our EntrEval framework. The exciting possibilities for EntrEval emerged after learning of three organisations’* struggles to measure the effectiveness and impact of leadership development programmes. Leadership development is an area of practice that shares with entrepreneurship development the problem of trying to facilitate change in complex, open-system environments. It too grapples with the complex challenges of evaluating the evidence and impact of this change.
Briefly, the EntrEval framework is guiding SAIE to gather hard evidence and more evocative stories and examples of programme impact at the levels of individuals, partner organisations and wider communities or society. These various forms of evidence will also be organised around three result types or levels of change:
Immediate or episodic: more predictable, tangible, short term results discernible immediately after programme intervention
Developmental: more open-ended, less predictable and less controllable changes that occur over time and represented as sequences in a dynamic process. They show differences in pace or rhythm among different individuals.
Transformative: fundamental shifts in values and perspective preceding fundamental shifts in behaviour and or performance; often the prize which programs aim for.
EntrEval progress in 2007 and early 2008 consisted mainly of undertaking rigorous assessment of SAIE’s current evidence collection processes and content. This has lead to significant reviews of indicators for each of SAIE’s programmes and to the development of a master list of key indicators. We have explored a range of appropriate data gathering methodologies and selected some to pilot test. Some questions and the gathering personal stories and examples of programme impact have been field-tested.
We have clarified the core importance of tracking changes in individuals. This has lead us to identify these as trainers or facilitators we equip to use our programmes and then of ‘end-user’ participants in SAIE programmes. The overall EntrEval framework has been changed to make it 4 levels instead of 3 (which take into consideration the splitting of the individual impact level into one for facilitators and one for participants).
An exciting element has been the establishment and holding of meetings with an excellent expert reference team to critique the emerging framework. These have lead to fruitful discussions around the core of entrepreneurship.
Major EntrEval development plans for 2008 are to:
Develop core reports and reporting options
Visit active programmes (field visits) to test the various data gathering methods
Get feedback from regular meetings of the expert reference team
Embed the emerging EntrEval framework into all SAIE forms and programmes
Train and support SAIE staff and partner organisations in the use of EntrEval
Start to embed EntrEval into the Management Information System.
We are acutely aware of the need to be able to give our donors feedback on the impact of their investments into entrepreneurship education development particularly. Many of the projects are long term with participants often starting as young as six years old. Our goal is to ensure that from as young as this, right through to adults, we can inculcate innovation and creative entrepreneurial, action-oriented mindsets. EntrEval is being designed to ensure that we are collecting evidence early enough to be able to make meaningful assessments as to the efficacies of our efforts.
*The Sustainable Leadership Institute, WK Kellogg Foundation and USAID developed a ‘best practice’ evaluation approach in the field of leadership development, EvaluLEAD, and, with permission, SAIE is adapting and contextualizing this approach for entrepreneurship development under the name EntrEval.