EntrEval
Evaluating the effects of entrepreneurship development programmes
What can we learn from the EvaluLEAD Guide? (Excellent evaluation practice from the leadership development field)
Entrepreneurship development is a complex process that defies simplistic forms of impact evaluation. Yet donors who fund entrepreneurship development programmes reasonably expect to find out whether or not their investments in people and institutions are making a positive impact. The South African Institute of Entrepreneurship (SAIE) has undertaken a range of impact evaluations over the past few years, some leading to a chapter on youth entrepreneurship in the South African version of the 2006 Global Entrepreneurship Monitor (GEM report) published by the University of Cape Town Graduate School of Business (Centre for Innovation and Entrepreneurship).
To measure programme impact, SAIE has used a combination of quasi-experimental design - randomized, pre-post questionnaire with different treatment groups administered by an external agency – and qualitative approaches.
- The quantitative element measured changes in students’ entrepreneurial knowledge, skills and attitudes.
- The qualitative approaches attempted to better understand how the programme works in classrooms and the kinds of effects it has on learners and Educators. Qualitative methods yield rich insights on the challenges Educators face and serve as a valuable guide to programme refinement for organisations like SAIE.
Quantitative methods reveal ‘what’ happened, while qualitative methods are more effective for helping to understand ‘why’ and ‘how’ things happened.
Pre test – post test studies remain the standard approach for evaluations of effects (impact evaluations); see, for example, Taylor and Vinjevold’s Getting Learning Right, 1999.
Measuring changes in students’ entrepreneurial knowledge, skills and attitudes using pre-post questionnaires is challenging, however, because classrooms are so complex. Complex environments require large scale quantitative studies to adequately control for extraneous variables, which is an expensive process and not feasible for an NGO to do.
Some of the complexity in classrooms includes: * teacher qualifications * teacher experience * teacher knowledge of entrepreneurship * teacher motivation and attitudes * classroom implementation of entrepreneurship curriculum * how the various materials were used * interaction of teachers, students, content and environments.
After a few years of developing and testing these impact evaluation approaches in over 75 schools, the SAIE team has concluded that we need to develop some richer, more sophisticated ways of measuring impact. A very useful evaluation tool called EvaluLEAD has emerged recently in the leadership development field and the South African Institute for Entrepreneurship is adapting it for use in entrepreneurship development.
In 2001 the Population Leadership Program (PLP) of the Public Health Institute (PHI), a project funded by the United States Agency for International Development (USAID), was wrestling with the question “What form of evaluation would best help us determine if and how leadership development activities make a difference?” At the same time the W.K. Kellogg Foundation was reviewing 55 varied leadership development programmes for good evaluation practices. The two groups collaborated in developing the PLP’s promising emerging framework into what became known as EvaluLEAD.
In early 2004, after some further concept development work, 17 organisations running a wide range of leadership development programmes were trained to use the EvaluLEAD methodology and field test it in their organisations throughout 2004, with external expert assistance throughout the year. The EvaluLEAD Guide is the result of the learnings and developments from these groups. More background information and the EvaluLEAD Guide itself are available at http://www.evalulead.net.
Some Key Concepts
Entrepreneurship development operates in a complex, open system, where many influences affect participants (family, school, peers, faith communities, role models, career options), and human elements (relationships, passions, viewpoints, freedom to make choices, etc.) make predictable outcomes difficult (see SAIE conceptual model).
Evaluative enquiries into entrepreneurship development programmes “… will be journeys of discovery and lead to findings that could not have been foreseen with clarity. The purpose of these explorations is threefold:
- To better understand the associations between results observed in the individual, organisational and societal domains;
- To gain an overall sense from observed patterns and examples of how a programme works to accomplish its short-term objectives and broader mission; and
- To share these understandings with key programme stakeholders and use them as a basis for modest to major programme enhancements.”**
(** EvaluLEAD: A Guide for shaping and evaluating leadership development programs, Grove, J.T., Kibel, B.M. & Haas, T., Sustainable Leadership Initiative, 2005.)
EvaluLEAD was formulated as an evaluation planning framework, but it also contains the processes for programme design and learning in order to improve programme activities. EvaluLEAD is more holistic and open ended than the logical framework approach.
Three levels of change should be explored:
- Episodic/Immediate: more predictable, short term, cause-and-effect results following a programme’s intervention;
- Developmental: more open-ended and less predictable changes occurring over time, including forward progress and setbacks, and affected by external influences and internal willingness and ability to change;
- Transformative: fundamental shifts in individual, organisational, or community values and perspectives that seed the emergence of fundamental shifts in behaviour and performance; often the ‘prize’ to which programmes aspire.
These three levels of change should be considered for three domains: individuals, organisations and the broader community being served. This approach leads to a three by three programme results map (see below). Two main forms of inquiry are considered – evidential (facts) and evocative (opinions) – for each square in the grid.
PROGRAMME RESULTS MAP
(This diagram originates from EvaluLead: A Guide for shaping and evaluating leadership development programs, Grove, J.T., Kibel, B.M. & Haas, T., Sustainable Leadership Initiative, 2005.)
The EvaluLEAD Guide was developed over three years by John T. Grove, Barry M. Kibbel and Taylor Hass of the Sustainable Leadership Initiative, a project of The Public Health Institute, Oakland, California, USA.
In 2001 the Population Leadership Program (PLP) of the Public Health Institute (PHI), a project funded by the United States Agency for International Development (USAID), was wrestling with the question “What form of evaluation would best help us determine if and how leadership development activities make a difference?” To answer this question PLP staff started with an extensive literature review around various theories and related tools for assessing changes in individuals and group leadership.
As no comprehensive or overarching leadership evaluation strategy was found, PLP staff developed some of the principles they found into a framework that was then workshopped over two days with a wide range of national and international evaluation experts and leadership development programme colleagues. This workshop led to a position paper that subsequently evolved into the EvaluLEAD methodology.
During this same period of time the W.K Kellogg Foundation was reviewing 55 leadership development programmes in different sectors to clarify their approaches to evaluation and outcome measurement. Seeing potential synergies, the W.K. Kellogg Foundation asked PLP staff to compare their emerging EvaluLEAD framework with the output and outcome measures in the 55 leadership programmes reviewed. The comparisons and analysis showed a lot of potential for the EvaluLEAD framework to help discern best practice models.
So the W.K. Kellogg Foundation formed a collaboration with PLP, PHI and USAID to fund the Sustainable Leadership Initiative (SLI) to develop the emerging methodology more fully. In early 2004, after some further concept development work, 17 organisations running a wide range if different leadership development programmes were trained to use the EvaluLEAD methodology and field test it in their organisations throughout 2004, with external expert assistance throughout the year. The EvaluLEAD Guide is the result of the learnings and developments from these groups. More background information and the EvaluLEAD Guide itself are available at http://www.evalulead.net.
