Evaluations that make a difference is a collection of 8 evaluation stories from around the world which is one of the first pieces of systematic research looking at factors that contribute to high quality evaluations that are used by stakeholders to improve programs and improve people’s lives. This initiative collected stories about evaluations that made a difference, not only from the perspective of the evaluators but also from the commissioners and users. The stories in this collection tell powerful stories about the findings in the evaluations and the ways the evaluations contributed to the impact of the programs. You may access the report and all the stories here, in English, Spanish, and French.
In these weekly posts, we will be sharing each story… Comments are very welcome!!
Like many countries, Kenya has an enormous population of unemployed young people. Depending on one’s perspective, this is either a ticking time bomb or an opportunity for affordable vocational training – relevant training that enables young people to get jobs, earn and save money, and move out of poverty.
The Community and Progress Youth Empowerment Institute (CAP YEI) opened its first training centre in Nairobi in 2011. In India, CAP had been successful in giving vulnerable youth a chance to acquire useful workplace skills through technical and vocational training. Now CAP was keen to learn how this approach could work in Kenya. Funded by The MasterCard Foundation as one of its Learn, Earn and Save initiatives, CAP YEI, in partnership with the University of Minnesota, evaluates its performance in both the short and the long term to make changes as it learns what works. The ongoing evaluations have helped to equip Kenyan participants with technical and psychological skills to get a job or to start their own business.
Douglas Moseti, CAP YEI Nairobi regional coordinator, has been with the programme since it began. ‘We target people between 18 and 25 years old who can demonstrate that they come from vulnerable backgrounds. Either they are orphans or they come from single-parent families or large families where a large number of people depend on a single breadwinner.’
Moseti knows what it means to need a leg up in life. He grew up poor and was forced to drop out of school several times when his mother could not pay his school fees. He finally finished high school, thanks to a lucky encounter at his church. But with a high school diploma, he could only find a job paying the equivalent of $1 a day. A 2-year vocational training course was out of his reach, at least for a time.
CAP YEI provides 3 months of training adapted to labour market demand. To determine which courses to offer, CAP YEI staff scan the local market, contacting companies and entrepreneurs to see what entry-level jobs are available. In its first centre in Nairobi – it now has nine there – they found that many hotels needed staff and security guards, and that construction firms need builders and electricians. As CAP YEI engages with potential employers to see what jobs they might have to offer their graduates, they also ask them to look at the proposed course content, to mentor CAP YEI trainees, and to offer them internships when they graduate from the programme.
Importantly, the training cycle doesn’t end with the classes as in most vocational training programmes. Rather, each cycle culminates when the graduates are placed in internships or jobs or are starting their own businesses. Moseti describes what happened with the first group of trainees.
There were 141 students in batch 1 and we were 6 staff who were very new. We were told that we’d get at least 80% of those 141 into employment opportunities. We didn’t hit the target for the first batch because the industry didn’t believe that a new employee could be trained in just 3 months, and we had no way to show potential employers that the trainees could do the work. But we did manage to place 67% of the trainees, including 5% who went to start their own businesses.
He then goes on to describe the evaluation that follows the placement of each graduating class.
Once a batch has completed everything, we review what we’ve finished before starting a new batch. Facilitators, the regional coordinator, and the programme director describe the whole process. We ask questions: Did we get the right people at the road shows? Did we give them the right skills? We look at student attendance at training and at work and ask: Was placement good? What course gave good placement? Where training does not lead to jobs, we change and get more industry people to help us, to tell us what’s missing and we supply that. In one centre where students had a hard time getting jobs in electrical and electronics, a new market scan was carried out and a curriculum was designed to offer training in building and construction. Elsewhere, garment manufacturing was replaced with training in security and management systems.
James Chepyegon, CAP YEI project manager, uses a management information system developed by CAP to collect data on course enrolments, on graduates per cohort, on the absorption rate of graduates into the job market, and on changes in students’ attitudes and lives after the training.
Students are surveyed before and after training to see how their attitudes have changed, how they made the transition to jobs, how they’ve created their own businesses, how their views of life have changed as a result of the training and employment, and how well they understand working relationships with employers.
This project data blends with the data collected annually by the University of Minnesota team that comes to Kenya to survey stakeholders, including participants and employers. Their data is compiled into a longitudinal study to see how the training changes the trajectory of participants’ livelihoods and lives. This impact evaluation is designed to emphasise learning in the monitoring, evaluation and learning portion of the programme to help CAP learn during implementation and respond to the needs of targeted youth.
One of the most striking findings of the evaluations has been the critical role of life skills in preparing young people for jobs. Professor Joan DeJaeghere, who leads the University of Minnesota team, cites her team’s annual on-site visits, surveys, and interviews with participants.
Year after year, CAP does life skills, a mix of engendering self-confidence, teaching students to be assertive, teaching them about job markets and how they work, where they are in the job market and what they can expect, and financial literacy. Youth say that the life skills training is what makes the difference. They say ‘I can get into something, start off, move to something new and manage finances.’
Dr Heidi Eschenbacher, another member of the team, confirms: ‘Our findings suggest that life skills combined with technical skills help youth to build up sufficient confidence to feel as though they can use their technical skills. In other words, they need life skills to build confidence to be efficacious in their technical skills.’
The life skills range from mentoring by successful business people, many of whom started out with little but managed to succeed, training in financial literacy, and training for young entrepreneurs looking to start their own businesses. Chepyegon confirms that the evaluations clearly show that the links between employers and life skills training work. ‘They are,’ he says, ‘the strongest point of success for CAP YEI in getting youth access opportunities.’
Their strength lies in part in the way that the life skills module has changed in response to the evaluations of participants’ comments over time. Adds Chepyegon:
The entrepreneurship department got stronger, to better account for entrepreneurship activities from the time individual potential entrepreneurs were identified to helping them start up their small business, including the creation and registration of entrepreneurship groups and support for the success of their small businesses. Nearly 200 entrepreneurs are currently functioning well on a full- and part- time basis.
A prominent person gave me electrical installation work at a very big building. As an electrical person God granted me favour with the boss. Whenever there was need for a driver and since I was qualified, I was the one to buy materials and run errands that required a driver. I became his point man on the construction site and ended up becoming the foreman of his building with both electrical and other construction responsibilities.
Moseti later joined his boss’ ICT firm, was promoted, and got a good raise.
Quite recently, Moseti was able to take the measure of the project’s success when he met with the first cohort of graduates in Nairobi. This cohort was the group followed by the University of Minnesota for its longitudinal study.
Seventy-eight of the 141 students came. All but one of them were working. I discovered that most of these young people (almost 40%) had moved into self-employment with big businesses. One lady had become an agent for large banks seeking to extend services to rural areas. She says that she’s earning 30,000 shillings per month from these mobile banking services, which include Mpesa, Equity Agent, and Cooperative Bank Agent. Another graduate imports clothes and shoes from Dubai and sells them in Kenya.
The learning and evaluation partnership at CAP YEI is working to help youth get ahead, while simultaneously learning what stands in their way. Reports Professor David Chapman, co-lead of the external evaluation:
There have been some surprises going into the fourth year of the 5-year field data collection. Where conventional wisdom might suggest that participants go through the training and then find or create a job, the reality is that they are multi-tasking to earn money. In their efforts to start their own businesses, these youths have trouble getting the necessary capital. That is slowly changing. Banks in Kenya have been in the forefront of youth bank products and have found that group lending programmes have a better repayment rate than they get with their more conventional customers.
CAP YEI, in response to this finding, has been pursuing a number of partnerships with financial service providers who are tracking key data points such as number of accounts opened, amount of funding released, average start-up capital, average earning post start-up, and average savings per client in order to make it easier for graduates to get the financial support they need to start their own businesses.
The ongoing programme improvement process is extremely helpful in assisting youths to get and keep good jobs, particularly in dynamic situations such as labour markets into which many youths are ready to enter.
Christopher Johnstone, Acacia Nikoi, and University of Minnesota research fellows conducted annual external evaluations of the project and presented data that helped refocus project programming and improve services for youth.
James Chepyegon, monitoring and evaluation executive, was responsible for conducting internal project monitoring, interpreting implications of M&E data, and building capacities of project staff .
Ashok Ankathi (project director at the time) provided programmatic evolutions influenced by evaluation.
Douglas Mose (Nairobi regional coordinator at the me) provided the context of the beneticiaries that helped shape the story.
Story writer: Deborah Glassman