Evaluations that make a difference is a collection of 8 evaluation stories from around the world which is one of the first pieces of systematic research looking at factors that contribute to high quality evaluations that are used by stakeholders to improve programs and improve people’s lives. This initiative collected stories about evaluations that made a difference, not only from the perspective of the evaluators but also from the commissioners and users. The stories in this collection tell powerful stories about the findings in the evaluations and the ways the evaluations contributed to the impact of the programs. You may access the report and all the stories here, in English, Spanish, and French.
In these weekly posts, we will be sharing each story… Comments are very welcome!!
Emily Chesne, a public health nurse in Changing River, won’t take credit for the improvements in her community’s prenatal health services that she helped implement.For years, Aboriginal mothers have had to travel long distances for prenatal checkups. And on the health care side, record keeping was far from what it should be. But thanks to a pilot project called Patient Wait Times Guarantee and people like Chesne who take their jobs seriously, the situation has changed dramatically for the better.
While Chesne is satisfied with her team’s success in transforming her community’s prenatal health services through the project, she downplays her role as an agent of change. She views her participation in the data collection, tool development,
and process mapping activities – all key components of the initiative – as simply part of her busy and demanding job. ‘I’m a community health nurse,’ she says, ‘and this goes back to my nursing background – when you start a project you need to finish it.’
Chesne’s partners in this groundbreaking initiative included colleagues, managers, community leaders, and researchers from Health Canada. The researchers were responsible for coordinating and evaluating a total of 19 pilot studies in Aboriginal communities across Canada.
The research team included the project’s leader, Judy Watson, a 30-year veteran of government health programming and research with Health Canada; Samir Khan, a senior researcher and Chesne’s main point of contact; and Stephanie Potter of the Whetstone Group, the programme’s lead evaluator.
Through supportive collaboration with these researchers, Chesne and her colleagues, as well as the nurses in other participating Aboriginal communities, were able to use the data collected as part of the initiative to make health services more accessible and effective. The data were important. Chesne knew that if she and her colleagues could get their hands on first-hand information, they could develop better services for their prenatal clients.
Changing River’s maternal and child health care needs are pressing. For one thing, the population of the community is very young, with a median age of just 21, and it’s growing quickly. For another, access to treatment is a major issue because the community is located 3 hours from the nearest urban centre.
The initiative – and importantly, its strong evaluation component– led to a realisation that would change the course of everything for the pregnant women of Changing River. Through the evaluation, Chesne and her colleagues learned that the reason data were lacking was because women were not attending their third trimester prenatal appointments.
It was not difficult to understand why. Previously, the women attended prenatal appointments and delivered their babies at a hospital 45 kilometers away. Recently, however, that hospital had not renewed its license to deliver babies. As a result, all third trimester appointments and deliveries were referred to a hospital a full 80 kilometers away – nearly twice as far.
Chesne knew that many local moms were young, impoverished, and uncomfortable in unfamiliar environments. She suspected that these circumstances made the 90-minute trek to the distant hospital too challenging. ‘We have a lot of young moms in our community. They are shy. They don’t feel comfortable with professionals, with doctors.’
But until data collection began, the community health team had no way of knowing if the women were attending their appointments once they were referred to the larger hospital. Through their involvement in the evaluation, they discovered that many at-risk mothers were showing up for delivery without having had prenatal care for several months.
Using the evidence at their fingertips
Armed with the evidence they needed, the team took immediate action by implementing on-site doctor visits for prenatal patients. Chesne: ‘We initiated getting one of the doctors from the delivering hospital to come to our clinic. Having the doctor here worked really well. It helped with gathering data.’
Improving third-trimester prenatal care rates led to the identification of more gaps in care and unrecognised health risks among the patients. Better access to the physician’s files helped the team realise that many prenatal patients were not getting their bloodwork done, including the glucose tolerance test for gestational diabetes. Says Chesne, ‘We’re trying to better educate our prenatals with what tests they need and why.’
Improving compliance with testing also led to the discovery that rates of gestational diabetes were much higher in the community than anyone realised. This is a serious health issue that, according to Chesne, could have long- term repercussions. By facilitating early diagnosis, such problems can be identified and remedied.
The team continued to use the initiative’s data to overhaul the community’s prenatal and postnatal health care, setting the stage for a healthier future. For the first time, the clinic was governed by a client-centred culture. According to Chesne, ‘Now moms come in and learn how to take some of their own prenatal tests. We have our doctor hold a sharing circle with clients as well as individual sessions.’
Chesne reflects, ‘We learned things weren’t happening that we thought were happening. Because of this project, we made changes.’
The initiative had an immediate and substantial effect on people’s lives because its leadership, especially Watson, chose to prioritise community engagement. Frontline health workers like Chesne were true collaborators in the evaluation, facilitating community ownership of the data and making needed changes. Without Once these changes had been made, many of Changing River’s women who had been at risk for pregnancy-related complications now had access to diagnosis and treatment.
Chesne’s community was not alone in taking ownership of the initiative’s data. Watson recalls a 2009 meeting with nurses from a Manitoba community even more remote and with fewer resources than Chesne’s clinic in Changing River. At the time, the swine flu crisis was raging and it was overwhelming the community. Watson wondered whether the nurses would have the time or motivation to continue with the evaluation. But her fears were allayed when the nurses all arrived for their meeting.
Watson was moved. ‘Two nurses had their study files under their arms – they didn’t even have briefcases. One nurse had lost a family member in the midst of all this, but they kept the study files organised in spite of the challenges. They really wanted to stay on track for themselves, for us, but also for the community. That’s how important the community is – counting and being part of the evaluation was so important to them.’ Potter recalls another community where nurses were able to use data to effect change. She was awed by the transformation of one group of nurses during the evaluation. At first, Potter found them apprehensive, as if things had gone poorly with past research.
By the end of the initiative, however, the health workers had shown Potter that even people without strong research backgrounds could ensure that their communities got the most out of the data to which they had access. She recalls:
They didn’t have computer databases, they only had pieces of paper. But by the end of the project, they made those pieces of paper mean something. I will always remember what they said at the end: Data is good. They learned that. They could see patterns and make changes in their communities even without sophisticated stats.
Community empowerment through stakeholder collaboration
At every level of participation, stakeholders point to the Patient Wait Times Initiative as a particularly powerful example of government-funded community programming to empower frontline workers to effect change. But what made it different from other initiatives?
Traditionally, health research in Aboriginal communities has been characterised by a top-down approach. Khan explains that usually such research is undertaken by scientists from outside the community. Rarely is community feedback sought. According to Potter, ‘The historical dynamic between the federal government and First Nations communities has not been positive. That’s the big context that all this work happens in and we can’t pretend it doesn’t exist.’
This project, however, fostered a truly collaborative spirit. In Khan’s words, ‘It’s usually hard to get people from different areas to collaborate. But not on this project.’ Watson agrees. She characterises Health Canada and its collaborator in the leadership of the project, the Assembly of First Nations, as true partners from the beginning to the end. Watson and Khan agree that the same can be said of the evaluation team, emphasising that with Potter’s and her colleagues they were able to form a genuine partnership, leveraging each other to inform each other. Potter refers to the initiative as her ‘touchstone’. In fact, her own approach to evaluation has been molded by her observations of Watson’s work in enacting Health Canada’s mission of community empowerment and willingness to take risks. Potter found that working with Watson and her team was refreshing, informative, and educational because they were ‘…so comfortable not knowing necessarily what was going to happen. They gave permission for evaluators to ask questions and find the answers.’
Questions and answers – from both sides
At the community level, the data collection tools and procedures were refined based on the evaluation results and direct feedback from the frontline. Both Watson and Khan found that communities were very comfortable coming to them with questions and suggestions. According to Khan, this feedback was vital to the success of the initiative: ‘We were getting data all along the way. We were trying to come up with explanations in Ottawa, but we could ask questions directly of the people providing the data. We could work together to find the answers much more readily and easily. We were getting explanations from the people who had the answers.’
Even more important, for Khan, this methodology demonstrated how having communities lead, and researchers finding the best ways to support, showed what is possible in government-sponsored research. The evaluation’s face-to- face group meetings with participating communities at the end of data collection were a particularly powerful learning experience for Khan.
As important as it was to hear first-hand about their challenges – travelling in winter, getting reimbursed for travel, staff turnover – it was critical to get real explanations for what we were learning. The real innovation was figuring out how to have First Nation communities lead, and then finding ways to support them.
In turn, the nurses collecting data found that their work was facilitated by the support of the programme’s research team. According to Chesne:
Meeting and speaking with the researchers, especially Khan, inspired you to go back and do better. It was very time-consuming and could be frustrating. Then I’d go to meetings or do teleconferences, and they’d tell me what’s going on and inspire me to finish the programme. It’s important to have the communication to be able to talk to the people processing the data.
This support reflects the collaborative spirit that marked the project under Watson’s leadership and the evaluation team’s facilitation.
Turning it around – for good
The Patient Wait Times evaluation had a long-lasting impact on all the stakeholders: the patients, the community health professionals, the government researchers, and the evaluation team. Watson, Khan, and Potter all agree that the evaluation affected how each stakeholder engaged in future research with at-risk communities.
Khan emphasises that the lessons he took from this initiative, including the importance of helping communities to take ownership of their own health care, were instrumental in shaping the rest of his career.
In Potter’s words, the project helped her learn that successful evaluation ‘…is all about building relationships and working in that indigenous community development way: community-led, community-paced. Whose project is it? Whose data? There has to be room for the people who are affected. It’s not just the government’s story.’
The Patient Wait Times initiative illustrates that an evaluation can be a story about all stakeholders working together to make positive changes for both individuals and communities – provided the researchers get out of the way and have the communities take the lead.
Note: The events described in this story are true. However, at the request of the resource persons involved, the names of people and places have been changed. Suffice to say the events transpired in a First Nations community in northern Manitoba.
——–
Stephanie Potter, Lead Evaluator, SP Consulting and the Whetstone Group
Samir Khan, former Senior Research Analyst, First Nations Inuit Health Branch, Health Canada
Judy Watson, former Project Lead, First Nations Inuit Health Branch, Health Canada
Sara Pederson conducted interviews and crafted the evaluation story.
Evaluations that Make a Difference 