Evaluations that make a difference is a collection of 8 evaluation stories from around the world which is one of the first pieces of systematic research looking at factors that contribute to high quality evaluations that are used by stakeholders to improve programs and improve people’s lives. This initiative collected stories about evaluations that made a difference, not only from the perspective of the evaluators but also from the commissioners and users. The stories in this collection tell powerful stories about the findings in the evaluations and the ways the evaluations contributed to the impact of the programs. You may access the report and all the stories here, in English, Spanish, and French.
In these weekly posts, we will be sharing each story… Comments are very welcome!!
—————————
For expectant mothers in the mountainous Pacific nation of Papua New Guinea (known popularly as PNG by its inhabitants), giving birth is an exercise in luck. If your labour is short, and you experience no complications, luck is in your favour. But if luck is against you, if your labour is difficult, if the baby is stuck, if you bleed too much – you can only hope that you and your baby will survive. Because you are very, very far from the nearest hospital.
For the tens of thousands of mothers who go into labour in remote villages each year, babies are born at home or at the local clinic. Almost all of the country’s population of 7 million, a third of whom are females of child-bearing age, are isolated from the major population centres. Rural health care workers do their best, but if something goes wrong, they may not know what to do.
The seriousness of the situation is illustrated by PNG’s maternal mortality rate, which is among the highest in the world. For every 100,000 live births, 733 women die. By comparison, in Australia – the country’s nearest neighbour – the death rate is only 6.8.
The Childbirth Emergency Phone programme was designed to tackle this grim problem. In 2007, PNG had only 50,000 phone lines for the entire country – one phone for every 140 persons. Seven years later, 2.7 million people had mobile phones – one for every 2.6 persons.
This was real progress. It meant that for the first time rural health workers could be reliably connected to specialists in the labour wards of major towns. A previous initiative to link rural clinics to major centres via radio had achieved some success, but this became unreliable when the radios broke down because the parts and expertise to repair them were lacking. These were uncharted waters for PNG. Could the promise of mobile technology really transcend the tyranny of distance? To find out, a pilot project was paired with a rigorous evaluation.
The idea for the project began with a medical specialist at the University of Papua New Guinea named Glen Mola. Professor Mola believed that the best place to run and evaluate a time-limited pilot project would be Milne Bay Province, where the maternal mortality rate was even higher than the national average. Because the province constitutes the mountainous eastern peninsula of PNG’s main island, as well as hundreds of remote islands, health care workers faced the challenge of extreme isolation.
After discussion with Mola, Australian researcher Dr Amanda Watson set up the pilot project in the labour ward of the Alotau Provincial Hospital in the provincial capital in late 2012. She and Billy Naidi, the chief executive officer of the province’s health authority, set up a dedicated landline phone in the middle of the bustling labour ward and sent the toll-free number to rural health care workers in remote locations. Importantly, solar panels were also provided to recharge the mobile phones. Calls started coming in almost immediately. Reports of breach births, excessive bleeding, eclampsia, ectopic pregnancies, retained placentas – all potentially lethal without the right information at the right time – could now be addressed in response to isolated, stressed health workers who needed help in managing difficult deliveries. Each month, an average of 17 calls were being made – each concerning a potentially life-threatening issue.
‘This project is saving two lives, mother and child, and we are thankful,’ says one rural health worker.
‘Once, I saw a mother die in front of me, simply because of lack of communication,’ says another. ‘Today, I can call anywhere, and it’s free.’
Alotau health extension officer Alice Siwawata fielded many of these calls. ‘It was very difficult previously, communications-wise,’ she says. ‘During emergencies now, it’s like having a labour ward doctor in the rural clinics.’
The Australian government made a comprehensive evaluation part of the pilot. ‘That’s not common in development work,’ says Watson. ‘But for this one, evaluation was very important. Australia wanted to know what could be taken from it.’
While the initial anecdotal feedback was positive, Billy Naidi needed some hard evidence that the emergency line was working. Ever since Watson had pitched the idea to him, he’d become interested in the health care possibilities opening up with the rapid expansion of PNG’s mobile phone network. Australia’s aid programme was funding the pilot project, but these funds would soon dry up. (Aid from Australia, PNG’s former colonial overseer, still accounts for around 13% of the nation’s revenues.) But for an initiative to become sustainably funded by the provincial health authority Naidi headed, he needed proof – proof that would come through a detailed evaluation of the pilot. With a thorough evaluation of the project in his pocket, he could justify taking over the funding of the phone line. ‘We wanted to see what has happened,’ says Naidi. Watson understood his position and agreed to switch roles and evaluate the project she’d set up.
Watson began the evaluation process by hiring a local field officer and evaluator named Gaius Sabumei, who came to know Milne Bay Province well. For months, Sabumei travelled across the province, interviewing rural health workers, recent mothers and village leaders in remote areas, as well as the labour ward staff at Alotau Provincial Hospital where the phone line was based. He returned from his foray through the province with 44 interviews in his pocket. These interviews almost universally praised the pilot.
One rural worker heard of the free call service just before a woman presented with a retained placenta. ‘If this project was not here and I didn’t have any means to communicate, we could have lost this mother because she was already septicaemic,’ said the worker. ‘From anywhere, at any point, I can seek advice and help the patients.’
Almost all the interviews reflected positive responses. And even when a few of the respondents had something critical to say, they readily admitted that things were working better than before. For example, one labour ward staff member said ‘Answering an emergency call when you are busy attending to a mother in delivery is sometimes frustrating, but it’s part of our work.’ Beyond that, criticism was muted. Another labour ward member said that the initiative helped build relationships between the metropolitan staff and the rural areas.
In addition, Watson sought external validation from a health professional from a different province. Dr Derick Bintol, a PNG national, listened to a sample of recorded phone calls and analysed the clinical notes from a sample of 68 calls.
The evaluation found that despite the challenges of phone network downtime and distaste for seeking help from clinics amongst some cultures on outlying islands, the phone line had saved lives. Dr Bintol wrote: ‘There is very good evidence from the cases that lives of both mothers and babies have been saved. It was also evident that information exchanges in the phone calls were very educational, especially for the rural health staff who may encounter a similar problem in the next event.’ The evaluation also demonstrated that the advice given over the phone was useful and appropriate in most cases, and inappropriate (and thus potentially harmful) in 16% of the cases. This led Watson and Sabumei to recommend follow-up training to address the types of cases where bad advice had been given.
When the evaluation was given to Naidi, he examined it carefully. ‘From the beginning, I liked the idea. I was excited,’ he says. What he wanted to know, though, was whether the line worked as well as the anecdotal feedback suggested, and whether the phone line was superior to the ailing radio network. While it was impossible to estimate how many lives had been saved, it was clear from the clinical evaluation that some women would have died without the service, and that rural health care workers felt the initiative reduced their isolation. ‘I wanted to know that it was effective. Eventually we were convinced, and we made a commitment to sustain it.’
And with that, the phone line’s future was secured. After two handover ceremonies, in Alotau and in the nation’s capital of Port Moresby, the Childbirth Emergency Phone line at last became an ongoing, sustainable reality.
What has surprised Watson has been the unexpected spin-offs from the evaluation. The lengthy report led to national and Australian media attention on maternal mortality. Newspapers, radio, and television all reported the evaluation’s findings. ‘The evaluation gave people ideas for the use of phone lines in health more broadly,’ she says. In the mountainous Western Highlands Province, for example, a broader scheme is now up and running – a call centre staffed by trained nurses who answer health questions from the general public.
For Billy Naidi, the value of the evaluation has also been demonstrated by proving the effectiveness of breaking down Papua New Guinea’s millennia-old isolation using modern technology. ‘With this phone number, you can be in a remote area, confronting a challenging situation, and you can be in an intimate conversation – as if you’re standing next to someone in the labour ward,’ he says. ‘After the evaluation, we were overwhelmingly happy!’
—————
The Childbirth Emergency Phone is a project of the Milne Bay Provincial Health Authority. Establishment and the evaluation were funded by the Government of Australia.
This story was writen by Jessica Kenway and Doug Hendrie. Drafts were reviewed by Billy Naidi, Chief Executive Officer of the Milne Bay Provincial Health Authority, and Amanda Watson, Mobile Communications Research Consultant, Economic and Public Sector Program.
Photographs were taken by Rawena Russell and Amanda Watson.
More information on the project is available at http://www.pngepsp.org/our-activities/mobile-4-development/maternal-hotline/maternal-hotline-overview.php.
Evaluations that Make a Difference 