Bright spots in a null RCT: Facility supplies and infrastructure

[From 2013 to 2018, the Transparency for Development (T4D) project — a joint effort of the Ash Center at the Harvard Kennedy School and Results for Development — evaluated the effectiveness of a transparency and participation program that sought to improve maternal, newborn and child health. The result of the randomized controlled trial component of the evaluation was null — meaning that, on average, the program did not measurably improve the targeted health outcomes. At the same time, the wealth of qualitative information we collected allowed us to dig deeper into the stories behind these results, which revealed some surprising examples of specific, community-level success. This Bright Spots blog series will highlight and discuss those exceptions, not to contradict the RCT results, but to further contextualize them. Read the previous introductory post on social actions on the ground. In this post, we specifically look at actions to improve things like medicine stocks, beds for patients, and water and electricity for health facilities.]

Residents of one rural Indonesian village that participated in T4D had many complaints about their local Puskesmas health facility. There were frequent medicine stock outs, a lack of privacy when giving birth, and a general perception that the fee for giving birth at the Puskesmas was too high. The story of their participation shares themes with many of our program’s bright spots, and provides three insights into community-level success.

First, we observed many communities choosing to work on problems that they experience themselves, rather than a problem that someone else hands to them to fix. The barriers noted above were personal for this village.

Second, “bright spots” does not necessarily mean “without challenges.” Instead, when community members hit these roadblocks, many were undeterred and found different pathways forward.

The group first tried to have an informal meeting with the midwives, but were not satisfied with the secondary midwife they managed to meet.  As one group member put it, “she did not quite know the problems.” So, the community representatives made a follow-up appointment with the main midwife, enlisting health volunteers who knew the main midwife to make sure they accommodated her schedule.

Finally, those who put the time and energy into the social action felt that changes were made.

That meeting with the midwife went surprisingly well when it finally happened.  Another representative reported that “Initially, we were afraid the midwife would get angry…but she was very welcoming.”  The midwife conveyed all of the requests to the head of the Puskesmas facility.

A few weeks after this meeting, all of the community representatives proudly reported that the midwife lowered the cost of facility births from 500,000 rupiah (about US$35) to 350,000 rupiah (about US$25), improved the stock of medicines, and procured a new privacy partition and an ambulance for the facility. According to the CRs, the only equipment missing was a vacuum pump, but they were confident it would be procured the following year. The midwife confirmed those changes, and added that as a result, “The community has felt the benefit of community action….”

While this story does not confirm whether the vacuum was procured eventually or whether the group continued working on different problems — it does provide some reason for optimism.

Bright spots in the context of ‘no impact’

Even with the above bright spot story, it is important to bear in mind that our randomized controlled trial was, on average, unable to make a statement of impact — a “null result.” As Table 1 (excerpted from our forthcoming in-depth analysis of more than 1,000 social actions) shows, only a little over half of the villages that attempted actions to improve facility supplies or infrastructure self-reported those actions as complete. And because these numbers are self-reported by community representatives, we might expect some inflation.

We are exploring the hypotheses for our null result over the coming months, but one partial explanation may be that not every group was able to persist in the face of challenges like the village above. In a different Indonesian case, the head of the facility dismissively told a group asking for improved medicine stocks, “there is enough medicine, if the medicine were insufficient, the facility would add more….” In other cases, representatives were only given the easier parts of what they wanted, hindering their action’s ability to really move the needle on maternal health. One group asked for a generator, more medicine and supplies, and a privacy door, but only received the door.

So, on average, there are plenty of cases where, for different reasons, actions did not lead to anything we could associate with better health services, utilization, or outcomes.

But ‘no average impact’ does not mean ‘no bright spots’

The lack of an average impact across all the villages, or the fact that middling successes were more common than transformative ones, should not obscure those (albeit) fewer cases where communities really changed their facilities. In fact, such variation should be expected: as mixed-method champion Michael Woolcock recently put it, the likely finding of a complex program like T4D “will be that… it worked wonderfully for some, had little effect on others, and was diabolically awful for still others. Even when carefully designed, fully supported (politically and financially), and faithfully implemented, complex interventions… interact with ‘contexts’ in inherently idiosyncratic ways.”

So, what else can we learn from those places where it “worked wonderfully”? Fortunately, our entire Phase 2 approach was designed to study context, so be sure to click subscribe above for those forthcoming results. But one bright spot shows that the effect of context is indeed “inherently idiosyncratic.” This action involved fundraising to procure a bed to be used when mobile clinic workers traveled to the village from a distant facility.

The community group went house to house asking for donations. The broader community resisted the action at first, because in the past corrupt village leaders had put some community donations for a development project into their own pocket. The representatives were able to overcome this context through educating the villagers on the importance of maternal health, and procuring a receipt book to reassure potential donors. After collecting 60,000 shillings (around US$26), “for almost 50 thousand we bought a bed and mattress, and we were left with 10 thousand savings”, one community representative explained.

The village executive officer designated part of his office for the mobile clinic, and praised the action: “already it has moved services closer to the people in contrast to the past when they were compelled to go to a nearby village.” T4D researchers were able to photograph the bed acquired, with the same community representative noting that “We didn’t expect it to be done in such a short time!”

The bed and mattress the community successfully procured.

So what?

Again, the above stories were exceptions to the broader lack of an average impact from the program. Even so, there were clearly bright spots that should remind us not to make any “stunning…jump from positive analysis to normative policy.” “No average impact” is not the end of the story; at most, it is the story’s climax that only makes sense if followed by a clarifying, exploratory denouement. As more social accountability replications and new RCTs roll in with disappointing findings, that reminder is only the more important for those of us in this field.

Of course, the rosiest optimists will be able to even more points of hope, like this quote from the midwife coordinator at a Puskesmas that was unable to comply with the community representative’s requests for additional equipment during the period of our program: “There is not yet a realization of the community proposals…[but] the braveness of the community has to be appreciated….” Such an intangible change in relationship may bear unmeasured fruit in the future.

 

Photo © Syane Luntungan/USAID EMAS