Blog Home

What can we learn from participants in transparency and accountability interventions?

T4D Learnings

By Stephen Kosack, an associate professor of public policy and governance at the University of Washington, Seattle, and a senior research fellow at the Harvard Kennedy School Ash Center for Democratic Governance and Innovation

Why do we sometimes hear that transparency and accountability programs empower citizens and community organizations to measurably improve the responsiveness and effectiveness of public services (here and here, for example), and sometimes we hear the opposite (see here, here, and here, for example)?

The effectiveness of development programs is generally assessed on objectively measurable outcome indicators: school attendance rates, test scores, per capita incomes, infant mortality rates, and other objective, comparable measures of impact. But perhaps we can also learn something about the implications of these programs from the experiences of those who participate.

In 2015 and 2016, several thousand people in 200 randomly-selected communities across four regions of Indonesia and Tanzania attended a series of facilitated meetings — as part of a mixed-method research project called Transparency for Development, to explore whether citizen engagement can improve maternal and newborn health outcomes. These meetings were carefully co-designed designed with civil society partners to encourage participants to work together to improve their maternal and newborn health care (see here, here, and here).

Like other evaluations, ours will assess the impact of this program on objective measures of maternal and newborn health and health care. But those improvements depend on the civic engagement of diverse participants in diverse communities. Thus, we are also attempting to understand the experiences of participants in this program, and the similarities and differences in their experiences. We are doing so from several perspectives, among them: 1) systematic qualitative observations of meeting discussions at the beginning and end of the program in 81 of these communities (41 in Indonesia and 40 in Tanzania), and 2) interviews with the people who participated before the first program meeting and immediately after the last in 32 communities (15 in Indonesia and 17 in Tanzania).

What are we learning so far? In a nutshell:

  1. Many people were willing to give the meetings and the civic activities they designed in them a try.
  2. Most, although far from all, seemed to think that they were getting somewhere.

Many people were willing to try

In most communities, many people showed up. Despite not receiving compensation or resources, an average of 14 people attended meetings over roughly three months of the program. Interviews and meeting observations suggest that in these meetings they engaged in sustained and generally self-directed discussions about problems with their maternal and newborn health care and deliberations about how they might alleviate those problems. Participants in almost all communities were also subsequently willing to try at least some of the civic activities they had planned. Here are a few indications:

It was rare for participants in meetings to sit quietly. At the first meeting, which was designed to provide space for participants to discuss and deliberate on information the facilitator presented on the state of maternal and newborn health care in the community (based on international standards and a survey of the community and its experiences of maternal and newborn health care in the previous weeks) and then to decide on the problems with that care that they would like to focus their efforts on, only 7 percent of participants made no distinctive contributions to the discussion. In the final meeting, when the facilitator asked participants to reflect on the progress they had made over the previous three months and to plan for the sustainability of their efforts once the facilitator had left for the final time, only 17 percent of participants made no distinct contributions to the discussion. In both countries, a majority of those who spoke in the discussions were women.

Generally, although not always, participants seemed to make decisions themselves rather than relying on the program facilitator. At several points in the discussions, observers also noted when participants relied on the facilitator or when their deliberations were controlled by the facilitator. In almost all communities, observers noted that in the majority of these discussions, the facilitator let those attending decide for themselves which problems to focus on or which approaches to take in resolving them. It was also uncommon for participants to rely on or defer to the facilitator: observers noted participants relying on the facilitator in most of the discussions in the first meeting in only five of the 81 communities, and most of the discussions in the last meeting in only 10 communities.

Participants were generally willing to try out what they planned, and to keep trying for several months. At the end of the program, approximately three months after the first program meeting, participants in almost every community still appeared engaged in at least some of the civic activities they had planned in the meetings to try to improve their maternal and newborn health care. How do we know? That’s a question we are exploring from a number of perspectives. But one indication from the meeting observations is that in all but one of the 81 communities, participants in the last program meeting engaged in detailed discussions reflecting on progress they had made and challenges they had faced in community activities that they discussed trying since the previous meeting about a month earlier — activities to improve knowledge or use of maternal and newborn health services, the responsiveness or effectiveness of health providers or officials, or to otherwise improve their care.

Was It Worth It? Optimism and Civic Efficacy

These meeting observations and interviews with individual participants also suggest that most thought their efforts were paying off. But far from all.

First, in discussions in the final program meeting, participants in 55 of the 81 communities where observers attended meetings seemed optimistic that the approaches they had designed and attempted outside the meetings were sustaining meaningful improvements in their maternal and newborn health care; this includes seven communities where participants seemed to begin the program skeptical that they would be able to improve their care. But in 26 of the 81 communities where observers attended meetings, participants seemed skeptical that they would be able to sustain improvements in their care. And context mattered: 21 of these 26 communities were in Tanzania.

We see something similar in interviews. In identical interviews with individual participants before and after the program, most seemed confident in their capacities to achieve the sort of improvements in their community’s public services that the program would encourage them to try, and significantly more told interviewers after the program that they were more confident in those capacities. These increases remain statistically significant even after we controlled for participants’ age, gender, education, community context, and used anchoring vignettes to adjust for differing interpretations participants might have of what it means to improve their communities.

Yet confidence was not universal, and, again, context mattered. In Tanzania, 14 percent of participants reported higher civic efficacy at the end of the program than before it began, but 19 percent reported lower civic efficacy. By contrast, in Indonesia, 46 percent of participants reported higher civic efficacy when the facilitator was leaving the community for the final time. Only 7 percent reported lower.

In addition to context, when we looked more closely at who seemed to lose confidence, we found that the people who said they were less confident at the end of the program tended to be those who did not participate as much: who spoke little or not at all at meetings and who did not engage in planned activities outside the meetings. If a person spoke an average amount or more, they tended to report higher civic efficacy at the program’s conclusion.

The Bottom Line

Overall, these observations and interviews suggest that a transparency and accountability program can encourage civic participation in a large number of diverse communities — but not everywhere.

Our team is now digging deeper, with in-depth ethnographic studies of what it was like for participants and their broader communities to be a part of this program, statistical comparisons of objective indicators of their maternal and newborn health and health care to another randomly selected group of communities who were not offered the program, and analyses of contextual differences that may have influenced differences in the program’s effects. More on them all soon.

If you want more details on our findings about participation and civic efficacy, you can read more here in an early draft of a working paper from our team.

Comments 6 Responses

  1. Aidan Eyakuze February 5, 2019 @ 8:06 am

    Thank you for this thoughtful and challenging post. We here at Twaweza are using the piece and its insights to reflect on “the challenge of demonstrating citizen agency and how we propose to address it in Tanzania, Uganda and Kenya.” I look forward to some actionable ideas emerging for us to experiment with and watch closely.

    Reply
    1. Courtney Tolmie February 5, 2019 @ 9:07 pm

      Hi Aidan — thanks for this message! We were really interested in the agency and empowerment results as well, especially what we can take from the differences in different settings. We are looking forward to sharing more results soon — and will definitely follow up as we have more to share!

      Reply
  2. Sue Cant January 29, 2019 @ 9:34 pm

    Thx Stephen, Courtney and R4D for how thorough you have been in consideration and design in this T4D work. As an adviser in this space, these are pearls of insight about the nature of the implementation required in a notoriously complex area to evaluate and it is very valuable for practitioners and INGOs. However, I don’t believe that it is an “either”/”or” scenario when it comes to the evidence, especially as national governments are, in any case, moving forward to institutionalise these mechanisms — including scorecards in Indonesia (2017 nutrition policy), Afghanistan (community health strategy and 3 pilot provinces), Armenia (nationwide participatory requirements for municipal governments). As a key global expert working for thought leading donors in this space said to me once — it is clear this work impacts services, the jury is still out on whether and how it impacts governance in a more systematic way. Looking forward to reading more.

    Reply
    1. Courtney Tolmie February 1, 2019 @ 6:20 am

      Thanks for this feedback Sue! You are right that these are complex issues and that the research won’t (and should not) point to a “black or white” explanation. The work in the Encouraging Participation paper is just a piece of the story, even for the T4D project, and we are interested to learn and share more about whether the complex picture we saw with empowerment and participation matches what we see in terms of impact (or lack thereof) on services. Our hope is that the evidence, while not a perfect map of how social accountability works, will provide some useful insights to advisors, practitioners and policymakers working on supporting, implementing, and integrating these approaches into their work. Looking forward to sharing more soon!

      Reply
  3. David Sasaki January 27, 2019 @ 9:17 am

    Steve, thanks for this interesting write-up. I appreciate the extra insights brought by the qualitative observation. The differences in “sense of progress” and “civic efficacy” between Tanzania and Indonesia are profound. I look forward to reading your analysis on how the five types of accountability contexts that you developed with Archon might help explain the differences. Are there other explanations that come to mind? Are you able to compare with other data, such as the world happiness report, about perceptions of optimism and pessimism in both countries over the past few years?

    Reply
    1. Steve Kosack January 30, 2019 @ 3:59 pm

      Hi David, Thank you for reading the post and for your thoughtful questions. It’s a good idea to look at recent trends in optimism or pessimism, and something we should think about as we fill out the rest of the story — what we’re describing here is just one part. We are working on implications for health and on ethnographic studies of what participation was like. We may then find that differences were partly about context (including the factors in that paper with Archon) — that is certainly my expectation. I do also want to note that some of the differences in perceptions of civic efficacy between participants in Tanzania and Indonesia may have to do with different ways of interpreting the question we asked (see Section 6 in the working paper). We look forward to sharing more findings from the project soon!

      Reply

Leave a Reply

Comment Guidelines

Your email address will not be published. Required fields are marked *

Fostering Digital Communities

We create and support global communities of innovators, funders and policymakers for continuous and iterative learning, knowledge generation, exchange and collaboration. Click on the logos below to explore our communities: