“Are children learning?” is a question that should inform all education policymaking. Yet in many countries, the answer to this question has remained largely unknown. The pursuit of an answer lies at the heart of the citizen-led assessment movement, initiated by Pratham in India in 2005. The movement is an attempt by civil-society organizations to gather evidence on learning and use it for two main purposes: first, to increase awareness of low learning outcomes and second, to stimulate actions that are intended to address the learning gap. This innovative approach to assessment has attracted interest and raised questions about the potential for non-traditional assessments to play a role in not only monitoring learning but advocating for more focus on educational outcomes within countries and at the international level.
In an effort to more deeply understand the nuts and bolts of the citizen-led assessment model and to evaluate its ability to measure learning, disseminate findings widely, and stimulate awareness and action, Results for Development (R4D), supported by The William and Flora Hewlett Foundation, evaluated four citizen-led assessments between May 2013 and November 2014: Annual Status of Education Report (ASER) in India, Beekunko in Mali, Jàngandoo in Senegal, and Uwezo in Kenya, Tanzania, and Uganda.
The evaluation aimed to answer three key questions, each of which was addressed by a separate but complementary component of the evaluation methodology:
| How well do citizen-led assessments measure learning? | Technical review of the testing tools, sampling design, and analytical processes used |
| How well do citizen-led assessment processes work? | Process evaluation |
| How well do citizen-led assessments stimulate awareness and action about learning outcomes? | Non-experimental evaluation of impact
|
What they measure, they measure well. Citizen-led assessments test a very constrained set of competencies in reading and mathematics. The testing tools they use yield valid results, which cast a spotlight on limited achievement in these basic competencies. In this way they are important and useful, but broadening the testing tools and strengthening comparability would allow citizen-led assessments to do more to inform policy and practice.A selection of the evaluation’s key findings are provided below:
- Evidence suggests volunteers are well-equipped to reliably assess children’s basic competencies. Inter-rater reliability studies of ASER and Uwezo indicate a high level of agreement in volunteers’ scoring of children’s responses.
- At the international level, both ASER and, more recently, Uwezo, have contributed to the critical focus on learning outcomes in global discourse and agenda-setting. Their contribution has included both providing evidence of the seriousness of the learning crisis (i.e., revealing major deficiencies in even the most basic competencies) and demonstrating how a low-resource model can be used to assess learning on a national scale.
- Increasing awareness of the learning crisis at the national level is one of the main successes of both ASER and Uwezo. But generating concrete action to improve learning outcomes on the part of key stakeholders has proven much more challenging for both initiatives.
- For both ASER and Uwezo, only sporadic evidence of impact at the district level was found. This can largely be attributed to the lack of resources available for systematic involvement of the network of district partner organizations in dissemination activities, and, relatedly, to the limited capacity of these organizations.
- One small-scale but significant way in which both ASER and Uwezo have triggered learning-focused action outside of the government is through the uptake of the testing tools in education programs run by NGOs and CSOs.
- The evaluation uncovered only limited anecdotal evidence that participation in the survey stimulates awareness or action at the community level.
Findings from the evaluation can be found in the full report available here.
R4D conducted this work with a number of key partners, including the Australian Council for Educational Research (ACER), Catalyst Management Services (CMS) in India, OWN & Associates Ltd. in East Africa, Mr. Abdoulaye Bagayogo in Mali, Mr. Souleymane Barry in Senegal, and Qdata Enterprises in Kenya.
