Q&A: Testing Assumptions in Real Time: Promoting literacy in India with mobile technology

[Editor’s Note: Duncan McCullough, a senior communications associate at R4D, recently caught up with Annya Crane, a project manager at Worldreader, to explore how R4D’s adaptive learning approach improved Worldreader’s understanding of the potential for mobile applications’ impact on literacy promotion. Together, R4D, Worldreader and Pearson piloted Read to Kids (R2K), an intervention in India that seeks to promote literacy by encouraging and empowering parents and caregivers to read to (and with) their young children by giving them access to a digital library of high quality, locally relevant children’s books via an app on their mobile phones. The findings will be published in a final report by R4D, Worldreader and Pearson in early 2018.]

Why literacy, and why mobile phones?

Annya: Literacy is transformative: it increases earning potential, decreases inequality, improves health outcomes and breaks the cycle of poverty. Yet there are 740 million illiterate people in this world and 250 million children of primary school age who lack basic reading and writing skills. To address this learning crisis, education programs need to support children’s learning long before the child reaches the classroom.

To that end, Worldreader partnered with Pearson (through its Project Literacy initiative) and R4D in 2015 to launch the Read to Kids program in Delhi, India. The program leverages the widespread mobile phone connectivity and declining cost of hardware devices in India to deliver e-books to parents and caregivers of children eight and under.

But just because you build and promote an app, doesn’t mean people will use it.

So Pearson and Worldreader worked with R4D’s Evaluation and Adaptive Learning team to put structured learning, rapid experimentation and feedback at the heart of their program design, pilot and scale up. The goal was to quickly learn what wasn’t working and iterate.

Designing and implementing a new program can be challenging. How has R4D engaged and worked with the Read to Kids pilot?

Annya: We got to know about R4D’s innovative M+E approach thanks to Pearson. We got rapidly interested in R4D’s evaluation and adaptive learning methodology, which focuses on co-designing rapid interventions, measuring progress and providing quick feedback in analyzing results and recommending whether to and how to redesign certain elements of the program.

Working with R4D, we were able to define the research framework and monitoring and evaluation (M&E) tools to measure our impact, and gather as many learnings as possible. The process included quarterly Learning Checks, in-person workshops where R4D shared data from the previous quarter and Worldreader and local partners discussed how the data would inform adaptations to their work in the next quarter. The Learning Checks helped our partners on the ground understand the impact of the work they were carrying out, and define the best programmatic intervention to reach our goal. Additionally, data analytics helped bring our backend app data to life and illustrated trends from work in the field.

What were the teams knowledge gaps early on in developing the intervention?

Annya: Prior to the Read to Kids program, we knew from previous research that a mobile phone could be a source of content to read to children. What we didn’t know was how to change parent behaviors — particularly, those who don’t see reading to children before they start school as a priority.

First, we needed to test several assumptions around competing priorities for the time and energy it takes to read with children, the value of reading to young children before they are literate themselves, awareness of best practices to read to children, and the willingness to use cellular data on their phone to read to children. The widespread use of mobile technology in India was playing in our favor. However, we knew access to cellular data for women (who spend most time with young children) would potentially be a barrier.

Our research framework included a pre-pilot phase, which allowed us to test assumptions about parents’ priorities and daily schedules. We also tested messaging and learned, for example, how talking about storytelling versus reading resonated much better among an Indian audience due to the long storytelling culture in the country. We also learned that we would need to engage respected community members and have a strong community presence since behavior change would be a greater barrier to reading than simple access to books.

Have there been any positive, but unexpected, impacts of the program so far?

Annya: Teachers were not the primary target audience, but as we began to implement the program, more and more teachers got involved in promoting reading to children amongst the parents they had access to. They were also interested in using the app in their own classrooms, due to the lack of supplemental reading materials available in preschools and government schools in Delhi.

Here’s the story of one of the teachers in our program who has been using the app over the past year to find content to use in her classroom.

The second unexpected positive impact was despite barriers in terms of access to mobile technology and data, more women became frequent readers than men by the end of the pilot. Additional qualitative findings highlighted that women’s confidence, demand and tech savviness increased thanks to the Read to Kids program.

What findings surfaced through the adaptive learning approach that may not have been found using traditional evaluation techniques?

Annya: A traditional M&E approach wouldn’t have allowed us to tweak our messaging and implementation strategy throughout the program and allow each local partner to learn from one another. But R4D’s evaluation and adaptive learning approach did.

Using the huge amount of data that mobile programing provides (combined with field monitoring data from local partners), we were able to assess what was happening and learn in real time. For example, when the program first launched we saw that users were accessing the app when they were asked to during official program-sponsored “Read to Kids” activities, but they were not coming back to the app to read to their children frequently at home. R4D brought us together with partners to brainstorm what could potentially not be working.

The implementing partners quickly agreed that parents needed further training and support to change their behaviors. We therefore decided to go back to the parents initially part of the program, focus our messaging on parents and communicating the importance of reading while also ensuring caregivers have the skills needed to use the app, select the content they feel both confident reading and their child will enjoy.

The Learning Checks also allowed us to identify the gaps in the training that was provided to our partner’s field staff, gather feedback from the book collection, the app and general qualitative feedback on how the program was going in the field.

And what results have those changes had? Are you seeing more parents use the app at home now?

Annya: The Read to Kids pilot program finished in June 2017. Overall, the program reached over 203,000 households, of which 15,000 were part of our implementation partners’ outreach in 177 low-income communities in Delhi. Thousands of parents in the program went from “non-readers” to frequent readers to their young children as a result of the program’s activities, but parents and caregivers’ attitudes and practices take time to shift. Though these engagement methods showed some promise, there are significant barriers to overcome. The most frequent barriers reported by our users were data costs, reliable connectivity and lack of time.

Finally and interestingly, the app was also used in a myriad of ways, those less literate readers prefered to narrate the story after having read it to themselves first and putting the phone down for fear of not being able to explain every word to their child; some older siblings read to younger children from the app; and some children even read the books alone.

Interesting. It sounds like a lot of learning took place — that media campaigns play a big role in building awareness and access to books, but behavior change requires follow-up with families over time from respected community partners. Thanks so much for the conversation, Annya! I look forward to reviewing the complete findings when the final report comes out next year.