Generating M&E data for continuous improvement efforts

Bebbo (Europe and Central Asia)

The Bebbo app, developed by UNICEF, provides personalized information and tips about child development and caregiving to parents of children aged 0-6. The app is currently available in 14 countries across Europe and Central Asia. UNICEF uses a comprehensive framework to regularly monitor progress and evaluate impact of the app.

Uptake of the app is monitored through data from the app store on the number of installations and deletions. Activities undertaken by country offices to promote the app (e.g., through services providers, social media) are also tracked to observe the strategies that result in more downloads. Further, UNICEF also has data indicators to analyze if app downloads lead to usage. Usage data, collected by Google Analytics for Firebase, is focused on the patterns, frequency and length of use. This includes the total number of unique users by day and month, data on user retention, how often users access the app, and their duration of use. Firebase also helps track how often different features of the app (e.g., games and activities, milestones trackers) are used.

An online dashboard accessed by UNICEF country and regional offices is used to analyze trends in the data. The dashboard can also be filtered to obtain interesting trends in user uptake and usage based on user profile characteristics which includes information on their country, language of choice, relationship to child, and age group of child. Further, the changes produced by any new features introduced on the app can also be observed on the dashboard.

UNICEF uses the data collected through these M&E practices to understand trends in user experience and to introduce new, responsive features in the Bebbo app.

An embedded user survey also provides key data for M&E such as: place of residence, level of education, how users heard about the app, and if they are able to practice the tips they learn from the app. While users who complete the survey are likely the most engaged and thus not representative of all users, these data have been used to understand the extent to which users’ awareness and practices across the thematic areas covered by the app have improved. Questions in the embedded user survey are changed on a regular basis to investigate patterns of the app use as well as potential impact of various features and the app in general. User surveys and feedback forms also generate valuable feedback on app performance, design and usability. To provide deeper understanding on the impact of the app on parenting practices, an online randomized control trial is being implemented in Bulgaria and Serbia.

More generally, UNICEF uses the data collected through these M&E practices to understand trends in user experience and to introduce new, responsive features in the app. For example, when it was discovered that engagement rates were initially low in a specific country, the team introduced translated content which led to an improvement in engagement rates. Similarly, before rolling out a new feature like push messages, different types of messages are tested to see what drives and increases engagement.

Parent Education Program (Jordan)

The Parent Education Program (PEP), aims at altering mothers’ behavior to better support their children’s learning and development in the critical first five years of life by equipping them with necessary awareness and practices. Delivered through Facebook Messenger, the chatbot is currently available in Jordan and neighboring countries. The Queen Rania Foundation for Education and Development (QRF), the developer of PEP, has designed frameworks for M&E to collect and analyze data that show the program’s effectiveness in achieving its direct results and objectives. QRF also gathers user feedback and uses research insights to refine and iterate the chatbot. These M&E efforts enable the Foundation to continuously improve the program and ensure that it meets the needs of its target audience.

Through research and evaluation, PEP has evolved to meet the needs of its users. For example, QRF conducted a feasibility assessment in 2021 – including focus group discussions and in-depth interviews – which confirmed that all participants (166 mothers) heard about the program through Facebook. At the time, the chatbot was still being used through Whatsapp, but later shifted to Facebook Messenger to accommodate its users. Another assessment in 2022 based on the delivery approach of the tool, with six focus group discussions and six key informant interviews, found that Facebook Messenger was perceived to be ‘efficient, simple, and direct’ (Ipsos, 2022). Through pre and post assessments, QRF also found that users wanted more interaction. Facebook groups and leaderboards for each cohort were created to give space for parents to share pictures and videos of their families doing the activities.

QRF also collaborates with MMIS Management Consultants to conduct tracer studies of PEP participants. A 2022 study, with four assessment periods over the duration of nine months, evaluated knowledge, skills, engagement/application, self-efficacy, confidence, and spillover to other family members. Using both quantitative and qualitative research methods, MMIS conducted four focus group discussions and four in-depth interviews, as well as two digital survey questionnaires. The final survey sample included 119 mothers. The study found a significant positive change to the participants’ knowledge and self-efficacy level between baseline and six months post-intervention. Moreover, the study revealed that users were motivated to engage in the program to acquire new skills and techniques to enhance their child’s development. This was facilitated by the convenience of online participation and the status of QRF as a trusted organization. At the same time, users found it challenging to absorb all of the information available in PEP. Based on MMIS’s recommendation to explain certain content areas, QRF updated some of the materials.

Through research and evaluation, Parent Education Program has evolved to meet the needs of its users.

Participants were also asked about the spillover effects of the program on the behavior of people around them. More than three-quarters (77.3 percent) of married respondents believe their husbands interact with their child/children more as a result of participating in the program. Almost all (95.3 percent) of participants whose family members practice the learning and development activities with their children reported that the effect persists even after the program ends. Moreover, half of the mothers interviewed reported that their older children became more involved in caring for younger siblings. However, others reported obstacles to spillover effects such as lack of time and social barriers. As an example of how QRF incorporated the participant feedback, PEP now allows fathers to register to help boost their involvement in their children’s development and learning. Additionally, QRF developed the course to be self-led which allows participants to learn at their own pace. In the future, there are plans to create a comprehensive app that provides parents with access to informative content, expert advice, and a milestone tracker, to offer further support.