Pilot Testing the Tools Sets Tone for Improving Immunization Data Quality and Use
By Robert Kindoli, Senior Monitoring & Evaluation Officer
Sep 25, 2014
Posted in Monitoring & Evaluation
Two primary outcomes for the BID Initiative include improving immunization data quality and utilization in demonstration countries, the first being Tanzania. To do this effectively we must establish a starting benchmark to monitor changes over time and a baseline survey is the perfect way of doing so. The BID Initiative baseline survey started with designing a health facility questionnaire to be administered to the head nurse of Reproductive and Child Health (RCH) and the manager of the health facility a district-level questionnaire for the District Immunization and Vaccine Officer (DIVO), and a regional-level questionnaire for the Regional Immunization and Vaccine Officer (RIVO). Each questionnaire has a Data Quality Assessment section adopted from the MEASURE Evaluation Routine Data Quality Assessment (RDQA) tool that assesses immunization data quality per EPI reporting indicators.
Poorly designed data collection tools can result in an unpleasant experience for the respondent and the researcher and can lead to poor quality findings. This is why pilot testing the tool before using it for actual baseline work becomes paramount. The pilot tests around the BID Initiative’s baseline questionnaires explored respondents’ understanding of the questions, time required to finish an interview, logical problems/flows of questions, and potential data sources for the DQA. The pilot testing was also envisioned to determine resources such as the size and organization of data collectors, training needs, and estimated costs. After each pilot test, we debriefed with respondents to get their opinions on the questionnaire and what questions should be added/removed to better understand data quality and use at each level of the health system.
With the BID Initiative already working in the Arusha region (in Meru district and Arusha city council) to test various interventions, pilot testing was done in sampled testing sites at regional, district and health facility levels where we have already built relationships. The health facilities for pilot testing were selected based on distance (to minimize the cost), geographical location (to include urban and peri-urban), and volume of the facility (to include both high and low volume facilities). At the district level, one DIVO with a high volume of data was selected and since we have only been working with one RIVO, that individual was selected at the regional level for pre-testing.
Through pilot testing, we identified many important changes to our data collection tools and procedures. For example, we realized that sections could be subdivided to allow the interview and DQA to be administered separately to more efficiently use our time with respondents. We also debriefed with respondents to get input and suggestions on how best the tool can be improved to suit all levels. In fact, through the pilot testing, we determined that extensive training should be done with data collectors, and discussed what topics should be covered in the process, and the importance of involving a person with extensive experience in vaccines. This will ensure that all data collectors are subjected to all necessary skills in reviewing immunization data source documents and assessing vaccine stock levels. For the BID Initiative team in Tanzania, pilot testing the tools has put us one step ahead in measuring the improvement in immunization data quality and use.