Locations of the first 42 Brilliance units sold
Measuring impact is at the core of D-Rev’s work. Not only does user feedback help us to iterate in the design process and continually improve our products for our users, but more than that impact is the heart of our mission – it justifies our work, publishing our impact data gets you, our supporters and colleagues, in on the action, and it allows us to be transparent and honest about both our achievements and our challenges.
We try, wherever we can, to integrate impact collection into our designs themselves. We don’t think of impact as something to tack on later, it’s not a nice-to-have, but an integral element in our design process and our work overall. With Brilliance, our phototherapy device to treat neonatal jaundice, we outfitted the device with an LCD screen and an internal counter that tracks the total time that the machine has been turned on. We also added a “reset button” and instructed doctors and nurses to use the reset button each time he or she begins treatment with a new baby.
Baby being treated with Brilliance in India
Except data collection is never that simple. Even when the technical infrastructure is in place how do we convince partners in the field, doctors and nurses with competing priorities, tight schedules, and limited resources, not only to care, but to take that care and build a habit around it, to commit themselves to a process focused not just on outputs, but on positive outcomes for their patients and for others? How do we best connect with them to extract the data that lives in the unit and also gain an understanding of their experiences with patients? This is something that we are still working on and would love the input of others tackling similar challenges.
Brilliance is our first formal data collection effort so over the next year we are piloting 3 methods of data collection: Phone calls, in-person visits, and SMS messaging. We are hoping to learn A LOT in these pilots – about what gives us the most informative and richest data, but also about what is easiest for a doctor or nurse and most likely to garner a response. For our first round of data collection – for the first 11 units sold into 9 hospitals – we called the mobile phones of the doctors who use each unit.
In order to compensate for the inevitable gaps in what we are able to receive from partners on the ground, we have developed mathematical models based on research with doctors and partners about the effects of jaundice. Even with this research in hand we are very conservative in our calculations and assumptions.
For Brilliance we are focused on 3 main data points:
1. Total Babies Treated with a Brilliance Device: 324
We can derive this from use of the reset button on the devices, reports from doctors and nurses, and by reading the total treatment time off of the machine and (using our field-confirmed average treatment time of 48 hours) determine the total number of babies who have been treated with each unit. We compare these data from different sources to determine the most accurate count.
Total Machine Time /48 hrs (average treatment time) = total babies treated
2. Babies treated who otherwise would not have received effective treatment: 283
Babies treated who otherwise would not have received effective treatment (or internally “Babies Otherwise”) is at the crux of what we do, this is where we can begin to trace the power of Brilliance. This number captures babies who are receiving care in a clinic or hospital that did not previously have effective treatment. We consider ineffective treatment to be either no phototherapy devices, or phototherapy devices that are below the treatment standard determined by the American Academy of Pediatrics (>30 µW/cm2/nm peak spectral irradiance at treatment distance).
Using published data that we have collected with our partners, we have determined three classifications for medical facilities (high or medium/low income, public or private, and rural or urban) that each have a different probability of being able to provide effective jaundice treatment. We expect to adjust these weights over time as we collect more data and expand to new markets. We apply these probabilities to the total number of babies treated at each hospital to determine how many babies would not have received effective treatment without Brilliance. To see some of the data we have collected from hospitals about the effectiveness of their existing phototherapy devices, see Phototherapy Device Effectiveness in Nigeria.
Total Babies Treated * probability of not receiving effective treatment without Brilliance = “babies otherwise”
3. Deaths and disabilities averted because of the treatment received: 7
Severe jaundice, when left untreated or ineffectively treated, can lead to severe brain damage, a condition called Kernicterus (KI) or death. Using published data by our research partners we determine the number of newborns who have avoided death and disabilities from ineffective treatment by multiplying the total newborns treated who otherwise would not have received treatment (2), by the kernicterus rate for the region – for our current units in India that is 2%. From our conversations with Dr. Vinod Bhutani of Stanford University, we have seen this number vary widely from as low as 2.2/100,000 births in North America to as high as 1,749 (95% confidence interval: 1,649–1,849) per 100,000 live births in Iraq.
Babies Otherwise * .02 = Deaths & Disabilities Averted
Despite the challenges of going beyond simple output measures for determining the success of its products, D-Rev is determined to do just that: to measure and communicate, in quantitative and qualitative ways, the human impact of our work and how that work is improving people’s lives around the world. Furthermore, we aim to do it in such smart, informative, and compelling ways that D-Rev becomes an exemplary model for impact assessment in our field – because impact assessment isn’t just about proving effectiveness, but also about demonstrating transparency in our operations and results, being held accountable for our work, and gaining and maintaining long-term credibility. We hope to be part of a new era of transparency.