How artificial intelligence and reinforcement learning can drive equitable care

One of the tragedies of modern healthcare is when treatments don’t reach everyone who can benefit from them. While there is much focus on what the future of AI can look like, there are some practical applications today that can help us innovate in care delivery through the identification of these gaps. Through reinforcement learning, we can better understand how to foster ongoing engagement with populations that are hard to reach, while operating within resource constrained environments. 

Our concept consists of applying a reinforcement learning approach to allocate resources toward care delivery dynamically, using a restless multi-armed bandit (RMAB) model. RMABs are a machine learning approach that experiments by allocating resources away from poorly-performing areas toward better performing ones to maximize resources and optimize outcomes. RMABs have been used by companies such as Spotify and StitchFix for random probability scheduling, and have recently been applied by researchers to health issues like computing optimal cancer screening regimens, improving maternal health with telehealth and planning hepatitis-C treatment delivery. Because RMABs are designed for reward-maximizing resource allocation strategies, they can lead to inadvertent inequities across demographic groups, resulting in disparate outcomes and exacerbating existing health inequities.

In the recent paper, “A New Approach to Equitable Intervention Planning to Improve Engagement and Outcomes in a Digital Health Program: Simulation Study”, published in JMIR Diabetes, authors from Verily, Google and Harvard applied the RMAB framework through a new solution, equitable RMAB (ERMAB), requiring that algorithmic policies take affirmative steps to distribute resources in a way that equalizes outcomes among pre-defined groups. This solution was applied in an extensive simulation study using publicly available statistics about Type 2 Diabetes management. The simulation study also applied a Pareto analysis to further analyze engagement-clinical outcome dynamics under different intervention strategies and perform sensitivity analyses.

When the ERMAB framework was applied to the optimization of intervention policies, it showed it could lead to 10% more patients reaching a healthy clinical outcome (defined by target A1C levels) after 12 months, with a 10% reduction in program engagement dropout compared to standard-of-care baselines.

This study is the first to formulate an RMAB model of digital health, with the novel characteristic of a multi-dimensional state space that factors in criteria such as engagement and clinical health. This outcome-based fairness formulation is the first of its kind for restless bandits.

This was a simulation exercise to validate a new methodological approach for equity. Further research applying ERMAB is needed in a real-world context in order to confirm the simulated results.

Our work demonstrated the feasibility and the importance of planning interventions in digital health that account not only for individual clinical outcome objectives, but also for long-term engagement dynamics, using an RMAB sequential decision-making framework. This is also the first RMAB with equity focused objectives, which viewed fairness through the lens of taking affirmative steps toward equitable outcomes. Co-author Yugang Jia, Head of Platform Data Science at Verily, noted that “we hope that ERMABs will add to the arsenal of tools available to practitioners addressing resource allocation problems in communities with unmet needs.”

There has often been the worry that bias and inequity can be a “side effect” of AI in healthcare. According to co-author and Head of Clinical Informatics at Verily, Erich Huang, MD, PhD, “we believe that AI can be designed to directly address the problem. So rather than ring-fencing an algorithm to minimize its untoward effects, why not explicitly design the AI to help solve the problem? As an industry, we have a strong obligation to develop techniques like ERMABs to solve these problems and amplify the potential of fit-for-purpose AI to be part of the solution.”

Authors

Jackson A. Killian of Harvard University, Verily Life Sciences and Google Research, Manish Jain of Google Research, Yugang Jia of Verily Life Sciences, Jonathan Amar of Verily Life Sciences, Erich Huang of Verily Life Sciences and Milind Tambe of Harvard University and Google Research.