Reinforcement Learning, a synthetic intelligence approach, has the potential to guide physicians in designing sequential treatment strategies for higher patient outcomes but requires significant improvements before it might be applied in clinical settings, finds a brand new study by Weill Cornell Medicine and Rockefeller University researchers.
Reinforcement Learning (RL) is a category of machine learning algorithms capable of make a series of selections over time. Chargeable for recent AI advances, including superhuman performance at chess and Go, RL can use evolving patient conditions, test results and former treatment responses to suggest the following best step in personalized patient care. This approach is especially promising for decision making for managing chronic or psychiatric diseases.
The research, published within the Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) and presented Dec. 13, introduces “Episodes of Care” (EpiCare), the primary RL benchmark for health care.
“Benchmarks have driven improvement across machine learning applications including computer vision, natural language processing, speech recognition and self-driving cars. We hope they may now push RL progress in healthcare,” said Dr. Logan Grosenick, assistant professor of neuroscience in psychiatry, who led the research.
RL agents refine their actions based on the feedback they receive, steadily learning a policy that enhances their decision-making. “Nonetheless, our findings show that while current methods are promising, they’re exceedingly data hungry,” Dr. Grosenick adds.
The researchers first tested the performance of 5 state-of-the-art online RL models on EpiCare. All five beat a standard-of-care baseline, but only after training on hundreds or tens of hundreds of realistic simulated treatment episodes. In the true world, RL methods would never be trained directly on patients, so the investigators next evaluated five common “off-policy evaluation” (OPE) methods: popular approaches that aim to make use of historical data (resembling from clinical trials) to bypass the necessity for online data collection. Using EpiCare, they found that state-of-the-art OPE methods consistently didn’t perform accurately for health care data.
“Our findings indicate that current state-of-the-art OPE methods can’t be trusted to accurately predict reinforcement learning performance in longitudinal health care scenarios,” said first creator Dr. Mason Hargrave, research fellow at The Rockefeller University. As OPE methods have been increasingly discussed for health care applications, this finding highlights the necessity for developing more accurate benchmarking tools, like EpiCare, to audit existing RL approaches and supply metrics for measuring improvement.
“We hope this work will facilitate more reliable assessment of reinforcement learning in health care settings and help speed up the event of higher RL algorithms and training protocols appropriate for medical applications,” said Dr. Grosenick.
Adapting Convolutional Neural Networks to Interpret Graph Data
In a second NeurIPS publication presented on the identical day, Dr. Grosenick shared his research on adapting convolutional neural networks (CNNs), that are widely used to process images, to work for more general graph-structured data resembling brain, gene or protein networks. The broad success of CNNs for image recognition tasks throughout the early 2010s laid the groundwork for “deep learning” with CNNs and the fashionable era of neural-network-driven AI applications. CNNs are utilized in many applications, including facial recognition, self-driving cars and medical image evaluation.
“We are sometimes all for analyzing neuroimaging data that are more like graphs, with vertices and edges, than like images. But we realized that there wasn’t anything available that was truly such as CNNs and deep CNNs for graph-structured data,” said Dr. Grosenick.
Brain networks are typically represented as graphs where brain regions (represented as vertices) propagate information to other brain regions (vertices) along “edges” that connect and represent the strength between them. This can also be true of gene and protein networks, human and animal behavioral data and of the geometry of chemical compounds like drugs. By analyzing such graphs directly, we are able to more accurately model dependencies and patterns between each local and more distant connections.
Isaac Osafo Nkansah, a research associate who was within the Grosenick lab on the time of the study and first creator on the paper, helped develop the Quantized Graph Convolutional Networks (QuantNets) framework that generalizes CNNs to graphs. “We’re now using it for modeling EEG (electrical brain activity) data in patients. We are able to have a net of 256 sensors over the scalp taking readings of neuronal activity — that is a graph,” said Dr. Grosenick. “We’re taking those large graphs and reducing them all the way down to more interpretable components to higher understand how dynamic brain connectivity changes as patients undergo treatment for depression or obsessive-compulsive disorder.”
The researchers foresee broad applicability for QuantNets. For example, also they are trying to model graph-structured pose data to trace behavior in mouse models and in human facial expressions extracted using computer vision.
“While we’re still navigating the security and complexity of applying cutting-edge AI methods to patient care, every step forward — whether it’s a brand new benchmarking framework or a more accurate model — brings us incrementally closer to personalized treatment strategies which have the potential to profoundly improve patient health outcomes,” concluded Dr. Grosenick.
