What’s the likelihood of dying in a plane crash? In keeping with a 2022 report released by the International Air Transport Association, the industry fatality risk is 0.11. In other words, on average, an individual would wish to take a flight day by day for 25,214 years to have a one hundred pc probability of experiencing a fatal accident. Long touted as one among the safest modes of transportation, the highly regulated aviation industry has MIT scientists pondering that it might hold the important thing to regulating artificial intelligence in health care.
Marzyeh Ghassemi, an assistant professor on the MIT Department of Electrical Engineering and Computer Science (EECS) and Institute of Medical Engineering Sciences, and Julie Shah, an H.N. Slater Professor of Aeronautics and Astronautics at MIT, share an interest within the challenges of transparency in AI models. After chatting in early 2023, they realized that aviation could function a model to be certain that marginalized patients are usually not harmed by biased AI models.
Ghassemi, who can also be a principal investigator on the MIT Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic) and the Computer Science and Artificial Intelligence Laboratory (CSAIL), and Shah then recruited a cross-disciplinary team of researchers, attorneys, and policy analysts across MIT, Stanford University, the Federation of American Scientists, Emory University, University of Adelaide, Microsoft, and the University of California San Francisco to kick off a research project, the outcomes of which were recently accepted to the Equity and Access in Algorithms, Mechanisms and Optimization Conference.
“I feel I can speak for each Marzyeh and myself after I say that we’re really excited to see sort of excitement around AI starting to come back about in society,” says first writer Elizabeth Bondi-Kelly, now an assistant professor of EECS on the University of Michigan who was a postdoc in Ghassemi’s lab when the project began. “But we’re also slightly bit cautious and wish to attempt to make certain that it’s possible we will have frameworks in place to administer potential risks as these deployments begin to occur, so we were searching for inspiration for methods to attempt to facilitate that.”
AI in health today bears a resemblance to where the aviation industry was a century ago, says co-author Lindsay Sanneman, a PhD student within the Department of Aeronautics and Astronautics at MIT. Though the Twenties were referred to as “the Golden Age of Aviation,” fatal accidents were “disturbingly quite a few,” based on the Mackinac Center for Public Policy.
Jeff Marcus, the present chief of the National Transportation Safety Board (NTSB) Safety Recommendations Division, recently published a National Aviation Month blog post noting that while quite a lot of fatal accidents occurred within the Twenties, 1929 stays the “worst 12 months on record” for probably the most fatal aviation accidents in history, with 51 reported accidents. By today’s standards that might be 7,000 accidents per 12 months, or 20 per day. In response to the high variety of fatal accidents within the Twenties, President Calvin Coolidge passed landmark laws in 1926 referred to as the Air Commerce Act, which might regulate air travel via the Department of Commerce.
However the parallels don’t stop there — aviation’s subsequent path into automation is analogous to AI’s. AI explainability has been a contentious topic given AI’s notorious “black box” problem, which has AI researchers debating how much an AI model must “explain” its result to the user before potentially biasing them to blindly follow the model’s guidance.
“Within the Nineteen Seventies there was an increasing amount of automation … autopilot systems that handle warning pilots about risks,” Sanneman adds. “There have been some growing pains as automation entered the aviation space when it comes to human interaction with the autonomous system — potential confusion that arises when the pilot doesn’t have keen awareness about what the automation is doing.”
Today, becoming a industrial airline captain requires 1,500 hours of logged flight time together with instrument trainings. In keeping with the researchers’ paper, this rigorous and comprehensive process takes roughly 15 years, including a bachelor’s degree and co-piloting. Researchers consider the success of intensive pilot training might be a possible model for training medical doctors on using AI tools in clinical settings.
The paper also proposes encouraging reports of unsafe health AI tools in the way in which the Federal Aviation Agency (FAA) does for pilots — via “limited immunity”, which allows pilots to retain their license after doing something unsafe, so long as it was unintentional.
In keeping with a 2023 report published by the World Health Organization, on average, one in every 10 patients is harmed by an opposed event (i.e., “medical errors”) while receiving hospital care in high-income countries.
Yet in current health care practice, clinicians and health care staff often fear reporting medical errors, not only due to concerns related to guilt and self-criticism, but additionally attributable to negative consequences that emphasize the punishment of people, equivalent to a revoked medical license, relatively than reforming the system that made medical error more more likely to occur.
“In health, when the hammer misses, patients suffer,” wrote Ghassemi in a recent comment published in Nature Human Behavior. “This reality presents an unacceptable ethical risk for medical AI communities who’re already grappling with complex care issues, staffing shortages, and overburdened systems.”
Grace Wickerson, co-author and health equity policy manager on the Federation of American Scientists, sees this latest paper as a critical addition to a broader governance framework that shouldn’t be yet in place. “I feel there’s rather a lot that we will do with existing government authority,” they are saying. “There’s other ways that Medicare and Medicaid pays for health AI that makes sure that equity is taken into account of their purchasing or reimbursement technologies, the NIH [National Institute of Health] can fund more research in making algorithms more equitable and construct standards for these algorithms that might then be utilized by the FDA [Food and Drug Administration] as they’re attempting to work out what health equity means and the way they’re regulated inside their current authorities.”
Amongst others, the paper lists six primary existing government agencies that might help regulate health AI, including: the FDA, the Federal Trade Commission (FTC), the recently established Advanced Research Projects Agency for Health, the Agency for Healthcare Research and Quality, the Centers for Medicare and Medicaid, the Department of Health and Human Services, and the Office of Civil Rights (OCR).
But Wickerson says that more must be done. Probably the most difficult part to writing the paper, in Wickerson’s view, was “imagining what we don’t have yet.”
Slightly than solely counting on existing regulatory bodies, the paper also proposes creating an independent auditing authority, much like the NTSB, that enables for a security audit for malfunctioning health AI systems.
“I feel that is the present query for tech governance — we’ve not really had an entity that is been assessing the impact of technology because the ’90s,” Wickerson adds. “There was once an Office of Technology Assessment … before the digital era even began, this office existed after which the federal government allowed it to sunset.”
Zach Harned, co-author and up to date graduate of Stanford Law School, believes a primary challenge in emerging technology is having technological development outpace regulation. “Nonetheless, the importance of AI technology and the potential advantages and risks it poses, especially within the health-care arena, has led to a flurry of regulatory efforts,” Harned says. “The FDA is clearly the first player here, they usually’ve consistently issued guidances and white papers attempting for example their evolving position on AI; nevertheless, privacy might be one other necessary area to observe, with enforcement from OCR on the HIPAA [Health Insurance Portability and Accountability Act] side and the FTC enforcing privacy violations for non-HIPAA covered entities.”
Harned notes that the world is evolving fast, including developments equivalent to the recent White House Executive Order 14110 on the secure and trustworthy development of AI, in addition to regulatory activity within the European Union (EU), including the capstone EU AI Act that’s nearing finalization. “It’s actually an exciting time to see this necessary technology get developed and controlled to make sure safety while also not stifling innovation,” he says.
Along with regulatory activities, the paper suggests other opportunities to create incentives for safer health AI tools equivalent to a pay-for-performance program, wherein insurance firms reward hospitals for good performance (though researchers recognize that this approach would require additional oversight to be equitable).
So just how long do researchers think it will take to create a working regulatory system for health AI? In keeping with the paper, “the NTSB and FAA system, where investigations and enforcement are in two different bodies, was created by Congress over many years.”
Bondi-Kelly hopes that the paper is a bit to the puzzle of AI regulation. In her mind, “the dream scenario could be that each one of us read the paper and are super inspired and in a position to apply a few of the helpful lessons from aviation to assist AI to stop a few of the potential harm that may come about.”
Along with Ghassemi, Shah, Bondi-Kelly, and Sanneman, MIT co-authors on the work include Senior Research Scientist Leo Anthony Celi and former postdocs Thomas Hartvigsen and Swami Sankaranarayanan. Funding for the work got here, partly, from an MIT CSAIL METEOR Fellowship, Quanta Computing, the Volkswagen Foundation, the National Institutes of Health, the Herman L. F. von Helmholtz Profession Development Professorship and a CIFAR Azrieli Global Scholar award.
