In biomedicine, segmentation involves annotating pixels from a very important structure in a medical image, like an organ or cell. Artificial intelligence models can assist clinicians by highlighting pixels which will show signs of a certain disease or anomaly.
Nonetheless, these models typically only provide one answer, while the issue of medical image segmentation is commonly removed from black and white. Five expert human annotators might provide five different segmentations, perhaps disagreeing on the existence or extent of the borders of a nodule in a lung CT image.
“Having options can assist in decision-making. Even just seeing that there’s uncertainty in a medical image can influence someone’s decisions, so it will be important to take this uncertainty into consideration,” says Marianne Rakic, an MIT computer science PhD candidate.
Rakic is lead creator of a paper with others at MIT, the Broad Institute of MIT and Harvard, and Massachusetts General Hospital that introduces a brand new AI tool that may capture the uncertainty in a medical image.
Generally known as Tyche (named for the Greek divinity of probability), the system provides multiple plausible segmentations that every highlight barely different areas of a medical image. A user can specify what number of options Tyche outputs and choose probably the most appropriate one for his or her purpose.
Importantly, Tyche can tackle recent segmentation tasks without having to be retrained. Training is a data-intensive process that involves showing a model many examples and requires extensive machine-learning experience.
Since it doesn’t need retraining, Tyche could possibly be easier for clinicians and biomedical researchers to make use of than another methods. It could possibly be applied “out of the box” for quite a lot of tasks, from identifying lesions in a lung X-ray to pinpointing anomalies in a brain MRI.
Ultimately, this technique could improve diagnoses or aid in biomedical research by calling attention to potentially crucial information that other AI tools might miss.
“Ambiguity has been understudied. In case your model completely misses a nodule that three experts say is there and two experts say is just not, that might be something it’s best to concentrate to,” adds senior creator Adrian Dalca, an assistant professor at Harvard Medical School and MGH, and a research scientist within the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).
Their co-authors include Hallee Wong, a graduate student in electrical engineering and computer science; Jose Javier Gonzalez Ortiz PhD ’23; Beth Cimini, associate director for bioimage evaluation on the Broad Institute; and John Guttag, the Dugald C. Jackson Professor of Computer Science and Electrical Engineering. Rakic will present Tyche on the IEEE Conference on Computer Vision and Pattern Recognition, where Tyche has been chosen as a highlight.
Addressing ambiguity
AI systems for medical image segmentation typically use neural networks. Loosely based on the human brain, neural networks are machine-learning models comprising many interconnected layers of nodes, or neurons, that process data.
After speaking with collaborators on the Broad Institute and MGH who use these systems, the researchers realized two major issues limit their effectiveness. The models cannot capture uncertainty and they need to be retrained for even a rather different segmentation task.
Some methods try to beat one pitfall, but tackling each problems with a single solution has proven especially tricky, Rakic says.
“If you should take ambiguity into consideration, you frequently must use a particularly complicated model. With the strategy we propose, our goal is to make it easy to make use of with a comparatively small model in order that it will possibly make predictions quickly,” she says.
The researchers built Tyche by modifying an easy neural network architecture.
A user first feeds Tyche a number of examples that show the segmentation task. As an illustration, examples could include several images of lesions in a heart MRI which were segmented by different human experts so the model can learn the duty and see that there’s ambiguity.
The researchers found that just 16 example images, called a “context set,” is enough for the model to make good predictions, but there isn’t a limit to the variety of examples one can use. The context set enables Tyche to resolve recent tasks without retraining.
For Tyche to capture uncertainty, the researchers modified the neural network so it outputs multiple predictions based on one medical image input and the context set. They adjusted the network’s layers in order that, as data move from layer to layer, the candidate segmentations produced at each step can “talk” to one another and the examples within the context set.
In this manner, the model can be certain that candidate segmentations are all a bit different, but still solve the duty.
“It’s like rolling dice. In case your model can roll a two, three, or 4, but doesn’t know you’ve got a two and a 4 already, then either one might appear again,” she says.
Additionally they modified the training process so it’s rewarded by maximizing the standard of its best prediction.
If the user asked for five predictions, at the top they’ll see all five medical image segmentations Tyche produced, although one is perhaps higher than the others.
The researchers also developed a version of Tyche that could be used with an existing, pretrained model for medical image segmentation. On this case, Tyche enables the model to output multiple candidates by making slight transformations to photographs.
Higher, faster predictions
When the researchers tested Tyche with datasets of annotated medical images, they found that its predictions captured the range of human annotators, and that its best predictions were higher than any from the baseline models. Tyche also performed faster than most models.
“Outputting multiple candidates and ensuring they’re different from each other really gives you an edge,” Rakic says.
The researchers also saw that Tyche could outperform more complex models which were trained using a big, specialized dataset.
For future work, they plan to try using a more flexible context set, perhaps including text or multiple sorts of images. As well as, they need to explore methods that might improve Tyche’s worst predictions and enhance the system so it will possibly recommend the very best segmentation candidates.
This research is funded, partially, by the National Institutes of Health, the Eric and Wendy Schmidt Center on the Broad Institute of MIT and Harvard, and Quanta Computer.
