New AI model revolutionizes cell structure segmentation in microscopy

Identifying and delineating cell structures in microscopy images is crucial for understanding the complex processes of life. This task is called "segmentation" and it enables a range of applications, such as analyzing the reaction of cells to drug treatments, or comparing cell structures in different genotypes. It was already possible to carry out automatic segmentation of those biological structures but the dedicated methods only worked in specific conditions and adapting them to new conditions was costly. An international research team led by Göttingen University has now developed a method by retraining the existing AI-based software Segment Anything on over 17,000 microscopy images with over 2 million structures annotated by hand. Their new model is called Segment Anything for Microscopy and it can precisely segment images of tissues, cells and similar structures in a wide range of settings. To make it available to researchers and medical doctors, they have also created μSAM, a user-friendly software to "segment anything" in microscopy images. Their work was published in Nature Methods.

To adapt the existing software to microscopy, the research team first evaluated it on a large set of open-source data, which showed the model's potential for microscopy segmentation. To improve quality, the team retrained it on a large microscopy dataset. This dramatically improved the model's performance for the segmentation of cells, nuclei and tiny structures in cells known as organelles. The team then created their software, μSAM, which enables researchers and medical doctors to analyze images without the need to first manually paint structures or train a specific AI model. The software is already in wide use internationally, for example to analyze nerve cells in the ear as part of a project on hearing restoration, to segment artificial tumor cells for cancer research, or to analyze electron microscopy images of volcanic rocks.

Analyzing cells or other structures is one of the most challenging tasks for researchers working in microscopy and is an important task for both basic research in biology and medical diagnostics. My group specializes in building tools to automate such tasks and we often get asked by researchers to help. Before the development of Segment Anything for Microscopy, we had to ask them to first annotate a lot of structures by hand – a difficult and time-consuming task. μSAM has changed this! Tasks that used to take weeks of painstaking manual effort can be automated in a few hours, because the model can segment any kind of biological structure with a few clicks and can then be further improved to automate the task with our tool. This enables many new applications, and we have already used it in a wide range of projects, ranging from basic cell biology to developing tools for treatment recommendation in cancer therapies."

Junior Professor Constantin Pape at Göttingen University's Institute of Computer Science