Insilico and NVIDIA unveil new LLM transformer for solving biological and chemical tasks

Insilico MedicineMay 20 2024

In a new paper, researchers from clinical stage artificial intelligence (AI)-driven drug discovery company Insilico Medicine ("Insilico"), in collaboration with NVIDIA, present a new large language model (LLM) transformer for solving biological and chemical tasks called nach0. The multi-domain and multi-task LLM was trained on a diverse set of tasks, natural language understanding, synthetic route prediction, and molecular generation, and works across domains to answer biomedical questions and synthesize new molecules. The findings were published in Chemical Science Journal.

While there are other LLMs designed for biomedical discovery, including BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) and SciFive, these datasets rely mainly on biomedical natural language texts, such as drugs, genes, and cell line names, but do not contain chemical structure descriptions. Those that have emerged with both text and chemical structure descriptions, such as Galactica, have not yet been trained for diverse chemical tasks. 

Nach0 seeks to bridge this gap for the first time. It draws from a dataset that includes abstract texts extracted from PubMed and patent descriptions derived from the U.S. Patent and Trademark Office related to the chemistry domain – 100 million documents that became 355 million tokens worth of abstracts and 2.9 billion patents, as well as molecular structures using simplified molecular-input line-entry system (SMILES). To train the system, researchers turned this chemical information into tokens as well – 4.7 billion – and then annotated these tokens with special symbols. 

Using this dataset, researchers trained nach0 to perform three key tasks: natural language processing, such as document classification and question answering; chemistry-related tasks, such as molecular property prediction, molecular generation, and reagent prediction; and cross-domain tasks, including description-guided molecule design and molecular description generation.

Nach0 represents a step forward in automating drug discovery through natural language prompts. In the future, we foresee the potential inclusion of protein sequences with their own special tokens as well as fine-tuning the model in order to accommodate new modalities and exploring the fusion of information from text and knowledge graphs."

Alex Zhavoronkov, PhD, Founder and CEO of Insilico Medicine

Nach0 is built on the NVIDIA BioNeMo generative AI platform, enabling training and scaling of drug discovery applications. Specifically, the training was performed using NVIDIA NeMo, an end-to-end platform for developing custom generative AI. The research team leveraged NLP capabilities to train and evaluate the new model's LMs. NVIDIA's memory-mapped data loader modules allowed researchers to manage large datasets with small memory footprints and optimal reading speed. 

"Generative AI and LLMs are transforming the landscape of scientific discovery in biology and chemistry," said Rory Kelleher, Global Head of Business Development for Life Sciences at NVIDIA. "Insilico's domain-specific nach0 model, powered by NVIDIA BioNeMo, is a significant step toward unlocking the full potential of LLMs for drug discovery."

Measured against other LLMs used for biomedical understanding, such as FLAN, SciFive, and MolT5, nach0 was found to have distinct advantages when performing molecular tasks using molecular data, and it significantly outperformed ChatGPT. 

Researchers tested nach0's capabilities in two case studies. The first was to generate molecules that could be effective against Diabetes mellitus. Researchers entered the prompt "discover biological targets with potential therapeutic activity, analyze the mechanism of action, generate molecular structure, propose one-step synthesis, and predict molecular properties." They generated 200 SMILES on the molecule generation prompt and selected one structure as the most promising from a chemical expert knowledge perspective. They also applied nach0 to a case study used as a demo for Insilico's Chemistry42 generative AI drug design platform, with the model returning 8 molecules satisfying the prompt in just 15 minutes for generation and 30 minutes for scoring in Chemistry42. 

"We anticipate that as nach0 evolves, it will require less supervision, and it will be able to simply generate and validate promising therapeutic options for medicinal chemists," says Maksim Kuznetsov, a senior research scientist at Insilico and one of the paper's lead authors. 

Insilico Medicine is a pioneer in using generative AI for drug discovery and development. The Company first described the concept of using generative AI to design novel molecules in a peer-reviewed journal in 2016. Then, Insilico developed and validated multiple approaches and features for its generative adversarial network (GAN)-based AI platform and integrated those algorithms into the commercially available Pharma.AI platform, which includes generative biology, chemistry, and medicine, and has been used to produce a robust pipeline of promising therapeutic assets in multiple disease areas, including fibrosis, cancer, immunology, and aging-related disease, several of which have been licensed. Since 2021, Insilico has nominated 18 preclinical candidates in its comprehensive portfolio of over 30 assets and has advanced six pipelines to the clinical stage. In March 2024, the Company published a paper in Nature Biotechnology that discloses the raw experimental data and the preclinical and clinical evaluation of its lead drug – a potentially first-in-class TNIK inhibitor for the treatment of idiopathic pulmonary fibrosis discovered and designed using generative AI currently in Phase II trials with patients.