Skip to main content
SpringerLink
Log in

Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented SMILES

  • Published:
Journal of Computer-Aided Molecular Design Aims and scope Submit manuscript

Abstract

Using generative deep learning models and reinforcement learning together can effectively generate new molecules with desired properties. By employing a multi-objective scoring function, thousands of high-scoring molecules can be generated, making this approach useful for drug discovery and material science. However, the application of these methods can be hindered by computationally expensive or time-consuming scoring procedures, particularly when a large number of function calls are required as feedback in the reinforcement learning optimization. Here, we propose the use of double-loop reinforcement learning with simplified molecular line entry system (SMILES) augmentation to improve the efficiency and speed of the optimization. By adding an inner loop that augments the generated SMILES strings to non-canonical SMILES for use in additional reinforcement learning rounds, we can both reuse the scoring calculations on the molecular level, thereby speeding up the learning process, as well as offer additional protection against mode collapse. We find that employing between 5 and 10 augmentation repetitions is optimal for the scoring functions tested and is further associated with an increased diversity in the generated compounds, improved reproducibility of the sampling runs and the generation of molecules of higher similarity to known ligands.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Code availability

Code was implemented in the proprietary codebase, GenAI, of the Odyssey Therapeutics generative drug discovery platform.

References

  1. Bjerrum EJ, Threlfall R (2017) Molecular generation with recurrent neural networks (RNNs). http://arxiv.org/abs/1705.04612

  2. Olivecrona M, Blaschke T, Engkvist O, Chen H (2017) Molecular de novo design through deep reinforcement learning. J Cheminform 9(1):48. https://doi.org/10.1186/s13321-017-0235-x

    Article  PubMed  PubMed Central  Google Scholar 

  3. Segler MHS, Kogej T, Tyrchan C, Waller MP (2018) Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Cent Sci 4(1):120–131. https://doi.org/10.1021/acscentsci.7b00512

    Article  CAS  PubMed  Google Scholar 

  4. Kadurin A, Nikolenko S, Khrabrov K, Aliper A, Zhavoronkov A (2017) DruGAN: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Mol Pharm 14(9):3098–3104. https://doi.org/10.1021/acs.molpharmaceut.7b00346

    Article  CAS  PubMed  Google Scholar 

  5. Gómez-Bombarelli R, Wei JN, Duvenaud D, Hernández-Lobato JM, Sánchez-Lengeling B, Sheberla D, Aguilera-Iparraguirre J, Hirzel TD, Adams RP, Aspuru-Guzik A (2018) Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent Sci 4(2):268–276. https://doi.org/10.1021/acscentsci.7b00572

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Bjerrum EJ (n.d.) Teaching computers molecular creativity. Cheminformania. https://www.cheminformania.com/teaching-computers-molecular-creativity/. Accessed 29 Jan 2023

  7. Segler MHS, Kogej T, Tyrchan C, Waller MP (2017) Generating focussed molecule libraries for drug discovery with recurrent neural networks. arXiv. http://arxiv.org/abs/1701.01329. Accessed 6 Feb 2023

  8. Blaschke T, Arús-Pous J, Chen H, Margreitter C, Tyrchan C, Engkvist O, Papadopoulos K, Patronov A (2020) REINVENT 2.0: an AI tool for de novo drug design. J Chem Inf Model 60(12):5918–5922. https://doi.org/10.1021/acs.jcim.0c00915

    Article  CAS  PubMed  Google Scholar 

  9. Guo J, Knuth F, Margreitter C, Janet JP, Papadopoulos K, Engkvist O, Patronov A (2022) Link-INVENT: generative linker design with reinforcement learning. https://doi.org/10.26434/chemrxiv-2022-qkx9f

  10. Fialková V, Zhao J, Papadopoulos K, Engkvist O, Bjerrum EJ, Kogej T, Patronov A (2021) LibINVENT: reaction-based generative scaffold decoration for in silico library design. J Chem Inf Model. https://doi.org/10.1021/acs.jcim.1c00469

    Article  PubMed  Google Scholar 

  11. Elton DC, Boukouvalas Z, Fuge MD, Chung PW (2019) Deep learning for molecular design—a review of the state of the art. Mol Syst Des Eng 4(4):828–849. https://doi.org/10.1039/c9me00039a

    Article  CAS  Google Scholar 

  12. Xu Y, Lin K, Wang S, Wang L, Cai C, Song C, Lai L, Pei J (2019) Deep learning for molecular generation. Future Med Chem. https://doi.org/10.4155/fmc-2018-0358

    Article  PubMed  Google Scholar 

  13. Wang M, Wang Z, Sun H, Wang J, Shen C, Weng G, Chai X, Li H, Cao D, Hou T (2022) Deep learning approaches for de novo drug design: an overview. Curr Opin Struct Biol 72:135–144. https://doi.org/10.1016/j.sbi.2021.10.001

    Article  CAS  PubMed  Google Scholar 

  14. Thomas M, O’Boyle NM, Bender A, de Graaf C (2022) Augmented hill-climb increases reinforcement learning efficiency for language-based de novo molecule generation. https://doi.org/10.26434/chemrxiv-2022-prz2r

  15. Bjerrum EJ (2017) SMILES enumeration as data augmentation for neural network modeling of molecules. arXiv. https://doi.org/10.48550/arXiv.1703.07076

  16. Arús-Pous J, Johansson SV, Prykhodko O, Bjerrum EJ, Tyrchan C, Reymond J-L, Chen H, Engkvist O (2019) Randomized SMILES strings improve the quality of molecular generative models. J Cheminform 11(1):1–13

    Article  Google Scholar 

  17. Neil D, Segler M, Guasch L, Ahmed M, Plumbley D, Sellwood M, Brown N (2018) Exploring deep recurrent models with reinforcement learning for molecule design. In: 6th International conference on learning representations, ICLR 2018—workshop track proceedings, 2018, pp 1–15

  18. Brown N, Fiscato M, Segler MHS, Vaucher AC (2019) GuacaMol: benchmarking models for de novo molecular design. J Chem Inf Model 59(3):1096–1108. https://doi.org/10.1021/acs.jcim.8b00839

    Article  CAS  PubMed  Google Scholar 

  19. Gao W, Fu T, Sun J, Coley CW (2022) Sample efficiency matters: a benchmark for practical molecular optimization. arXiv. https://doi.org/10.48550/arXiv.2206.12411

  20. Renz P, Van Rompaey D, Wegner JK, Hochreiter S, Klambauer G (2019) On failure modes in molecule generation and optimization. Drug Discov Today Technol 32–33:55–63. https://doi.org/10.1016/j.ddtec.2020.09.003

    Article  PubMed  Google Scholar 

  21. Kotsias P-C, Arús-Pous J, Chen H, Engkvist O, Tyrchan C, Bjerrum EJ (2020) Direct steering of de novo molecular generation with descriptor conditional recurrent neural networks. Nat Mach Intell 2(5):254–265

    Article  Google Scholar 

  22. Bjerrum EJ, Sattarov B (2018) Improving chemical autoencoder latent space and molecular de novo generation diversity with hetero encoders. Biomolecules 8(4):131

    Article  PubMed  PubMed Central  Google Scholar 

  23. Irwin R, Dimitriadis S, He J, Bjerrum EJ (2022) Chemformer: a pre-trained transformer for computational chemistry. Mach Learn Sci Technol 3(1):015022. https://doi.org/10.1088/2632-2153/ac3ffb

    Article  Google Scholar 

  24. Sumner D, He J, Thakkar A, Engkvist O, Bjerrum EJ (2020) Levenshtein augmentation improves performance of smiles based deep-learning synthesis prediction. https://doi.org/10.26434/chemrxiv.12562121.v1

  25. Polykovskiy D, Zhebrak A, Sanchez-Lengeling B, Golovanov S, Tatanov O, Belyaev S, Kurbanov R, Artamonov A, Aladinskiy V, Veselov M, Kadurin A, Johansson S, Chen H, Nikolenko S, Aspuru-Guzik A, Zhavoronkov A (2020) Molecular sets (MOSES): a benchmarking platform for molecular generation models. Front Pharmacol. https://doi.org/10.3389/fphar.2020.565644

    Article  PubMed  PubMed Central  Google Scholar 

  26. Margreitter S (2022) ChemCharts. https://github.com/SMargreitter/ChemCharts. Accessed 9 Sep 2022

  27. Blaschke T, Engkvist O, Bajorath J, Chen H (2020) Memory-assisted reinforcement learning for diverse molecular de novo design. J Cheminform 12(1):1–17. https://doi.org/10.1186/s13321-020-00473-0

    Article  CAS  Google Scholar 

  28. ReinventCommunity (Jupyter Notebook Tutorials for REINVENT 3.2) (2022) https://github.com/MolecularAI/ReinventCommunity. Accessed 9 Sep 2022

  29. Wang S, Che T, Levit A, Shoichet BK, Wacker D, Roth BL (2018) Structure of the D2 dopamine receptor bound to the atypical antipsychotic drug risperidone. Nature 555(7695):269–273. https://doi.org/10.1038/nature25758

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Berman HM, Westbrook J, Feng Z, Gilliland G, Bhat TN, Weissig H, Shindyalov IN, Bourne PE (2000) The protein data bank. Nucleic Acids Res 28(1):235–242. https://doi.org/10.1093/nar/28.1.235

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Friesner RA, Banks JL, Murphy RB, Halgren TA, Klicic JJ, Mainz DT, Repasky MP, Knoll EH, Shelley M, Perry JK, Shaw DE, Francis P, Shenkin PS (2004) Glide: a new approach for rapid, accurate docking and scoring. 1. Method and assessment of docking accuracy. J Med Chem 47(7):1739–1749. https://doi.org/10.1021/jm0306430

    Article  CAS  PubMed  Google Scholar 

  32. Guo J, Janet JP, Bauer MR, Nittinger E, Giblin KA, Papadopoulos K, Voronov A, Patronov A, Engkvist O, Margreitter C (2021) DockStream: a docking wrapper to enhance de novo molecular design. J Cheminform 13(1):89. https://doi.org/10.1186/s13321-021-00563-7

    Article  PubMed  PubMed Central  Google Scholar 

  33. Sun J, Jeliazkova N, Chupakhin V, Golib-Dzib J-F, Engkvist O, Carlsson L, Wegner J, Ceulemans H, Georgiev I, Jeliazkov V, Kochev N, Ashby TJ, Chen H (2017) ExCAPE-DB: an integrated large scale dataset facilitating big data analysis in chemogenomics. J Cheminform 9(1):17. https://doi.org/10.1186/s13321-017-0203-5

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay É (2011) Scikit-Learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    Google Scholar 

  35. RDKIT: open source cheminformatics. http://www.rdkit.org. Accessed 8 Sep 2022

  36. McInnes L, Healy J, Melville J (2020) UMAP: uniform manifold approximation and projection for dimension reduction. arXiv. https://doi.org/10.48550/arXiv.1802.03426

  37. Richards R, Groener A (2022) Conditional β-VAE for de novo molecular generation. https://doi.org/10.26434/chemrxiv-2022-g3gvz

  38. Di L, Kerns EH (2016) Drug-like properties: concepts, structure design and methods from ADME to toxicity optimization, 2nd edn. Elsevier, Amsterdam

    Google Scholar 

  39. Papadopoulos K, Giblin KA, Janet JP, Patronov A, Engkvist O (2021) De novo design with deep generative models based on 3D similarity scoring. Bioorg Med Chem 44:116308. https://doi.org/10.1016/j.bmc.2021.116308

    Article  CAS  PubMed  Google Scholar 

  40. Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, Bates R, Žídek A, Potapenko A, Bridgland A, Meyer C, Kohl SAA, Ballard AJ, Cowie A, Romera-Paredes B, Nikolov S, Jain R, Adler J, Back T, Petersen S, Reiman D, Clancy E, Zielinski M, Steinegger M, Pacholska M, Berghammer T, Bodenstein S, Silver D, Vinyals O, Senior AW, Kavukcuoglu K, Kohli P, Hassabis D (2021) Highly accurate protein structure prediction with AlphaFold. Nature 596(7873):583–589. https://doi.org/10.1038/s41586-021-03819-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Wong F, Krishnan A, Zheng EJ, Stärk H, Manson AL, Earl AM, Jaakkola T, Collins JJ (2022) Benchmarking AlphaFold-enabled molecular docking predictions for antibiotic discovery. Mol Syst Biol 18(9):e11081. https://doi.org/10.15252/msb.202211081

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We want to acknowledge the Data Science Team at Odyssey Therapeutics for their helpful feedback and discussions, and, especially, Dr. Atanas Patronov and Dr. Kostas Papadopoulus for their REINVENT 2.0 expertise. We also want to thank Sophie Margreitter for the helpful discussions regarding ChemChart code modifications.

Funding

The research contribution of Raquel López-Ríos de Castro in this study was supported by the Biotechnology and Biological Sciences Research Council (BB/T008709/1) through the London Interdisciplinary Doctoral Programme (LIDo) under Grant No. BB/T008709/1.

Author information

Authors and Affiliations

  1. Odyssey Therapeutics, Cambridge, MA, USA

    Esben Jannik Bjerrum, Christian Margreitter, Thomas Blaschke, Simona Kolarova & Raquel López-Ríos de Castro

  2. Department of Physics and Department of Chemistry, King’s College, London, UK

    Raquel López-Ríos de Castro

Authors
  1. Esben Jannik Bjerrum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Margreitter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Blaschke
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simona Kolarova
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Raquel López-Ríos de Castro
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

EJB made the initial approach suggestion, implemented the prototype and production code for GenAI, carried out and analyzed the similarity and docking tasks, wrote the draft manuscript and had overall supervision of the project. RL-RdC ported the augmented Hill-Climb code modifications to GenAI, developed the D2R QSAR model, and carried out and analyzed the QSAR task and the comparison of the AHC algorithm variations. CM and TB provided help with the setup of the REINVENT 2.0 algorithm and scoring functions and offered helpful discussions and feedback on results. SK offered helpful discussions and feedback, and extensively helped with the writing of the manuscript. All authors read, edited, and approved the final paper.

Corresponding author

Correspondence to Esben Jannik Bjerrum.

Ethics declarations

Conflict of interest

Authors are employees at Odyssey Therapeutics, which has a commercial interest in utilizing generative modelling of prospective drug candidates.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 776 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bjerrum, E.J., Margreitter, C., Blaschke, T. et al. Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented SMILES. J Comput Aided Mol Des 37, 373–394 (2023). https://doi.org/10.1007/s10822-023-00512-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10822-023-00512-6

Keywords

Search

Navigation