Foundation model

A foundation model, also known as large AI model, is a machine learning or deep learning model that is trained on a broad dataset so it can be applied across a wide range of use cases.[1] Generative AI applications like Large Language Models are often foundation models.[1]

Building foundation models is often highly resource-intensive, with the most expensive models costing hundreds of millions of dollars to pay for the underlying data and compute required.[2] In contrast, adapting an existing foundation model for a specific use case or using it directly is much less expensive.

Early examples of foundation models are language models (LMs) like OpenAI's GPT series and Google's BERT.[3] Beyond text, foundation models have been developed across a range of modalities—including DALL-E and Flamingo[4] for images, MusicGen[5] for music, and RT-2[6] for robotic control. Foundation models are being built for astronomy,[7] radiology,[8] genomics,[9] music,[10] coding,[11] times-series forecasting,[12] mathematics,[13] and chemistry.[14]

  1. ^ a b Competition and Markets Authority (2023). AI Foundation Models: Initial Report. Available at: https://assets.publishing.service.gov.uk/media/65081d3aa41cc300145612c0/Full_report_.pdf
  2. ^ Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark, and Raymond Perrault, "The AI Index 2023 Annual Report," AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2023.
  3. ^ Rogers, Anna; Kovaleva, Olga; Rumshisky, Anna (2020). "A Primer in BERTology: What we know about how BERT works". arXiv:2002.12327 [cs.CL].
  4. ^ Tackling multiple tasks with a single visual language model, 28 April 2022, retrieved 13 June 2022
  5. ^ Copet, Jade; Kreuk, Felix; Gat, Itai; Remez, Tal; Kant, David; Synnaeve, Gabriel; Adi, Yossi; Défossez, Alexandre (7 November 2023). "Simple and Controllable Music Generation". arXiv:2306.05284 [cs.SD].
  6. ^ "Speaking robot: Our new AI model translates vision and language into robotic actions". Google. 28 July 2023. Retrieved 11 December 2023.
  7. ^ Nguyen, Tuan Dung; Ting, Yuan-Sen; Ciucă, Ioana; O'Neill, Charlie; Sun, Ze-Chang; Jabłońska, Maja; Kruk, Sandor; Perkowski, Ernest; Miller, Jack (12 September 2023). "AstroLLaMA: Towards Specialized Foundation Models in Astronomy". arXiv:2309.06126 [astro-ph.IM].
  8. ^ Tu, Tao; Azizi, Shekoofeh; Driess, Danny; Schaekermann, Mike; Amin, Mohamed; Chang, Pi-Chuan; Carroll, Andrew; Lau, Chuck; Tanno, Ryutaro (26 July 2023). "Towards Generalist Biomedical AI". arXiv:2307.14334 [cs.CL].
  9. ^ Zvyagin, Maxim; Brace, Alexander; Hippe, Kyle; Deng, Yuntian; Zhang, Bin; Bohorquez, Cindy Orozco; Clyde, Austin; Kale, Bharat; Perez-Rivera, Danilo (11 October 2022). "GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics". bioRxiv 10.1101/2022.10.10.511571.
  10. ^ Engineering, Spotify (13 October 2023). "LLark: A Multimodal Foundation Model for Music". Spotify Research. Retrieved 11 December 2023.
  11. ^ Li, Raymond; Allal, Loubna Ben; Zi, Yangtian; Muennighoff, Niklas; Kocetkov, Denis; Mou, Chenghao; Marone, Marc; Akiki, Christopher; Li, Jia (9 May 2023). "StarCoder: may the source be with you!". arXiv:2305.06161 [cs.CL].
  12. ^ Se, Ksenia; Spektor, Ian (5 April 2024). "Revolutionizing Time Series Forecasting: Interview with TimeGPT's creators". Turing Post. Retrieved 11 April 2024.
  13. ^ Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; Santos, Marco Dos; McAleer, Stephen; Jiang, Albert Q.; Deng, Jia; Biderman, Stella; Welleck, Sean (30 November 2023). "Llemma: An Open Language Model For Mathematics". arXiv:2310.10631 [cs.CL].
  14. ^ "Orbital".