Building foundation models is often highly resource-intensive, with the most expensive models costing hundreds of millions of dollars to pay for the underlying data and compute required.[2] In contrast, adapting an existing foundation model for a specific use case or using it directly is much less expensive.
Early examples of foundation models are language models (LMs) like OpenAI's GPT series and Google's BERT.[3] Beyond text, foundation models have been developed across a range of modalities—including DALL-E and Flamingo[4] for images, MusicGen[5] for music, and RT-2[6] for robotic control. Foundation models are being built for astronomy,[7] radiology,[8] genomics,[9] music,[10] coding,[11]times-series forecasting,[12] mathematics,[13] and chemistry.[14]
^Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark, and Raymond Perrault, "The AI Index 2023 Annual Report," AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2023.
^Rogers, Anna; Kovaleva, Olga; Rumshisky, Anna (2020). "A Primer in BERTology: What we know about how BERT works". arXiv:2002.12327 [cs.CL].
^Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; Santos, Marco Dos; McAleer, Stephen; Jiang, Albert Q.; Deng, Jia; Biderman, Stella; Welleck, Sean (30 November 2023). "Llemma: An Open Language Model For Mathematics". arXiv:2310.10631 [cs.CL].