PaLM

PaLM
Developer(s)Google AI
PredecessorLaMDA
SuccessorGoogle Gemini
Available inEnglish
TypeLarge language model
Websiteai.google/discover/palm2/ Edit this on Wikidata

PaLM (Pathways Language Model) is a 540 billion-parameter transformer-based large language model (LLM) developed by Google AI.[1] Researchers also trained smaller versions of PaLM (with 8 and 62 billion parameters) to test the effects of model scale.[2]

PaLM is capable of a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation.[2][3][4][5] When combined with chain-of-thought prompting, PaLM achieved significantly better performance on datasets requiring reasoning of multiple steps, such as word problems and logic-based questions.[1][2]

The model was first announced in April 2022 and remained private until March 2023, when Google launched an API for PaLM and several other technologies.[6] The API was initially available to a limited number of developers who joined a waitlist before it was released to the public.[7]

Google and DeepMind developed a version of PaLM 540B (the parameter count, 540 billion), called Med-PaLM, that is fine-tuned on medical data and outperforms previous models on medical question answering benchmarks.[8][9] Med-PaLM was the first to obtain a passing score on U.S. medical licensing questions, and in addition to answering both multiple choice and open-ended questions accurately, it also provides reasoning and is able to evaluate its own responses.[10]

Google also extended PaLM using a vision transformer to create PaLM-E, a state-of-the-art vision-language model that can be used for robotic manipulation.[11][12] The model can perform tasks in robotics competitively without the need for retraining or fine-tuning.[13]

In May 2023, Google announced PaLM 2 at the annual Google I/O keynote.[14] PaLM 2 is reported to be a 340 billion-parameter model trained on 3.6 trillion tokens.[15]

In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.[16]

  1. ^ a b Narang, Sharan; Chowdhery, Aakanksha. "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance". ai.googleblog.com. Retrieved 17 March 2023.
  2. ^ a b c Chowdhery, Aakanksha; Narang, Sharan; Devlin, Jacob; et al. (2022). "PaLM: Scaling Language Modeling with Pathways". arXiv:2204.02311 [cs.CL].
  3. ^ Cite error: The named reference venturebeat was invoked but never defined (see the help page).
  4. ^ Bastian, Matthias (5 April 2022). "Google PaLM: Giant language AI can explain jokes". the decoder. Retrieved 17 March 2023.
  5. ^ "Google: Why Is No One Talking About PaLM". seekingalpha.com. 12 December 2022. Retrieved 17 March 2023.
  6. ^ Vincent, James (14 March 2023). "Google opens up its AI language model PaLM to challenge OpenAI and GPT-3". The Verge. Retrieved 17 March 2023.
  7. ^ Huffman, Scott; Woodward, Josh. "PaLM API & MakerSuite: an approachable way to start prototyping and building generative AI applications". Retrieved 17 March 2023.
  8. ^ Singhal, Karan; Azizi, Shekoofeh; Tu, Tao; et al. (2022). "Large Language Models Encode Clinical Knowledge". arXiv:2212.13138 [cs.CL].
  9. ^ "MedPaLM: New Chatbots Will Soon Be Better Than Waiting For A Doctor". The Medical Futurist. 17 January 2023. Retrieved 17 March 2023.
  10. ^ Matias, Yossi; Corrado, Greg (14 March 2023). "Our latest health AI research updates". Google. Retrieved 17 March 2023.
  11. ^ Driess, Danny; Xia, Fei; Sajjadi, Mehdi S. M.; et al. (2023). "PaLM-E: An Embodied Multimodal Language Model". arXiv:2303.03378 [cs.LG].
  12. ^ Driess, Danny; Florence, Pete. "PaLM-E: An embodied multimodal language model". ai.googleblog.com. Retrieved 17 March 2023.
  13. ^ Edwards, Benj (7 March 2023). "Google's PaLM-E is a generalist robot brain that takes commands". Ars Technica. Retrieved 17 March 2023.
  14. ^ Lardinois, Frederic (May 10, 2023). "Google launches PaLM 2, its next-gen large language model". TechCrunch. Archived from the original on May 10, 2023. Retrieved May 10, 2023.
  15. ^ Elias, Jennifer (16 May 2023). "Google's newest A.I. model uses nearly five times more text data for training than its predecessor". CNBC. Retrieved 18 May 2023.
  16. ^ "AudioPaLM". google-research.github.io. Retrieved 2023-06-30.