 | This article may be too technical for most readers to understand. (October 2021) |
Modern Hopfield networks[1][2] (also known as Dense Associative Memories[3]) are generalizations of the classical Hopfield networks that break the linear scaling relationship between the number of input features and the number of stored memories. This is achieved by introducing stronger non-linearities (either in the energy function or neurons’ activation functions) leading to super-linear[3] (even an exponential[4]) memory storage capacity as a function of the number of feature neurons. The network still requires a sufficient number of hidden neurons.[5]
The key theoretical idea behind the modern Hopfield networks is to use an energy function and an update rule that is more sharply peaked around the stored memories in the space of neuron’s configurations compared to the classical Hopfield network.[3]
- ^ Ramsauer, Hubert; et al. (2021). "Hopfield Networks is All You Need". International Conference on Learning Representations. arXiv:2008.02217.
- ^ "Hopfield Networks is All You Need". hopfield-layers. 2020-08-25. Archived from the original on 26 Mar 2023. Retrieved 2023-05-04.
- ^ a b c Krotov, Dmitry; Hopfield, John (2016). "Dense Associative Memory for Pattern Recognition". Neural Information Processing Systems. 29: 1172–1180. arXiv:1606.01164.
- ^ Demircigil, Mete; et al. (2017). "On a model of associative memory with huge storage capacity". Journal of Statistical Physics. 168 (2): 288–299. arXiv:1702.01929. Bibcode:2017JSP...168..288D. doi:10.1007/s10955-017-1806-y. S2CID 119317128.
- ^ Krotov, Dmitry; Hopfield, John (2021). "Large associative memory problem in neurobiology and machine learning". International Conference on Learning Representations. arXiv:2008.06996.