# Long short-term memory

A peephole LSTM block with input, output, and forget gates. The exit arrows from the ct node actually denote exit arrows from ct-1 except for the single right-to-left arrow. There are many other kinds of LSTMs as well.[1]

Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture (an artificial neural network) proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber[2] and further improved in 2000 by Felix Gers et al.[3] Like most RNNs, a LSTM network is universal in the sense that given enough network units it can compute anything a conventional computer can compute, provided it has the proper weight matrix, which may be viewed as its program. Unlike traditional RNNs, an LSTM network is well-suited to learn from experience to classify, process and predict time series when there are time lags of unknown size and bound between important events. Relative insensitivity to gap length gives an advantage to LSTM over alternative RNNs and hidden Markov models and other sequence learning methods in numerous applications. Among other successes, LSTM achieved the best known results in natural language text compression,[4] unsegmented connected handwriting recognition,[5] and in 2009 won the ICDAR handwriting competition. LSTM networks have also been used for automatic speech recognition, and were a major component of a network that in 2013 achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.[6] As of 2016, major technology companies including Google, Apple, Microsoft, and Baidu are using LSTM networks as fundamental components in new products.[7][8] For example, Google uses LSTM for speech recognition on the smartphone,[9][10] for the smart assistant Allo,[11] and for Google Translate.[12][13] Apple uses LSTM for the "Quicktype" function on the iPhone[14][15] and for Siri.[16] Amazon uses LSTM for Amazon Alexa.[17]

## ArchitectureEdit

A LSTM network is an artificial neural network that contains LSTM units instead of, or in addition to, other network units. A LSTM unit is a recurrent network unit that excels at remembering values for either long or short durations of time. The key to this ability is that it uses no activation function within its recurrent components. Thus, the stored value is not iteratively squashed over time, and the gradient or blame term does not tend to vanish when Backpropagation through time is applied to train it.

LSTM units are often implemented in "blocks" containing several LSTM units. This design is typical with "deep" multi-layered neural networks, and facilitates implementations with parallel hardware. In the equations below, each variable in lowercase italics represents a vector with a size equal to the number of LSTM units in the block.

LSTM blocks contain three or four "gates" that they use to control the flow of information into or out of their memory. These gates are implemented using the logistic function to compute a value between 0 and 1. Multiplication is applied with this value to partially allow or deny information to flow into or out of the memory. For example, an "input gate" controls the extent to which a new value flows into the memory. A "forget gate" controls the extent to which a value remains in memory. And, an "output gate" controls the extent to which the value in memory is used to compute the output activation of the block. (In some implementations, the input gate and forget gate are combined into a single gate. The intuition for combining them is that the time to forget is when a new value worth remembering becomes available.)

The only weights in a LSTM block (${\displaystyle W}$  and ${\displaystyle U}$ ) are used to direct the operation of the gates. These weights occur between the values that feed into the block (including the input vector ${\displaystyle x_{t}}$ , and the output from the previous time step ${\displaystyle h_{t-1}}$ ) and each of the gates. Thus, the LSTM block determines how to maintain its memory as a function of those values, and training its weights causes the LSTM block to learn the function that minimizes loss. LSTM blocks are usually trained with Backpropagation through time.

Initial values: ${\displaystyle c_{0}=0}$  and ${\displaystyle h_{0}=0}$ . The operator ${\displaystyle \circ }$  denotes the Hadamard product (entry-wise product).

{\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}

Variables

• ${\displaystyle x_{t}}$ : input vector
• ${\displaystyle h_{t}}$ : output vector
• ${\displaystyle c_{t}}$ : cell state vector
• ${\displaystyle W}$ , ${\displaystyle U}$  and ${\displaystyle b}$ : parameter matrices and vector
• ${\displaystyle f_{t}}$ , ${\displaystyle i_{t}}$  and ${\displaystyle o_{t}}$ : gate vectors
• ${\displaystyle f_{t}}$ : Forget gate vector. Weight of remembering old information.
• ${\displaystyle i_{t}}$ : Input gate vector. Weight of acquiring new information.
• ${\displaystyle o_{t}}$ : Output gate vector. Output candidate.

Activation functions

• ${\displaystyle \sigma _{g}}$ : The original is a sigmoid function.
• ${\displaystyle \sigma _{c}}$ : The original is a hyperbolic tangent.
• ${\displaystyle \sigma _{h}}$ : The original is a hyperbolic tangent, but the peephole LSTM paper suggests ${\displaystyle \sigma _{h}(x)=x}$ .[18][19]

### Peephole LSTMEdit

Peephole LSTM with forget gates.[18][19] ${\displaystyle h_{t-1}}$  is not used, ${\displaystyle c_{t-1}}$  is used instead in most places.

{\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}

### Convolutional LSTMEdit

Convolutional LSTM.[20] ${\displaystyle *}$  denotes the convolution operator.

{\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\circ c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\circ c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\circ c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}

## TrainingEdit

To minimize LSTM's total error on a set of training sequences, iterative gradient descent such as backpropagation through time can be used to change each weight in proportion to its derivative with respect to the error. A major problem with gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events, as first realized in 1991.[21][22] With LSTM blocks, however, when error values are back-propagated from the output, the error becomes trapped in the memory portion of the block. This is referred to as an "error carousel", which continuously feeds error back to each of the gates until they become trained to cut off the value. Thus, regular backpropagation is effective at training a LSTM block to remember values for very long durations.

LSTM can also be trained by a combination of artificial evolution for weights to the hidden units, and pseudo-inverse or support vector machines for weights to the output units.[23] In reinforcement learning applications LSTM can be trained by policy gradient methods, evolution strategies or genetic algorithms.

## ApplicationsEdit

Applications of LSTM include:

## Large scale industrial applications of LSTMEdit

• LSTM is the core technology for voice search and spoken commands in the Google App for smartphones (on Android and iOS), and for dictation on Android devices: Google Voice Transcription[9] and Google voice search[10]
• LSTM is used by Google's smart assistant Allo[11]

## ReferencesEdit

1. ^ Klaus Greff; Rupesh Kumar Srivastava; Jan Koutník; Bas R. Steunebrink; Jürgen Schmidhuber (2015). "LSTM: A Search Space Odyssey". arXiv:.
2. ^ a b Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276.
3. ^ a b Felix A. Gers; Jürgen Schmidhuber; Fred Cummins (2000). "Learning to Forget: Continual Prediction with LSTM". Neural Computation. 12 (10): 2451–2471. doi:10.1162/089976600300015015.
4. ^ "The Large Text Compression Benchmark". Retrieved 2017-01-13.
5. ^ A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009.
6. ^ a b Graves, Alex; Mohamed, Abdel-rahman; Hinton, Geoffrey (2013). "Speech Recognition with Deep Recurrent Neural Networks". Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on: 6645–6649.
7. ^ "With QuickType, Apple wants to do more than guess your next text. It wants to give you an AI.". WIRED. Retrieved 2016-06-16.
8. ^
10. ^ a b Google Research Blog. Google voice search: faster and more accurate. September 24, 2015. By Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays and Johan Schalkwyk – Google Speech Team http://googleresearch.blogspot.co.uk/2015/09/google-voice-search-faster-and-more.html
11. ^ a b Google Research Blog. Chat Smarter with Allo. May 18, 2016. By Pranav Khaitan, Google Research http://googleresearch.blogspot.co.at/2016/05/chat-smarter-with-allo.html
12. ^ a b Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (26 Sep 2016): Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean. https://arxiv.org/abs/1609.08144
13. ^ a b "An Infusion of AI Makes Google Translate More Powerful Than Ever." Cade Metz, WIRED, Date of Publication: 09.27.16. https://www.wired.com/2016/09/google-claims-ai-breakthrough-machine-translation/
14. ^ a b "Apple’s Machines Can Learn Too." By Amir Efrati, The Information, Jun. 13, 2016 https://www.theinformation.com/apples-machines-can-learn-too
15. ^ a b "iPhone, AI and big data: Here's how Apple plans to protect your privacy." By Steve Ranger, ZDNet, June 14, 2016. http://www.zdnet.com/article/ai-big-data-and-the-iphone-heres-how-apple-plans-to-protect-your-privacy
16. ^ a b "iOS 10: Siri now works in third-party apps, comes with extra AI features." Chris Smith, BGR, June 13th, 2016 http://bgr.com/2016/06/13/ios-10-siri-third-party-apps/
17. ^ a b "Bringing the Magic of Amazon AI and Alexa to Apps on AWS." All Things Distributed, by Werner Vogels on 30 November 2016: http://www.allthingsdistributed.com/2016/11/amazon-ai-and-alexa-for-all-aws-apps.html
18. ^ a b c Gers, F. A.; Schmidhuber, J. (2001). "LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages" (PDF). IEEE Transactions on Neural Networks. 12 (6): 1333–1340. doi:10.1109/72.963769.
19. ^ a b c Gers, F.; Schraudolph, N.; Schmidhuber, J. (2002). "Learning precise timing with LSTM recurrent networks" (PDF). Journal of Machine Learning Research. 3: 115–143.
20. ^ Xingjian Shi; Zhourong Chen; Hao Wang; Dit-Yan Yeung; Wai-kin Wong; Wang-chun Woo (2015). "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting". Proceedings of the 28th International Conference on Neural Information Processing Systems: 802–810.
21. ^ S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut f. Informatik, Technische Univ. Munich, 1991.
22. ^ S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
23. ^ Schmidhuber, J.; Wierstra, D.; Gagliolo, M.; Gomez, F. (2007). "Training Recurrent Networks by Evolino". Neural Computation. 19 (3): 757–779. doi:10.1162/neco.2007.19.3.757.
24. ^ H. Mayer, F. Gomez, D. Wierstra, I. Nagy, A. Knoll, and J. Schmidhuber. A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks. Advanced Robotics, 22/13–14, pp. 1521–1537, 2008.
25. ^ J. Schmidhuber and D. Wierstra and F. J. Gomez. Evolino: Hybrid Neuroevolution / Optimal Linear Search for Sequence Learning. Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, pp. 853–858, 2005.
26. ^ Graves, A.; Schmidhuber, J. (2005). "Framewise phoneme classification with bidirectional LSTM and other neural network architectures". Neural Networks. 18 (5–6): 602–610. doi:10.1016/j.neunet.2005.06.042.
27. ^ S. Fernandez, A. Graves, J. Schmidhuber. An application of recurrent neural networks to discriminative keyword spotting. Intl. Conf. on Artificial Neural Networks ICANN'07, 2007.
28. ^ D. Eck and J. Schmidhuber. Learning The Long-Term Structure of the Blues. In J. Dorronsoro, ed., Proceedings of Int. Conf. on Artificial Neural Networks ICANN'02, Madrid, pages 284–289, Springer, Berlin, 2002.
29. ^ Schmidhuber, J.; Gers, F.; Eck, D.; Schmidhuber, J.; Gers, F. (2002). "Learning nonregular languages: A comparison of simple recurrent networks and LSTM". Neural Computation. 14 (9): 2039–2041. doi:10.1162/089976602320263980.
30. ^ Perez-Ortiz, J. A.; Gers, F. A.; Eck, D.; Schmidhuber, J. (2003). "Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets". Neural Networks. 16 (2): 241–250. doi:10.1016/s0893-6080(02)00219-8.
31. ^ A. Graves, J. Schmidhuber. Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks. Advances in Neural Information Processing Systems 22, NIPS'22, pp 545–552, Vancouver, MIT Press, 2009.
32. ^ A. Graves, S. Fernandez,M. Liwicki, H. Bunke, J. Schmidhuber. Unconstrained online handwriting recognition with recurrent neural networks. Advances in Neural Information Processing Systems 21, NIPS'21, pp 577–584, 2008, MIT Press, Cambridge, MA, 2008.
33. ^ M. Baccouche, F. Mamalet, C Wolf, C. Garcia, A. Baskurt. Sequential Deep Learning for Human Action Recognition. 2nd International Workshop on Human Behavior Understanding (HBU), A.A. Salah, B. Lepri ed. Amsterdam, Netherlands. pp. 29–39. Lecture Notes in Computer Science 7065. Springer. 2011
34. ^ Hochreiter, S.; Heusel, M.; Obermayer, K. (2007). "Fast model-based protein homology detection without alignment". Bioinformatics. 23 (14): 1728–1736. doi:10.1093/bioinformatics/btm247. PMID 17488755.
35. ^ Thireou, T.; Reczko, M. (2007). "Bidirectional Long Short-Term Memory Networks for predicting the subcellular localization of eukaryotic proteins". IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB). 4 (3): 441–446. doi:10.1109/tcbb.2007.1015.