深度学习之门控RNN（LSTM）

LSTM

$$\begin{array}{rlrl}{\displaystyle {\mathit{g}}^{(t)}}& {\displaystyle =\sigma \text{\hspace{0.17em}⁣}({\mathit{b}}^g+{\mathit{W}}^g{\mathit{h}}^{(t-1)}+{\mathit{U}}^g{\mathit{x}}^{(t)})}& {\displaystyle }& {\displaystyle \text{(输入门)}}\\ {\displaystyle {\mathit{f}}^{(t)}}& {\displaystyle =\sigma \text{\hspace{0.17em}⁣}({\mathit{b}}^f+{\mathit{W}}^f{\mathit{h}}^{(t-1)}+{\mathit{U}}^f{\mathit{x}}^{(t)})}& {\displaystyle }& {\displaystyle \text{(遗忘门)}}\\ {\displaystyle {\mathit{q}}^{(t)}}& {\displaystyle =\sigma \text{\hspace{0.17em}⁣}({\mathit{b}}^q+{\mathit{W}}^q{\mathit{h}}^{(t-1)}+{\mathit{U}}^q{\mathit{x}}^{(t)})}& {\displaystyle }& {\displaystyle \text{(输出门)}}\\ {\displaystyle {\tilde{\mathit{s}}}^{(t)}}& {\displaystyle =\tanh \text{\hspace{0.17em}⁣}({\mathit{b}}^s+{\mathit{W}}^s{\mathit{h}}^{(t-1)}+{\mathit{U}}^s{\mathit{x}}^{(t)})}& {\displaystyle }& {\displaystyle \text{(候选状态)}}\\ {\displaystyle {\mathit{s}}^{(t)}}& {\displaystyle ={\mathit{f}}^{(t)}\odot {\mathit{s}}^{(t-1)}+{\mathit{g}}^{(t)}\odot {\tilde{\mathit{s}}}^{(t)}}& {\displaystyle }& {\displaystyle \text{(细胞状态更新)}}\\ {\displaystyle {\mathit{h}}^{(t)}}& {\displaystyle ={\mathit{q}}^{(t)}\odot \tanh \text{\hspace{0.17em}⁣}\left({\mathit{s}}^{(t)}\right)}& {\displaystyle }& {\displaystyle \text{(隐状态读出)}}\\ {\displaystyle {\mathit{o}}^{(t)}}& {\displaystyle ={\mathit{b}}^y+\mathit{V}{\mathit{h}}^{(t)}}\\ {\displaystyle {\widehat{\mathit{y}}}^{(t)}}& {\displaystyle =\mathrm{softmax}\text{\hspace{0.17em}⁣}\left({\mathit{o}}^{(t)}\right)}\end{array}$$

其中：

• 遗忘门控制长期记忆信息 ${\mathit{s}}^{(t-1)}$（cell state）的比例（实现“只记住有用的信息”）
• 输入门控制短期记忆（上一步的隐藏层输出）${\mathit{h}}^{(t-1)}$ 的比例
• 输出门控制隐藏层输出的比例（将“记住什么”和“输出什么”解耦）

如图：

图中绿色的线为短期记忆，蓝色的线是长期记忆。