ML - Recurrent Neural Network(RNN)

RNN

• Type of ANN in which a hidden node is connected to an edge with a direction to form a circulating structure.
• It is suitable for processing data that sequentially appears along the time axis, such as voice or characters.
• The I/O length is free.
• Structure can be made diversely and flexibly as needed.

FNN vs RNN

RNN considers pass result.

Basic Structure of RNN

 

From Vailla RNN to LSTM

• It's Activation function(tanh) causes Vanishing gradients thr process of Backpropagation

LSTM

σ means Sigmoid

Forget gate

Sequence-to-Sequence Model (Seq2Seq)

Typical Many-to-Many Encoder-Decoder Model.

Encoder abstracts inputs to one accumulated vector.

Decoder uses the accumulated vector that encoder produces to make outputs.

Attention

There are too many words that have similar meaning from input words.

So, To make decoder choices more relative words with encoder's inputs, Attention was created.

In its process, Inner product of vectors -> Softmax -> One-hot encoding are used.

모든 사진의 출처는 건국대학교 컴퓨터공학부 김학수 교수님의 강의자료 일부입니다.