2023-Q3-AI 7.Language models - LSTM

7.1. Video / Materials

Video (21 Jul 2023, 10:00): https://youtube.com/live/5WG6mJtQKLo?feature=share

Jamboard: https://jamboard.google.com/d/1525UGA3rOthYMUOr2hgAvtS7QHw3u5utk9B6NjPwtvs/edit?usp=sharing

Materials:
https://calvinfeng.gitbook.io/machine-learning-notebook/supervised-learning/recurrent-neural-network/recurrent_neural_networks/ https://lanwuwei.github.io/courses/SP19/3521_slides/11-Recurrent_Neural_Networks_2.pdf https://danijar.com/tips-for-training-recurrent-neural-networks/ https://medium.com/datadriveninvestor/attention-in-rnns-321fbcd64f05 https://arxiv.org/abs/1610.09513 https://wiki.pathmind.com/word2vec

Ir iedota pieeja jamboard un ar OBS jāveic screen streaming uz šādu setting

Youtube video key: h5bk-pucs-ft1w-e902-fcf9

Par katru uzdevumu dodam 100 punktus

Saturs:

RNN (shared weights)
Language modeling task
Embedding dict
[END] token
⚠️ [ANY] būtu labi pielikt token any pie visu vārdu vietā, kuri atkārtojas mazāk kā 3 reizes, lai saglabātu teikumu skaitu lielāku datu kopā
Train VS Inference many-to-one (inference), many-to-many (training)
Izstāstīt dažādu garumu teikumu apstrādi vienā batch

Iepriekšējā gada lekcija

Video https://youtu.be/-nuoRn1ohzI

Jamboard: https://jamboard.google.com/d/1nEQLzDVjXrK7RfkxifA9jlyVBLTA-x3vQPIFOYuQ1UU/edit?usp=sharing

7.2. Implement Vanilla RNN

Following the instructions in video 7.1., implement a Vanilla RNN without using built-in RNN functions from torch.nn.

Submit the source code and screenshots of the results.

Template:

http://share.yellowrobot.xyz/quick/2023-4-3-3F68F1D8-DF36-4D8C-BB56-F0407D2A6512.zip

7.3 Implement GRU

Using the template from task 7.1 and instructions from task 7.2, implement a GRU model. Replace the RNN cell with your own creation. You are not allowed to use built-in functions like torch.nn.GRU etc. Submit the source code and screenshots of the results.

GRU equation: http://share.yellowrobot.xyz/upic/8f34c76492d8b3a520255d023e962dc9_1680532330.jpg

7.4 Homework - Implement LSTM

Using the template from task 7.1 and instructions from task 7.2 and 7.3, modify the source code to implement the following:

Create LSTM. Replace the RNN cell with your own creation. You are not allowed to use built-in functions like torch.nn.LSTM etc.
Implement weight storage at the lowest test_loss value
Implement a separate script where you can load the model weights and the user can enter the beginning of a sentence from several words in the console and the model will predict the end of the sentence
Implement the built-in torch.nn.LSTM model and compare the results with your own model
Submit the code and screenshots of the training and rollout results

LSTM equation: http://share.yellowrobot.xyz/upic/70d53425be0fec7c7dc0ebb246b6fecb_1680532356.jpg