2023-Q4-AI-EN 12.Time-Series, Recurrent Neural Networks, RNN, LSTM

12.1. Video / Materials

Video: https://youtube.com/live/xpLDWDdqGQ4?feature=share

Jamboard: https://jamboard.google.com/d/16pII0AOesnL8Igf27idEVNdpu4xZuVYgRcx169TDfs0/edit?usp=sharing

Materials:
https://calvinfeng.gitbook.io/machine-learning-notebook/supervised-learning/recurrent-neural-network/recurrent_neural_networks/ https://lanwuwei.github.io/courses/SP19/3521_slides/11-Recurrent_Neural_Networks_2.pdf https://danijar.com/tips-for-training-recurrent-neural-networks/ https://medium.com/datadriveninvestor/attention-in-rnns-321fbcd64f05 https://arxiv.org/abs/1610.09513 https://wiki.pathmind.com/word2vec

Jamboard given rights to: amir.zkn85@gmail.com Youtube OBS stream key: 2jfa-fqm7-g3sz-kaad-b8ys

Finished source code: http://share.yellowrobot.xyz/quick/2023-11-26-DB83680C-31DB-41B5-8DB7-46071B507C22.zip

Previous Year lecture (English): https://youtube.com/live/3tMxReZ7wrQ

Jamboard (previous): https://jamboard.google.com/d/1ryRdI08rThos-KY-2aV2jLxL8T0wt808dAc3V2iRR_A/edit?usp=sharing

Content:

RNN (shared weights)
Language modeling task
Embedding dict
[END] token
[ANY] It would be good to add the any token instead of all words that repeat less than 3 times, to keep the sentence count larger in the data set
Train VS Inference many-to-one (inference), many-to-many (training)
Explain the processing of different length sentences in one batch

12.2. Implement Vanilla RNN

Following the instructions in video 12.1, implement a Vanilla RNN, without using built-in RNN functions from torch.nn.

Submit the source code and screenshots with the results.

Template: http://share.yellowrobot.xyz/quick/2023-4-3-3F68F1D8-DF36-4D8C-BB56-F0407D2A6512.zip

12.3 Implement GRU

Using the template from task 12.1 and the directions from task 12.2, implement a GRU model. Replace the RNN cell with your own creation. You must not use built-in torch.nn.GRU etc. Submit the source code and screenshots with the results.

GRU equation: http://share.yellowrobot.xyz/upic/8f34c76492d8b3a520255d023e962dc9_1680532330.jpg

12.4 Homework - Implement LSTM

Using the template from task 12.1 and the directions from tasks 12.2 and 12.3, modify the source code so that the following is implemented:

Create an LSTM. Replace the RNN cell with your own creation. You must not use built-in torch.nn.LSTM etc.
Implement weight saving at the lowest test_loss value
Implement a separate script where you can load the model weights and the user can enter the beginning of a sentence from several words and the model will predict the end of the sentence
Implement the built-in torch.nn.LSTM model and compare the results with your own model
Submit the code and screenshots with training and rollout results

LSTM equation: http://share.yellowrobot.xyz/upic/70d53425be0fec7c7dc0ebb246b6fecb_1680532356.jpg