RTU HPC usage Documentation - 2021-10-04

TORQUE 4.0 Resource management system (uzdevumi pa rindām pēc prioritātēm): https://docs.adaptivecomputing.com/torque/4-0-2/help.htm

Tutorial

Latvian Tutorial video: https://youtu.be/tr6y-w0tnm4

Jamboard 11. sesija: https://jamboard.google.com/d/1p_TZ1bdndv5HtCKqxzbHIEzuuSl0CFQ2GgBA7YXh5J0/edit?usp=sharing

Templates:

http://share.yellowrobot.xyz/1628158950-vea-rtu-course-2020-q1/tensorboard_utils.py http://share.yellowrobot.xyz/1628158950-vea-rtu-course-2020-q1/11_1_tensorboard_template.py http://share.yellowrobot.xyz/1628158950-vea-rtu-course-2020-q1/11_2_classification_template.py http://share.yellowrobot.xyz/1628158950-vea-rtu-course-2020-q1/11_3_hpc_template.sh

Manual: http://share.yellowrobot.xyz/1628158950-vea-rtu-course-2020-q1/rokasgramata_RTU.pdf

For data storage use folder (different than the one used in tutorial) /mnt/beegfs2/home/abstrac01/your_name

Examples how to automate task generation task: http://share.yellowrobot.xyz/1631791795-hpc/task_gen_example_hpc.zip

Available GPUs

Ir pieejams jauns aprēķinu mezgls ar 4 x NVIDIA A100 GPU (wn44): qsub -l feature=a100 (detalizētāk: https://hpc.rtu.lv/hpc/klastera-rokasgramata/klasteru-tehniskais-apraksts/#rudens ).
wn59 compute-gpu Centos 7 192 GB Gold 6130 32 4 /scratch dell vasara gpu v100 “wn60 compute-gpu Centos 7 192 GB Gold 6130 32 4 /scratch dell vasara gpu v100

Only 2 available GPU combinations

K40 - first try with this GPU


1
1
#PBS -l nodes=1:ppn=12:gpus=1:shared,feature=k40

V100


1
1
#PBS -l nodes=1:ppn=8:gpus=1:shared,feature=v100

Script that works for V100 and K40


15
1
#!/bin/sh -v
2
#PBS -e /mnt/home/abstrac01/logs
3
#PBS -o /mnt/home/abstrac01/logs
4
#PBS -q batch
5
#PBS -p 1000
6
#PBS -l nodes=1:ppn=12:gpus=1,feature=k40
7
#PBS -l mem=40gb
8
#PBS -l walltime=96:00:00
9

10
eval "$(conda shell.bash hook)"
11
conda activate conda_env
12
export LD_LIBRARY_PATH=/mnt/home/abstrac01/.conda/envs/conda_env/lib:$LD_LIBRARY_PATH
13

14
cd /your/task/dir
15
python taskgen.py -learning_rate 1e-4 1e-3

Trouble shooting

When setting up conda environment if you get "no space left" disk error, use


1
1
pip3 --cache-dir /home/tmp

When installing something very important to use "pip3" not "pip" Must check that last entry is loaded from current environment


2
1
whereis pip
2
which -a pip

If you get missing libs errors, need to link specific verrsion in conda_env


2
1
export LD\_LIBRARY\_PATH=~/.conda/envs/conda\_env/lib:$LD\_LIBRARY\_PATH
2
ln -s ~/.conda/envs/conda\_en/lib/libstdc++.so.6.0.28 ~/.conda/envs/conda\_env/lib/libstdc++.so.6

Simple example

Tasks must be run with flag:


1
1
qsub script.sh -A ditf_ldi

Example run.sh (notice with and signs at the end run will execute in parallel, do not forget at the end wait command) Also very important that you do not waste resources - try to run single run and look into showq, then see if your run is exectuing for example on wn60, then type "ssh wn60" then inside that node "nvidia-smi" to look how much GPU resources are you using - I expect that you could run 16 runs in parallel. I also advise you to make automatic script generator so you do not have to type in all of hyper parameters by hand.


21
1
#!/bin/sh -v
2
#PBS -e /mnt/home/abstrac01/evalds\_urtans/logs
3
#PBS -o /mnt/home/abstrac01/evalds\_urtans/logs
4
#PBS -q batch
5
#PBS -l nodes=1:ppn=16:gpus=2:shared,feature=v100
6
#PBS -l mem=40gb
7
#PBS -l walltime=10:00:00
8

9
module load conda
10
eval "$(conda shell.bash hook)"
11
conda activate conda\_env
12

13
export TEMP=$HOME/tmp
14
export TMPDIR=$HOME/tmp
15
export LD\_LIBRARY\_PATH=/mnt/home/abstrac01/.conda/envs/conda\_env/lib:$LD\_LIBRARY\_PATH
16

17
cd /mnt/home/abstrac01/evalds\_urtans
18
python ./11\_2\_classification\_finished\_tmp.py -learning\_rate 1e-3 -is\_cuda True &
19
python ./11\_2\_classification\_finished\_tmp.py -learning\_rate 1e-4 -is\_cuda True &
20
python ./11\_2\_classification\_finished\_tmp.py -learning\_rate 1e-5 -is\_cuda True
21
wait

Execute it by using:


xxxxxxxxxx
1
1
qsub ./run.sh

Credits left


xxxxxxxxxx
1
1
mam-balance

Check status


xxxxxxxxxx
1
1
qstat -a ditf_ldi

HPC commands

To see all tasks and nodes operating
And also usage


xxxxxxxxxx
1
1
showq -r

our own jobs


xxxxxxxxxx
3
1
qstat 
2
qsub ./script.sh
3
qdel ID-from-qstat

command to activate conda


xxxxxxxxxx
2
1
module load conda
2
module load cuda

Before executing scripts you need to create your own conda environment


xxxxxxxxxx
5
1
conda create -n conda_env
2
conda activate conda_env
3
conda list
4
conda install -c anaconda numpy
5
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

GPU settings (do not use other combinations)

nodes=1:ppn=12:gpus=1,feature=k40
^ use this one

nodes=1:ppn=8:gpus=1,feature=v100

CPU settings for testing
nodes=1:ppn=1
nodes=1:ppn=8

Very simple script