View source for UMIACS Servers

Notes on using UMIACS servers


==Modules==
Use modules to load programs you need to run.<br>
;Notes
* You can load modules in your <code>.bashrc</code> file

<syntaxhighlight lang="bash">

# List loaded modules
module list

# Load a module
module load [my_module]

# List all available modules
module avail
</syntaxhighlight>

Some useful modules in my <code>.bashrc</code> file
<syntaxhighlight lang="bash">
module load tmux
module load cuda/10.0.130
module load cudnn/v7.5.0
module load Python3/3.7.6
module load git
</syntaxhighlight>

==Python==
Do not install anaconda. You will run out of space.<br>
Load the Python 3 module adding the following to your .bashrc file
<syntaxhighlight lang="bash">
module load Python3/3.7.6
export PATH="${PATH}:$(python3 -c 'import site; print(site.USER_BASE)')/bin"
</syntaxhighlight>

Then run the following to get pip installed
<syntaxhighlight lang="bash">
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py --user
</syntaxhighlight>

;Notes
* You will need to install things with <code>pip --user</code>
* You may need to add your local site-packages to your PYTHONPATH environment variable
** Add this to .bashrc: 
** <code>export PYTHONPATH="${PYTHONPATH}:/nfshomes/$(whoami)/.local/lib/python3.7/site-packages/"</code>
* You can also install using <code>pip install --target=/my-libs-folder/</code>

===Install PyTorch===
<syntaxhighlight lang="bash">
pip install --user torch===1.3.1 torchvision===0.4.2 -f https://download.pytorch.org/whl/torch_stable.html
</syntaxhighlight>

===Installing Packages to a Directory===
<syntaxhighlight lang="bash">
pip install geographiclib -t /scratch1/davidli/python/
</syntaxhighlight>

==MBRC Cluster==
See [https://wiki.umiacs.umd.edu/umiacs/index.php/MBRC UMIACS MBRC]<br>

===SLURM Job Management===
See [https://docs.rc.fas.harvard.edu/kb/convenient-slurm-commands/ https://docs.rc.fas.harvard.edu/kb/convenient-slurm-commands/]<br>

; 1 GPU
<pre>
srun --pty --gres=gpu:1 --mem=16G --qos=high --time=47:59:00 -w mbrc00 bash
</pre>

; 2 GPUS mbrc00
<pre contenteditable="true">
srun --pty --gres=gpu:2 --mem=16G --qos=default --time=23:59:00 -w mbrc00 bash
</pre>

; CPU-only on default QOS
<pre>
srun --pty --mem=16G --qos=default --time=23:59:00 -w mbrc00 bash
</pre>

;Notes
* You can add <code>-w mbrc01</code> to pick mbrc01
* <code>-c 4</code> for 4 cores

====See Jobs====
; See my own jobs
<pre>
squeue -u <user> -o "%8i %10P %8j %10u %10L %5b"
</pre>
; Formatting
* <code>%L</code> is remaining time
* <code>%b</code> is the number of GPUs

; See all jobs
<pre>
squeue
</pre>

===SFTP===
Note: If you know of an easier way, please tell me.

On your PC  
Start an sshd for forwarding. You can do this in a docker container for privacy purposes.

On the cluster:  
Generate an sshd host key:
<pre>
ssh-keygen -t ed25519 -a 100 -f /nfshomes/dli7319/ssh/ssh_host_ed25519_key
</pre>
Create the following <code>sshd_config</code> file
<pre>
#	$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
Port 5981
HostKey /nfshomes/dli7319/ssh/ssh_host_ed25519_key
AuthorizedKeysFile	.ssh/authorized_keys
Subsystem	sftp	/usr/libexec/openssh/sftp-server
</pre>
Start the sshd daemon and proxy the port to your local sshd.
You can make a script like this:
<pre>
#!/bin/bash

LOCAL_PORT=5981
REMOTE_PORT=22350
REMOTE_SSH_PORT=22450
REMOTE_ADDR=192.168.78.34

/usr/sbin/sshd -D -f sshd_config & \
ssh -R $REMOTE_PORT:localhost:$LOCAL_PORT root@$REMOTE_ADDR -p $REMOTE_SSH_PORT 
</pre>

On your PC:  
Proxy the sshd from the local docker to your localhost.  
Connect to the the sshd on the cluster

==<code>.bashrc</code>==
{{hidden | My .bashrc |
<syntaxhighlight lang="bash">
#PS1='\w$ '
PS1='\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$'

if test -f "/opt/rh/rh-php72/enable"; then
    source /opt/rh/rh-php72/enable
fi

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

command_exists() {
  type "$1" &> /dev/null ;
}


# Modules
if command_exists module ; then
  module load tmux
  module load cuda/10.1.243
  module load cudnn/v7.6.5
  module load Python3/3.7.6
  module load git/2.25.1
  module load gitlfs
  module load gcc/8.1.0
  #module load gcc/6.3.0
  module load ffmpeg

fi
if command_exists python3 ; then
  alias python=python3
fi

if command_exists python3 ; then
  export PATH="${PATH}:$(python3 -c 'import site; print(site.USER_BASE)')/bin"
fi
export PYTHONPATH="${PYTHONPATH}:/nfshomes/dli7319/.local/lib/python3.7/site-packages/"

export PATH="${HOME}/bin/:${PATH}"
</syntaxhighlight>
}}

==Software==
===git===
The MBRC cluster has an git available in the modules.<br>
Then you can download [https://github.com/git-lfs/git-lfs/releases git-lfs compiled] and drop it in <code>~/bin/</code>.<br>
Make sure <code>${HOME}/bin</code> is in your path and run <code>git lfs install</code><br>

;Notes
* Make sure you have a recent version of git
** E.g. <code>module load git/2.25.1</code>

==Copying Files==
There are 3 ways that I copy files to the scratch drives:
* For small files, you can copy to your home directory under <code>/nfshomes/</code> via SFTP to mbrcsub00. I rarely do this because the home directory is only a few gigs.
* For large files, I typically use [[rclone]] to copy to my terpmail Google Drive and then copy back to the scratch drives with a cpu-only job. Do not do this with thousands of small files; it will take forever since Google Drive has a limit on files per second. Also note that Google Drive has a daily limit of 750GB in transfers.
* For mounting, I have a convoluted system where I start SSHD in a job and port forward the SSH port to my local PC. See above for more details.