1. Use the following command to connect to PACE via ssh. This will ask for a password. Use same you used to login into canvas, BuzzPort, etc.

ssh <gtid>@login-<cluster>.pace.gatech.edu

# Example:
ssh mandrade9@login-ice.pace.gatech.edu
Copy

If you are not familiar with ssh and want to learn more, there are plenty of good tutorials on the web (example)

You can proceed to (2) for the slurm workflow. The following two sections are optional (though highly recommended)

Follow the steps below to avoid entering a password every time you run the ssh command or connect via VSCode ssh extension. These steps should work for Linux, MAC and Windows (use PowerShell or the Terminal app).

1. Generate an SSH key. This will ask for a location, passphrase, etc. You can just press enter; no need to input anything.

ssh-keygen -t rsa
Copy

The following two steps will ask for a password; use the same you use in Buzzport.

2. Create a directory in the remote machine to hold ssh keys

ssh mandrade9@login-ice.pace.gatech.edu mkdir -p .ssh
Copy

3. Append the ssh key from local machine to the remote machine authorized keys

cat .ssh/id_rsa.pub | ssh mandrade@mandrade9@login-ice.pace.gatech.edu 'cat >> .ssh/authorized_keys'
Copy

Connecting to PACE via VSCode is recommended if you use the IDE. It gives an experience similar as if you were coding in your local machine, and provides the convenient features from VSCode such as code highlighting, a visual file explorer, integrated terminal sessions pointing to multiple nodes, etc.

NOTE: before connecting via VSCode, ensure you are connected to GT VPN.

To connect with VSCode:

1. Search for ssh in the extensions window and download this extension:

2. Open the command pallet (windows: ctrl+shift+p, MAC command+shif+p) and type ssh. Select the option Remote-SSH:Connect to Host...

3. Select Add New SSH Host... then type the same command you would use in the terminal: ssh mandrade9@login-ice.pace.gatech.edu

4. Enter your password (the same you use for canvas, buzzport, etc)

5. A new VSCode window like below should appear. The ssh: login-ice.pace.gatech.edu in the left corner indicates you are connected to PACE.

The next time you try to connect as in (2), you will see something like below. That is, you don't need to enter ssh ... again. VSCode saves the ssh configurations for you.

(to check/change ssh configurations: command pallet + Remote-SSH:Open SSH Configuration File)

Use pace-quota to find the location of your scratch directory

[mandrade9@login-ice-3 ~]$ pace-quota

# Output:
# Filesystem                                             Usage (GB)    Limit
# Home:/home/hice1/mandrade9                                   21.3     30.0  70.9%
# Scratch:/storage/ice1/9/2/mandrade9                          50.7    300.0  16.9%
Copy

You can create a scratch folder in your home directory and link it to the path above.

mkdir ~/scratch 
ln -s /storage/ice1/9/2/mandrade9 ~/scratch
Copy

Now use ~scratch or /storage/ice1/9/2/mandrade9 to store big files and install python environments.

When you log into PACE, you will receive a terminal pointing to a "login node", like below:

~ ssh mandrade9@login-ice.pace.gatech.edu
[mandrade9@login-ice-3 ~]$
Copy

The login node has little not computing power and is accessed by many people at once. You should not execute heavy programs on it, or else performance for all users may be affected!

To run heavy programs, you should request a computing node, like below:

salloc -N1 -t0:15:00 --cpus-per-task 8 --ntasks-per-node=1 --gres=gpu:V100:2 --mem-per-gpu=32G
Copy

In this example, I requested:

• one computing node (-N1 )
• for 15 minutes (-t0:15:00)
• With 8 CPU cores (--cpus-per-task)
• With two V100 GPUs (--gres=gpu:V100:2), with 32GB of memory each (--mem-per-gpu=32G)

Notes:

• check hardware available to request here; details on slurm options here,here.
• For simpler use cases, you will mostly tweak the GPU and CPU specs relative to the command above.
• For more compute-intense jobs, you may want a distributed and multi-GPU solution, which requires more tweaking. Check this guide on PyTorch DDP with slurm.

The terminal should point to a computing node now, like below.

This means you have access to a machine with the specs above. Executing nvidia-smi you can confirm details of the GPUs provided:

The computing nodes initializes with the bare minimum of software. You have to load them to do any work.

PACE has commonly used programs pre-installed. You can "load" these modules via the command:

module load <module_name>
Copy

To find which software is available and their identifiers, use the spider command:

module spider <partial_module_name>
Copy

For example, to see which versions of anaconda are available:

This tells that I can load anaconda via:

module load anaconda3/2022.05.0.1
Copy

NOTE: software versions are not the same among PACE nodes.

• For instance, if you request a A100 GPU you may receive a node with only conda versions 2023+
• This gives a little headache when using different nodes, which is not so rare, as not always the same nodes are available.
• For anaconda in particular, I tend to create one environment for each node to avoid problems.

1. Load the anaconda module

module load anaconda3/2022.05.0.1
Copy

2. Create the virtual environment in your scratch directory and activate it.

Obs.: Do not continue if you don't know what is the scratch directory. Go to (2) first.

conda create --prefix ~/scratch/<envName> python=<3.x>
conda activate ~/scratch/<envName>
Copy

All environment files will be saved in ~/scratch/<envName> instead of the default directory ~/.conda

Alternatively, you can set as conda default to store packages in specific directories inside scratch with the commands below. config will add the paths to the ~./condarc file and you don't need to run it again.

# Change conda default path to save env files
conda config --add envs_dirs ~/scratch/conda_venvs
conda config --add pkgs_dirs ~/scratch/conda_pkgs

# Create environment
conda create -n vlm python=3.11 
conda activate vlm
Copy

3. Install your packages.

Typical packages

pip3 install numpy
Copy

For PyTorch, there is one additional step:

(i) Check and load CUDA module:

module spider cuda 
# Output: [...]
# Versions: # cuda/11.8.0 # cuda/12.1.1 
module load cuda/11.8.0
Copy

(ii) Check Pytorch's website for the appropriate command. In this case (Linux, Pip, CUDA11.8):

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Copy

NOTES

(i) You can confirm CUDA is loaded and it's version with nvcc --version

nvcc --version
# nvcc: NVIDIA (R) Cuda compiler driver
# Copyright (c) 2005-2022 NVIDIA Corporation
# Built on Wed_Sep_21_10:33:58_PDT_2022
# Cuda compilation tools, release 11.8, V11.8.89
# Build cuda_11.8.r11.8/compiler.31833905_0
Copy

• You can also use Python to check CUDA and GPU availability:

import torch
torch.cuda.is_available() # True if any CUDA device ready for use
# > True
torch.cuda.device_count() # Number of CUDA devices ready for use
# > 2
Copy

(ii) I tend to use pip3; using conda in slurm caused bugs for me in the past.

(iii) It is good to check if pip is pointing to scratch folder with which pip3 before installing anything. If it is not, check the troubleshooting section below

$ which pip3
~/scratch/VLM_project/bin/pip
Copy

• click on files -> home directory and upload your local files with the upload button.

Terminal interface

1. On the dashboard, click on clusters -> ICE Shell Access

Desktop interface
This provides a desktop-like view with interactive access to the computing node resources (like salloc1 but with a desktop too)

1. On the dashboard, click on clusters -> interactive apps -> interactive Desktop
2. Choose the cluster (ICE), and hardware desired for the computing node
3. Click on launch

This provides a VSCode session with interactive access to the computing node resources (similar to the ssh via VSCode as in (1))

1. On the dashboard, click on clusters -> interactive apps -> VS Code
2. Choose the cluster (ICE), hardware desired for the computing node, and the modules to load.
3. Click on launch