Distributed Training

Here are instructions for using distributed/multinode Training with Intel® Transfer Learning Tool.

Prerequisites

Participating nodes should have Intel® oneAPI Base Toolkit installed. Verify the files under /opt/intel/oneapi
Participating nodes should have passwordless SSH setup. Instructions to set up are given below.

Passwordless SSH setup

Use an existing (or create an) SSH key pair.
- Check under ~/.ssh and see if they exist. If present, make sure they have default names (id_rsa.pub id_rsa) and they don't have any passphrase.
- To remove passphrase, type ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile] by replacing new_passphrase with a blank space.
How to create SSH key pair:
- Get to your .ssh directory cd ~/.ssh (if this gives you an error, change the permissions: chmod u+x ~/.ssh)
- Run the command: ssh-keygen -t rsa
- The first prompt will ask you what you want to call your key files (id_rsa.pub id_rsa). Press <enter> to use the default key names.
- The second prompt will ask for passphrase. Do not enter any passphrase, just press <enter>.
Locate the two ssh key pair files in your .ssh directory (id_rsa.pub, id_rsa):
- Open the Public Key in an editor like vi/vim/nano/pico (this is the .pub file)
- The ending of the public key may say <your_idsid>@<hostname.domain>, edit this file to omit the "@<hostname.domain>" at the end. The result will be your idsid only.
- Create a file in your .ssh directory called authorized_keys
- Paste your entire public key into this file
- Make sure your new ssh key pair files AND authorized_keys files are read-write only for yourself with no permissions for anyone else (chmod 600 file1 file2 file3)
Test the SSH ssh <ip_or_hostname.domain>

IMPORTANT NOTE: You have to make sure the authorized_keys file exists on all of the target systems that will participate in running the workload (in your local home dir in your .ssh directory) with contents of public key inside as well.