TACC#
Installing and Setting Up for TACC#
You will need to create a new TACC user account. Once you are granted an account you will need Dr. Kockelman to add you to one of the projects (allocations). After this step, you will be able to access your account on any of the supercomputers the team has allocations on. You can access a login node via the following three programs:
Figure xx shows how to connect to the Lonestar6 allocation with your TACC login credentials. TACC requires two-step authorization, and you will be asked to provide a 6-digit number from a mobile app. Figure xx shows the highlighted button that is accessible only after downloading PuTTY. This connection provides command line access and is highly inaccessible for beginner programmers. It is recommended you use WinSCP as this set-up provides an easy side-by-side screen between your local machine and the TACC folder system. If you are cloning and building dependencies for the first time, you might take advantage of this dual screen feature to ensure LINUX mimics Windows. Second, you can upload and download files to and from TACC with this side-by-side window feature. Figure xx shows the process of submitting a DCV remote desktop session, which will count against the allocations the team has. If you prefer, you can submit a VNC remote desktop version, but would require another program like TigerVNC. Regardless, once the job is submitted and running you will be provided with a session ID (e.g., ls6.tacc.utexas.edu:37806) and will need to use this and your TACC credentials to visually access your TACC folder structure.
Build POLARIS in TACC (including getting depedencies from Murthy’s $WORK folder)#
Before starting this step, we encourage you to explore the user guide of the resource you have access to TACC Documentation Website. There are three file systems: $HOME, $WORK, and $SCRATCH. Figure xx provides an overview of the key features of all three. $HOME and $WORK files are not purged but only $HOME is regularly backed up. Most POLARIS regional inputs are greater than 10 GB, so we do not use the $HOME file structure. You will need to request an Argonne Gitlab Account to access the online repository through this website. We will be cloning the Gitlab repository and accessing the dependency folder in $WORK. As $SCRATCH is purged frequently, we do not recommend storing input or output data in this file structure for more than 10 days at a time.
Create a folder called “Polaris_code” in your $WORK folder and clone the polaris-linux repository into it. To do this, you will need to use the command “git clone [address]” in the PuTTY terminal when you are inside the Polaris folder (check syntax once). If you are in a DCV/VNC session, just right click to open the terminal in a specific folder. You will need to use your Gitlab Access Token (not your password) when performing any Git action.
After cloning the git repository located at the address into the current directory in the PuTTY terminal, change the current directory (“Polaris_code”) to the cloned directory (“polaris-linux”).
When you clone a repository, Git will automatically check out the default branch for that repository, which is named ‘develop’. This default branch is set by the repository owner and may contain the stable or production version of the project. However, development often happens in other branches. If you want to work with or see the latest changes in a different branch, you need to switch to that branch using the git checkout command. For a remote branch, you may need to fetch the branch first using git fetch before you can check it out, if that branch hasn’t been tracked locally yet.
To switch to the remote branch named tracking-tnc from your current branch (in this case develop), you need to fetch the remote branches and then create and switch to a new local branch that tracks the remote branch. When you create a new local branch that tracks a remote branch using the command git checkout -b <local_branch_name> origin/<remote_branch_name>, Git will automatically pull the latest changes from the remote branch into your new local branch. So you do not need to manually pull from the remote branch immediately after creating your new local branch.
# Fetch all remote branches
git fetch origin
# Create and switch to a new local copy that tracks the remote one
git checkout -b <local_branch_name> origin/<remote_branch_name>
However, if others are also working on the remote branch and pushing changes to it, you will need to periodically pull these changes into your local branch to keep it up to date. You can do this using the git pull command:
git pull origin <remote_branch_name>
Remember, before you pull, make sure you have committed any changes you have made in your local branch, as the git pull command could potentially overwrite uncommitted changes.
Now that you have pulled the latest changes from the remote branch into your new local branch, run the following commands on the terminal from inside the deps folder
module load gcc
module load python3
module load cmake
Check to see what modules are loaded by running the command “module list” in any terminal window. You should see the following modules:
gcc/11.2.0
mvapich2/2.3.7
python3/3.9.7
cmake/3.24.2
We can build POLARIS in an interactive python environment under the cloned polaris-linux directory (still in the PuTTY terminal) by running the following command:
python build.py -i
This will prompt you to specify the folder with dependencies, which are located in the following directory:
/work/04579/gkmurthy/POLARIS_OPT/
If you come across an error/warning related to patchelf, you need to use the following command to configure and build again in the PuTTY terminal:
python build.py -cb --patchelf=n
After successfully building, your Integrated Model executable will be located in a bin folder whose directory you can trace from the figure inserted as you run the “python build.py -i” command. This bin file can then be copied to your $SCRATCH folder where you will run your scenarios. You could copy it manually, or use the following command:
cp -r build/linux/gcc/release/bin $SCRATCH/folder_name
“build/linux/gcc/release/bin” is the path inside the cloned repository directory to the folder containing executable you just built. Essentially the full path would be:
/work/08773/payebare/Polaris_code/polaris-linux/build/linux/gcc/release/bin $SCRATCH/folder_name
The build folder is created after running the “python build.py -i” command, even if it does not build successfully.
Running scenarios on TACC#
We recommend using the WinSCP program to open up a console to run your .slurm files. Figure xx shows how to run a .slurm file, which is the TACC method of running a scenario file. The UT Austin team can share an example slurm script that contains the scenario.json file and other necessary TACC information like job and nodes. New users are encouraged to read TACC’s user guide for more details. The command to run a .slurm file is “sbatch [name].slurm”. To check on the status of the job, use “showq -u”. If you need to cancel a job, just use “scancel job#” where the job number comes from the console terminal window. You can see your priority by using “squeue – u username.” To see the progress log as opposed to opening & re-loading the file, just open the command line in the log folder and type “tail -f polaris_progress.log”