The main compute platform for this class is Stampede, which is located at TACC, The Texas Advanced Computing Center. Stampede contains over 6400 Dell PowerEdge server nodes each with 2 Intel Xeon E5 (Sandy Bridge) 8-core processors and an Intel Xeon Phi Co-processor (MIC Architecture). In addition, 128 of the compute nodes contain a single NVIDIA Tesla K20 GPU (“Kepler,” hardware capability 3.5, currently running CUDA 5.5) The nodes are interconnected by FDR Infiniband. Note that 40 of the nodes are available for short duration, interactive use, which comes in handy when debugging. Here is a handy description of the hardware’s capablities:https://www.microway.com/hpc-tech-tips/nvidia-tesla-k20-gpu-accelerator-kepler-gk110-up-close
The authoritative source of information for accessing and using Stampede is the The Stampede User’s Guide, which is maintained by TACC. If you have questions, post to the Stampede Piazza Folder.
To access Stampede, ssh to the front end, stampede.tacc.utexas.edu, using your assigned XSEDE credentials.
We’ll be using the bash shell. You may change shells if you wish, however, all instructions will be given in terms of the bash shell (It is not difficult to convert between the two, so post to the Stampede folder on Piazza if you have aliases to donate)
Login to the front end and. Next, modify .bashrc, which is located in your home directory. In particular, insert the file BASHRC, located in /home1/00660/tg458182/cse262-wi15/Profiles into your .bashrc file, immediately after the following text appearing in .bashrc:
#####################################################################
# **** Place any else below. ****
#####################################################################You will now have various handy command aliases at your disposal. You’ll also have a pre-defined environment variable, called PUB, where all public files for the course are located.
The front end enables you to gain access computational resources: the GPUs, Intel Phis, and to run on multiple nodes (e.g. with MPI). All of these nodes have Sandy bridge processors, too. Use the front-end for code editing, compilation and the like, but not for heavy duty number crunching. Stampede provides both interactive and batch access. Use interactive mode for code development and tuning, batch to conduct timing measurement for turnin.
The incrArr example has been set up in $PUB/Examples/CUDA/incrArr Follow the instructions to build this test program. To run interactively, use the special inter-gpu alias that the BASHRC file set up for you. To submit a batch job, use the sbatch command as in
sbatch batchscript.slurm
You’ll get output which includes a message at the end informing you that your job has been submitted:
Submitted batch job 4683723You can check on the status of this job with the showq command:
showq
SUMMARY OF JOBS FOR USER: <tg458182>
ACTIVE JOBS--------------------
JOBID JOBNAME USERNAME STATE CORE REMAINING STARTTIME
================================================================================
4683723 INCR-ARR tg458182 Running 16 0:04:52 Fri Jan 9 13:18:03
WAITING JOBS------------------------
JOBID JOBNAME USERNAME STATE CORE WCLIMIT QUEUETIME
================================================================================
Total Jobs: 1 Active Jobs: 1 Idle Jobs: 0 Blocked Jobs: 0 Since we specified that we wanted output (see the slurm submission file for the details), output is available where we submitted the job from: INCR-ARR.o4683723