The production platform for this class–where you’ll conduct benchmarking studies–is Stampede, which is located at TACC, The Texas Advanced Computing Center. Stampede contains over 6400 Dell PowerEdge server nodes each with 2 Intel Xeon E5 (Sandy Bridge) 8-core processors and an Intel Xeon Phi Co-processor (MIC Architecture). In addition, 128 of the compute nodes contain a single NVIDIA Tesla K20 GPU (“Kepler,” hardware capability 3.5, different from Sorken’s K80’s which have hardware capability 3.7 and contain two devices each). The nodes are interconnected by FDR Infiniband. Note that 40 of the nodes are available for short duration, interactive use, which comes in handy when debugging. Here is a handy description of the hardware’s capablities:https://www.microway.com/hpc-tech-tips/nvidia-tesla-k20-gpu-accelerator-kepler-gk110-up-close
The authoritative source of information for accessing and using Stampede is the The Stampede User’s Guide, which is maintained by TACC. If you have questions, post to the Stampede Piazza Folder.
To access Stampede, ssh to the front end, stampede.tacc.utexas.edu, using your assigned XSEDE credentials.
We’ll be using the bash shell. You may change shells if you wish, however, all instructions will be given in terms of the bash shell (It is not difficult to convert between the two, so post to the Stampede folder on Piazza if you have aliases to donate)
Login to the front end and. Next, modify .bashrc, which is located in your home directory. In particular, insert the file BASHRC, located in /home1/00660/tg458182/cse260-fa15/Profiles into your .bashrc file, immediately after the following text appearing in .bashrc:
#####################################################################
# **** Place any else below. ****
#####################################################################It’s important that you add the commands in his file to your .bashrc, as, among other things, it will load version 7.0 of the CUDA environment, which we are also using on Sorken (The default on Stampede is v6.5 at the time of this writing). You will now have various handy command aliases at your disposal. You’ll also have a pre-defined environment variable, called PUB, where all public files for the course are located.
The front end enables you to gain access computational resources: the GPUs, Intel Phis, and to run on multiple nodes (e.g. with MPI). All of these nodes have Sandy bridge processors, too. Use the front-end for code editing, compilation and the like, but not for heavy duty number crunching. Stampede provides both interactive and batch access. Use interactive mode for code development and tuning, batch to conduct timing measurement for turnin.
The incrArr example has been set up in $PUB/Examples/CUDA/incrArr Follow the instructions to build this test program. To run interactively, use the special inter-gpu alias that the BASHRC file set up for you. To submit a batch job, use the sbatch command as in
sbatch batchscript.slurm
You’ll get output which includes a message at the end informing you that your job has been submitted:
Submitted batch job 4683723You can check on the status of this job with the showq command:
showq -u tg458182
SUMMARY OF JOBS FOR USER: <tg458182>
ACTIVE JOBS--------------------
JOBID JOBNAME USERNAME STATE CORE REMAINING STARTTIME
================================================================================
4683723 INCR-ARR tg458182 Running 16 0:04:52 Fri Jan 9 13:18:03
WAITING JOBS------------------------
JOBID JOBNAME USERNAME STATE CORE WCLIMIT QUEUETIME
================================================================================
Total Jobs: 1 Active Jobs: 1 Idle Jobs: 0 Blocked Jobs: 0 Since we specified that we wanted output (see the slurm submission file for the details), output is available where we submitted the job from: INCR-ARR.o4683723
Use the scancel command to remove a previously submitted job, where the argument is the job number returned when the job was submitted (or found via the showq command): scancel 4683723
As with Bang, on Stampede, MPI will only run on compute nodes. Access an interactive node with the idev command (our provided .bashrc suggestion set up a handy alias, inter-mpi), where the -m argument is in minutes.
idev -m 30You will run in a new shell. Be sure to exit from this shell when it is no longer needed, so other users can have access to the node.
The command to run an MPI job on Stampede is called ibrun. The following command runs the hello executable on 4 processes (cores):
ibrun -np 4 ./helloVarious example programs with Slurm scripts are found in $PUB/Examples/MPI
Valgrind is available on Stampede. Build your code by specifying valgrind=1 on the make command line. Run your code as follows:
ibrun -np <# processors> valgrind [valgrind options] a.out <program options>Valgrind has valuable tool called memcheck that can identify memory memory errors such as illegal frees, dangling pointer references, memory leaks and other common errors.
valgrind --tool=memcheck --leak-check=full --error-limit=no --log-file=file.log ibrun -np <# proc> a.out <args>The output is voluminous and the program will output spurious errors, including errors buried deep inside MPI that can be ignord. Specifying a log file enables you to examine the output more conveniently. The memcheck manual will tell you much more. If you are going to use this tool frequently. be sure to learn how to use suppression lists to filter spurious errors.
DDT is a debugger for MPI programs. You’ll need to compile your code with the -O0 -g options enabled. To do this, rebuild your code with the gdb=1 on the make command line (This has been set up in the arch file).
Since ddt uses x-windowing, you’ll need to login in via with tunneling enabled, e.g. ssh -X
PAPI is a performance monitoring tool that enables you to count events such as L1 cache misses under program control.
I’ve set up an example in $PUB/Examples/PAPI
You’ll need to load papi before using it
module load papi
To see which counters are available run: papi_avail
Not all events can be counted simultaneously. To learn about how to determine which counters can be collected simultaneously see https://www.cs.uoregon.edu/research/tau/docs/old/bk01ch02s04.html
For more details, see the following
There are various commands that tell ll you which counters are available and what there names are. Run papi.slrm to see the full listings