Getting Started with Triton

Changelog Date Description 31-Oct-10 Original posting 11-Nov-10 Added an envioronment variable setting

The Triton Compute Cluster (TCC) provides 256 Appro gB222X Blade Servers, each with dual-socket quad-core Intel Xeon E5530 (Nehalem, 2.40 GHz) processors and 24 GBytes of memory, for a total of 2048 cores. Triton applications may employ MPI, OpenMP or pthreads, or any combination. Triton provides a variety of development tools for debugging, locating memory leaks, measuring cache performance, and many others. Consult the documentation.

The purpose of this Primer is to get you started quickly. However, SDSC's Triton documentation provides more information and is the authoritative source. Other resources include Frequently-Asked Questions for Triton users.

Be sure to look over the Quick Start Guide to get you started with job submission. If you have questions about Triton that you’d like to submit to SDSC Staff, be sure to join the triton-discuss mailing list. (You can only submit questions if you are a member of the list).

Each of you has been given a time allocation in service units or SUs. If you get a message about a job being deleted when you attempt to run interactively, then your account has run out of time. Contact SDSC and they will top off your account.

qsub: waiting for job 422405.triton-42.sdsc.edu to start
qsub: job 422405.triton-42.sdsc.edu apparently deleted

% gbalance -u sbaden
Id  Name      Amount Reserved Balance CreditLimit Available 
--- --------- ------ -------- ------- ----------- --------- 
120 sbaden       226        0     226           0       226 

triton-login.sdsc.edu

Use the front-end node to develop code, and to submit jobs via the batch subsystem. The front-end should not be used to run jobs interactively; such jobs should be run using the procedure described below.

You may obtain information about Triton’s processors by looking at the file /proc/cpuinfo, or even more detailed information in the following files

/sys/devices/system/cpu/cpu*/cache/index*/*

Be sure you are to run in batch mode (see below) when you collect this information, as the hardware on the login nodes may be different from that of the compute nodes.

Compiling and Running on Triton

Triton uses the "module" system for setting up paths to compilers, etc. This is similar to softEnv, which we use on Lincoln. Before you start compiling any code, add the following 3 lines to your .bashrc or .bash_profile file:

module purge
module load intel
module load openmpi_mx

You should also add the following enivironment variable setting

export MX_RCACHE=0

A public directory has been set up for you on Triton:

~sbaden/cse260-fa10

To establish that your environment has been set up correctly to build and run MPI programs, compile and run the provided parallel "hello world" program, which prints "Hello World'' from each process along with the process ID. The code for this program is found in the directory $PUB/Examples/MPI/Basic/hello. Be sure to use the Makefile that we've supplied so you'll get the correct compiler and loader flags for Triton. Code built to run with MPI must use the arch.mpi.intel version of the arch file. You’ll see that this has been set up in the Makefile.

Note that any command line arguments come in the usual position, after the name of the executable, and are followed by the arguments to mpirun. Thus, to run the Ring program (found in $PUB/Examples/MPI/Ring) on 16 CPUs with command line arguments -lin 0 1024 64, enter the following:

mpirun -np  16 -lin 0 1024 64
mpirun -np  16 -lin 0 1024 64

To run interactively, use the qsub command with the -I option as follows:

qsub -I -V -l walltime=00:10:00,nodes=1:ppn=1

This will give you access to 1 interactive node for 10 minutes and is appropriate for running single processor or multithreaded jobs.

Depending on the activity on the machine, you may have to wait for resources to become available.

qsub: waiting for job 2509.triton-42.sdsc.edu to start
qsub: job 2509.triton-42.sdsc.edu ready

This is likely to happen as assignment deadlines draw near, so the preferred method of debugging your code is to develop it on other resources.

You’ll notice that the qsub -I command forks off a new shell. This shell will exit after your reservation has exhausted its time. If you are done before then, be sure to use the exit command to free up resources for other users. Here is the on-line man page for qsub.

When interacting with applications that provide a graphical display (e.g. gnuplot), you also need to enable X11 forwarding on the qsub command line, which you specify with the -X option:

qsub -I -V -X -lwalltime=00:30:00,nodes=1:ppn=8

When running MPI jobs, keep the ppn parameter set to 8. If running hybrid MPI-OpenMP jobs, set ppn=1. While you may run with more than 1 node interactively, note that interative jobs run at high prioirty, which depletes your bank account more quickly. Note that your job will be charged for all cores on a node no matter how many cores you are using. Your account is billed according to the formula.

# CPUs × # nodes × walllclock time 

When you are ready to collect measurements, make your production runs using one of the batch queues. These batch queues provide dedicated access to resources—though the switch will be shared with other jobs. They implement a technique called space sharing, whereby each user gets a portion of the hardware until they are done using it. Compare this with time sharing, in which users get time slices of the available resources. We will use space sharing when collecting performance measurements, since the mechanism provides dedicated access.

While batch jobs provide dedicated access, the caveat is that you will have to wait for them to return. This method of submitting jobs may be new to you, and it takes some getting used to. However, if you think about it, it would be very expensive to give each user their own dedicated resources for an indefinite period of time. Most heavily subscribed high performance computing platforms employ space sharing.

To submit and manage batch jobs, we will use the Torque resource manager (also known by as the PBS batch system, its historical name). To specify a batch job, you need a batch script. We’ve set up a sample script file run.sh in $(PUB)/Examples/MPI/Basic, which will run the applications in Basic. You will need to make some minimal changes to the script, as noted in the file, in particular, to set up the correct address for email notification that the job has completed (qsub scripts indicate options with a # mark. These are not comments unless there are two or more # in a row.)

As with interactive jobs, we batch jobs using qsub: documentation on qsub command:

qsub run.sh

If successful, the command will reply by printing a message identifying the job, where 413985 is the job number.

Job passed jobfilter, writing script to STDOUT ... 

413985.triton-42.sdsc.edu

The provided script is set up to write stdout and stderr to files with distinguished names. This name contains two parts: the first part is a Job ID that you specify in the batch submission script (or on the command line via the -N command, which takes precedence). The second part is the job number. For example, the job described above will generate two files for stdin and stdout, respectively:

Basic.o413985
Basic.e413985

You may also have the output mailed to you; see the provided batch submission script for the details.

Note that if there are problems with the job, you’ll be notified. You can then examine the stderr file for the cause. For example, the following message indicates that the program could not be found

hello: command not found

This might have been a result of not setting the correct working directory in your batch script. By default, the batch subsystem puts your job in your home directory (including interactive mode). So, you must cd to the correct working directory, as set up in the provided job submission script, where $PBS_O_WORKDIR is defined to be the directory from which the batch job was submitted via qsub:

cd $PBS_O_WORKDIR

Once your job has been submitted, it will wait in the queue until the specified resources are available. You may check on the status of your job using the qstat command. If you specify your user ID, you'll see only your jobs:

qstat -u sbaden
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
90.triton-46              PBStest          hocks                  0 R batch
91.triton-46              PBStest          hocks                  0 Q batch
92.triton-46              PBStest          hocks                  0 Q batch
2367.triton-42            SimpleThreads    sbaden          00:00:00 C batch

You can remove jobs with the qdel command, and you only need to specify the leading number in the job ID:

qdel 2367

More on Queues

To get an idea of when your job will finish, use the showstart command. To run showstart, provide a job number as the single command line argument:

showstart 607580
job 607580 requires 32 procs for 00:30:00
Estimated Rsv based start in 00:00:00 on Fri Oct 31 21:40:35
Estimated Rsv based completion in 00:30:00 on Fri Oct 31 22:10:35

You can also get a prediction of when your job will run via the TeraGrid portal’s HPC queue prediction service (You must be logged into the Teragrid portal service). The portal also provides a system monitor.

You may run your jobs using various batch queues, which vary according to factors such as expected job length and the number of nodes requested.

The Intel compilers process the OpenMP pragmas, provided the -openmp compiler flag has been set:

icc -c -openmp -openmp-report2 file.c

To establish that you can compile build OpenMP code with the Intel compiler suite, build a copy of your code from Assignment 1. Change the first two lines of the Makefile as follows:

PUB = /home/sbaden/cse260-fa10
include $(PUB)/Arch/arch.intel.omp

If you don’t want to compile with OpenMP, i.e. you are building serial code, use the arch.intel "arch" file.

To run an OpenMP program, you must set the environment variable OMP_NUM_THREADS to the number of threads you require, e.g.

export OMP_NUM_THREADS=2

Here is a Tutorial on OpenMP, complete with code examples. Also see C++ Examples by Jon Burkardt Example OpenMP programs taken from the tutorial have been installed on Triton in $PUB/Examples/OpenMP complete with Makefile and batch submission script.

Here are some articles to help get you started, all published by Intel, though note that the compiler flags are different from those expected by the PGI compilers.

Here are two tutorial presentations from the Ohio Supercomputing Center

PDF, Troy Baer, April 2007
PDF, Jim Giuliani, November 2003

Software environment

How are job charges determined?  

If I create a batch job, where I set a wall time limit of for instance 1 hour on 8 nodes, but my job only runs for 30 minutes, will my account get charged for 1 hour (the time I requested) or 30 minutes (what I actually used)? All charges on Triton are based on one-hour Service Units. Due to rounding, a 30-minute, single-CPU job will be charged for one full CPU-hour, or one SU. If a job runs for 29 minutes, the charge would be zero SUs.

In your case (assuming TCC nodes, not PDAF/M), using eight nodes with eight cores each running for 30 minutes, the CPU charges would be calculated on 64 cores times 30 minutes, and the rounding is applied after this calculation. Also, most TCC nodes charge for the entire node regardless of how many cores are used by a job, so the longest-running core on each node would be the determining factor for all eight cores on that node. Only the "small" queue has a per-core charge basis. Please see the following sections of the Triton Resource website for more details:

• Charge Policies for Triton Compute Jobs (Rounding Charges): http://tritonresource.sdsc.edu/policies.php#rounding
• Job Charging examples: http://tritonresource.sdsc.edu/policies.php#examples
• Triton Resource FAQ: What accounting and charging policies are in use? http://tritonresource.sdsc.edu/faq.php#acct6

Documentation about the Nehalem Processor

• Next Generation Intel® Microarchitecture Nehalem, by Paul G. Howar
• www.microway.com
• Inside Nehalem: Intel's Future Processor and System, by David Kanter
• realworldtech.com
• First Look at Nehalem Microarchitecture, by Ilya Gavrichenkov, xbitlabs.com.

We have set up a Moodle forum called "Triton." Post questions and discussions about the machine and its software to the forum. There are no specific known problems at this time, but watch the Triton Forum for any developments.

Copyright © 2010 Scott B. Baden.