A Primer for Honest Abe

Changelog Date Description 19-Oct-08 Original posting 28-Oct-08 Additional information about the programming environment 28-Oct-08 How to get started with OpenMP 02-Nov-08 Added discussion about setting up mpd.conf for MPI. 03-Nov-08 Added discussion about the showstart command for predicting job completion. 03-Nov-08 Added discussion about the MKL library 07-Nov-08 PSRun is restricted to single processor runs. 11-Nov-08 Added references to vectorization and SSE instructions.

Introduction

Abe is a parallel computer located at the NCSA. The machine is a Dell PowerEdge 1955 Cluster interconnected by a high performance Infiniband switch. Each processing module contains 8 cores constructed from dual-socket, quad-core Intel 64 (Clovertown, E5345) 2.33GHz processors.

The purpose of this Primer is to get you started quickly. However, NCSA’s extensive User Documentation will provide considerably more information, and is the authoritative source.

Abe

You may obtain detailed information about Abe’s processors by looking at the file /proc/cpuinfo.
To use Abe, log in to the front-end machine called

A public directory has been set up for you on Abe.

To establish that your environment has been set up correctly, compile and run Assignment 0, which has been set up in $(PUB)/A0 (CSE 260: this directory is found in ~baden/cse160-fa08). Be sure to use the Makefile that we've supplied so you'll get the correct compiler and loader flags for Abe.

You’ll notice that the Makefile includes a file called arch.pthread. This file configures the Makefile to use the Intel compiler suite (C/C++/Fortran), with various compiler and loader flags set appropriately to build a pthread job. The "arch" file should not normally be changed. If you do modify it, be sure to document any changes you make.

To run interactively, use the qsub command with the -I option as follows:

This will give you access to 1 interactive node for 10 minutes and is appropriate for running single processor or multithreaded jobs.

You’ll notice that the qsub -I command forks off a new shell. This shell will exit after your reservation has exhausted its time. If you are done before then, be sure to use the exit command to free up resources for other users.

Depending on the activity on the machine, you may have to wait for resources to become available

qsub: waiting for job 1244.abem5.ncsa.uiuc.edu to start
qsub: job 1244.abem5.ncsa.uiuc.edu ready

This is likely to happen as assignment deadlines draw near, so the preferred method of debugging your code is to develop it on other resources, as with the previous assignments.

Here is some more Documentation on qsub

Running Batch Jobs

When you are ready to collect measurements, make your production runs using one of the batch queues. These batch queues provide dedicated access to resources—though the switch will be shared with other jobs. They implement a technique called space sharing, whereby each user gets a portion of the hardware until they are done using it. Compare this with time sharing, in which users get time slices of the available resources. We will use space sharing when collecting performance measurements, since the mechanism provides dedicated access.

While batch jobs provide dedicated access, the caveat is that you will have to wait for them to return. This method of submitting jobs may be new to you, and it takes some getting used to. However, if you think about it, it would be very expensive to give each user their own dedicated resources for an indefinite period of time. Most heavily subscribed high performance computing platforms employ space sharing.

We will use the PBS batch system. Here is some documentation on PBS. Submit your batch job using the qsub command:

where the run.sh is a job submission script containing the appropriate environment settings and one or more runs that you wish to make.

A message will be printed at the screen

We’ve set up a script file in $(PUB)/A0, which will run Assignment #0. You will need to make only minimal changes to the script, as noted in the file. (Qsub scripts indicate options with a # mark. These are not comments unless there are two or more # in a row.)

Once your job has been submitted, it will wait in the queue until the specified resources are available. You may check on the status of your job using the qstat command. If you specify your user ID as in qstat -u baden you'll see only your jobs.

                                                                   Req'd  Req'd   Elap
Job ID               Username Queue    Jobname    SessID NDS   TSK Memory Time  S Time
-------------------- -------- -------- ---------- ------ ----- --- ------ ----- - -----
579997.abem5.ncsa.ui baden    debug    Ring          --      4   1    --  00:05 Q   -- 
579998.abem5.ncsa.ui baden    debug    Ring          --      4   1    --  00:05 Q   -- 

You may also monitor the machine via this url. You can remove jobs with the qdel command:

llcancel 579998

The provided script is set up to write all output to a file whose name contains a unique identifier, in Basic.o578801. You also get an error output file with the same prefix, i.e. Basic.e578801

You can ask to be notified by email when your job has finished. See the provided batch script and be sure to set the correct email address. You may also have the output mailed to you; see the documentation.

Note that if there are problems with the job you’ll be notified. Some output appears in the "error" file, other in the "output" file. For example, the following message came from the "output" file, and it indicates that the program could not be found

This might have been a result of not setting the correct directory in the cd command which appears in the provided job submission script. By default, the batch subsystem puts your job in your home directory (this includes interactive mode). So, you must cd to the correct working directory:

More on Queues

  To get an idea of when your job will finish, use the showstart command. To run showstart, provide a job ID as the single command line argument:

showstart 607580
job 607580 requires 32 procs for 00:30:00
Estimated Rsv based start in 00:00:00 on Fri Oct 31 21:40:35
Estimated Rsv based completion in 00:30:00 on Fri Oct 31 22:10:35

qsub run.sh

This job will be charged to account: kmi (TG-CCR070001)
607580.abem5.ncsa.uiuc.edu

You can also determine when job will start via the teragrid portal’s (https://portal.teragrid.org) resource tab at https://portal.teragrid.org/gridsphere/gridsphere and then drilling down to HPC queue prediction.

Programming Environment

  A particularly useful library is the Intel Math Kernel Library MKL, which provides high performance implementations for BLAS, Fast Fourier Transforms and a variety of other mathematical kernels. These have been heavily optimized, and it will be difficult (though not impossible) for you to build more efficient implementations. If you are using MKL, you need to add the following to the end of your loader commmand line

-L/usr/local/intel/mkl/9.0/lib/em64t -lmkl -lguide -lpthread -lg2c -lm

+intel-mkl

You can find documentation and sample code for the Intel compilers on Abe in the directory tree rooted at /usr/local/intel/10.0.026 Also see Intels C++ Compiler for Linux Systems User’s Guide (hosted at NERSC) with additonal information about vectorization and OpenMP.

Performance Measurement

There are various tools for measuring performance. A simple way to measure times in your program is to use a high resolution timer. The gettimeofday() routine will give you approximately 1 microsecond resolution, but has a 1 microsecond overhead. If you are measuring timings of 100 microseconds or more, this method should give you reasonable accuracy. Code has been posted in $(PUB) showing how to use the timer.

Another method is to use PerfSuite, a set of tools which can measure performance within th memory hierarchy, e.g. cache. For example, you may run a program with psrun, which produces an xml file containing various statistics about the program.

psrun Jacobi3D

7-Point Point Jacobi with N = 128
Process geometry 1 x 1 x 1
iterations: 10

...

     N    Px   Py   Pz   It       Time         Gflops
#>   128    1    1    1   10    5.67385e-01    0.29569
honest3 [135]: ls -l *.xml

-rw-------  1 baden kmi 4922 Oct 19 01:32 Jacobi3D.16547.honest3.xml
psprocess Jacobi3D.16547.honest3.xml > report.txt

Here is a book published by Intel Press: The Software Optimization Cookbook.

OpenMP

The Intel compilers process the OpenMP pragmas provided the -openmp compiler flag has been set:

		icc -c -openmp -openmp-report2 file.c
		

To run an OpenMP program, you must set the environment variable OMP_NUM_THREADS to the number of threads you require, e.g.

setenv OMP_NUM_THREADS 4
./a.out

Here is a Tutorial on OpenMP, complete with code examples. Also see C++ Examples by Jon Burkardt Example OpenMP programs taken from the tutorial have been installed on Abe in $(PUB)/Examples/OpenMP complete with Makefile and batch submission script.

Here is a short document on Using OpenMP, including a pointer to the use of environment variables. Also look at Optimizing Applications (found within the above Intel compiler document). On the left hand pane of the window, drill down to Optimizing Applications. The outline tree will expand at this point; from there drill down to Increasing Performance Using Parallelism → OpenMP Support. (You can also perform this navigation from within the right hand pane, but the titles are slightly different). If you prefer, you can obtain a PDF of Intel® C++ Compiler User and Reference Guides (255 pages)

Here are some articles to help get you started, all published by Intel.

Here are two tutorial presentations from the Ohio Supercomputing Center

PDF, Troy Baer, April 2007
PDF, Jim Giuliani, November 2003

As mentioned previously, Abe uses the Intel E5345 (Clovertown) processor. To optimze for SSE3 instructions use the -axT option as recommended by the Intel documentation (you may also use -msse3). Be sure to generate a vectorzation report with the -vec_report3 command line option in order to learn about what the compiler was not able to vectorize. Here are additional sources to consult on vectorization and SSE instructions

MPI Users (CSE 260)

  Before you get started, create the file $HOME/.mpd.conf if you do not already have one. The mpd daemons (MPI) require this file.

    $ echo "MPD_SECRETWORD=mywordabc" > $HOME/.mpd.conf
    $ chmod 600 $HOME/.mpd.conf

To establish that your environment has been set up correctly to run with MPI, compile and run the provided parallel "hello world" program, which prints "Hello World'' from each process along with the process ID. The code for hello world! is found in $(PUB)/Examples/Basic/hello. Be sure to use the Makefile that we've supplied so you'll get the correct compiler and loader flags for Abe. Code built to run with MPI must use the arch.mpi version of the arch file.

Note that any command line arguments come in the usual position, after the name of the executable, and are followed by the arguments to mpirun. Thus, to run the Ring program (found in $(PUB)/Examples/Ring) on 16 CPUs with command line arguments -lin 0 1024 64, enter the following:

You may vary the qsub parameters nodes and ppn up to values limited by the machine configuration (Recall that nodes have 8 CPUs). You may also set up the configuration with environment variables; consult the on-line instructions for the details.

If you do need to run for more than a few minutes, use the normal queue. Keep the job time limit low until you understand the performance of your application. Then adjust the time limit carefully by modifying your script. Longer jobs may have to wait longer in the queue, but depending on other activity your wait times may vary.

When collecting timings, you may use the gettimeofday() timer as stated above, but you may also use the built in timer MPI_Wtime().

A variety of tools are available on Abe. In addition, I’ve also installed fpmpi which tallies various message passing statistics.

To use this tool, simply load the fpmpi library. to do this add the following line to your Makefile:

Porting code to Abe

If you have difficulty compiling code, post to the web board. There are no specific known problems at this time, but watch this space for any developments.