Changelog |
|
| Date | Description |
| 11-Oct-10 | Original posting |
| 30-Oct-10 | Various editing changes |
| 01-Nov-10 | SSH Access to Lincoln |
|
|
Lincoln is a GPU cluster located at the NCSA. The machine consists of a collection of Dell PowerEdge 1950 servers connected with InfiniBand SDR. Each server contains 8 cores constructed from dual-socket, quad-core Intel 64 (Clovertown, E5345) 2.33GHz processors, together with 2 Tesla processors connected via PCI-e Gen2 X8 slots. The hardware thus provides a platform for writing applications that run under message passing, threads, or hardware acceleration, or any combination of these models.
The purpose of this Primer is to get you started quickly.
However,
NCSA’s extensive
User Documentation will provide considerably more information,
and is the authoritative source.
Lincoln
To use Lincoln, log in to the front-end machine called
You may access Lincoln via the Teragrid portal; if you want to access Lincoln via SSH, follow these instructions.
Use the front-end node to compile code, and to submit jobs via the batch subsystem. The front-end has no direct access to the GPUs, though there is a mechanism for gaining interactive access.
You may obtain information about Lincoln’s processors by looking at the file /proc/cpuinfo, or even more detailed information in the following files
/sys/devices/system/cpu/cpu*/cache/index*/*
Be sure you are to run in batch mode (see below) when you collect
this information, as the hardware
on the login nodes may be different
from that of the compute nodes.
Compiling and Running on Lincoln
A public directory has been set up for you on Lincoln:
~baden/cse260-fa10
The public directory contains source code you’ll use in your assignments and other goodies. From now on we’ll refer to this directory as $(PUB).
We will program using Nvidia’s CUDA environment. To use CUDA on Lincoln, be sure to read NCSA’s instructions on the Tesla Compilers
To establish that you can compile, build, and run a CUDA application, make a copy, in your home directory, of the incrArray application from $PUB/Examples/CUDA and type make.
If you see error messages about include files or libraries not found, your environment is not set up correctly. To set up our environment, we’ll rely on SoftEnv, which loads modules specified in a file called .soft, residing in your home directory. For example, to set up CUDA, put the following two entries in your .soft file Also specify version 4.4.3 of the Gnu compilers.
+cuda-3.0 +nvidia-sdk-cuda-3.0 +gcc-4.4.3
These changes will not take effect until you next login, so logout and then logout again before continuing. Do not explicitly set the environment variables as with other Unix like systems (such as Lilliput). SoftEnv will take care of all that for you. If you want to learn more about how SoftEnv works, consult the SoftEnv User’s Guide.
Having built the code, you are now ready to run it. To run interactively on the Tesla processors, specify the lincoln queue with ppn=2 (there are 2 GPUs on a node):
qsub -I -V -q lincoln -lwalltime=00:10:00,nodes=1:ppn=2
Qsub will notify you that it found a valid account (computer time) and that it is waiting for an available resource:
This job will be charged to account: djy (TG-CCR080007) qsub: waiting for job 3304027.abem5.ncsa.uiuc.edu to start
You will usually need to wait for resources to become available, which will be indicated as shown below, after which point you will get a shell command line prompt. During peak times, you may experience long delays, so plan accordingly. You may also submit batch jobs, as described below.
qsub: job 3304027.abem5.ncsa.uiuc.edu ready ---------------------------------------- Begin Torque Prologue (Thu Oct 14 17:28:35 2010) Job ID: 3304027 Username: baden Group: djy Job Name: STDIN Limits: ncpus=1,walltime=00:10:00 Job Queue: normal Account: djy Nodes: abe1075 /dev/sda2 on /tmp type ext2 (rw) End Torque Prologue ----------------------------------------
Interactive mode is costly and also ties up resources. When you are done, release the resources to others by typing
exit
This command will exit the shell running under the interactive session. You will still be logged in to Lincoln, but will no longer have access to the GPU.
Cudaprof is a handy GUI-based tool for profiling your code, whether written in CUDA or with PGI's accelerator extensions. When you use ssh to connect to Lincoln, be sure to enable X11 forwarding:
ssh -X lincoln.ncsa.illinois.edu
Cudaprof provides an interactive graphical interface, thus it is accessible via Lincoln’s interactive queue only. You also need to enable X11 forwarding on the qsub command line, which you specify with the -X option:
qsub -I -V -X -q lincoln -lwalltime=00:30:00,nodes=1:ppn=2
When you are ready to collect measurements, make your production runs using one of the batch queues. There is an interactive batch queue that will enable you to take measurements interactively, but you may experience long wait times during peak usage periods. In this case, submit batch jobs that will be staged automatically when the resources become available.
Batch processing takes some getting used to. You cannot interact with the job. You will have to wait for the job to make its way to the top of the queue, a time duration known as queue latency. At this point the job will run, and you can elect to be notified by email when the job has completed.
To submit and manage batch jobs, we will use the PBS batch system. To specify a batch job, you need a batch script. We’ve set up a sample script file run.sh in $(PUB)/incrArray, which will run the incrArr application. You will need to make some minimal changes to the script, as noted in the file, in particular, to set up the correct address for email notification that the job has completed (qsub scripts indicate options with a # mark. These are not comments unless there are two or more # in a row.)
We submit
batch jobs using the
qsub command (here is some
documentation on qsub command):
qsub run.sh
If successful, the command will reply by printing a message identifying the job, where 2196961 is the job number.
2196961.abem5.ncsa.illinois.edu
The provided script is set up to write stdout and stderr to files with distinguished names. This name contains two parts: the first part is a Job ID that you specify in the batch submission script (or on the command line via the -N command, which takes precedence). The second part is the job number. For example, the job described above will generate two files for stdin and stdout, respectively:
Increment.o2196961 Increment.e2196961
To have the output mailed to you, see the provided batch submission script for the details.
Note that if there are problems with the job, you’ll be notified. You can then examine the stderr file for the cause. For example, the following message indicates that the program could not be found
increment: command not found
This might have been a result of not setting the correct working directory in your batch script. By default, the batch subsystem puts your job in your home directory (including interactive mode). So, you must cd to the correct working directory, as set up in the provided job submission script, where $PBS_O_WORKDIR is defined to be the directory from which the batch job was submitted via qsub:
cd $PBS_O_WORKDIR
Once your job has been submitted, it will wait in the queue until the specified resources are available. You may check on the status of your job using the qstat command. If you specify your user ID, you'll see only your jobs:
qstat -u baden
abem5.ncsa.illinois.edu:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
------------------ ------- -------- ---------- ------ ----- --- ------ ----- - -----
2196961.abem5.nc baden lincoln Increment -- 1 1 -- 00:05 R --
You may monitor the job queue via a web portal or monitor the entire machine. Remove jobs with the qdel command:
qdel 2196961
Depending on demand, you may experience long queuing delays. Keep this in mind when as deadlines approach. Generally speaking, short jobs at higher priority will run sooner than longer jobs running at lower priority. To get an idea of when your job will finish, use the showstart command. To run showstart, provide a job number as the single command line argument:
showstart 607580 job 607580 requires 32 procs for 00:30:00 Estimated Rsv based start in 00:00:00 on Fri Oct 31 21:40:35 Estimated Rsv based completion in 00:30:00 on Fri Oct 31 22:10:35
Note that it can take a several minutes for showstart to notice that a job has been submitted, so if the queue has a short turnaround, you might not get the prediction until after the job has started!
You can also get a prediction of when your job will run via the TeraGrid portal’s HPC queue prediction service (You must be logged into the Teragrid portal service). The portal also provides a system monitor.
You may run your jobs using various batch queues, which vary according to factors such as expected job length and the number of nodes requested.
When debugging, your jobs should not normally use more than a few minutes of wallclock time. If you do need to run for more than a 10 minutes, use the normal queue. Keep the job time limit low until you understand the performance of your application and can make informed decisions about adjusting a time limit upward. Longer jobs may have to wait longer in the queue, but depending on other activity your wait times will vary.
We’ll be using the Intel compilers SoftEnv for further information.> By default, we is also available via SoftEnv
The Intel compilers process the OpenMP pragmas, provided the -openmp compiler flag has been set:
icc -c -openmp -openmp-report2 file.c
For more details, see the NCSA’s posting of general information on compilers, including Intel C/C++ compiler documentation
To run an OpenMP program, you must set the environment variable OMP_NUM_THREADS to the number of threads you require, e.g.
setenv OMP_NUM_THREADS 4 ./a.out
Since OpenMP assumes the existence of shared memory, OpenMP runs are restricted to a single node--8 processors. Thus, effective upper bound on OMP_NUM_THREADS is 8 unless hyper-threading has been enabled. Depending on the application, maximum performance may be achieved on fewer than 8 threads.
To establish that you can compile build OpenMP code with the Intel compiler suite, build a copy of your code from Assignment 1. Change the first two lines of the Makefile as follows:
PUB = /u/ac/baden/cse260-fa10 include $(PUB)/Arch/arch.intel.omp
If you don’t want to compile with OpenMP, i.e. you are building serial code, use the arch.intel "arch" file.
Example OpenMP programs taken from Blaise Barney’s tutorial have been installed on in $(PUB)/Examples/OpenMP, complete with Makefile and batch submission script (run.sh)
Here is a short document on Using OpenMP, including a pointer to the use of environment variables.
Also look at Optimizing Applications (found
within the above Intel compiler document).
On the left hand pane of the window, drill down to Optimizing Applications. The outline tree will expand at this point;
from there drill down to Increasing Performance Using Parallelism
→
OpenMP Support. (You can also perform this
navigation from within the right hand pane, but the titles are slightly
different).
If you prefer, you can
obtain a PDF
of Intel® C++ Compiler User and Reference Guides (255 pages)
Vectorization and SSE instructions
Lincoln’s Intel E5345 (Clovertown) processors support SSE3 instructions. The compiler can target these instructions if you specify the -axT option as recommended by the Intel documentation (you may also use -msse3). Be sure to generate a vectorization report with the -vec_report3 command line option in order to learn about what the compiler was not able to vectorize. Here are additional sources to consult on vectorization and SSE instructions
We have set up Moodle forums called "CUDA" and "Lincoln" Direct questions pertaining to coding issues to CUDA, machine or installation issues to Lincoln. There are no specific known problems at this time, but watch the Lincoln Forum for any developments.
Copyright © 2010 Scott B. Baden.