| Date | Description |
| 6-Apr-11 | Original posting |
| 7-Apr-11 | Will use SUMMA instead of MKL for the CPU version |
| 10-Apr-11 | Added path for MPI. |
| 11-Apr-11 | Updates to Intel tools and on-line documentation |
| 12-Apr-11 | Added references to Fermi. |
| 13-Apr-11 | Added documentation about the Nehalem processor. |
| 19-Apr-11 | Added Discussion about Trestles |
| 9-May-11 | Added Discussion about Kraken |
| 10-May-11 | Added note about setting Intel compilers for backend of mpicc and mpicxx |
| 16-May-11 | Added notes Teragrid access |
This programming assignment contains 3 parts, and there is there are 2 intermediate milestones, i.e. with due-dates, to ensure progress. You'll be provided with a a CUDA implementation of Matrix multiplication and an executable that utilizes multiple cores (SUMMA) on Lilliput's CPUs.
In part I of the assignment, you'll compare the performance of these two codes, running the CUDA code both on Lilliput's GPU (C1060, 200 series) and one of the Fermi GPUs (GTX 570, cseclass0{1,2}). You'll run SUMMA code on 4 cores of the CPU.
In part II, you'll optimize the CUDA implementation for the c1060's shared memory, as discussed in the Kirk and Hwu text. Next, you'll optimize the code for Fermi. You'll be on your own for the Fermi optimizations.
For the GPU coding, be sure to refer to the paper by Vasily Volkov and James W. Demmel:
"Benchmarking GPUs to tune dense linear algebra" Proc. 2008 ACM/IEEE Conf. on Supercomputing(SC '08).
In Part III, you'll implement matrix multiplication in MPI, using Canon's algorithm, and run on Trestles and/or Kraken. You'll be provided with MPI starter code, and you should use the provided SUMMA binary to assess performance.
The milestones (with due dates) are as follows:
| Part I | 12-Apr-11 |
| Part II | 21-Apr-11 |
| Part III | 3-May-11 |
For both Parts I and II.
1/4 L1, 3/4 shared memory
3/4 L1, 1/4 shared memory
$PUB is defined on Trestles as /home/baden/cse262-sp11
$PUB is not yet set up on Kraken as
Document your work in each part with a well-written report of about 5 pages, not including drawn figures and pseudocode. Your writeup should discuss decisions you made in implementing an optimizing your algorithm. Provide pseudocode to help describe how your code works, but do not include any source code. That will appear in your electronic turnin. Read over these Tips for your writeup.
Your turnin must include a self-evaluation, which is available HERE. (Not needed for Part I).
Put a hard-copy of your report in my mailbox on the 2nd floor of the CSE Building (EBU3B).
Also transmit an electronic copy by emailing a single archive file, e.g. .tgz, containing your writeup and source code-- and the self-evaluation. Note: submitting the electronic copy establishes the turnin time of your assignment.
Be sure to delete all .o files and executables before creating the tar file. Email the archive file as an attachment to the following address, and put the string cse262:Part1 in the Subject: line:
| baden | @ | ucsd. |
edu |
The name of your archive file should start with your login ID, followed by an underscore '_' e.g. baden_p1.tgz. When your archive file is extracted, a new directory should be created with the same name as the archive file, excluding the filename extension, e.g. baden_p2 The directory should contain two subdirectories
| src | All source code including your makefile, so we can build the software. |
| report | Your report including any image files, screen shots, or other output you want me to see. |
Various hardware platforms are available in the course. Lilliput is a server with a quad core Intel Nehalem processor (E5504, "Gainsboro") running at 2.0 GHz, 16 GB of memory, and 4 NVIDIA C1060 Tesla GPUs. Here is Documentation about Lilliput. Lilliput runs Ubuntu Linux and it is possible to program in MPI, pthreads, OpenMP, CUDA, or some combination of these models. We have installed some basic documentation in
http://cseweb.ucsd.edu/groups/hpcl/scg/guide.html
On Lilliput as well as other Linux systems you can find useful information about the processor in the file /proc/cpuinfo. For example, here is part of Lilliput's cpuinfo file:
cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5504 @ 2.00GHz stepping : 5 cpu MHz : 1600.000
The contents of cpuinfo enables you to identify the specific processor, and the identifying information may be used to obtain other details from on-line resources. This information confirms that Lilliput is a quad-core 2.0GHz Intel Xeon E5504 processor. There is additional information in procinfo: family, model, and stepping. So in this case the family is 6, the model is 26, and the stepping is 5. This will allow you to explore smaller scale changes to the processor’s design that occurred over time.
We may learn more by accessing Intel's ARK Database. Once at the ARK web site, search on "E5504" in the search window at the top of the page. You'll obtain various information; for example, the processor has 4 cores, and a 4 MB L3 cache shared among all cores. (The 2.0 GHz clocks speed matches that listed in the cpuinfo file). The L1 and L2 cache sizes, not listed, are: 32KB (per core; separate instruction and data caches) and 256KB (per core; unified), respectively.
Unfortunately, Intel dropped the handy processor finder web site (CPU World maintains similar functionality if you know the S-Number). However, you may obtain more detailed information about the processor in another way. There is set of files providing detailed characteristics of all levels of cache (though there are many files):
/sys/devices/system/cpu/cpu*/cache/index*/*
To learn more about the Nehalem processor consult one of the following soruces
The definitive source of information is Intel 64 and IA-32 Architectures Optimization Reference Manual, where you can find detailed information about the processor.
In addition to Lilliput, two workstations are also available: cseclass01 and cseclass02. Each workstation has a hex-core AMD Phenom II X6 1090T Processor (800 MHz) and a NVIDIA GeForce GTX 570 "Fermi" GPU.
To access Teragrid resources, generate a private/public key pair each machine you'll use to access those resources. A private key is secret, so don't share it or else anyone would be able to log in as you.
Let's say that you want to access resources from your laptop. To create the key pair, first open a terminal window and type (on Linux/Unix systems)
ssh-keygen
In response to the questions, use the default location ($HOME/.ssh/id_rsa) and choose a password. You will obtain a public key, which will be stored in the file id_rsa.pub, and a private (secret) key, that will be stored in id_rsa (both reside in $HOME).
Next, log on to portal.teragrid.org using the password provided in the letter you received from Teragrid. Click on the "My Teragrid" tab, and from there on "Accounts" right below the tab. Click on the Login link to the right of the entry for the server you wish to log into (e.g. Trestles). A ssh terminal will then open up for the server.
In the terminal window you created on your laptop, open up the file ~/.ssh/authorized_keys in an editor.
Paste the content of the
Also read this Discussion.
In your .bashrc file on Lilliput, define the environment variable PUB as follows (cseclass0{1,2} share Lilliput's file system)
/class/public/cse262-sp11
The installation for Trestles is discussed below (Kraken is in progress).
You've been provided with various example codes in $PUB/Examples. (This directory structure is set up on all course hardware testbeds). You'll notice that the Makefile provided with each example includes an "arch" file of the form $(PUB)/Arch/arch.* (or arch.*.*.) This arch file contains various settings needed to compile and built executables and generally it should not be changed. Avoid using a private copy in case (rarely) we need to make a patch to an arch file. (Many of the Makefiles in the provided codes use conditional statements that enable the Makefile to be portable onto the platforms identified in the course.) Arch files for serial, MPI, and on machines with a GPU, CUDA, are provided in $(PUB)/Arch/
The CUDA compiler is called nvcc and your path should already be set up to find it. The Increment code resides in $PUB/Examples/CUDA
To access the Intel tools and their man pages, add the following to your .bashrc file
export PATH=/opt/intel/bin/:$PATH export MANPATH=$MANPATH:/opt/intel/man/en_US
You also need to agument the loader path:
export LD_LIBRARY_PATH=/opt/intel/lib/intel64:$LD_LIBRARY_PATH
To set up MPI, be sure to augment your LD_LIBRARY_PATH:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/mpi/openmp/lib
We have installed Intel's C++ Composer XE 2011, which includes
The C and C++ compilers are called icc and icpc, respectively, and are found in /opt/intel/bin. The Intel Debugger idb also resides here. To get help use the -help option or use man
To view a table of getting started documents: for the Intel products, run the command line version of Chromium Browser
chromium-browser /opt/intel/composerxe-2011.2.137/Documentation/en_US/documentation_c.htm
You must login to Lillput with X tunneling enabled. If you are connecting to Lilliput from a Linux platform using ssh, use the -X option:
% ssh -X yourlogin@lilliput.ucsd.edu
If you are connecting from a Windows platform, and are using a 'secure shell client' AND 'exceed' (or xwin32, or cygwin) to log into Lilliput, then you must ensure that 'X11 tunneling' is set. To do this, you must have installed Cygwin /X with X11.
The above documentation is also available from the following web site, and you can run from your favorite browser without connecting to Lilliput:
www.fkf.mpg.de/edv/docs/intel_composer/Documentation/en_US/documentation_c.htm
Movies and additional training are available on Intel's website at
www.intel.com/software/products
Intel has also posted the Intel Guide for Developing Multithreaded Applications
Documentation can be found on-line
software.intel.com/en-us/articles/intel-c-composer-xe-documentation/#lin
Documentation for C++ compiler
software.intel.com/sites/products/documentation/hpc/composerxe/en-us/cpp/lin
Compiler man page
software.intel.com/sites/products/documentation/hpc/composerxe/en-us/mans/icc.txt
Math Kernel Library (MKL)
In addition, we have installed VTune Amplifier XE 2011, with documentation located at
software.intel.com/en-us/articles/intel-vtune-performance-analyzer-for-linux-documentation/
For additional information, including forums, consult the Intel Knowledge base:
http://software.intel.com/en-us/articles/tools/
Lilliput and the cseclass machines are running version 3.2 for the CUDA SDK.
NVIDIA’s Developer Zone hosts a good deal of
resources, especially at the CUDA Zone.
See
http://developer.nvidia.com/cuda-toolkit-32-downloads
to obtain documentation for CUDA v. 3.2.
Of note:
Be sure to use the documentation for the Linux port. The web site provides (free) downloads for the drivers, toolkit, and SDK, together with downloadable applications.
There isn't a lot published specifically about Fermi. In fact, the CUDA C Programming Guide, version 3.2, describes most of the details. (The Documentation is here, as described above). But these NVIDIA documents are also useful.
A nice feature of CUDA is that it is free, so you may install it on your own machine and develop your GPU code off-line. Here are Fred Lionetti’s notes (somewhat outdated), on how to install CUDA under MacOS or Ubuntu. The definitive source of information and software is the Developer Zone, mentioned above.
Code examples reside in $PUB/Examples/CUDA
Follow Hallgeir's instructions for logging in to Trestles for the first time.
Log in to trestles and then do
cd /home/baden/cse262-sp11/Profiles
Copy the files BASHRC and BASH_PROFILE to the end of the .bashrc and .bash_profile files you found in your home directory when you logged in.
Log out and log in. In your home directory run the following commands
cp -p -r $PUB/Examples/MPI/Basic . cd Basic make qsubI 4
Entering this command will allocate 4 processors on an interactive node, which is shared with other users. Our account will be charged for the cores used, so allocate only the necessary number of cores. (This is not true in batch mode, when you get dedicated access to all 32 cores running on the normal queue). You'll see the following output
qsub: waiting for job 22955.trestles-fe1.sdsc.edu to start qsub: job 22955.trestles-fe1.sdsc.edu ready
Wait until you receive a command prompt. Enter the directory containing the source code and type make
cd Basic make
Enter the following commands one at a time, they will run the two MPI applications (mpi-run doesn't handle applications that have command line arguments; for this you will need to use mpirun_rsh; see below for the details).
mpi-run 4 message mpi-run 4 hello
On some systems, the output from the hello or message programs may be garbled. If so, take a look at the code and try to figure out what is happening.
Note: At various times, you'll see messages about unloading a module, and you may ignore them.
Unloading compiler-dependent module mvapich2/1.5.1p1
If your program has command line arguments, use mpirun_rsh to launch it:
mpirun_rsh -np 4 -hostfile $PBS_NODEFILE ring -s 8192
The general form is as follows, where anything not shown within angle brackets must be typed exactly as shown:
mpirun_rsh -np <# processors> -hostfile $PBS_NODEFILE
Log in to kraken.nics.utk.edu and then do
cd /nics/d/home/baden/cse262-sp11/Profiles
Copy the files BASHRC and BASH_PROFILE into the files .bashrc and .bash_profile, that you found in your home directory when you logged in. Also copy the file MODULES into a new file called .modules
Lastly, create a symbolic link to the scratch file system area (On Kraken, jobs can only be run from scratch space).
cd $HOME ln -s /lustre/scratch/$USER Scratch
Log out and log in. In your home directory run the following commands
cp -p -r $PUB/Examples/MPI/Basic . cd Basic make qsubI 12
Entering this command will allocate 12 cores on an interactive node, which is shared with other users. You'll see the following output
qsub: waiting for job 1213203.nid00016 to start qsub: job 1213203.nid00016 ready
Wait until you receive a command prompt. You will find yourself in your home directory, $HOME. To run a program, you must copy the executable and the batch script file over to your scratch file area and cd to that space, as jobs may only be run from /lustre/scratch
cd Basic cp hello message run.pbs $HOME/Scratch cd $HOME/Scratch
Kraken uses the aprun command to run MPI jobs:
aprun -n 4 message aprun -n 4 hello
The output from both programs will be garbled on Kraken, whereas, on Trestles, the output from message was not garbled. Take a look at the code and try to figure out what is happening.
Note: At various times, you'll see messages about unloading a module, and you may ignore them.
Unloading compiler-dependent module mvapich2/1.5.1p1
If your program has command line arguments, put them after the name of the executable, as with mpirun on Lilliput:
aprun -n 4 ring -s 8192
The batch script is found in run.pbs. You may ignore the following messages that appear in the job output file
ModuleCmd_Switch.c(137):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
Some sample MPI applications have been installed in $(PUB)/Examples/MPI, and an "arch" file arch.intel.mpi has been installed in $(PUB)/Arch.
The Makefile for the present assignment uses the MPI version of the "arch" file, as we will use special versions of the compiler set up for MPI (the compiler commands are actually wrappers to gcc). Be sure to that your Makefile properly defines the PUB macros, and that it includes the correct "arch" file as follows.
PUB = /class/public/cse262-sp11 include $(PUB)/Arch/arch.intel.mpi
The MPI compilers (e.g. mpicc) are actually wrappers to the compiler back end (e.g. icc). By default, the back end compilers are the Gnu compilers. To use the Intel compilers, be sure to add the following to your .bashrc file in $HOME.
# Set the Intel compilers as the back end for mpicc and mpicxx export OMPI_CC=icc export OMPI_CXX=icpc
To run an MPI program, use the mpirun command.
mpirun -np 4 ring -t 10000 -s 16
The mpirun command provides the -np flag to specify the number of processes (Since Lincoln has 8-way nodes, you'll allocate them accordingly.) The value of this parameter should not exceed the number of available physical processors; the program will run, but very slowly, as it is multiplexing resources . The next parameter (ring) is the name of the executable. Any command line arguments come in the usual position, after the name of the executable.
To establish that your environment has been set up correctly, copy, compile and run the parallel "hello world" program. This program prints "Hello World'' from each process along with the process ID. It also reports the total number of processes in the run. The hello world program is found in $(PUB)/Examples/Basic/hello. To run the program use mpirun as follows:
mpirun -np 2 hello
Here is some sample output:
# processes: 2 Hello world from node 0 Hello world from node 1
To take timings, use MPI's timer function MPI_Wtime() to measure wall clock time (like a stopwatch). On Lilliput, your timings will reflect contention from other jobs. On Triton, you'll have dedicated access to processing nodes, but not the switch. Thus, to avoid the effects of switch contention on Triton, we’ll use a technique to filter out the variations. Run the program at least 3 times (for each input size and number of cores, or any variable that affects the running time) until you see a consistent timing emerge, such that a majority agree within 10%. If you observe a lot more variation try doubling the number of runs to see if you can obtain a better consensus. Exclude outlying timings, but note them in your report. Report the minimal running time rather than the average, as previously discussed in class.
To measure time, take the difference in times sampled by two successive calls to MPI_Wtime():
double t0 = MPI_Wtime();
// some code....
double t1 = MPI_Wtime();
double time = (t1-t0);
MPI_Wtime() reports time in seconds so there is no need to scale the value returned by the timer (resolution of MPI_Wtime using MPI_Wtick()). The man-page for MPI_Wtime() is found on line.
Recall that the running time of the application is that of the last processor to finish. Don't time individual iterations, but rather, time an entire run, but excluding initialization, calls to MPI_Init() and MPI_Finalize() and any other activities that are not representative of a run with many iterations. communication time.
It can be difficult to measure communication times accurately, owing to the short time durations involved. Therefore, we will use an indirect measurement technique. We do this by disabling communication and subtracting the resultant time from that obtained from a run that includes communication. To this end, you'll need to add an option to your program to disable communication calls like MPI_Send. To this end, implement the the -k command line option.
While the PPF package discussed in class can help sort out some I/O problems, it can only synchronize a single line of output. When you have multiple lines, and you are using PPF_Print() or just printf(), you will find that fflush() does not sort out the problem of garbled I/O.
The only sure way of getting I/O to stdout (or stderr) to properly synchronize is to use sprintf() to generate string output, and then collect via Gatherv() at the root. In other words, only 1 process should actually perform the I/O.
There is an example of how to manage I/O in the Ring example code:
$PUB/Examples/MPI/Ring/getHost.C
There is a nice tutorial with some worked examples. Man pages for the MPI calls may be accessed with the man command and they are also available On-line. More extensive documentation can be found at http://www-cse.ucsd.edu/users/baden/Doc/mpi.html. The on-line man pages for MPI calls used in this assignment appear below.
| MPI _Init | Initialize the MPI environment. Call this prior to making any MPI calls. |
| MPI_Finalize | Exit the MPI environment. Call this function before exiting your program. |
| MPI_Comm_size | Queries the total number of processes use in this program invocation. |
| MPI_Comm_rank | Queries this process's unique identifier, an integer called the rank |
| MPI _Send | Sends a message to another process, a return signifies that the message buffer may be reused. |
| MPI_Recv | Receives a message from another process, a return signifies that the message has been received. |
| MPI_Irecv | Immediate form of Receive. A return does not signify that the message has been received. To receive this assurance, you need to call MPI_Wait(). |
| MPI _Isend | Non blocking variant of Send(). A return does not signify that the message buffer may be reused. To receive this assurance, you need to call MPI_Wait(). |
| MPI_Wait | Return indicates that a previously posted Irecv (or Isend) has completed |
| MPI_Wtimer | Timer. |
| MPI_Gather | Gathers data from all processes onto the designated root process. All processes contribute the same amount of data. A collective routine, which all processes in the communicator must call. This may be be useful in debugging. |
| MPI_Barrier | Blocks calling process until all processes have checked in. A collective routine, which all processes in the communicator must call. |
Maintained by [Mon Mar 21 20:38:20 PDT 2011]