CSE160 Wi16 - Getting Started With Bang



The main computing platform for the course is the CSME program’s Bang cluster. This cluster consists of 35 Dell PowerEdge 1950 servers, which are interconnected with an infiniband switch. Each compute server has two quad-core Intel Xeon E5345 (“Clovertown”) 2.33GHz CPUs, for a total of 8 cores per compute node, and 16GB of RAM. Bang runs Rocks Cluster Linux 6.1, and the Rocks-bundled “SGE” batch job scheduler. We will use the Gnu compiler suite, version 4.8.4, which implements the C++11 standard, notably including C++11 threads.

Access

Bang’s processing hardware includes 3 components:

To access Bang, Login to your account on the cluster’s login server node (not the front end server node):

ssh YourADLogin@ccom-bang-login.ucsd.edu

Use your current campus Active Directory (AD) login and password; these are the same as you use for the campus UCSD-PROTECTED wireless network, campus VPN service, and most of the ACMS servers/labs.

Students may reset their passwords with the “ACMS Global Password Change Tool” at

http://acms.ucsd.edu/students/gpasswd.html

Faculty and staff may reset their AD password at: http://adpass.ucsd.edu

Setting up your environment

The first time you log in to Bang, you’ll be asked to set up your ssh key:

    It appears that you have not set up your ssh key.
    Enter passphrase (empty for no passphrase):

In response to the question, use an empty passphrase, that, just hit return without any other input. If you use a non-empty passphrase, then the compute nodes cannot authenticate the login, resulting in an error message.

We’ll be using the bash shell. You may change shells if you wish, however, all instructions will be given in terms of the bash shell (It is not difficult to convert between the two, so post to the Bang folder on Piazza if you have aliases to donate)

To get your environment set up properly, modify your .bash_profile. In particular, append the file BASH_PROFILE, located in /share/class/public/cse160-wi16/Profiles, to the end of the .bash_profile file, located in your home directory, immediately after the following text appearing in .bashrc:

#####################################################################
# **** Place any else below.                                     ****
#####################################################################

You will now have various handy command aliases at your disposal. You’ll also have a pre-defined environment variable, called PUB, where all public files for the course are located (except for the assignment starter codes, which will be made available via GitHub):

/share/class/public/cse160-wi16

Now, you are ready to start running jobs. To run jobs on Bang, read Running jobs on the Bang Cluster.

Obtaining Multicore Processor Characteristics

Here is a description of the memory hierarchy characteristics of Bang’s multicore processors. More generally, here is how to Explore your processor’s attributes.

Useful software

Valgrind Tool Suite

On Bang, you may use the valgrind tool suite to check for memory leaks, and also obtain detailed cache memory performance statistics, down to the individual line of code. The man page for valgrind gives detailed information but the on-line page Valgrind User Manual is more accessible. For cache statistics, use cachegrind. Here is a short synopses to get you started.

Cachegrind can be run by compiling your code with the -g option. We’ve set up the “arch” file so that you can achieve this effect by building your code with valgrind=1 on the make command line, or putting the following line inside your makefile before the point where the “arch” file is included (optimization will be retained at whatever level you set it at):

valgrind = 1

Next, remove all old binaries and recompile as

make valgrind =1

Run the binary under control of valgrind:

valgrind --tool=cachegrind ./lu -n 1024 -l -t 1
You will obtain output giving summary statistics, which among other things tell us that the L1 miss rate for data reads is 6.4% (92,331,844/1,442,308,403). The Last Level cache’s (in the case of the Clovertown processor, which has only 2 levels of cache, this is L2) read miss rate is 4.2%. See the on-line documentation of cachegrind for more details. 
==7448== 
==7448== I   refs:      4,349,513,529
==7448== I1  misses:            1,904
==7448== LLi misses:            1,873
==7448== I1  miss rate:          0.00%
==7448== LLi miss rate:          0.00%
==7448== 
==7448== D   refs:      2,161,554,669  (1,442,308,403 rd   + 719,246,266 wr)
==7448== D1  misses:       92,874,408  (   92,331,844 rd   +     542,564 wr)
==7448== LLd misses:       62,100,061  (   61,571,458 rd   +     528,603 wr)
==7448== D1  miss rate:           4.2% (          6.4%     +         0.0%  )
==7448== LLd miss rate:           2.8% (          4.2%     +         0.0%  )
==7448== 
==7448== LL refs:          92,876,312  (   92,333,748 rd   +     542,564 wr)
==7448== LL misses:        62,101,934  (   61,573,331 rd   +     528,603 wr)
==7448== LL miss rate:            0.9% (          1.0%     +         0.0%  )

You’ll notice an output file of the form cachegrind.out.<pid> (<pid> is the program’s process ID). This output file can be used by the cg_annotate command to obtain line-by-line cache performance statistics as follows (cachegrind.out.7448 was the output for the above run)

cg_annotate --auto=yes cachegrind.out.7448
Here is a portion of the output, edited for clarity purposes.
We can see that the one line of code that performs the rank-1 update, accounts for nearly all the data reads and level-1 cache misses as well. 
--------------------------------------------------------------------------------
           Ir               Dr       D1mr       DLmr          Dw   file:function
--------------------------------------------------------------------------------
4,349,513,529    1,442,308,403 92,331,844 61,571,458 719,246,266   PROGRAM TOTALS
4,321,765,796    1,435,345,996 91,784,271 61,067,684 715,831,318   elim.cpp:elim()
elim.cpp
--------------------------------------------------------------------------------
           Ir               Dr         D1mr         DLmr      Dw
....
    2,113,536           10,240      2,670        908           0     for ( i = k+1; i < cb.N; i++ ) {
   19,902,564        2,095,104  1,173,520    222,286           0         double Aik = A[i][k];
            .    .          .          .           .                     double *Ai = A[i];
        2,156                0          0          0           0         for ( j = k+1; j < cb.N; j++ )
4,291,309,352    1,430,083,512 89,441,868 60,636,776 714,781,694              Ai[j] -= Aik * A[k][j];
            .    .          .          .           .           .     }

Use cachegrind with care as it will slow down your program significantly. You may need to run a batch job, as the time limit for interactive use may be too short.