Getting started with Valkyrie

Change Log

  • Modified discussion about setting up environment [5/7/04]
  • Added discussion about removing authentication warnings [4/7/04]
  • Modified discussion about machine files [4/7/04]

    • Valkyrie

    The hardware platform for the course is a Beowulf cluster named Valkyrie. Valkyrie is managed by the ACS and runs the Rocks software developed at the San Diego Supercomputer Center. The system consists of 16 dual 1 GHz Pentium III CPUs, each with 1GB of RAM and running Linux. (Here is a web page telling you more about Valkyrie's CPUs.) A Myrinet switch provides low latency connectivity between the nodes.

    Valkyrie should only be used for parallel program development and measurement. If you need to use a Linux or UNIX system, please use your student account on the Solaris machines in the Advanced Programming Environment (APE) lab, located in APM 6426.

    ACS has set up a web page providing documentation on  how to use Valkyrie: http://acs.ucsd.edu/info/rocks.php

    Here are additional instructions about software we've added to simplify the use of the machine.
     

    Issues

  • Valkyrie is running, but with one caveat:
  • You may run with not more than 11 processes unless you use a machinefile. See discussion below.
  • There is a debugger on Valkyrie in ~/../public/debug-example. See the README file for details.
  • When you run an MPI program with mpirun, you may see warning messages about authentication: 
    Warning: No xauth data; using fake authentication data for X11 forwarding.
/usr/X11R6/bin/xauth:  error in locking authority file /home/cs160x/cs160x1/.Xauthority
    These messages may be ignored. If you want them to go away, then create the file config in ~/.ssh/ and include the following contents (A copy of this file is in ~/../public): 
    Host compute-0-0
  ForwardX11 yes
  Host *
    ForwardX11 no
  • Runaway processes
  • The Intel Math Kernel Library MKL has not yet been installed.

    • SSH

    If you haven't set up your environment for the secure shell, then you'll get the following message printed on your screen:

        It doesn't appear that you have set up your ssh key.
        This process will make the files:
        /home/cs160s/<your account>/.ssh/identity.pub
        /home/cs160s/<your account>/.ssh/identity
        /home/cs160s/<your account>/.ssh/authorized_keys

    Generating public/private rsa1 key pair.

    You will then be asked 3 questions shown below. Be sure to hit carriage return (entering no other input) in response to
    each question:

        Enter file in which to save the key (/home/cs160s/<your account>/.ssh/identity):
        Created directory '/home/cs160s/<your account>/.ssh'.
        Enter passphrase (empty for no passphrase):
        Enter same passphrase again:
        Your identification has been saved in /home/cs160s/<your account>/.ssh/identity.
        Your public key has been saved in /home/cs160s/<your account>/.ssh/identity.pub.
        The key fingerprint is:
        <several 2 digit hex numbers separated by :> <your account>@valkyrie.ucsd.edu

    Environment

    We'll be using the bash shell. Set up two files in your home directory: .profile and .bashrc There are default copies of these files stored in ~/../public: profile and bashrc (that is without the leading "dot").

    We'll use a special version of the Gnu C++ compiler that incorporate the MPI libraries. To use these compilers, you must have the following environment variable set: 

    export PATH=/opt/mpich/myrinet/gnu/bin:$PATH

    You may also want to set the MANPATH so you can access the MPI manual pages: 

    MANPATH=/opt/mpich/myrinet/gnu/man:$MANPATH

    These are set for you in the default profile we provided. (The default profile also defines the PATH to include /home/cs160x/public/bin.) Note: the ACS web page at http://rocks.npaci.edu/papers/rocks-documentation/book1.html is out of date. The correct PATH and MANPATH for MPI are as described here.

    Getting started

    We've set up some code to get your started in the directory ~/../public/examples. These will help acquaint you with process of running an MPI program. Compile and run the two programs in the subdirectory called Basic. Be sure to use the Makefile that we've supplied so you'll get the correct compiler and loader flags. The Makefile includes an "arch" file that defines appropriate command line flags for the needed to compile and load on Valkyrie. (If you want to run on other machines let us know)

    Running a program with mpirun

    Run your program with the mpirun command. (Be sure to run this program in your own directory) The command provides the -np flag so you can specify how many nodes to run on. There are 16 nodes numbered 0 through 15.

    To establish that your environment has been set up correctly, compile and run the parallel "hello world" program. This program prints "Hello World'' from each process along with the process ID. It also reports the total number of processes in the run. The hello world program is found in ~/public/examples/Basic/hello. To run the program use mpirun as follows: 

    mpirun -np 2 ~/../public/examples/Basic/hello

    Here is some sample output: 

    # processes: 2
Hello world from node 0
Hello world from node 1
    Note that any command line arguments come in the usual position, after the name of the executable. Thus, to run the Ring program(found in ~/../public/examples/Ring) on 4 processes with command line arguments -t 5 and -s 1024, type:
    mpirun -np 4 ~/../public/examples/Ring/ring -t 5 -s 1024

    Using machine files

    Sometimes you'll want to specify particular nodes to run on. To do this you need to specify a machines file listing the names of the physical nodes.    The command line sequence takes the following form

       mpirun -np <# NODES> -machinefile <MACHINE_FILE>

    The machine file contains a list of physical node names, one per line. The nodes are numbered from 0 to 15, and are named compute-0-0 through compute-0-15. (Each node contains 2 CPUs, but in effect you may use only 1 CPU per node.) Thus, to run the ring program with nodes 6, 7, 11, 14 as logical processes; 0-3, create the following file, say mfile: 

      compute-0-6
  compute-0-7
  compute-0-11
  compute-0-14
    To, run, type 
    mpirun -np 4 -machinefile mfile ~/../public/examples/Ring/ring -t 5 -s 1024

    Note: if you previously had set up a config file, you must ensure that the first entry in your machine file matches the first entry in the config file.

    Note, we have provided a program to generate randomized machine files. It is found in ~/../public/bin/randMach.py. (A python script). The command line argument specifies the number of processors in the machine file. For example, the command randMach.py 7 > mach was used to generate the following 7-line machine file: 

    compute-0-8
compute-0-0
compute-0-1
compute-0-15
compute-0-4
compute-0-9
compute-0-2

    Intel Math kernel library

    The Intel Math Kernel Library (MKL) provides high performance implementations of common numerical kernels like Matrix Multiplication and FFT.

    To link with MKL use the following on your load line 

    ${MKLPATH}/libmkl_lapack.a ${MKLPATH}/libmkl_ia32.a ${MKLPATH}/libguide.a -lpthread
    where ${MKLPATH} has been set as follows (i.e. in your .bash_profile file): 
    export MKLPATH=/opt/intel/mkl61/lib/32

    If you want to use the FFT or DFT, add -lm to your link line.

    Documentation can be found at http://developer.intel.com/software/products/mkl/docs/mklqref/index.htm or at http://www.phys.ksu.edu/~esry/Computing/Documentation/NewIntelLib/

    Runaway processes

    Sometimes  runaway processes will persist after a run. This might occur if you break a run using control/C.
    If you feel that the machine is slowing down run the command

                ganglia load_one | sort -n -k 2

    which will shows sorted load on each node.  Since there are 2 CPU per node the load should be not more that
    2.5 or so (i.e., there are maximum 2 PBS jobs running per node) unless there are other processes running on the node.
    This behavior could be due to valid user jobs or due to runaway processes. .

    If you observe a load average higher than 3  run the following command to check your processes on all nodes: 

    	cluster-ps <username>
    If you see that some processes are running, use the cluster-kill command to delete them: 
    	cluster-kill <username>
    If any nodes are down, and the commands will detect this (there should be no delay)

     

    Next, re-run the ganglia command shown above.  If the load average is still high, then check the status of all users
    on valkyrie by running the following

    cluster-fork 'ps awx' | sed -f ~/../public/bin/cl.sed

    The sed script will clear out a lot of unnecessary status information.

    As matter of etiquette, be sure and run cluster-ps before logging out. If you plan to be on the machine for a long time, it would be a good idea to run this command occasionally, and before you start a long series of benchmark runs.

    MPI

    MPI documentation is found at at http://www.cse.ucsd.edu/classes/sp04/cse160/testbeds.html#MPI. You can obtain man pages for the MPI calls used in the example programs described here at http://www-unix.mcs.anl.gov/mpi/www/www3/:


     Copyright © 2003 Scott B. Baden. Last modified: 11/03/03 10:30 AM