The login node should be used to edit, transfer files and compile code. It should not be used to run computational jobs; the class can easily overwhelm the login node if several users run on it simultaneously.
The 32 compute nodes are for running computationally intensive jobs and are divided into interactive and non-interactive subsets. Use a node interactively to debug your code, non-interactively to tune your code or to conduct experiments for your assignments.
An interactive job may use 1 node, (max 8 cores) at a time. A user may run only 1 interactive job at a time. (If you attempt a 2nd session, you'll get the message: "Your "qlogin" request could not be scheduled, try again later.") Request an interactive shell session on a compute node by typing "qlogin". You can not only run jobs interactively, but also edit, debug, compile files etc. However, it is not possible to submit a non-interactive job during a qlogin session. You must submit non-interactive jobs via the SGE scheduler.
If your jobs are short, you could run interactively, to collect timings. But, since you will be repeating jobs multiple times and running with different settings, you'll likely prefer to submit non-interactive jobs.
A batch queue provides dedicated access to resources, enabling you to obtain highly reproducible measurements when running on a single node. (Note however, that multi-node MPI jobs will see switch contention from other jobs which share the switch). Thq queue manager implements a technique called space sharing, whereby each user gets a portion of the hardware until they are done using it. Compare this with time sharing, in which users get time slices of the available resources. We will use space sharing when collecting performance measurements, since the mechanism provides dedicated access.
While batch jobs provide dedicated access, the caveat is that you will have to wait for them to return. This method of submitting jobs may be new to you, and it takes some getting used to. However, if you think about it, it would be very expensive to give each user their own dedicated resources for an indefinite period of time. Most heavily subscribed high performance computing platforms employ space sharing.
We will use the Sun Grid Engine resource manager (SGE).
Submit your
batch job using the
qsub command:
qsub batch_bang.sh
where the batch_bang.sh is a job submission script containing the appropriate environment settings and one or more runs that you wish to make. (A sample job script (with comments) is provided in $PUB/Examples/batch_bang.sh)
A message will be printed at the screen
Your job 257 ("TESTJOB") has been submitted
where 257 is the job number.
You will need to make only minimal changes to the script, as noted in the file. (qsub scripts indicate options with a # mark. These are not comments unless there are two or more # in a row.)
Once your job has been submitted, it will wait in the queue until the specified resources are available. You may check on the status of your job using the qstat command. If you specify your user ID as in qstat -u sbaden you'll see only your jobs. The command output shows the job Ids and queues, for example:
job-ID prior name user state submit/start at queue slots ja-task-ID
---------------------------------------------------------------------------------------------
257 0.00000 TESTJOB baden qw 10/01/2012 08:29:45 1
You can remove jobs with the qdel command, and you only need to specify the leading number in the job ID:
qdel 257
Here is the full documentation for SGE, the batch system.
The provided script is set up to write all output to a file whose name contains a unique identifier, in TESTJOB.o257. (Stderr is merged with Stdout, but you can disable this option so that you get a separate stderr file e.g. TESTJOB.e257.)
If you modify the line with the -M option, you will be notified by email when your job started, finished or exited abnormally. (See the provided batch script and be sure to set the correct email address!) You may disable any of these 3 options independently as described in the script.
Note that if there are problems with the job you’ll be notified. For example, the following message indicates that the program could not be found
./e1: Command not found.
This might have been a result of not setting the correct directory in the cd command which appears in the provided job submission script. By default, the batch subsystem puts your job in your home directory (this includes interactive mode). So, you must cd to the correct working directory.
You many submit batch jobs from the login node only. If you attempt to submit a job from a compute node, you'll get an error message
Unable to run job: denied: host "compute-0-17.local" is no submit host. Exiting.
The job scheduler manages resources via batch queues. 2 queues have been set up for the course: "debug.q," and "normal.q" and you may run in either queue. The queues differ according to the resources they provide:
Both INTERACTIVE and BATCH jobs
A job can use any available compute node
A maximum of 1 node = 8 CPU cores
A maximum of 1 job per user can be run in this queue at a time
No limits for memory usage
Maximum runtime = 30 minutes
Only BATCH jobs
A job can use any available compute nodes
A maximum of 2 nodes = 16 CPU cores
A maximum of 1 job per user can be run in this queue at a time
No limits for memory usage
Maximum runtime = 10 minutes
After the specified hard runtime limit, Sun Grid Engine aborts the job using the SIGKILL signal. If the similar soft limit is reached, Sun Grid Engine warns the job by sending the job the SIGUSR1 signal. This behavior is effective only if the job catches and handles that warning signal.
Maintained by [Mon Feb 18 17:50:19 PST 2013]