The login node should be used to edit, transfer files and compile code. It should not be used to run computational jobs; the class can easily overwhelm the login node if several users run on it simultaneously.
The 32 compute nodes are for running computationally intensive jobs and are divided into interactive and non-interactive subsets. Use a node interactively to debug your code, non-interactively to tune your code or to conduct experiments for your assignments.
To run interactively, request an interactive compute node via the qlogin command. Once you've been granted your node, you'll see the following output, though your job number and assigned node number will differ:
Your job 239 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 239 has been successfully scheduled.
Establishing builtin session to host compute-0-26.local ...
You are now running in a shell and have exclusive access, for 20 minutes, to a compute node with 8 cores.
Wait until you receive a command prompt. You will now be running in a new shell, within your $HOME directory. You may run commands on the compute node, edit and compile code, but you won't be able to submit non-interactive jobs.
As a courtesy to other users, exit the interactive session when no longer needed.
An interactive job may use 1 node, (max 8 cores) at a time. A user may run only 1 interactive job at a time. (If you attempt a 2nd session, you'll get the message: "Your "qlogin" request could not be scheduled, try again later.") To request an interactive shell session on a compute node, type the "qlogin" command. You can not only run jobs interactively, but also edit, debug, compile files etc. However, it is not possible to submit a non-interactive job during a qlogin session. You must submit non-interactive jobs via the SGE scheduler.
If your jobs are short, you could run interactively, to collect timings. But, since you will be repeating jobs multiple times and running with different settings, you'll likely prefer to submit non-interactive jobs.
When you are ready to collect measurements, make your production runs using the the batch queue. If your jobs are short, you could run interactively, but since you will be repeating jobs multiple times and running with differ settings, you'll likely decide to submit non-interactive jobs. A batch queue provides dedicated access to resources, enabling you to obtain highly reproducible measurements when running on a single node. (Note however, that multi-node MPI jobs will see switch contention from other jobs which share the switch). Thq queue manager implements a technique called space sharing, whereby each user gets a portion of the hardware until they are done using it. Compare this with time sharing, in which users get time slices of the available resources. We will use space sharing when collecting performance measurements, since the mechanism provides dedicated access.
While batch jobs provide dedicated access, the caveat is that you will have to wait for them to return. This method of submitting jobs may be new to you, and it takes some getting used to. However, if you think about it, it would be very expensive to give each user their own dedicated resources for an indefinite period of time. Most heavily subscribed high performance computing platforms employ space sharing.
We will use the Sun Grid Engine resource manager (SGE).
Submit your
batch job using the
qsub command:
qsub batch_bang.sh
where the batch_bang.sh is a job submission script containing the appropriate environment settings and one or more runs that you wish to make. (A sample job script (with comments) is provided in $PUB/Examples/batch_bang.sh)
A message will be printed at the screen
Your job 257 ("TESTJOB") has been submitted
where 257 is the job number.
You will need to make only minimal changes to the script, as noted in the file. (qsub scripts indicate options with a # mark. These are not comments unless there are two or more # in a row.)
Once your job has been submitted, it will wait in the queue until the specified resources are available. You may check on the status of your job using the qstat command. If you specify your user ID as in qstat -u baden you'll see only your jobs. The command output shows the job Ids and queues, for example:
job-ID prior name user state submit/start at queue slots ja-task-ID
---------------------------------------------------------------------------------------------
257 0.00000 TESTJOB baden qw 10/01/2012 08:29:45 1
You can remove jobs with the qdel command, and you only need to specify the leading number in the job ID:
qdel 257
Here is the full documentation for SGE, the batch system.
The provided script is set up to write all output to a file whose name contains a unique identifier, in TESTJOB.o257. (Stderr is merged with Stdout, but you can disable this option so that you get a separate stderr file e.g. TESTJOB.e257.)
If you modify the line with the -M option, you will be notified by email when your job started, finished or exited abnormally. (See the provided batch script and be sure to set the correct email address!) You may disable any of these 3 options independently as described in the script.
Note that if there are problems with the job you’ll be notified. For example, the following message indicates that the program could not be found
./e1: Command not found.
This might have been a result of not setting the correct directory in the cd command which appears in the provided job submission script. By default, the batch subsystem puts your job in your home directory (this includes interactive mode). So, you must cd to the correct working directory.
You many submit batch jobs from the login node only. If you attempt to submit a job from a compute node, you'll get an error message
Unable to run job: denied: host "compute-0-17.local" is no submit host. Exiting.
The job scheduler manages resources via batch queues. 1 queue has been set up for the course: "debug.q" ("normal.q" has been removed) The debug queue provides the following capabilities:
Both INTERACTIVE and BATCH jobs
A job can use any available compute node
A maximum of 1 node = 8 CPU cores
A maximum of 1 job per user can be run in this queue at a time;
thus it is not possible for interactive and batch jobs to be running simultaneously
Maximum runtime = 20 minutes
After the specified hard runtime limit, Sun Grid Engine aborts the job using the SIGKILL signal. If the similar soft limit is reached, Sun Grid Engine warns the job by sending the job the SIGUSR1 signal. This behavior is effective only if the job catches and handles that warning signal.
There are various reasons why a job won't run. For example, there may be other jobs ahead of it. But this situtation cannot persist indefinately. To see all the jobs in the queue, run the following command
$ qstat -u "*" job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- 2960 0.55500 MSORT baden qw 10/26/2013 22:41:31 1
Job 2960 is waiting ("qw"). To see why, we use the qstat commmand:
$qstat -j
Jobs can not run because the resource requirements can not be satisfied
2960
We then look at the job more closely
$ qstat -j 2958
we got back a lot of output. Among the output is a line that says that we can't run in "debug.q"
(-l h_rt=1800) cannot run in queue "debug.q" because of cluster queue
The string "h_rt" stands for the "requested max runtime limit". We are asking for 1800 seconds or 30 minutes. But our limit is 20 minutes, or 1200 seconds. The batch system should be more forthcoming about this, so be sure to double check your batch files.
If in doubt about queue limits, you can view the queue's configuration parameters via the qconf command:
qconf -sq debug.q
Or better yet, search for the line of output that reports h_rt for the queue, using the "grep" command
$ qconf -sq debug.q | grep h_rt h_rt 00:20:00
Maintained by [Thu Jan 9 21:24:41 PST 2014]