Programming Lab 3: Cardiac Electrophysiology Simulation using MPI

Due: Monday, November 23, 2015 at 9pm


Changelog

Date Description
6-Nov-15 Original posting


Contents


Part 0. The Assignment

In this assignment you'll develop a high performance MPI implementation of the heart electrophysiology simulator discussed in class. You'll also optimize the inner loops, and optionally use SSE intrinsics to realize vectorization (extra credit) Conduct experiments on Stampede to assess performance. For development, you may also use the bang cluster, which has more plentiful interactive resources. The assignment includes a report, ~10 pages long (including figures), which is important part of the assignment.

Your grade will based on 4 factors: performance attained by your implementation; correctness, well-displayed results together with comments and explanations and insight into how you realized your reported performance. In determining your grade for the assignment, explanations and insight will be emphasized more heavily than performance itself, unless there was little improvement in performance over the provided code.

In the spirit of friendly competition, post your performance results to the HW3 Piazza folder.

Part 1. Getting Started with GitLab

All assignments will be turned in using UCSD's GitLab and the instructions are identical for the previous assignments. The turnin procedure is described below in the section Turning in your assignment. In this section, we'll go over how to set up your repository.

You must follow these steps to get the starter code and to get set up to do assignment 3

Refer to the link below for instructions on how to obtain the starter code to get started on your assignment

A3 Git Instructions

Part 2. Assignment Checklist

  1. Parallelize your code using MPI.
    1. Your code must support arbitrary one- and two-dimensional processor geometries consistent with the command line flags -x -y that specify the geometry (see the discussion below about cmdLine.cpp). However, you may find it easier to start with one-dimensional partitions, which give reasonable performance for small numbers of cores (say, not more than 8) and then add two-dimensional partitionings. This is consistent with the experiments described below. 2D decompositions introduce additional complexity to ghost cell handling, but 1D decompositions will enable you to focus on learning MPI.
    2. You need support square global meshes only. However, when you partition a square mesh using a rectangular processor geometry, the resultant submeshes will be rectangular therefore handle this case, too.
    3. Your code must correctly handle the case when the processor geometry doesn't divide the mesh evenly. (It may be difficult to ensure that, for certain nearby values of N, performance may vary by 25% or more. You may explore techniques for improving stability of the running times across nearby N, but don't get bogged down.) In particular, it must divide the mesh as evenly as possible using partitionings having the following property:

      The amount of work assigned to the most heavily and most lightly-loaded process along a given dimension will differ by not more than one row (or column) of the solution array.

  2. Optimize your code to improve performance. Performance goals will be discussed in class along with some useful performance programming techniques. To help you assess your performance, we've provided reference binary executables (see below).
    1. Optimize communication to enable the application to scale strongly: for a sufficiently large problems, performance should be proportional to the number of cores.
    2. Optimize the inner loops. There are two possibilities.
      1. Improve cache locality
      2. Get the code to vectorize (work extra credit)
      You may combine both optimizations. See below for details about vectorization and extra credit.

  3. Conduct strong scaling studies. You'll run over different ranges of parallelism:
    1. 1 to 8 cores
    2. 16 to 64 cores
    3. up to 1024 cores

    Each of the above ranges should use a different value of N, which we'll call N0, N1 and N2. Do not attempt to run beyond 64 cores unless you have observed robust strong scaling results on 64 cores (goal #2). Similarly, do not attempt the 16 to 64 core goal unless you have obtained robust strong scaling results on 8 cores of Bang. This procedure is intended to save our allocation, and it's important to never run a poorly performing program on larger numbers of cores, especially at the highest end of 1024 cores (When user's apply for large allocations on supercomputers, they are. required to demonstrate good scaling)

    1. Start on Bang. Set N0=400 and # iterations (niter)=500; the provided code will complete in about 2 seconds on one core of Bang. (There is a batch script called batch_serial.sh) Compare the single processor performance of your parallel MPI code against the performance of the original provided code to asses any overheads introduced in parallelizing the code. These overheads should be negligible; if not, investigate the source of the overhead and eliminate it.
    2. Next, set niter=2000. Conduct a strong scaling study: observe the running time as you successively double the number of cores, starting at p=1 core and ending at 8 cores, while keeping N0 fixed. We've picked a small value of N0 so that your runs should scale nearly perfectly to 8 cores. Your code should run in under 10 seconds on 1 core of bang and run in about 1 second on 8 cores. (There is a batch script called p8-bang.sh) Use a 1D processor geometry to avoid the need to search for the optimal geometry. Your goal here is to gain insight into using MPI effectively and to collect and collate your performance data.
    3. Measure communication overhead on 8 cores. To do this, use the indirect method, described below, for timing the simulation without communication. You should find that there is no communication overhead. Why is this the case?
    4. Repeat steps (b) and (c). Conduct a strong scaling study on 16 to 64 cores on Stampede and measure communication overhead. Keeping i=2000, set N1=1800. This value of N ensures that runs on 64 cores will complete n about 1 second on Stampede. (Note that for our simulator, each Stampede core runs our simulator at about 3 times the rate of Bang's cores.) We have set up a batch file called N1_stampede.slurm.

      Once you are running beyond 8 cores, you will need to use 2D processor geometries. For each value of p, determine the optimal geometry, and use that geometry when reporting performance. You may find that more than one geometry yields the maximum performance. For the largest value of p (64) report all geometries that yield the top performance. For our purposes, we'll treat the top contenders as those coming within 10% of the top performer(s). Can you see a pattern emerge in the choice of geometry? This may help you prune the search for optimal geometry at the next level of parallelism (step (e), coming next). It should not be necessary to conduct an exhaustive study of optimal geometry, as you may be able to eliminate obvious non-performers, in particular 1D geometries. Avoid these and save the time for more interesting studies! Develop an intuition about geometry selection based on empirical observations and share your insight on Piazza.

      You will now see a noticeable increase in the cost of communication. Why is this the case?

    5. Next attempt to scale to 128, 256, 512 and 1024 cores. Get as far as you can, but proceed carefully as our bank account is charged for # cores * time for each run! If you get to 512 cores you should be running at more than a Teraflop/sec (1000 GFlops/sec). Start with P=128. Keeping i=2000, set N2=8000. This ensures that runs on 128 cores will complete in under 10 seconds. We have set up a batch file called N2-128_stampede.slurm. Unlike the previous runs (step (d)), which ran all jobs over the range of 16 to 64 cores from one batch script, you must run with only one value for the number of cores. This is necessary to avoid wasting processors to ensure that all cores get used by the job. Do not allow the time limit for the job to exceed 2 minutes. This limit has been set up in the job script and should not be changed.

      Use your intuition in studying 16 to 64 processors to develop a strategy for picking the optimal geometries. Avoid exhaustive search and do not attempt 1D decompositions at scale. Before attempting this study, be sure that your code scales strongly on up to 64 cores and that all known bugs have been fixed. Display the performance (in GFlops/s) on up to the larger number of cores for which your code ran correctly.

  4. Extra credit (25%) You'll notice that we've provided both fused and unfused versions of the main computational loops in solve.cpp. As discussed in class, the C++ compiler cannot vectorize this code as written. One way of inducing the code to vectorize is to use the SSE intrinsics explored in the first programming assignment. The other way is to use the feedback from the compiler's optimization report as a guide to help you determine what is preventing vectoriztion. Such modifications may include adding special Intel-defined pragmas along with the restrict keyword for pointer arguments (see A Guide to Vectorization with Intel C++ compilers, §6.2.2, at http://tinyurl.com/qc46qjs.) The optimization report is enabled via the compiler option -qopt-report=n and you'll get a file called solve.optrpt after compiling solve.cpp. Our "arch" file defines n=1, but values up to n=5 are supported and may tell you more. You can change the value of n as shown in the provided Makefile.

    One obvious required change (with either approach to vectorization) is that the accesses to memory must align on 16 byte boundaries. You can see that when storage is allocated in helper.cpp that it is forced to align on 16 byte boundaries. This is a requisite step. But when N is not divisible by 16, you'll have to come up with some way of ensuring that loops neither begin nor end on 16 byte boundaries. One way is to pad the arrays, but there may be other solutions that avoid the storage overhead of padding.

    We will award partial credit of 25% if you can improve performance by 50% on 64 cores. You may assume that N is an even number, as this avoids a time-consuming corner case. Note that, the compiler will attempt to vectorize certain code, however it will not succeed in vectorizing the most computationally intensive parts of the application and this is where your hand optimizations will make a significant difference in performance.

    Report any performance anomalies or outstanding issues in your code. We will provide some partial credit for test cases your code doesn't pass, if we receive a carefully documented description. Negative results can be a valuable means of learning, so we are giving you an incentive to explore this venue. To save scarce computing time, we've set the number of cores to 64 for this performance goal. But if you can show 60% improvement on 1 core, we'll award some partial credit (this is a hint that once you've vectorized the code, you may need to improve the scaling properties of your code by reducing communication overhead) Note that, while Sorken provides a slightly more advanced SSE than Stampede, it is a useful alternative to developing your SSE code, and may provide better turnaround.

    Not all code is amenable to optimization and code that consumes little of the application's running time won't make a significant difference even if it can take advantage of SSE instructions. The code used to to set up the ghost cells is case an point. Your efforts will be much more rewarding if you optimize the PDE and ODE code. Vectorize these two parts on at a time. We have set up a simple vectorization example to help get you started. However, in the example code the stencils are one dimensional and there are no alignment issues. In the PDE and ODE loops, you'll have two dimensional stencils and array strides are not integral powers of two in length.

    The SSE coding for this assignment will be different what you use to hand vectorize for matrix multiply. The reason is that the data access patterns used by the applications are fundamentally different. The Aliev-Panfilov code operates on nearest neighbors rather than small matrix multiplies. You'll still want to work with blocks of registers, however, so there are some common practices.

  5. Report and comment on your results. Document your work in a well-written, report, of roughly 10 pages, including plots. Present and discuss your experimental results and your optimization strategies and offer insight into how your code realized the performance your observed.

    The general requirements for the document are described in §8.2. Discuss how you modified the code to improve performance and giving a clear evaluation of your code's design, including bottlenecks of the implementation. If you improved performance of your code in a sequence of steps, document the process. It may help clarify your explanation if you summarize the information in a list or a table in your report. Include pseudo code listings to illustrate your discussions as needed, but do not include full code listings as these will be in your electronic turnin.

    Thoroughly explain the results presented in your plots, relating them to how you designed the code. Discuss the design decisions you made to improve performance, including any negative results. If you have negative results, discuss them, too. They may give insight into performance bottlenecks especially if the bugs prevented you from collecting all your data.

    As needed, devise models, describe experiments and plot data to explain what you've observed. In explaining optimal processor geometry, consider the two components of performance: the cost of computation and the cost of communication. In general, the optimal geometry effects a balance between these two costs.

  6. Requirements.

Part 3. Cardiac Electrophysiology Simulation

Simulations play an important role in science, medicine, and engineering. A cardiac electrophysiology simulator like the one we are using in this assignment can be used for clinical diagnostic and therapeutic purposes. To reduce computational costs, simplifying assumptions are often made to make the simulations feasible. In our clinical example, time may be of the essence in determining an appropriate life-saving treatment.

Cell simulators entail solving a coupled set of a equations: a system of Ordinary Differential Equations (ODEs) together with a Partial Differential Equation (PDE). If you are interested in learning more about cardiac modeling peruse www.labheart.org or thevirtualheart.org or visit the web page of the UCSD Cardiac Mechanics Research Group and the Simula Computational Cardiac Modeling Department

Our simulator models the propagation of electrical signals in the heart, and it incorporates a cell model describing the kinetics of the membrane of a single cell. There can be different cell models, of varying degrees of complexity. We will use a model known as the Aliev-Panfilov model, that maintains 2 state variables, and solves one PDE (The PDE couples multiple cells into a system). This simple model can account for complex behavior such as how spiral waves break up and form elaborate patterns. Spiral waves can lead to life threatening situations such as ventricular fibrillation.

Our simulator models electrical signals in an idealized biological system in which voltages vary at discrete points in time, called timesteps, at the discrete positions of a mesh. The simulator is an iterative procedure that updates the simulation state variables (on the mesh) many times over a sequence of discrete timesteps. A simulation will run for some amount of simulated time, which is a real number. (For our purposes, the units of time are unimportant.) The timesteps subdivide the simulation time line that starts at t=0 and ends at some specified time t=T. The unit of subdivision is called Δt, or the time discretization. It is important to keep in mind that when we use the term "time," we need to specify whether we are talking about real-valued simulation time, or the discrete timestep. We will use the fully qualified name to avoid ambiguity. By convention, we number the discrete timesteps with integers. Thus, if we say that we are at timestep k of the simulation, then the current simulation time is kΔt.

To discretize space, we'll use a uniformly spaced two-dimensional mesh, shown in the left side of the figure (Irregular meshes are also possible, but are more difficult to parallelize).

[Mesh] [5-pt stencil]

At each time step, the simulator updates the voltage according to nearest neighboring positions in space and time. This is done first in space, and next in time. Nearest neighbors in space are defined on the 4 Manhattan directions only: north, south, east and west, as shown on the right side of the figure.

In general, the finer the mesh — and hence the more numerous the points — the more accurate the solution, but at an increased computational cost. In a similar fashion, the smaller the timestep, the more accurate the solution, but at a higher computational cost. Thus, if h is the mesh spacing, Δt is the time discretization, then the running time is proportional to (Δt×h2)-1. However, to simplify our performance studies, we will run our simulations for a given number of iterations rather than a given amount of simulated time.

The simulator algorithm

The simulator maintains two state variables characterizing the cell electrophysiology model: excitation (E) and recovery (R). Since we are using the method of finite differences to solve the problem, we discretize the variables E and R by considering the values only at a regularly spaced set of discrete points. We represent the variables as 2D arrays: E and R, respectively. (our notation will use a pseudo C syntax, E[i,j] is synonymous with E[i][j]) The PDE solver also maintains the voltage at the previous timestep, Eprev.

Here is the formula for solving the PDE, where the superscripts t and t-1 refer to the current and previous timestep, respectively, and the constant α is defined in the simulator (see apf.cpp and solve.cpp):

Et[i,j] = Et-1[i,j] + α*(Et-1[i+1,j] + Et-1[i-1,j] + Et-1[i,j+1] + Et-1[i,j-1] - 4*Et-1[i,j])

Here are the formulas for solving the ODE:

E = E - Δt*(k*E * (E-a) * (E-1)+E * R); R = R + Δt*(ε+ μ1*R / ( E+μ2)) * (-R-k*E * (E-b-1));

The variables k, a, b, ε, μ1 and μ2 are constant floating point scalars, and are defined in the simulator. References to E and R correspond to whole arrays. Expressions involving whole arrays (that is, array names without subscripts) are pointwise operations on corresponding array elements. We interpret a pointwise whole array operation as follows. If A, B and C are 2D arrays of size N × N, then A = B + C is equivalent to the following loop nest:

for (i = 0; i < N; i++)
    for (j = 0; j < N; j++)
        A[i][j] = B[i][j] + C[i][j];

When an array is combined with a constant, e.g. (E-a), the constant is extended to cover each element of the array, i.e. we compute the E[i][j] - a for all i and j.

To organize our work, we observe the following bookkeeping conventions. We divide each axis into N-1 subintervals using N points positioned uniformly along the line at positions x=i×h, i=0, 1, 2, ..., N-1. In order to properly handle updates next to a boundary, we pad the boundaries with a cell on each side. (This padding will simplify the code and you'll see comments about it in solve.cpp.) This is shown in below, for 1 axis of the mesh drawn above, where the mesh spacing h = 1/(N-1). The points labeled 'o' correspond to the physical boundary, the points labeled 'P' are padding.

i=0   i=1    i=2                           i=N-1   i=N   i=N+1
P------o------|------| ...... |------|------|------o------P
       ^                                           ^
       |                                           |
      x=0.0  x=h    x=2h                 x=1.0-h   x=1.0

Consider the case where we set N=64. We allocate an array of size 66 × 66, but we solve the problem on a central 64 × 64 submesh. Thus, our loops range from 1 to N, that is, 1 to 64. Points 0 and 65 correspond to padding points.


Part 4. The provided starter code

We've provided you with starter code that implements the Aliev-Panfilov solver, which was discussed in class. We have also provided an executable (stripped) so that you may compare your results with our reference. The file is found in $PUB/HW/A3/apf-ref on both Stampede and Bang. It is possible to improve on the performance of the reference, since it doesn't implement vectorization via SSE, nor does it include any significant optimizations of memory locality.

All of your changes will made to just two files:

solve.cpp       helper.cpp

Solve.cpp contains the solver, helper.cpp provides various helper functions such as storage allocation. (You may therefore change the storage allocation scheme if you wish.) The solver is computationally intensive, and so the focus of your parallelization and performance programming effort will be on this routine. You'll notice that the solver includes both the fused and unfused versions of the main computational loop. You can control which version to use via the FUSED macro, defined in solve.cpp.

To build your code, use the provided Makefile, which sets the appropriate flags for the compiler via the "arch" file located in $(PUB)/Arch/arch.gnu-c++11.generic. The Makefile produces a target called apf. Do not change the name of this target. While you may modify any files not listed above, remember that we will substitute fresh copies when we grade your assignment, so don't really on any custom changes as they will be lost during the grading process.

To get you started, we have included some basic calls to MPI in the provided code. These calls are conditionally compiled under control of the _MPI_ C-preprocessor macro. To facilitate conditional compilation, the Makefile will cause this macro to be asserted when you set the mpi=1 on the Make command line.

make mpi=1

You can achieve the same effect by uncommenting the Makefile variable definition in the provided Makefile:

# If you want to compile with MPI enabled,
# uncomment this line
# mpi = 1

Additional code is included to show you how to restrict output to one MPI process, for example, when printing information to stdout. We have also included some very basic code to support SSE intrinsics, which is enabled via the vec=1 flag (which asserts the SSE_VEC C pre-processor macro, enabling conditionally compiled code). A small but complete example code showing the use of SSE intrinsics is found in $PUB/Examples/Vec.

We have also provided batch scripts, batch_strong_p8.sh, for conducting the strong scaling study on Bang, as well as various slurm scripts for Stampede. The stampede scripts all end in stampede.slurm. Be sure to use the correct batch file for the capability you require; for example, the parallel batch job won't run correctly if you haven't enabled MPI when building your code. It will deadlock, tying up resources for the specified duration of the job. The starter code enables you to input the processor geometry via two command line flags: -x and -y. The code will report an error if the product of x and y do not equal the number of cores you requested when involving MPI (-np for mpirun or ibrun)

You'll notice that the provided code uses a different naming convention for the bounds of the computational box. In particular, while we use the single letter N in this document, the code uses serrate bounds for the x and y axes: the variables m and n. This convention enables the code to accommodate non-square local problem sizes that arise when the data has been partitioned on non-square processor geometries. Also, we "flatten" the arrays so that all solution arrays are 1-dimensional and are subscripted with just 1 index.

Also note the conventions used to index the 2D solution arrays: the indices are swapped, with the i index being the most rapidly varying index, that is, indexing along the last dimension of the array. Thus, array elements in the same row have different i indices and elements i-1, i and i+1 are adjacent in memory.

The simulator has various options, controlled via command line arguments (parsed in in cmdLine.cpp), with default values set in apf.cpp:

-n <int> Number of mesh points in the x and y dimensions
-i <int> Duration of simulation in iterations
-s <int> Print statistics, L2 and L∞ norms of the excitation variable at specified regular intervals (in timesteps).
This feature is useful when debugging.
-p <int> Plot the excitation variable, at the specified regular intervals (in timesteps). Useful when debugging.
-x <int>
-y <int>
x and y-axis of the the processor geometry (Used only for your MPI implementation)
-k Disable communication (Used only for your MPI implementation)
-d Enable debugging output. Sets a variable called `debug` that you may use to debug your code.

Thus, the following command line will conduct the simulation on a 150 × 150 box, run for 400 iterations, plot the excitation variable every 20 iterations, and print out summary values of the excitation variable every 10 iterations.

    ./apf -n 150 -i 400 -s -10 -p 20

If we leave off the -s and -p flags, then the output is as follows. The last line of output, beginning with the string "@" outputs performance data along with relevant parameters, that is suitable for automated extraction from a sequence of runs.

[Simulation begins] Wed Nov  4 14:30:41 2015

dt= 0.0639292, # iters = 400
m x n = 150 x 150
processor geometry: 1 x 1

Compiled with MPI ENABLED

[Simulation completes] Wed Nov  4 14:30:41 2015

End at iteration 400
Max norm: 9.928374e-01, L2norm: 6.125795e-01
Running Time: 0.216294 sec. [1.17 GFlop/sec]

   M x N   px x py Comm?   #iter  T_p, Gflops        Linf, L2
@ 150 150    1 1      Y     400  0.2163 1.165    9.92837e-01 6.12580e-01

The plotting capability, controlled via the -p flag, uses gnuplot, and may be useful in debugging. Do not enable plotting when you are measuring running times for your performance studies. The plotter capability only works on a single core and is discussed in more detail later in this writeup.


Part 5. Testing your code

Your code will be tested for correctness and performance against various combinations of the N, niter and processor geometry. Test your code by comparing against the provided implementation. You can catch obvious errors using the graphical output (see below), but a more precise test is to compare two summary quantities reported by the simulator: the L2 norm and the L∞ norm, the maximum value (magnitude) of the excitation variable. (These are included in the 1 line performance output). The former value is obtained by summing the squares of all solution k values on the mesh, dividing by the number of points in the mesh (called normalization), and taking the square root of the result. For the latter measure, we take the "absolute max," that is the maximum of the absolute value of the solution, taken over all points. While there may be slight variations in the last digits of the reported quantities, variations in the first few digits indicate an error.

For example, you may find that after adding a new performance optimization that your code indeed ran faster but now the L2 of the errors differ in the 5th decimal place, while the Max errors are the same. There is probably an error in your code. The L2 norm is giving a clearer picture of what is going on here, as it takes the square root of the sums of the squares of the differences (look at the code to see how the norms are computed) the max takes the max of the absolute values of those differences. So what's happening is you are probably introducing small errors in many elements of the solution mesh, but these are masked by some maximum error value and hence not visible with the max norm.
The error is small, so any penalty (if we even assess in this case) would not be severe. Write up your results and take some time time to ask 3 questions, which you should add as addendum to your plots:

  1. Where is the error coming from? Make your best guess by examining the code. Show us code fragments as needed to clarify
  2. How severe is it? Try successively doubling the number of iterations with the -i flag, say for 2000, 4000 and 8000 iterations and see how much the errors grows.
  3. How much of an increase in performance did you get at a cost of introducing this error? Was it worth it? If you got say only 10% performance improvement, probably not worth it. But if you went much faster, could be worth it.

The code and Makefile have been set up to enable you to compile without MPI, which may come in handy when debugging. To compile without MPI, be sure that the following line has been commented in the Makefile:

mpi=1 

This setting defines the _MPI_ macro which causes certain code to be conditionally compiled. To take advantage of this feature in any code you write, follow the convention used in the provided starter code.

Plotting capability

As mentioned previously, the code includes with a plotting capability to observe the simulation dynamics which can be helpful when debugging.

The simulations will produce a spiral pattern that evolves with time. However, if you set the number of iterations (-i) to a sufficiently large value, the organized spiral pattern will break up. This indicates non-physical behavior, and the length of time that the simulator produces physically meaningful results depends on the mesh size N.

Be sure that, while collecting performance data, that you have not enabled plotting (-p) or statistics reporting (-s).

The plotter (Plotting.cpp) interfaces with gnuplot, which is available on all platforms in use by this course (and is freeware). Remote visual display can be slow, so either work from a campus machine, or increase the frequency to at least 20 (-p 20). The summary statistics may be more useful than plotting in some cases. If a display window does not appear on your screen, then your ssh display settings are not set correctly. If you are connecting to the remote machine from a Linux platform using ssh, use the -X option:

$ ssh -X yourlogin@ccom-bang-login.ucsd.edu

If you are connecting from a Windows platform, and are using a 'secure shell client' AND 'exceed' (or xwin32, or cygwin) to log into a remote machine you must ensure that 'X11 tunneling' is set. To do this, you must have installed Cygwin /X with X11.

Due to an interaction between OpenMPI and Unix Pipes, by default you will get a large warning message beginning with the following text: "An MPI process has executed an operation involving a call to the "fork()" system call to create a child process." To avoid this message set the following shell environment variable before running mpirun (e.g. via your shell startup scripts):

export OMPI_MCA_mpi_warn_on_fork="0"


Part 6. How to take timings

Use the non-interactive batch queues to collect timings. Use the interactive queues (when available) to tune your code's performance. On Bang, use an interactive qlogin node to debug your code and tune performance (Interactive nodes are limited to 8 cores). Use the MPI batch queues to collect timings. Instructions for submitting jobs via the batch queue are described in

http://cseweb.ucsd.edu/classes/fa15/cse260-a/static/Bang/runningJobs.html

and similar instructions for Stampede are found here: http://cseweb.ucsd.edu/classes/fa15/cse260-a/Stampede/runningJobs.html

Take timings using the provided timer code, which measures wall clock time. With MPI enabled, the timer uses MPI_Wtime() else gettimeofday(), as in the first assignment. Both timers collect wall clock time. Instructions for how to use the timer to take timings in MPI are described in the document Using MPI on Bang. The man page for MPI_Wtime is available using the man command.

To measure time, take the difference in times sampled by two successive calls of the timer. For example, for MPI_Wtime():

        double t0   = MPI_Wtime(); 
        // some code....
        double t1   = MPI_Wtime();
        double time = (t1-t0);

The indirect method for measuring communication

Recall that the running time of the application is that of the last processor to finish. Don't time individual iterations, but rather, time an entire run. Exclude initialization, or any other one-time costs that are not representative of a run of many iterations, e.g calls to MPI_Init() and MPI_Finalize() storage allocation and initialization.

It can be difficult to measure communication times accurately, owing to the short time durations involved. Therefore, we will use an indirect measurement technique. We do this by disabling communication of ghost cells and subtracting the resultant time from that obtained from a run that includes communication. The -k command line option (cmdLine.cpp) can be used to measure communication using the indirect method; when the option is asserted, your code should disable all MPI calls that handle ghost cell communication (e.g. Send, Isend, Ireceive and Receive), as well as any code needed to pack and unpack message buffers. Do not disable the calls to MPIInint and MPFinalize as these are needed to enable your program to run under MPI. Similarly don't remove any other message passing calls that don't involve the ghost cell communication directly but are essential for the correct operation of your code. Note that, when you shut off ghost cell communication, your code will produce incorrect results. This is to be expected and won't introduced errors in measured performance, unless numerical exceptions are generated. (You'll know when this happens; the L2 and Linf errors will appear as nan or -nan).

Although Bang and Stampede provide exclusive access to allocated processors, the Infiniband interconnect is not exclusive; it is a shared resource. Thus, if you run on one node, you'll observe no contention from other jobs, but on two or more nodes you'll observe the effects of switch contention. To reduce the effects of switch contention run the program at least 3 times (for each input size and number of cores, or any parameter that affects the running time) until you see a consistent timing emerge, such that a majority of the timings agree to within 10%. If you observe greater variation, then try doubling the number of runs to see if you can obtain a better consensus. If this fails to mask the variation there may be another cause, and this should be investigated. Exclude outlier timings in your plots, but note them in your report if significant. Report the minimum running time rather than the average, as discussed in class.

When you are measuring running times for your performance studies, do not enable plotting (-p) nor output statistics using the -s option, as doing so could affect your observations. Adjust the iteration count so that the running time never drops below 1 or 2 seconds.


Part 7. Development Environment

To build the starter code, be sure that your environment has been set up as explained in Computing Resources document.

Both Bang and Stampede support interactive and non-interactive access. Use Bang for development but do performance experiments at scale on Stampede. (Stampede's interactive queues can be busy, but job turnaround is usually good.) To run interactively on Stampede, use the handy alias inter-mpi set up for you when you added the provided aliases to your .bashrc.

It is not possible to run MPI jobs on a login node on either Bang or Stampede; any attempt will result in and error message reported by the provided code. Be courteous to other users signing of an interactive node when you no longer nee ed it. You can always tell when you using a login node because the host name will begin with the word "login" as in

$ hostname
ccom-bang-login.ucsd.edu
login2.stampede.tacc.utexas.edu

When you access a compute node , the host name will be different, e.g.

$hostname
442-804.stampede.tacc.utexas.edu
baden@compute-0-32

MPI Documentation

There is a nice tutorial with some worked examples. Man pages for the MPI calls may be accessed with the man command and they are also available On-line. More extensive documentation can be found at http://www-cse.ucsd.edu/users/baden/Doc/mpi.html.

SSE/Vectorization Documentation


Part 8. Turning in your assignment

8.1 Submission via GitLab

When you have completed the assignment, you must submit it. Do this by pushing your changes to GitLab. Follow these steps, which are more or less the same as the steps from the "pushing changes to GitLab" section above:

  1. Run a ’git status’ to see what files have been changed.
  2. Use ’git add <files>’ to stage certain files for a commit, use ’git add .’ to add all files in the working directory to the commit (hopefully you will not be doing this at this late stage—all files should already be in your repository!)
  3. Use ’git commit -m "FINAL SUBMISSION"’ to commit the changes to your repository and indicate that you are done and this is your final submission.
  4. To push your changes onto your repository on GitLab run ’git push origin master’ Don’t forget this push! You must do this to get credit.
  5. As an added safety check, you should log on to GitLab on the web and verify that your push went through.

Some of the finer points about this process:

8.2. Required files in the submission

8.2.1. Source code

Even though we will be substituting fresh copies for all but the required files, include all code files your worked with, in case we need to unravel a problem with your turnin. For this reason, also include your Makefile.

8.2.2. 2 text documents

Your turnin must also complete two important documents, which accompany the provided code. These documents are required and we will not report a grade for your assignment without them. A 10% penalty will be assessed in cases of missing forms, empty forms, or forms not properly filled out.

  1. A completed self-evaluation discussing your work; if in a team this form address the division of labor and other aspects of how you worked together as a team.
  2. A file named DECLARATION, that states that the work was your own, and that you did not violate the Academic Integrity policies in effect for the course. Modify the DECLARATION by filling in your correct login ID(s) and the date only. Don't change anything else.

8.2.3. Lab Report, a PDF document

This file must have the following format. The stem part of the file name (before the .pdf) should begin with "A2-report" and end with your repository name, e.g. A3-report-baden-powell.pdf.

An important part of your report is your analysis. That is, to provide insight that explains your results. For this reason, if you are running out of time, put more effort into the writeup and your analysis rather than trying yet another performance enhancement. It can take time to collect, tabulate and plot results, as well as offer a reasonable explanation. We prefer a deeper analysis involving a few program optimizations over a shallower analysis over a larger number of optimizations. Spend more time discussing the optimizations that made a difference and less on those that made less of a difference. Negative results are also especially valued if you carefully document them, with the underlying causes, or best guesses if the causes are unclear.

Do not include full code listings in your writeup as these will be in your turnin.

Take care in how you plot your data, plotting only what's necessary to get your points across. Choose log and linear plots carefully, to take advantage of the plotting space. You must also include any plotted data in tabular form. Tabular data may be included electronically (in your turnin) as a spreadsheet or other document called, e.g. Data.xlsx, Data.txt. Be sure to provide a caption for each plot or table that describes the experiment you conducted, e.g. "strong scaling."

Cite any written works or software as appropriate in the text of the report. Citations should include any software you used that was written by someone else, or that was written by you for purposes other than this class. In such cases you must talk with us (and any code co-authors) prior to get approval.


9. Grading

There are 10 possible points on the assignment and your grade will depend on 4 factors: performance attained by your code, correctness, well-displayed results and comments. You must not only put comments inside your code, but every time you commit to your repository. There should be a steady stream of comments that enable us to retrace your development steps.

Your performance score will be based on the two Stampede performance goals described earlier. (The bang portion is intended to simplify code development, and won't be graded) The first goal will be worth more than the 2nd.

  1. Strong scaling on 16 to 64 cores
  2. Performance (strong scaling) beyond 64 cores and up to 1024 cores. Equivalently, how many cores can you run on, an still observe strong scaling (up to 1024 cores)?
  3. Extra credit using SSE or autovectoriztion (25%). Your code will be tested for performance and correctness. We will test even values of N only. To get all your performance points, your code must show a performance improvement, on 64 cores, of 50% over your MPI code that doesn't employ vectorization. Your code must use the provided SSE_VEC macro to enable us to conditionally compile coded into vectorizing and non-vectorizing forms (Use this macro even if using the autovectorization method, and not SSE intrsinsics). If you are unable to realize a 50% improvement on 1 core, we may award some performance points if you can realize a 60% improvement on 1 core. Be sure to report any performance anomalies or outstanding issues in your code.

    For correctness, even if your code fails, we may be able to award some partial credit provided you (1) meet the performance goal and (2) provide a carefully documented discussion in your report that explains or speculates on the possible cause of the error.

We will grade your code on Stampede using our own Makefile with the same compilation flags as in the provided starter code. To get full credit, design, style and comments (including gitlab commit comments) count.


☛ Important.

Don't wait until the last minute to attempt a turnin for the first time. Try out the turnin process early on so you are sure that you you are familiar with the process, and that all the required additional files are present.

You will receive 0 points for this assignment if your code does not compile, including cases of non-compilation due to merge conflicts. If you developed your code on another machine you are responsible for properly testing on bang, else you may lose significant credit, possibly all 10% depending on the circumstances. As mentioned previously, you must ensure that you resolve all of your merge conflicts and that your code compiles and runs correctly IMMEDIATELY BEFORE you push to GitLab.

For this assignment staff will attempt intervene manually to fix problems. However, the policy for such manual intervention is as follows. If any manual intervention is required by staff you will lose at least 2 points. Included are: incorrectly naming your GitLab repository, or misspelling the words FINAL SUBMISSION in the commit message. Lastly, the only files you should be pushing to GitLab are text files: source code and ASCII test files. Do not push large binaries such as the executables or media files, as doing so could lead to problems such as losing the ability to push to GitLab. Staff will not intervene under these circumstances and you you may be unable to complete your assignment by the deadline. If you commit binaries, you are doing so at your own risk.

Frequently asked questions

  1. Q. What do you mean by "intuition" in choosing processor geometries?
    A. When running on 8 or 64 cores, notice if certain geometries with certain shapes are performing much better or much worse that others. Now, as you grow the number of cores, the general shapes will remain the same, but there will be certain formulas relating the x, y and z values in the geometry that can tell you if a geometry will perform well (or not). Eliminate the poor performers, and favor the strong ones. Be include your rationale in your report.

  2. Q. Why is the binary reference returning a different result than the provided code.
    A. For the binary reference, the effective value of n is one smaller than for the provided code. If, for example, your ran with n=400 in the provided code, use n=399 in the binary reference.

  3. Q. What about partial credit for using SSE or autovectoriztion?

    A. Your code will be tested for performance and correctness. We will test even values of N only. To get all your performance points, your code must show a performance improvement, on 64 cores, of 50% over your MPI code that doesn't employ vectorization. Your code must use the provided SSE_VEC macro to enable us to conditionally compile coded into a vectorizing and non-vectorizing forms (Use this macro even if using the autovectorization method, and not SSE intrsinsics). If you are unable to realize a 50% improvement on 1 core, we may award some performance points if you can realize a 60% improvement on 1 core. Be sure to report any performance anomalies or outstanding issues in your code.

    For correctness, even if your code fails, we may be able to award some partial credit provided you (1) meet the performance goal and (2) provide a carefully documented discussion in your report that explains or speculates on the possible cause of the error.

  4. Q. How can I use the statistics output (L2 and Linfo) to help me debug my code.
    A. Your first step after partitioning the data structures and setting up the initial conditions is to install statistics reporting as they provide a diagnostic that you can compare with the provided code or the reference. Having done this, you can test that you've set up the initial conditions correctly, and that your solver runs correctly on the local partitions. Do this before implementing ghost cell communication as you might otherwise be trying to find and repair multiple bugs at once!

    Though the solution will not be computed correctly, as long as you get no arithmetic exceptions, your code should be consistent with the reference implementation. (This is one of the reasons we provided a reference).

    Once you have established that the solver works correctly without the ghost cells, add the ghost cell communication. Start with 1D partitioning, as communication is simpler than in 1D. Be sure to implement the -k option so that you can shut off ghost cell communication to check that you haven't broken something else, in the course of managing the ghost cells!

  5. Q. What performance are you looking for?
    A. Follow the guidelines in the check list. We don't specifying a specific target floating point rate, rather we are more interested in scalability. However, we do tell you, that on various numbers of cores of Stampede, your code should complete within a certain period of time for a certain value of -n and -i. These are rough guidelines. Even more importantly, your code must scale. What do we mean by "scale?" That performance increases as you increase P. A reasonable rule of thumb is that performance should increase by more than 50% each time you double P, the more the better. At some point, you may hit a plateau. If you do, it's time to investigate the cause of the plateau. If you are short on time, explain the plateau in your writeup. The writeup is worth 60% of the assignment. A good explanation with experimental observations backing up that explanation may be worth more than trying to hunt down an elusive bug.

  6. Q. Will our code be tested for odd values of N as well? The Grading section says this for extra credits: "We will test even values of N only"
    A. Your code should work correctly whether or not N is odd or even. The only time we won't test for odd values of N is for the SSE extra credit option (see FAQ #3). But that is another matter and unrelated to correct operation of the provided code. You should expect that we will test odd and even values when grading your assignment without SSE enabled. (FAQ # 3, explains how to enable conditional compilation if you are attempting the extra credit)

    To see what the matrix looks like for odd and even values of N, modify the provided routine printMat() [helper.cpp] so it prints out larger matrices. Redirect to a file. Or, if you like, shut off the test in apf.cpp so that the code will run when N is <= 25 and use a small matrix like 8 or 9. This output is easily readable at the terminal. (Remember to turn back on that check when done.)



[Fri Nov 20 12:12:04 PST 2015]