CSE 260 Schedule (Winter 2014)

This schedule is subject to change, so check frequently
When references to the texts are cited, we use the following naming conventions:

Kirk and Hwu: Programming Massively Parallel Processors: A Hands-on Approach, Second Edition, by David Kirk and Wen-mei Hwu, Morgan Kaufmann Publishers (2012).
Pacheco: An Introduction to Parallel Programming, by Peter Pacheco, Morgan Kaufmann (2011).

Lecture 1, 1/6/14 (Tuesday)
Introduction

Lecture slides posted
Today's reading:
- Pacheco: Chapter 1 (ALL)
- On-line reader
- Little's Law and high performance computing, David Bailey (1997).
For background reading on memory hierarchies (optional), including virtual memory, see §2.2.1-2.2.4 of Pacheco. Also consult one of the following references
- John L. Hennessy and David A. Patterson, Computer Architecture, A Quantitative Approach, 5th Ed., Morgan Kaufmann, 2012, Chapter 2, esp. §2.1 and Appendix B, §B.1 and B.4. (If you have the 4th edition, the corresponding sections are §5.1, C.1 and C.4). The 5th Edition is available electronically via the UCSD library at http://roger.ucsd.edu/record=b7202878~S9 (UCSD Active Directory login required).
- What Every Programmer Should Know About Memory (Ulrich Drepper, on-line)

Lecture 2, 1/8/12 (Thursday)
Address space organization, shared memory and memory locality optimization.

Lecture slides posted
Programming Lab #1
Today's reading:
- Pacheco, Chapter 2: pp. 15-18 (§ 2.1), 32-37 (MIMD systems, shared memory interconnect), 43-46 (cache coherence, snooping cache and false sharing), Chapter 4 (pp. 151-153).
- Designing fast linear algebra kernels in the presence of memory hierarchies Jim Demmel, UC Berkeley
- For additional background material (optional) on shared memory architecture, consult Hennessy and Patterson, Chapter 5, esp §5.1 and §5.2 through page 357.(This material is also discussed in the 4th Ed., Chapter 4, §4.1-2, §4.6.) Also see Overview of Recent Supercomputers, § 2.1-2, 2.4-5 (Or the HTML version, § 3.)

Lecture 3, 1/14/14 (Tue)
☛ Note time and room change: 6.30-7.50pm, WLH 2205
Memory hierarchy optimization; Vectorization

Lecture slides Posted
Today's reading:
- Pacheco. Chapter 2: pp. 19-25 (review for those who have taken graduate level computer architecture); §2.3.1, pp 29-32 (SIMD Systems)
- Chellappa, S., Franchetti, F., and Püschel, M. 2008. How To Write Fast Numerical Code: A Small Introduction, Lecture Notes in Computer Science 5235, 196–259, §4-5 (15 pp.)
- Optimizing Software in C++, Agner Fog, §12: pp 105-113 (skip §12.1, §12.2), 117-118, 122-123
- Hennessy Patterson, 5th Ed., §4.3, pp. 282-285. Available electronically via the UCSD library at http://roger.ucsd.edu/record=b7202878~S9 (UCSD Active Directory login required)
To probe further
- HPC Mac OS X(How to obtain software and documentation)
- Automatic Intra-Register Vectorization for the Intel® Architecture, A. Bik et al., Int. J. of Parallel Programming, vol 30, issue 2, pp 65-98 (2002). DOI
- Programming Guidelines for Vectorizing C/C++ Compilers, A. Bik et al. Dr. Dobb’s Journal, 2/1/03.
- Vectorization (wikipedia)
- Streaming SIMD Extensions (Wikipedia)

Lecture 4, 1/16/14 (Thu): Multicore programming with OpenMP; Parallel Performance

Lecture slides updated
Today's reading:
- Pacheco, Chapter 2: pp. 47-53, 58-62
- Pacheco, Chapter 5: pp. 209-232
To probe further (optional)
- Getting Started with OpenMP, Intel.
- OpenMP: A Portable Solution for Threading, by Shameem Akhter and Jason Roberts, Dr. Dobb's Journal (July 12, 2010). [You'll need to sign up for a free subscription to view this article]
- Resources for OpenMP (including tutorials)

Lecture 5, 1/21/14 (Tue): Introduction to computing with Graphical Programming Units

Lecture slides Posted
Today's reading:
- Scalable Parallel Programming with CUDA, J. Nickolls et al., ACM Queue, 6(2):40-53, March/April 2008. Read through page 48 (If reading the html version, read through Related Work)
- Programming Massively Parallel Processors: A Hands-on Approach, by David Kirk and Wen-mei Hwu, 2nd Ed. Morgan Kaufmann Publishers (2012). Chapter 1 (all).

Lecture 6, 1/23/14 (Thu):
CUDA programming; Matrix Multiplication in CUDA

Lecture slides Posted
Today's reading: Kirk and Hwu, Chapter 3 (all), Chapter 4 (pp. 63-81).

Lecture 7, 1/28/14 (Tue):
Using shared memory, performance programming and the memory hierarchy

Lecture slides Updated
Today's reading:
- Kirk and Hwu, Chapter 4 (81-91), Chapter 5 (all).

Lecture 8, 1/30/14 (Thu):
More on Coalescing; Bank Conflicts; Further performance programming for matrix multiply

Lecture slides Posted
Today's reading:
- V. Volkov and J. Demmel. "Benchmarking GPUs to tune dense linear algebra," Proc. 2008 ACM/IEEE Conf. on Supercomputing, Austin, TX, Nov. 15 - 21, 2008. PDF (Read through §4)
- Kirk and Hwu, Chapter 6, §6.2.
- Shared memory for capability 2.0 devices CUDA C Programming Guide, NVIDIA.
To probe further (optional). "NVIDIA Tesla: A Unified Graphics and Computing Architecture," by Erik Lindholm, John Nickolls, Stuart Oberman, and John Montrym. IEEE Micro 28(2):39-255, March 2008, DOI (This paper is a little outdated but it discusses warp scheduling)

Lecture 9, 2/4/14 (Tue):
Divergence; ILP; Thread scheduling.

Lecture slides Posted
Today's reading:
- Kirk and Hwu, Chapter 6 (§6.1, 6.3-6.4).
- Hennessy Patterson, §C.7, pp C-70 to C-80 (4th Ed.: Appendix A, § A.7 (pp. A-65 to A-75)), "Pipelining, Crosscutting Issues." (If you wish to probe further, read § C.1 - C.4)
  Available electronically via the UCSD library at http://roger.ucsd.edu/record=b7202878~S9 (UCSD Active Directory login required):
- Hennessy and Patterson, Chapter 4, §4.4, pp 288-298, 300-303, 305-308, "Graphical Processing Units"

Lecture 10, 2/6/14 (Thu):
Stencil methods, Limits to performance.

Lecture slides Posted
Today's reading:
- Kirk and Hwu, Chapter 8, skip §8.3.
- Maruyama, N. and Aoki, T. "Optimizing Stencil Computations for NVIDIA Kepler GPUs." HiStencils 2014 (2014):89.
- Hennessy Patterson, 5th Ed., §4.3, pp. 285-288 ("Roofline Model") Available electronically via the UCSD library at http://roger.ucsd.edu/record=b7202878~S9 (UCSD Active Directory login required)
- To dig further into the roofline model (optional): "Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures", Comm. ACM (CACM), April 2009, doi: 10.1145/1498765.1498785

Lecture 11, 2/11/14 (Tue)
☛ Note time and room change: 6.30-7.50pm, EBU3B 4140
Floating point; Particle Methods;

Lecture slides Posted
Today's reading:
- Kirk and Hwu, Chapter 7 (ALL).
- Notes on Simulating Particle Systems (Jim Demmel, UC Berkeley)
To read further about IEEE Floating Point Arithmetic
- Precision & Performance: Floating Point and IEEE 754 Compliance for NVIDIA GPUs, by Nathan Whitehead and Alex Fit-Florea
- How Futile are Mindless Assessments of Roundoff in Floating-Point Computation, W. Kahan. Professor Kahan was the leading force behind the IEEE standard, and he later received the A. M. Turing Award (in 1989).
- IEEE 754: Standard for Binary Floating-Point Arithmetic, a web page with more resources.
- 754-2008 IEEE Standard for Floating-Point Arithmetic, 55 pages (2008))

Lecture 12, 2/13/14 (Thu): Message Passing Programming Model; Introduction to MPI;

Lecture slides Posted
Today's reading:
- Pacheco. Chapter 2: §2.3.2 (pp. 32-35), §2.4.4 (pp. 53-54).
- Pacheco. Chapter 3: §3.1-3.2 (pp. 83-97), §3.4-3.4.4 (pp. 101-106)
Tutorial on MPI(To explore further)
Tutorial material about numerical integration in MPI (To explore further, optional)
- Using point-to-point communication
- Using collective communication

Lecture 13, 2/18/14 (Thu):
Under the hood of MPI; Asynchronous, non-blocking communication;
Debugging; Message Passing Performance

Lecture slides Posted
Today's reading:
- Pacheco. Chapter 3: §3.3 (pp. 97-101), §3.4.5 (pp. 106-109), §3.6 (pp. 123-127)
- Reader on Stencil Methods
- Modeling the performance of an iterative method pdf
Parallel print function. PPF is the Parallel Tools consortium's parallel print facility. For more information, consult the PPF web page. The software is installed in $(PUB)/lib/PPF, examples in $(PUB)/examples/PPF (See the README file for important information about using the software).

Lecture 14, 2/20/14 (Tuesdsay): Matrix multiplication:
Cannon's Algorithm, Communication avoiding Matrix multiply, SUMMA.

Lecture slides Posted
Today's reading:
- Cannon's algorithm on a 2D Mesh, Jim Demmel's notes from from CS267, UC Berkeley.
- Edgar Solomonik and James Demmel, Communication-optimal parallel 2.5D matrix multiplication and LU factorization algorithms, Tech Report UCB/EECS-2011-10, February 2011. Skip LU factorization (§4-6, and parts of §7-8)
To probe further
- R. Van de Geign and J. Watts, SUMMA: Scalable universal matrix multiplication algorithm, Concurrency: Practice and Experience, 9:255-74 (1997)
- To probe further into Communication-avoiding algorithms, visit the Bebop web site (UC Berkeley) or Jim Demmel’s Course on Communication-Avoiding algorithms

Lecture 15, 2/25/12 (Tuesday): Collective communication and its applications

Lecture slides Posted
Today's reading:
- Pacheco, pp. 37-40, 110-116.
- Optimization of Collective Communication Operations in MPICH, by R. Thakur, R. Rabenseifner, and W. Gropp. Int’l. J. of High Performance Computing Applications, (19)1:49-66, Spring 2005. Under the hood of MPICH collectives.
To probe further
- G. E. Blelloch et al, A comparison of sorting algorithms for the connection machine CM-2. Proc SPAA '91.
- Make Barnett et al., Building a high-performance collective communication library, Proc. Supercomputing '94, pp 107-116. Discusses van de Geijn's linear time broadcast algorithm for long messages.
- Detailed discussions about MPI collective communication (For Reference). MPI: The Complete Reference, by Marc Snir et al.

Lecture 16: 3/1/12 (Thursday): Sorting; Gaussian Elimination

Lecture slides Posted
Today's reading:
- Highly Scalable Parallel Sorting, E. Solomonik and L. Kalé, IEEE Intl Parallel and Distrib. Proc. Sym. (IPDPS), 2010, 12 pp.
- Notes on Gaussian Elimination, Jim Demmel, UC Berkeley. Read through Modeling the Performance of Distributed Gaussian Elimination
A detailed explanation of Gaussian Elimination with worked examples http://mathworld.wolfram.com/GaussianElimination.html

Lecture 17: 3/4/14 (Tuesday): Irregular Problems: partitioning and load balancing

Lecture slides Posted
Today's reading:
- Reader on Load Balancing and Data Decomposition
- Using Space-filling Curves for Multi-dimensional Indexing by J. K. Lawder and P. J. H. King, 4th July 2000.
For further reading on spacefilling curves
- Dynamic partitioning of non-uniform structured workloads with spacefilling curves, J. R. Pilkington and S.B Baden. IEEE Trans. Parallel Distrib. Sys. 7(3):288-300 (Mar 1996).
- Multidimensional space-filling curves
- Fractal and space-filling curves
- Plane filling curves
- Space-filing Curves by Zbigniew Fiedorowic, Ohio State University.
- Hilbert Curve (Wolfram Research).
- The Peano Curve and Fractal Curves. Excerpt from from the 30th Edition of the CRC Standard Mathematical Tables and Formulas, 1995, CRC (transcribed by Silvio Levy).

Lecture 18: 3/6/14 (Thursday): Irregular Problems - continued: sparse matrices;
NUMA architectures and programming.

Lecture slides Posted
Today's reading:
- Matlab reader about the pragmatics of optimizing fill in also at http://tinyurl.com/nxaprzl.
- "The SGI Origin: a ccNUMA highly scalable server," J. Laudon and D. Lenoski, Proc. 24th ISCA, pp 241-251, 1997. DOI
- Notes on shared memory
- For background material, see Hennessy and Patterson, Computer Architecture A Quantitative Approach, 4th Ed., Morgan Kaufmann: Chapter 4, esp. §4.1, §4.4, § 4.6, on reserve in the S&E library.
Supplemental reading
- Origin 2000 and Onyx2 Performance Tuning and Optimization Guide, Document No. 007-3430-003, SGI, 2001.
  - Chapter 1. Understanding SN0 Architecture
  - Chapter 2. SN0 Memory Management
  - Chapter 8. Tuning for Parallel Processing: read sections "Tuning Parallel Code for SN0," "Scalability and Data Placement," "Using Data Distribution Directives," but only read through "Understanding the AFFINITY clauses for threads (Example 8-11). There is a convenient table of contents at the beginning of the section.
- Presentation Materials on the Origin 2000 (David Culler, UC Berkeley)

Lecture 19, 3/12/14 (Tuesday): Parallel Programming Languages

Lecture slides Posted
Today's reading:
- M. Frigo, C. E. Leiserson, and K. H. Randall. "The implementation of the Cilk-5 multithreaded language." SIGPLAN Notices 33(5):212-223 (May. 1998). DOI (Also see HPCC06 Challenge Class 2 in Cilk)
  - To read further (optional)
  - How to Survive the Multicore Software Revolution (or at Least Survive the Hype, Charles Leiserson and Ilya Mirman, CILK Arts (Intel web site, optional)
  - Cilk web page
- O. Tardieu et al., "X10 and APGAS at Petascale," Proc 19th ACM SIGPLAN Symp. Principles & Practice Par. Prog. (PPoPP '14), pp. 53-66. DOI.
  - To read further (optional)
  - For background on the APGAS model: V. Saraswat et al., The asynchronous partitioned global address space model, Proc. First Workshop Adv. Message Passing, 2010.
  - For background on PGAS, see PGAS Tutorial (OSC).
  - A Brief Introduction to X10
  - http://x10-lang.org/, X10 web site.

Lecture 20, 3/14/14 (Thursday): The future

Lecture slides Posted
Today's reading:
- The future of microprocessors, by Shekhar Borkar and Andrew A. Chien, Commun. ACM 54(5):67-77, May 2011.
- AMD Announces Server Targeted ARM Part, James Hamilton's BLog
To probe further
- Exascale computing study: technology challenges in achieving exascale systems, by P. Kogge et al, September 2008, DARPA.
- The International Exascale Software Project roadmap, by J. Dongarra et al., Intl J. of High Performance Computing Applications 25(1):3-60, February 2011.
- Exascale.org
- Advanced Archtiectures and Critical Technoglogies for Exascale Computing. US Dept of Energy, 2011. Many presentations. Start with William Harrod’s Overview