CSE 262 Schedule

When references to the texts are cited, we use the following naming conventions:

• Kirk and Hwu: Programming Massively Parallel Processors: A Hands-on Approach, by David Kirk and Wen-mei Hwu, Morgan Kaufmann Publishers (2010).
• Pacheco: Parallel Programming with MPI, by P. Pacheco, Morgan Kaufmann Publishers (1997).

Lecture 1, 3/29/11 (Tue): Introduction

Lecture slides Assignment #0 Today's reading: Kirk and Hwu: Chapter 1 (ALL) On-line reader For background reading on shared memory hierarchies, see one of John L. Hennessy and David A. Patterson, Computer Architecture, A Qualitative Approach, 4th Ed., Morgan Kaufmann, 2007, Chapter 4. For a discussion of NUMA support, also see What Every Programmer Should Know About Memory (Ulrich Drepper)

Lecture 2, 3/31/11 (Thu): Data Parallelism.

Lecture slides Today's reading: Definition of a Parallel Random Access Machine (PRAM) [Wikipedia] Intel SSE Wikipedia article on Vectorization. To probe deeper (optional) PRAM Vectorization with the Intel Compilers (Part I). Includes code examples. Automatic Intra-Register Vectorization for the Intel® Architecture, A. Bik et al., Int. J. of Parallel Programming, vol 30, issue 2, pp 65-98 (2002).         DOI Programming Guidelines for Vectorizing C/C++ Compilers, A. Bik et al. Dr. Dobb’s Journal, 2/1/03.




Lecture 3, 4/5/11 (Tue): Computing with Graphical Processing Units

• Lecture slides
• Today's reading:
  • Scalable Parallel Programming with CUDA, J. Nickolls et al., ACM Queue, 6(2):40-53, March/April 2008. Read only through page 48
  • CUDA Supercomputing for the Masses, by Rob Farber, part II of a 20-part series.

Lecture 4, 4/7/11 (Thu): Programming with CUDA

Lecture slides Today's reading: Programming Massively Parallel Processors: A Hands-on Approach, by David Kirk and Wen-mei Hwu, Morgan Kaufmann Publishers (2010). Chapter 3 (all), Chapter 4 (59-68)

Lecture 5, 4/12/11 (Tue): Performance Programming

Lecture slides Today's reading: Kirk and Hwu, Chapter 4 (68-75), Chapter 5 (All)

Lecture 6, 4/14/11 (Thu): Programming with Warps, the memory hierarchy, and a look at Fermi

Lecture slides Today's reading: Fermi Architecture Whitepaper Inside Fermi: Nvidia's HPC Push, David Kanter, Real World Technologies, 09-30-2009.

Lecture 7, 4/19/11 (Tue): Message Passing

Lecture slides Today's reading: A User's Guide to MPI, by Peter Pacheco, pp. 1-17. (Or from from Peter Pacheco's Parallel Programming with MPI. pp. 41-76). pdf by Grama, Gupta, Karypis, and Kumar, Addison-Wesley Publisher, 2003: Introduction to Parallel Computing 2nd Ed, pp. 233-240. Code examples from the Pacheco textbook are available in $PUB/Examples/MPI/Pacheco Quadrature code (trapezoidal rule): available in $PUB/Examples/MPI/Pacheco/ppmpi_c/chap04

Lecture 8, 4/21/11 (Thu): Communicators, collective communication, Cannon's Algorithm

Lecture slides Today's reading: A User's Guide to MPI, by Peter Pacheco, pp. 29-36.     (Or from from Peter Pacheco's Parallel Programming with MPI. pp. 111-121). pdf Introduction to Parallel Computing 2nd Ed, by Grama, Gupta, Karypis, and Kumar, Addison-Wesley Publisher, 2003: pp. 345-349.

Lecture 9, 4/26/11 (Tue): Communication avoiding algorithms; SUMMA matrix multiplication algorithm

Lecture slides Hallgeir Lien's slides Today's reading: Edgar Solomonik and James Demmel, Communication-optimal parallel 2.5D matrix multiplication and LU factorization algorithms, Tech Report UCB/EECS-2011-10, February 2011. Skip LU factorization (§4-6, and parts of §7-8) R. Van de Geign and J. Watts, SUMMA: Scalable universal matrix multiplication algorithm, Concurrency: Practice and Experience, 9:255-74 (1997) Background on collective communication implementations - to probe deeper Optimization of Collective Communication Operations in MPICH, by R. Thakur, R. Rabenseifner, and W. Gropp. Int’l. J. of High Performance Computing Applications, (19)1:49-66, Spring 2005. Under the hood of MPICH collectives. Make Barnett et al., Building a high-performance collective communication library, Proc. Supercomputing '94, pp 107-116. Discusses van de Geijn's linear time broadcast algorithm for long messages. Detailed discussions about MPI collective communication (For Reference). MPI: The Complete Reference, by Marc Snir et al.

Lecture 10, 4/28/11 (Thu): Project proposal presentations; Performance Programming on GPUs: Dense linear algebra

Yajaira Gonzalez' slides Today's reading: Benchmarking GPUs to tune dense linear algebra, by V. Volkov and J. Demmel. Proc. 2008 ACM/IEEE Conf. on Supercomputing, Austin, TX, Nov. 15 - 21, 2008. Read §1-4 only; we'll cover §5-6 another time.

Lecture 11, 5/3/11 (Tue): Heterogenerous architectures; Matrix Factorization

Lecture slides on Gauss elimination Julian Bui's slides     PDF     PPT Today's reading: Grey Ballard, James Demmel, and Andrew Gearhart, Communication bounds for heterogeneous architectures Tech Report, UC Berkeley (January 2011) Notes on Gaussian Elimination, Jim Demmel, UC Berkeley. Skip "Overview of LAPACK and ScaLAPACK" (but read the 3rd paragraph about "Matlab Notation"); "Performance of LAPACK LU Decomposition"; "Software Structure of ScaLAPACK." Additional materials on Gaussian Elimenation (for background if you need it). Numerical Recipies in C, 2nd Ed., by Press et al., Cambridge University Press, Chapter 2, pp. 32-33, §2.3 (pp. 41-48). Also On line. A detailed explanation with worked examples http://mathworld.wolfram.com/GaussianElimination.html Applet for 3x3 systems. Choose "Gaussian Elimination."

Lecture 12, 5/5/11 (Thu): Translation and parallelization.

Eric Christiansen's lecture slides Mickey Ju's lecture slides Today's reading: S. Krishnamoorthy et al., Effective automatic parallelization of stencil computations, Proc 2007 ACM SIGPLAN conference on Programming language design and implementation (PLDI '07), 2007. U. Bondhugula, J. Ramanujam and P. Sadayappan, Automatic mapping of nested loops to FPGAs, Proc. ACM SIGPLAN symposium on Principles and practice of parallel programming (PPoPP '07), 2007.

Lecture 13, 5/10/11 (Tue): GPU Translation

Yajaira Gonzalez' lecture slides Didem Unat's slides Today's reading: OpenMPC: Extended OpenMP Programming and Tuning for GPUs, S. Lee and R. Eigenmann, Proceedings 2010 ACM/IEEE Conf. Supercomputing. D. Unat, X. Cai and S. B. Baden, Mint: Realizing CUDA performance in 3D Stencil Methods with Annotated C 25th Intl. Conf. on Supercomputing (ICS 2011), Tuscon, Arizona, May 31-June 4, 2011, to appear.

Lecture 14, 5/12/11 (Thu): GPU algorithms and performance limitations

Eric Christiansen's lecture slides Micky Ju's lecture slides Today's reading: S. Venkatasubramanian and R. W. Vuduc, Tuned and wildly asynchronous stencil kernels for hybrid CPU/GPU platforms, Proc. ACM Int'l. Conf. Supercomputing (ICS), 2009. R. Vuduc et al. On the limits of GPU acceleration. Proc. 2nd USENIX conference on Hot topics in parallelism (HotPar'10),, 2010.

Lecture 15, 5/17/11 (Tue): Project Progress reports

Lecture 16, 5/19/11 (Thu): Sparse Matrices

Danni Als' slides Mike Folkert's slides Today's reading: S. Williams et al., Sparse Matrix-Vector Multiplication on Multicore and Accelerators, Scientific Computing with Multicore and Accelerators, pp. 83-109, Chapter 5, Dec 2010. (Danni Als) J. W. Choi, A. Singh, and R. W. Vuduc. Model-driven autotuning of sparse matrix-vector multiply on GPUs, Proc 15th ACM SIGPLAN symposium on Principles and practice of parallel programming (PPoPP '10), pp 115-126. (Mike Folkerts)

Lecture 17, 5/24/11 (Tue)

Julian Bui's slides     PDF     PPT Hallgeir Lien's slides Today's reading: A. Kerr, D. Campbell and M. Richards, QR decomposition on GPUs, Proc 2nd Workshop on General Purpose Processing on Graphics Processing Units (GPGPU-2), pp. 71-78 (2009). (Hallgeir Lien) V. T. Ravi, W. Ma, D. Chiu, and G. Agrawal, Compiler and runtime support for enabling generalized reduction computations on heterogeneous parallel configurations, Proc. ICS '10, 2010. (Julian Bui)

Lecture 18, 5/26/11 (Thu): Massively Parallel Processing  

Danni Als' slides Today's reading: S. Kamil, L. Oliker, A. Pinar and J. Shalf, Communication Requirements and Interconnect Optimization for High-End Scientific Applications, IEEE Trans. Parallel Distrib. Sys., pp. 188-202, Feb., 2010 (Mike Folkerts). F. Petrini, D. J. Kerbyson and S. Pakin, The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q, Proc. 2003 ACM/IEEE conference on Supercomputing (SC '03), 2003 (Danni Als).

Lectures 19 and 20 (5/31/11): CSE 262 Symposium (Room 4140)


Schedule TBA

Paper list


J. Dinan et al., Scalable work stealing, Proc SC '09, 2009. A. Rahimian et al., Petascale Direct Numerical Simulation of Blood Flow on 200K Cores and Heterogeneous Architectures, Proc SC '10. U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayappan, A practical automatic polyhedral parallelizer and locality optimizer, Proc. 2008 ACM SIGPLAN conference on Programming language design and implementation (PLDI '08), pp. 101-113. N. Bell and M. Garland, Implementing a sparse matrix-vector multiplication on throughput oriented processors, Proc. ACM/IEEE Conf. Supercomputing (SC), Portland, OR, Nov. 2009. Efficient Sparse Matrix-Vector Multiplication on CUDA N. Bell and M. Garland, NVIDIA Technical Report NVR-2008-004," Dec. 2008. Richard W. Vuduc. Automatic performance tuning of sparse matrix kernels. PhD Thesis, Univ. of Calif., Berkeley, CA, USA, Tech Report UCB/CSD-04-1335, January 2004. S. Williams, Auto-tuning Performance on Multicore Computers, Ph.D. Thesis, Univ. of Calif., Berkeley, EECS Tech. Rep. EECS-2008-164, Dec. 2008. [pdf] I. Lashuk et al., A massively parallel adaptive fast multipole method on heterogeneous architectures, Proc. ACM/IEEE Conf. Supercomputing (SC), Portland, OR, Nov. 2009. S. B. Baden and S. J. Fink, Communication overlap in multi-tier parallel algorithms, Proc the 1998 ACM/IEEE Conference on Supercomputing, San Jose, CA, Nov 1998, pp. 1-20. Communication avoiding algorithms, see http://bebop.cs.berkeley.edu/#pubs M. Anderson, G. Ballard, J. Demmel, and K. Keutzer, Communication-avoiding QR decomposition for GPUs, UC Berkeley Technical Report EECS-2010-131, 2010. J. Demmel, M. Hoemmen, M. Mohiyuddin , and K. Yelick. Minimizing communication in sparse matrix solvers, SC09, pp. 1-12, 2009. [Algorithm for computing matrix powers kernel when A and M have similar sparsity patterns, requires background reading of QR] Graph Partitioning U.V. Catalyurek, C. Aykanat, and B. Ucar. On two-dimensional sparse matrix partitioning: Models, methods, and a recipe. SIAM J. Sci. Comput., 32(2):656-683, 2010. Hypergraph-partitioning-based decomposition for parallel sparse- matrix vector multiplication. IEEE Trans. Parallel Distrib. Syst., 10(7):673-693, 1999. Load Balancing (Collected by Keshav Pingali, UT Austin) Computing with Scan S. Sengupta, M. Harris, Y. Zhang and J. D. Owens. Scan primitives for GPU computing, Proc. 22nd ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware (GH '07), Aire-la-Ville, Switzerland, Switzerland, pp. 97-106. G. E. Blelloch, M. A. Heroux and M. Zagha. Segmented Operations for Sparse Matrix Computation on Vector Multiprocessors, Tech. Rep.. Carnegie Mellon Univ., 1993.