CSE160 Wi16 - Lectures and Homework Schedule

This schedule is subject to change, so check frequently

When references to the texts are cited, we use the following naming conventions:

• Pacheco: An Introduction to Parallel Programming, by Peter Pacheco, Morgan Kaufmann, 2011.

• Williams: C++ Concurrency in Action: Practical Multithreading, by Anthony Williams, Manning Publications, 2012. Here are the errata for the Willams text

• P&H: *Computer Organization and Design: The Hardware/Software Interface, 4th Edition. by Patterson & Hennessy, Morgan Kaufmann. This book is available on-line at http://site.ebrary.com/lib/ucsd/reader.action?docID=10509203. You must have a UCSD network login (AD) to access this resource.

Here are the slides from section: https://goo.gl/tA92TA

Lecture 1: Introduction to parallel computing (1/5/16 Tue)

• slides
• Today’s readings
  • Pacheco: Chapter 1 (All), Chapter 2 (pp 15-18).
  • Williams: Chapter 1 (All)
  • Introduction to Parallel Computing, by Blaise Barney. §1-2.
  • SUPERCOMPUTER vs your computer in bang-for-buck battle iPad 2 owns Cray-2?, by Dan Olds, Posted in HPC, 8th March 2012.

Lecture 2: Hardware organization; software execution models; threads Programming; performance

1/7/16 Thu

• slides
• Today’s readings
  • Pacheco. Chapter 2: pp. 29-37 (read through Shared Memory Interconnects, skip vector processors on pp. 30-32), pp 47-53, 56-65, 74-76.
  • Williams. Chapter 2: pp. 15-19 (through §2.1.2), 23-25 (§2.2), 28-32 (§2.4-2.6)

Lecture 3: Synchronization; the Mandelbrot set computation (1/12/16 Tue)

• slides
• Today’s readings
  • Williams. Chapter 3: pp. 33-40 (through § 3.2.2).
  • Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers. 4pp.
  • Wikipedia entry on the Mandelbrot Set.
    Read the Introductory section, as well as sections 1, 8.1 and 9. Look at the Image Gallery (§ 5.1)
  • Mandelbrot video (YouTube)

Lecture 4: Applying synchronization; finding and avoiding synchronization errors (1/14/16 Thu)

• slides
• Today’s readings
  • Williams. Chapter 10: pp. 300-309 (through brute force testing).
  • Williams. Chapter 3, § 3.2.2-3: pp. 39-47.

Lecture 5: Processor self scheduling; the memory hierarchy (1/19/16 Tue)

• slides
• Today’s readings
  • Reader on Load Balancing
  • Pacheco. Chapter 2: pp. 18-25 (through § 2.2.4)
  • Optional reading for further background on memory hierarchies. See one of:
    • Computer Organization and Design: The Hardware/Software Interface, Patterson & Hennessy, Morgan Kaufmann, 4th Edition, Chapter 5, pp. 450-468.
      Available electronically via the UCSD library at http://goo.gl/T52ZSV. Requires UCSD Active Directory Login.
    • What Every Programmer Should Know About Memory Ulrich Drepper. Chapter 2: CPU caches § 3.1-3.2, Also see Chapter 3, Virtual memory

Lecture 6: Cache consistency and coherence (1/21/16 Thu)

• slides
• Today’s readings
  • Pacheco. Chapter 2: pp. 43-46
  • Pacheco. Chapter 5: pp. 251-256. (Note that the effect of the pragma over the i loop on p. 253 is to BLOCK partition iterations of the loop over cores.)

  • Computer Organization and Design: The Hardware/Software Interface, Patterson & Hennessy, Morgan Kaufmann, 4th Edition, Chapter 5, §5.8 (pp. 534-8) Available electronically via the UCSD library at http://goo.gl/T52ZSV. Requires UCSD Active Directory Login.

  • Optional reading
    • Eliminate False Sharing, by Herb Sutter (Dr. Dobb’s Journal)
    • Memory Hierarchy Design, by J. L. Hennessy and D. A. Patterson. 6 part excerpt from their text Computer Architecture
    • Reader on Paging: http://wiki.osdev.org/Paging

Lecture 7: Parallel Merge, Barrier Synchronization (1/26/16 Tue)

• slides
• Today’s readings
  • Videos
    • 15 Sorting Algorithms in 6 minutes. YouTube video.
    • What different sorting videos sound like. YouTube video. +Odd-Even Sort Video. YouTube video.
  • The Art of Concurrency: A Thread Monkey’s Guide to Writing Parallel Applications, Chapter 8, pp. 146-160. Read Bubble Sort and Odd-Even Transposition Sort, stopping at A Concurrent Code…
    Available with UCSD sign on only

Optional Reading

• To dig deeper into parallel merge sort, including an analysis of running time, see Chapter 27, §27.3 of Introduction to Algorithms, (3rd Ed) by Cormen, Leiserson, Rivest and Stein, a classic algorithms text. This chapter is available publicly from MIT at https://goo.gl/AtiqdP

• Leiserson’s lecture slides, includes Merge Sort (MIT course CS 6.172): http://goo.gl/ECKh83

Lecture 8: OpenMP (1/28/16 Thu)

• slides
• Today’s readings
  • Pacheco. Chapter 5: pp. 209-239 (through § 5.7.2), re-read pp. 251-256 (§ 5.9)
  • OpenMP: A Portable Solution for Threading, by Shameem Akhter and Jason Roberts, Dr. Dobb’s Journal (July 12, 2010), Read Effective Use of Reductions only. (A good tutorial complete with code examples)

Tutorials and documentation

• LLNL Tutorial on OpenMP, complete with code examples.
• More work sharing with OpenMP (Intel)
• Getting Started with OpenMP
• 32 OpenMP traps for C++ Developers (Intel)

Lecture 9: OpenMP, condition variables (2/2/16 Tue)

• slides
• Williams. §4.1 (pp 67-76).

Lecture 10: Midterm Review 2/4/16 Thu

• slides There will be only 1 section this week: on Friday 2/5/16

There will be no reading quiz for today’s lecture

Midterm 2/9/16 Tue

Lecture 12: Stencil Methods and C++ atomics (2/11, Thu)

• slides
• Today’s reading
  • Stencil method reader
  • Aliev Panfilov Method, Part 2.
  • Williams. Chapter 5, §5.1-5.2: pp. 103-119.

Lecture 13: The C++11 memory model; implementing synchronization (2/16, Tue)

• slides
• Today’s readings
  • Williams. Chapter 5, §5.3: pp. 119-132, through Understanding Relaxed Ordering.
  • Williams. Chapter 5, §5.3.5: pp. 143-147, through sequentially consistent ordering.
  • Wikipedia entry for Compare and Swap
    

For further reading (optional)

• Compare and swap
  • Wait-Free Synchronization, by Morris Herlihy, ACM TOPLAS, 13(1):124-149 (1991).
  • C++11 compare and Exchange
  • CMPXCHG instruction, Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M, pp. 3-188 through 190.
• Memory models
  • S. V. Adve and K. Gharachorloo, “Shared memory consistency models: a tutorial,” Computer, vol.29, no.12, pp.66-76, Dec 1996 doi: 10.1109/2.546611, url: http://goo.gl/hzF5Pe Research article
  • H. Sutter, “Atomic Weapons: The C++ Memory Model and Modern Hardware” http://herbsutter.com/2013/02/11/atomic-weapons-the-c-memory-model-and-modern-hardware, 2013-02-11
  • Threads Basics, by Hans-J. Boehm.
  • “Memory Models: A Case for Rethinking Parallel Languages and Hardware,” by Sarita V. Adve and Hans-J. Boehm, Commun. ACM 53(8): 90-101. http://cacm.acm.org/magazines/2010/8/96610-memory-models-a-case-for-rethinking-parallel-languages-and-hardware/fulltext

Lecture 14: The C++11 memory model: continued; implementing synchronization, SSE vector processing (2/18, Thu)

• slides

• Pacheco. Chapter 2: pp. 29-32 (§ 2.3.1)

• Getting Started with SSE Programming

• SSE Introduction. Short introduction

To read further about SSE - pragmatics (optional)

• Motion Estimation with Intel Streaming SIMD Extensions 4

• SSE Intrinsics Tutorial

• The Architecture of the Nehalem Processor Nehalem-EP SMP Platforms. This processor is newer than Bang’s CPUs, but most of the SSE discussions are relevant.

• Wikipedia entries on Vectorization and Streaming SIMD Extensions

• Automatic Intra-Register Vectorization for the Intel Architecture, A. Bik et al., Int. J. of Parallel Programming, 30(2):65-98 (2002). DOI

To read further about other optimizations

• Optimizing Software in C++, Agner Fog, §12: pp 105-113 (skip §12.1, §12.2), 117-118, 122-123
• HPC Mac OS X. How to obtain software and documentation

Lecture 15: Floating Point Arithmetic (2/23, Tue)

• slides

• Programming Massively Parallel Processors: A Hands-on Approach, by David Kirk and Wen-mei Hwu, Morgan Kaufmann Publishers (2010), Chapter 7 (Floating Point). (9 pp). Available on-line in draft form

• For further reading (optional)
  • IEEE-754 Floating-Point Converter portal
  • Background material on floating point
  • The Baleful Effect of Computer Languages and Benchmarks upon Applied Mathematics, Physics and Chemistry, by W. Kahan. John von Neumann Lecture, 1997.

Lecture 16: Optimizing for the memory hierarchy (2/25, Thu)

• slides

• Motivation for Improving Matrix Multiplication, by Jim Demmel.

• For further reading (optional): Designing fast linear algebra kernels in the presence of memory hierarchies, by Jim Demmel, UC Berkeley. (Skip the last section: Fast serial matrix multiplication algorithms)

Lecture 17: NUMA architecture and programming (3/1, Tue)

• slides

• Reader on shared memory and NUMA

• Best practices for programming with openMP on NUMA systems, by Joshua Mora. Look over the examples on slides 19-26

• To explore further (optional) + What Every Programmer Should Know About Memory Ulrich Drepper. Chapter 4: NUMA Support http://lwn.net/Articles/254445,
  • The SGI Origin: a ccNUMA highly scalable server by J. Laudon and D. Lenoski, Proc. 24th ISCA, pp. 241-251, 1997.
    
  • Hot Chips Presentation on Magny Cours (2009).

Lecture 18: Pipelining and Instruction Level Paralellism (3/3, Thu)

• slides

• Pacheco: Chapter 2, § 2.2.5 (pp 25-28).
• “Instruction Pipelining,” Wikipedia. en.wikipedia.org/wiki/Instruction_pipelining
• “Out of Order Execution,” Wikipedia. en.wikipedia.org/wiki/Out-of-order_execution

• “Register Renaming,” Wikipedia. https://en.wikipedia.org/wiki/Register_renaming

For Further Reading (Optional)

• Hennessy and Patterson, §C.7, pp C-70 to C-80, “Pipelining, Crosscutting Issues.”
  (To probe further, read § C.1 - C.4)
  Available electronically via the UCSD library at http://roger.ucsd.edu/record=b7202878~S9
  UCSD Active Directory login required.

• Hennessy and Patterson, Chapter 3, sect;3.1-3.4. “Instruction Level Paralelism”

Lecture 19: Graphical Processing Units (3/7, Tue)

• slides

• Multicore and GPU Programming, by G. Barlas. Elsevier (2015). Available electronically via the UCSD library. Requires UCSD Active Directory Login.

  • Chapter 1, §1.1, §1.2, §1.3 (but skip §1.3.1, 1.3.3-4) dx.doi.org/10.1016/B978-0-12-417137-4.00001-0

  • Chapter 6, through §6.3: dx.doi.org/10.1016/B978-0-12-417137-4.00006-X

For Further Reading (Optional)

• CUDA Education and Training https://developer.nvidia.com/cuda-education-training. CUDA is free for download from the developer site, but you will require appropriate hardware (i.e an NVIDIA GPU).

• CUDA Documentation appears at docs.nvidia.com/cuda Of particular interest are these documents.
  • Programming Guide
  • Best Practices Guide
  • Kepler Tuning Guide
  • CUDA Runtime API

Lecture 20: Final exam review 3/10/16 Thu

• slides

Final

3/15/16 Tue (3:00pm to 5:59pm) WLH 2005 New