The URL for the class home page is
Important announcements will be made via this page, so watch for changes regularly.
The course has two required texts
The prerequisite for CSE 260 is graduate standing. I recommend that you
have undergraduate-level preparation in
computer architecture and operating systems and that you are comfortable programming in C or C++.
This background
may also be met by equivalent
work experience. Students from other departments are welcome; contact me if
you have any questions about your academic background, especially
if you are not comfortable with C (for example, you may have a Fortran background).
I can point you
to resources that will give you a solid CSE background.
The purpose of this course is to learn how to use high performance parallel computers to solve problems in computational science and engineering. Parallel computation has a rich and varied history, and the techniques for solving problems on high performance parallel computers are both intriguing and intellectually appealing. We'll develop a sound basis for understanding the theory of operation of parallel computers and the performance tradeoffs involved in using them, including Graphical Processing Units (GPUs), multicore processors, and Clusters constructed from these building blocks. We’ll explore the available software techniques including parallelization and performance programming. We’ll develop a methodology for understanding salient application requirements and determining appropriate implementation and problem-solving techniques meeting those requirements.
The course is divided into five main topics as follows.
Various hardware platforms will be used in the class, including GPU workstations in CSE Department, a multicore cluster in the Math Department, and the Dirac GPU cluster located at the National Energy Research Scientific Computing Center (NERSC). These resources should be used for CSE 260 coursework only. For other tasks, use your student account in your home department.
The lecture reading schedule is linked from the main course web page. Readings will come from the two texts, class handouts and and on-line resources. All on-line resources are accessible from any machine on the UCSD campus; if off campus, use the UCSD Web proxy to enable you to access restricted content from non-UCSD Internet service providers. (UCSD Active Directory Login required.) In addition, here are some recommended texts.
Here is a web page resources available to the class:
On particular note:
Historically, processing rates increased at an exponential rate of about 50% per year, doubling about every 18 months. This phenomenon, a consequence of Moore's Law, was primarily dependent on exponentially increasing clock rates. But power consumption increases linearly with clock frequency, and ultimately, power densities became unmanageable. As a result, processor clock speeds began flattening out a few years ago. In response, the industry began replicating CPUs (cores) on the processor as a way of mitigating a curtailing of clock speeds. The multi-core processors became widespread, with multiple CPUs are integrated onto a single chip. It is hoped that multi-core processing will enable a return to historical trends. (If you are interested in learning more, contact my colleagues Michael Taylor or Dean Tullsen). Today, conventional multicore processors support up to 12 cores. Graphical processing units (GPUs) support a few thousand simple cores. For some applications GPUs can outperform a conventional CPU by an order of magnitude or more and deliver a higher floating point rate for a given power budget.
However, even the extraordinary improvements due to multicore processors and processor accelerators are not sufficient to meet the needs of some applications, which can require more memory and I/O capacity than a single computer can accommodate. Today, the fastest supercomputer can deliver up to 50 PetaFlops/second (5×1016) and the leading edge research is looking forward a 20× increase to an Exaflop (1018), sometime in the coming decade. Power consumption imposes a significant design constraint in Exaflop computing, and is a significant concern among today’s supercomputing centers. Another concern is that the amount of memory per core is decreasing, which prevents the user from scaling the problem size and the amount of parallelism together. Such weak scaling is deemed essential if we are to increase performance to Exaflop levels and thus raises a paradox. In addition to these constraints, it will become important to limit data motion costs by increasing the amount of computation for each data byte moved.
Though Exascale computing is available to a select community of users only, parallelism is no longer exclusive to the scientific and engineering research communities. Parallel computers are available to everyone, and at a low cost, and this will no doubt change the world. A daunting side effect of this development is that users are now compelled to use parallelism: "use it or lose it." But parallelism is difficult to manage, especially if we are to fully utilize the processing power that a parallel computer can offer. The principal reason is that compilers can handle only simple cases, and for many applications, the user is left to their own devices, often programming with a low-level interface. This reality challenge even the expert: the details can depend on the hardware, the application and even the input. Unless we train users to manage the array of available implementation challenges, so that they are no longer daunting, then parallel computing technology will be unable to assume its expected role, and it will remain the province of a small select community.
In this course, we focus on parallelism rather than concurrency. The goal in employing parallelism is to improve performance. The goal in managing concurrency is to ensure correctness. Thus, we define parallel processing as the decomposition of a workload onto simultaneously executing physical resources that cooperate in order to increase the capacity to compute more ambitious workloads, or reduce the time to compute existing workloads. The resources comprise a parallel computer: a collection of processing elements that execute simultaneously, communicating and synchronizing with one another. The processing elements co-operate to solve a related set of tasks comprising a single problem.
For example, an automobile assembly line is an example of parallel processing; each stage of the assembly line may be overlapped with the other stages. Parallelism is attractive because it provides a way to deliver increased machine performance, memory, and storage, though memory capacity per core is beginning to shrink. At present, parallelism is also an effective way of dealing with the physical limitations of micro-electronics, i.e. power consumption. Thus, it is possible to have a hierarchically constructed parallel computer comprising many multi-core processors.
It follows that if we have 1 computer capable of computing 5 Billion FlOating Point Operations per second--5 gigaflops/second (109/sec)--then a team of 1000 computers should run at 1000 times the rate, that is, 5000 Gigaflops/sec = 5 Teraflops/sec. In practice, this expectation is optimistic unless the algorithm and its implementation can map optimally onto the hardware. In fact, some machines may deliver only 5% of their peak performance for certain applications. Attaining even this level of performance can still tax the programmer, as application code may require extensive tuning and recoding with each tenfold increase in parallelism. The study of parallel computation helps us to understand the guiding principles and practice that can lead to an effective solution.
We may employ parallel processing without being aware of it, and we may write a parallel program without having a parallel computer. In the first case, we may rely on a compiler to take care of the details or we may use an application that has been parallelized for us. In the second case we may rely on our system to emulate parallelism in a way that allows us to debug our software in a familiar environment. We note that multiprogramming, which is supported by time-sharing operating systems, treats parallelism by interleaving the execution of multiple instruction streams running in separate processes.
Parallelism can occur at many different levels, often referred to as granularity. Here are several possibilities, ranging from fine to coarse grained.
In this course, we will focus on parallel computing rather than distributed or "cloud" computing, which are geographically distributed and hence more loosely coupled. Compared with distributed computation, our parallel computations
Parallel computers are often used to perform elaborate numerical simulations, which can provide cost-effective alternative to expensive "wet lab" experimentation. The Boeing 777 jetliner, for example, was designed by computer, avoiding costly wind tunnel experiments. The reliance on computers to solve problems effectively has stimulated increased resource demands. (For example, see Jim Demmel's notes "Motivation for Parallel Computing" at http://www.eecs.berkeley.edu/~demmel/cs267/lecture01.html).
Our success in the above endeavor depends on our ability to split up the problem effectively across multiple processors. Parallel processing is cost effective when the running time and programming effort are competitive with alternative computing solutions. Note that there are often many different ways of solving a problem on a parallel computer, and that the highest performing implementation may depend on the hardware and algorithm used to solve the problem. Indeed, an algorithm that runs well on a parallel computer may perform poorly on a single core. It might perform more numerical operations than the "best" single processor algorithm, but due to the parallel speedup, the extra costs can be justified. But a parallel computer can only deliver a constant speedup and cannot make up for a higher complexity operation count, i.e. O(N2) vs O(N).
Programming a parallel computer involves juggling many more factors than on a single processor computer, and is correspondingly more challenging. To get a rough idea of why parallel computing can be more challenging, consider that most of us tend to take the operating system and compiler for granted. These tools often handle many decisions about memory hierarchy management (e.g. cache, virtual memory and registers) and scheduling for us. What if we had to manage cache locality and virtual memory ourselves? Parallel processing introduces other activities that are often part and parcel of OS programming: communication and synchronization.
There are generally two reasons for using a parallel computer
While both viewpoints are valid, users are often attracted to the opportunity offered by improved capability. The reason is that a user's primary activity is to engage in scientific discovery. While performance is important, it is often not the primary goal. Scientists are less concerned if they got the answer in 10 hours or 20. Why is this the case? There are two reasons.
First, the difference between 10 and 20 hours may not be the factor in
determining whether or not a scientist makes a discovery (though it may impact
their CPU time budget!). On a mainframe system, job turnaround may
include long queuing delays--think of what happens when everyone is trying to
meet a conference deadline at the same time, and you get the idea. Second, it
takes time to improve performance, and the added software development time
generally increases with the level of parallelism and hence performance. This is
especially true if the user has old "legacy" or "dusty desk" code, and may have
difficulty in upgrading their software.
On the other hand, performance programming is vital in maintaining scalability,
whereby we expect performance to increase as we increase the level of
parallelism.
The moral of the story is that efficient programs may benefit a
computer center more than its users,
but the equalizer is that computing time is a finite resource, which
gives the users the incentive to optimize their code to improve performance.
With increased levels of performance comes the ability to generate prodigious amounts of data and the need to manipulate them. We will briefly discuss data-intensive applications.
Scalable parallel computers are generally constructed from servers which may be embellished with hardware accelerators. Thus, they employ a hierarchical organization: each computational node is a shared memory multiprocessor. The server "nodes" are interconnected with switch fabric like InfiniBand or Myrinet though some vendors (like IBM and Cray) may also build custom interconnect. A high end machine provides services like user accounting, job submission services, and a full range of numerical libraries. One dilemma with production machines is their upgrade path. Generally a machine is designed around a particular chip, and it may take some time before newly designed chips can be incorporated into an existing system. The lifetime of a mainframe is 3 to 4 years. During this time, lower cost platforms such as laptops and Desktops will show an increase in performance. As a result, some low cost clusters can outperform a mainframe at lower levels of parallelism.
While processing speeds were rising at an astronomical rate as predicted by Moore's law, memory speeds increased far more slowly. During the 1990s, this "processor-memory gap" had a dramatic effect on processor design and on the difficulty of obtaining acceptable machine performance. Design techniques such as instruction level parallelism, hyperthreading and deeper memory hierarchies have historically played an important role. However, these techniques also consume large amounts of power and they consume valuable "real estate" on the chip. Streaming processors, such as GPUs, which we’ll use in this course, use simplified designs, freeing up space for more computational resources and storage to drive them. In exchange, the user has to manage a partially exposed memory hierarchy and avoid branching when possible. For some workloads, these programming tasks are justified by the boost in performance. As a result, new processing technologies can deliver a higher physical density of processing power or a lower power consumption cost per flop.
In dealing with the memory hierarchy, we may have multiple levels of cache to contend with, in addition to main memory and virtual memory (Even the processing pipeline may be included). Our goal is to improve memory locality whereby we access memory most frequently at the lowest levels of the hierarchy (i.e. registers and primary cache which is on the processor). Serial computers are more forgiving about poor memory locality than parallel computers, though the gap is narrowing due to the effects of Moore's Law. When commercial parallel computers began appearing in the early to mid 80's, we were not as far along on the processing speed curve as we are today. So main memory wasn't so "far away" from the processor as it is today. But in the intervening decades processing speeds ramped up more rapidly than memory speeds, and the situation has changed. The moral of the story: perhaps parallelism provided a prescient image of what was to come.
[Tue Sep 22 08:25:02 EDT 2015]
