The URL for the class home page is
Important announcements will be made via this page, so watch for changes regularly.
The course has one recommended
The prerequisite for CSE 262 is graduate standing. This is not a first course in parallel processing, and you must have prior experience writing parallel programs, either in threads, MPI, or CUDA.
The purpose of this course is to learn how to employ high performance parallel computers to solve problems in computational science and engineering. Parallel computation has a rich and varied history, and the techniques for solving problems on high performance parallel computers are both intriguing and intellectually appealing. We'll develop a sound basis for understanding the theory of operation of parallel computers and the performance tradeoffs involved in using them, including Graphical Processing Units (GPUs), multicore processors, and Clusters constructed from these building blocks. We’ll explore the available software techniques including parallelization and performance programming. We’ll develop a methodology for understanding salient application requirements and determining appropriate implementation and problem-solving techniques meeting those requirements.
The course is divided into five main topics as follows.
Various hardware platforms will be used in the class, including an NVIDIA GPUs and other machines available via the NSF Teragrid. These high performance resources should be used for parallel program development and experimentation only. For other tasks, use your student account in your home department.
Please visit the following web page with extensive documentation, papers, software repositories, and other goodies:
Here is a web page resources available to the class:
Historically, processing rates increased at an exponential rate of about 50% per year, doubling about every 18 months. This phenomenon, a consequence of Moore's Law, was primarily dependent on exponentially increasing clock rates. However, because power consumption increases as the square of the clock frequently, power densities became unmanageable. As a result processor clock speeds began flattening out a few years ago. In response, industry began developing multi-core processors, where multiple CPUs are integrated onto a single chip. It is hoped that multi-core processing will enable a return to historic trends. (If you are interested in learning more, contact my colleagues Steve Swanson, Michael Taylor, or Dean Tullsen). Today, conventional processors support up to 12 cores. Graphical processing units (GPUs) support hundreds of simple cores and for some applications, and for selected workloads, can outperform a conventional CPU by an order of magnitude or more.
However, even the extraordinary improvements due to multicore processors and processor accelerators are not sufficient to meet the needs of some applications, which can require more memory and I/O capacity than a single computer can accommodate. Today, the fastest supercomputers are coming close to delivering a peak speed of a PetaFlop/second (1015) and the leading edge research is looking forward a 1000 × increase to an Exaflop (1018), sometime in the next 10 years. Power consumption is seen as a significant impediment to Exaflop computing, and is a significant concern among today’s supercomputing centers. Another possible stumbling block is that the amount of memory per core is decreasing with time, and this can prevent the user from scaling the problem size and the amount of parallelism together, which is known as weak scaling. Weak scaling is essential if we are to enable application performance to increase by an order (or orders) of magnitude.
We should note that such high levels of performance are useful to a select community of users only. On the other hand, as the result of the recent technological trends we have just noted, parallelism is no longer exclusive to the scientific and engineering research communities. Parallel computers are available to everyone, and at a low cost, and this will no doubt change the world. A daunting side effect of this development is that users are now compelled to use parallelism. This is true because processor clock speeds are no longer decreasing, so the only way to improve performance is through increased parallelism. But parallelism is difficult to manage, especially if we are to fully utilize the processing power that a parallel computer can offer. The principal reason is that compilers can handle only simple cases, and the user is left to their own devices, often programming with a low-level interface. This is hard even for experts and the details can depend both on the hardware, the application and even the input. Unless we train users to manage the daunting array of available implementation challenges, so that they are no longer daunting, then parallel computing technology will be unable to assume its expected role, and it will remain the province of a small select community.
We define parallel processing as the decomposition of a workload onto simultaneously executing physical resources that cooperate in order to increase the capacity to compute more ambitious workloads, or reduce the time to compute existing workloads.
It follows that a parallel computer is a collection of processing elements that are able execute simultaneously, communicating and synchronizing with one another. The processing elements co-operate to solve a related set of tasks comprising a single problem. For example, an automobile assembly line is an example of parallel processing; each stage of the assembly line may be overlapped with the other stages. Parallelism is attractive because it provides a way to deliver increased machine performance, memory, and storage, in a manner that compounds improvements in processor technology. At present, it is also an effective way of dealing with the physical limitations of micro-electronics, i.e. multi-core CPUs. Thus, it is possible to have a hierarchically constructed parallel computer comprising many multi-core processors.
It follows that if we have 1 computer capable of computing 5 Billion FlOating Point Operations per second--5 gigaflops/second (109/sec)--then a team of 1000 computers should run at 1000 times the rate, that is, 5000 Gigaflop/sec = 5 Teraflops/sec. In practice, this expectation is optimistic unless the algorithm and its implementation can map optimally onto the hardware. In fact, some some machines may deliver only 10% of their peak performance for certain applications. Attaining this level of performance can still tax the programmer, as application code may require extensive tuning and recoding with each tenfold increase in parallelism. The study of parallel computation helps us to understand the guiding principles that can lead to an effective solution.
We may employ parallel processing without being aware of it, and we may write a parallel program without having a parallel computer. In the first case, we may rely on a compiler to take care of the details or we may use an application that has been parallelized for us. In the second case we may rely on our system to emulate parallelism in a way that allows us to debug our software in a familiar environment. We note that multiprogramming, which is supported by time-sharing operating systems, treats parallelism by interleaving the execution of multiple instruction streams running in separate processes.
Concurrency can occur at many different levels, often referred to as granularity. Here are several possibilities, ranging from fine to coarse grained.
In this course, we will focus on parallel computation rather than distributed, "cloud," or "grid" computations, which are geographically distributed and hence more loosely coupled. "grids" or "clouds." This implies that, compared with distributed computation, our parallel computations
While our computations may not need to incorporate knowledge about the processor interconnect structure, this situation is not true for some architectures, in particular, for the high end "leadership class" platforms. Time will tell if this trend will continue. Leadership class platforms also introduce the need to tolerate hardware failures, a capability that had been largely ignored historically. For lower end computing, fault tolerance is less of an issue, though it certainly would be in a critical application.
Parallel computers are often used to perform elaborate numerical simulations, which can provide cost-effective alternative to expensive "wet lab" experimentation. The Boeing 777 jetliner, for example, was designed by computer, avoiding costly wind tunnel experiments. The reliance on computers to solve problems effectively has stimulated increased resource demands. (For example, see Jim Demmel's notes "Motivation for Parallel Computing" at http://www.eecs.berkeley.edu/~demmel/cs267/lecture01.html).
Our success in the above endeavor depends on our ability to split up the problem effectively across multiple processors. Parallel processing is cost effective when the running time and programming effort are competitive with alternative computing solutions. Note that there are often many different ways of solving a problem on a parallel computer, and that the highest performing implementation may depend on the hardware we are using and similarly for the algorithm used to solve the problem. In fact, an algorithm that runs well on a parallel computer may perform poorly on a single core. It might perform more numerical operations than the "best" single processor algorithm, but due to the parallel speedup, the extra costs can be justified. But a parallel computer can only deliver a constant speedup and cannot make up for a higher complexity operation count, i.e. O(N2) vs O(N).
Programming a parallel computer involves juggling many more factors than on a single processor computer, and is correspondingly more challenging. Managing software development costs is vital. To get a rough idea of why parallel computing can be more challenging, consider that most of us tend to take the operating system and compiler for granted. These tools often handle many decisions about memory hierarchy management (e.g. cache, virtual memory and registers) and scheduling for us. What if we had to manage cache locality and virtual memory ourselves? Parallel processing introduces other activities that are often part and parcel of OS programming: messages and synchronization.There are generally two reasons for using a parallel computer
While both viewpoints are valid, users are often attracted to the opportunity offered by improved capability. The reason is that a user's primary activity is to engage in scientific discovery. While performance is important, it is often not the primary goal. Scientists are less concerned if they got the answer in 10 hours or 20. Why is this the case? There are two reasons.
First, the difference between 10 and 20 hours may not be the factor in
determining whether or not a scientist makes a discovery (though it may impact
their CPU time budget!). Moreover, on a mainframe system, job turnaround may
include long queuing delays--think of what happens when everyone is trying to
meet a conference deadline at the same time, and you get the idea. Second, it
takes time to improve performance, and the added software development time
generally increases with the level of parallelism and hence performance. This is
especially true if the user has old "legacy" or "dusty desk" code, and may have
difficulty in upgrading their software.
On the other hand, performance programming is vital in maintaining scalability,
whereby we expect performance to increase as we increase the level of
parallelism.
The moral of the story is that
computer centers may have more of an incentive than the user in seeing
that user programs run efficiently, but the equalizer is that computing
time is a finite resource.
Data management is also an important concern to many users. With increased levels of performance comes the ability to generate prodigious amounts of data. We will briefly discuss data-intensive applications and software support issues.
Parallel computers are generally constructed from servers which may be embellished with hardware accelerators. Thus, they employ a hierarchical organization: each computational node is a shared memory multiprocessor. The server "nodes" are interconnected with switch fabric like InfiniBand or Myrinet though some vendors employ custom interconnect. A high end machine provides services like user accounting, job submission services, and a full range of numerical libraries. One dilemma with production machines is their upgrade path. Generally a machine is designed around a particular chip, and it may take some time before newly designed chips can be incorporated into an existing system. The lifetime of a mainframe is 3 to 4 years. During this time, lower cost platforms such as laptops and Desktops will have improved in performance. As a result, some low cost clusters can outperform a mainframe at lower levels of parallelism.
While processing speeds were rising at an astronomical rate as predicted by Moore's law, memory speeds increased far more slowly. During the 1990s, this "processor-memory gap" had a dramatic effect on processor design and on the difficulty of obtaining acceptable machine performance. Design techniques such as instruction level parallelism, hyperthreading and deeper memory hierarchies have historically played an important role. However, these techniques also consume large amounts of power and they consume valuable "real estate" on the chip. Streaming processors (such as GPUs and STI Cell) use simplified designs, freeing up space for more computational resources and storage to drive them. In exchange, the user has to manage a paritally exposed memory hierarchy and avoid branching when possible. For some workloads, these programming tasks are justified by the boost in performance. As a result, new processing technologies can deliver a higher physical density of processing power or a lower power consumption cost per flop.
In dealing with the memory hierarchy, we may have multiple levels of cache to contend with, in addition to main memory and virtual memory (Even the processing pipeline may be included). Our goal is to improve memory locality whereby we access memory most frequently at the lowest levels of the hierarchy (i.e. registers and primary cache which is on the processor). Serial computers are more forgiving about poor memory locality than parallel computers, though the gap is narrowing due to the effects of Moore's Law. When commercial parallel computers began appearing in the early to mid 80's, we were not as far along on the processing speed curve as we are today. So main memory wasn't so "far away" from the processor as it is today. But in the intervening decades processing speeds ramped up more rapidly than memory speeds, and the situation has changed. The moral of the story: perhaps parallelism provided a prescient image of what was to come.
Another technogical factor that has changed the face of high peformance computing is power consumption. Since power consumption varies as the square of the clock frequency, manufacturers have adapted by employing multiple cores on a chip, rather than to continue to increase the clock frequency. As a result, we all have a parallel computer at our finger tips. But as there is no universal strategy for parallelizing single processor codes, these machines incur higher programming overheads.
As core counts increase, the amount of memory per chip (espcially fast on-chip memory) will descrease. This trend will affect scalability; traditionally, weak scaling has been employed as a way of amortizing fixed overheads. But with descreasing memory per core, weak scaling may no longer be a possibility.
Another trend is for processor designs to become more streamlined, enabling the manufacturer to increase core density and reduce power consumption per flop. The impact, however, is to increase the complexity of performance programming. Streamlined designs add constraints such as SIMD execution, and these are not compatible with workloads that are irregular, that exhibit data dependent exeuction times, often involving adaptivity with pointer based data structures.
Copyright © 2011 Scott B. Baden.