CSE 221 Fall 2004 Programming Project: System Measurement
Due at the beginning of the last class meeting.
In building an operating system, it is important to be able to
determine the performance characteristics of underlying hardware (CPU,
RAM, disk, network, etc.), and to understand how their performance
influences or constrains operating system services. Likewise, in
building an application, one should understand the performance of the
underlying hardware and operating system, and how they relate to the
user's subjective sense of that application's "responsiveness". While
some of the relevant quantities can be found in specs and
documentation, many must be determined experimentally. While some
values may be used to predict others, the relations between lower- and
higher-level performance are often subtle and non-obvious.
In this project, you will create, justify, and apply a set of
experiments to a system to characterize and understand its
performance. In addition, you may explore the relations between some
of these quantities. In doing so, you will study how to use
benchmarks to usefully characterize a complex system. You should also
gain an intuitive feel for the relative speeds of different basic
operations, which is invaluable in identifying performance bottlenecks.
You may work either alone or in two-person groups. If you work alone,
you will be responsible only for the system measurement part of the
project. If you work with a partner, you will be required to do both
parts. The group project is intended to require about as much time as
two individual projects. However, since some of the operations are
serialized, it may take somewhat longer to finish, so you should start
Your report should contain a reasonably detailed description of the
test machine(s). The relevant information should be available either
from the system (e.g.
sysctl on BSD,
on Linux, System Profiler on Mac OS X), or online. You will not be graded on
this part, and it should not require much work, but in explaining and
analyzing your results, you will find these numbers useful; for
example, you might compare advertised and observed RAM bandwidths.
You should report at least the following quantities:
- Processor: model, cycle time, cache sizes (L1, L2, instruction,
- Bus speed.
- RAM size.
- Disk: size, RPM, controller cache size.
- Network card speed.
- Operating system (including version/release)
Properties to Measure and/or Report
Both individuals and groups should do this part. Individuals may
omit some parts, noted below.
For each property, you should clearly explain the experiment you
perform, describe how you intend it to measure the quantity of
interest, and evaluate its success. In all cases, you should run your
experiment multiple times, for long enough to obtain repeatable
measurements, and average the results. Often, the property of
interest is an interesting function data size; in such cases (noted
below), you should plot a curve showing time as a function of size,
and explain any interesting features of this curve.
- Scheduling and OS services
- System call overhead
i.e. what is the cost of a minimal system call?
- Task creation time
If available, compare processes to system threads.
- Time to write-protecting a segment
Report as a function of segment size.
- Virtual Memory
- RAM access time
Note: you will need to account for L1 and L2 caches;
- RAM bandwidth (both read and write)
- Page fault service time
For each quantity, you should compare remote and loopback
interfaces, and compare TCP (stream) and (groups only)
UDP (packet) protocols. For your remote tests, either provide a
machine description for the second machine (as above), or use
two identical machines.
- Connection setup and tear-down.
- Peak bandwidth
- File System
- Size of file cache
Note: this may be very sensitive to other load on the
- Sequential and random-access read time vs. size.
Note: Accurate local measurement may require root access
to a Unix machine. If you do not have this, use an
ordinary file as in the remote case, and explain the sense
in which your "sequential" access might not be sequential.
- (groups only) Synchronous and asynchronous
(sequential) write time
- Time to create, fill, and delete a file vs. size
Only groups should do this part.
From the point of the user of a system, a key question is how the
individual operations measured above relate to application-level
performance. Given the above measurements, one wants to predict how
user tasks will scale with hardware size. To answer this question,
you will measure the performance of three different application-level
operations as a function of size. Before running each test, predict
which of the quantities measured above will have a significant effect.
Afterwards, state whether or not your predictions were correct, and if
they were not, explain which factors turned out to dominate.
- Application startup (interactive responsiveness)
Choose several applications of varying size and construction
(e.g. echo, vi, Emacs, Netscape). For each application, measure
the time it takes to start and immediately quit, distinguishing
between the first and subsequent launches.
- Numerical computation
In Matlab (or equivalent), measure the time to multiply two
matrices, to add two vectors, and to add two sparse matrices
(i.e. matrices created with
sparse()), as a
function of size.
- Web server performance
Set up a web server on your benchmark machine serving a file
statically, and serving that same file through a CGI script that
simply reads and outputs the file. Create a client program that
makes repeated requests for one method or the other. Report the
steady-state response rate for both methods as a function of the
file size and of the number of simultaneous clients. Make sure
your client machine(s) is (are) not a bottleneck.