Bill and Paul's Excellent UCSD Benchmarks for Java (version 1.1)

There is now a host of Java compilers and interpreters, and we need a way of telling which compilers produce better code, and which interpreters execute them faster. To meet this need, we have produced a small set of simple benchmarks written in Java. These were inspired by Jon Bentley's books Programming Pearls and Writing Efficient Programs.

Using These Benchmarks

• Run benchmarks as applet (compiled with Sun's 1.1.4 JDK):
  • Compiled under Sun javac, no optimization. (Some versions of Sun's appletviewer and Netscape have had troubles with garbage collection benchmark.)
  • Compiled under Sun javac, with optimization (-O). (Some versions of Sun's appletviewer and Netscape have had troubles with garbage collection benchmark.)
  • Compiled under Sun javac, no optimization, no GC benchmark.
  • Compiled under Sun javac, with optimization (-O), no GC benchmark.

• Download the 1.1 benchmarks. Note that these benchmarks are copyrighted by the Regents of the University of California. See the file headers for details. Downloading gives you greater flexibility, such as compiling the benchmarks with a compiler that we don't have. Your choices: a tar file, a gzip'd tar file, a zip archive.

• Results we've gathered from a variety of system configurations (1.0 benchmarks).
• Results we've gathered from a variety of system configurations (1.1 benchmarks).

Benchmark-Related Links

• Doug Bell's Benchmark applet
• Jonathan Hardwick's slick Java optimization site
• The SPEC Web site
  • SPECweb96 Benchmarks
• Some C-language SPEC benchmark results

Caveats and Details

• These benchmarks are simplistic and do not cover all interesting performance issues. In particular, the loop test employs an empty loop, and the methods and threads are trivial as well. A really good optimizer could detect that the results of calculations are not used, etc., and then not perform the calculation at all!

• Many of these benchmarks repeatedly perform an activity and then report the total time. This time inevitably includes the looping time required to repeat the activity. You can use the time returned from the loop benchmark to factor out this effect.

• Two types of GC measurements have been performed. The first requests that the runtime perform a full GC. Both the time and the space recovered (and perhaps added) are reported. The second ``randomly'' allocates several thousand objects into a small array (the time for looping and array access are probably a small part of the overall time.), causing most to be made available for reclamation by the garbage collector. Because, for example, Sun's Java runtime has an incremental collector, many of these may be collected without a full GC occuring.

• The results you get will be dependent on all aspects of your system, including the load on the system at the time which you run the benchmarks. This factor can complicate comparisons.

• For maximum portability, timings are peformed with System.currentTimeMillis(). This function records wall-clock time, so all activities of the computer are included in this time. Your results will be most accurate if you are running on an unloaded system. If you are running on a UNIX system, using time or uptime will give you an idea of how much time your process is actually getting.

• We have gotten non-intuitive results in some cases, suggesting that the Java runtime system occasionally does something (like garbage collection or thread switching) without the benchmark specifically requesting it. The garbage collection benchmark seems to have widely varying behavior depending upon the environment, including crashing (Sun's appletviewer) and thrashing/hanging (Netscape).

Authorship and Contacts