Rate this Entry

Benchmarking (Part 3): Java JVM Performance with DaCapo

Posted March 23rd, 2008 at 17:37 by chrisplough
Updated April 22nd, 2008 at 07:41 by chrisplough (Updating benchmark results for new hardware)

Hello!

I wanted to give a quick tutorial on how to benchmark JVM performance with the DaCapo Benchmarks. This is a great benchmark for determining the computational (algorithm crunching) power of different Java JVMs on various platforms. Also, since this is a highly computational, it does a good job of providing relative OTM Bulk Planning performance. Best of all, it fits into the MavenWire Benchmark Toolkit, because it works well. As an added bonus for the community - it's free!

First, some links you'll need:

DaCapo Benchmark
http://dacapobench.org/

BEA JRockit JVM
http://commerce.bea.com/products/web...t_prod_fam.jsp

Sun Java JVM
http://java.sun.com/javase/downloads/

HP Java JVM
http://www.hp.com/java/

IBM Java JVM
http://www.ibm.com/developerworks/java/

To get started, download and install DaCapo, then download the JVMs that you'd like to test. Keep in mind that OTM v5.0 and up uses Java 1.4.x. For our tests here, I'm going to benchmark JRockit 1.4.2_12.

Go ahead and install DaCapo to /opt/benchmarks/dacapo. Now install your JDKs, per the directions given for each. After installing, I like to copy each one under the /opt/benchmarks directory, so that I have everything in one place and can easily tar it up and move it to other servers. For instance, I'd copy JRockit into /opt/benchmarks/jrockit1.4.2_12. Ensure that the JDK you'd like to test is in your path and then run the following to verify:

Code:

[benchmark@perftest02 dacapo]$ ./java -version
java version "1.4.2_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_12-b03)
BEA JRockit(R) (build R27.1.0-109-73164-1.4.2_12-20061129-1418-linux-ia32, compiled mode)
[benchmark@perftest02 dacapo]$

Next, extract the xalan.jar file, so the benchmark will run correctly:

Code:

[benchmark@perftest02 dacapo]$ jar -xvf dacapo-2006-10-MR2.jar xalan.jar

Now, in the DaCapo installation directory, create a script like the following to run the benchmark and process the log files:

run_dacapo.sh

Code:

#!/bin/bash

export CLASSPATH=./lib:$CLASSPATH; export CLASSPATH

echo "** Starting Benchmarks"
echo " * This will take several minutes"
java -Xms256m -Xmx256m -Xbootclasspath/p:xalan.jar -jar dacapo-2006-10-MR2.jar -s large antlr bloat 
fop hsqldb jython lusearch luindex pmd xalan > output.txt 2>&1
echo "** Benchmarks Completed"
grep '=====' output.txt > results.txt
echo " * Benchmark results are in results.txt"

Now that the installation and configuration are complete, let's run some tests!

Open up one terminal session into your server. Run the following command to run the benchmark:

Code:

[benchmark@perftest01 dacapo]# chmod +x run_dacapo.sh
[benchmark@perftest01 dacapo]# ./run_dacapo.sh

The benchmark will take anywhere from 5-10 minutes to run.

Once it is complete, check the output.txt file and ensure there weren't any errors. Certain JVMs aren't able to run all of the tests (I've noticed this with the IBM JDK), so you may have to exclude various tests. Once you're sure that you're benchmark runs are clean, then you can reliably run the benchmark and simply pull the timing results from the results.txt file.

Example Results from results.txt:

Code:

===== DaCapo antlr starting =====
===== DaCapo antlr PASSED in 4630 msec =====
===== DaCapo bloat starting =====
===== DaCapo bloat PASSED in 61779 msec =====
===== DaCapo fop starting =====
===== DaCapo fop PASSED in 2226 msec =====
===== DaCapo hsqldb starting =====
===== DaCapo hsqldb PASSED in 5079 msec =====
===== DaCapo jython starting =====
===== DaCapo jython PASSED in 24531 msec =====
===== DaCapo lusearch starting =====
===== DaCapo lusearch PASSED in 3396 msec =====
===== DaCapo luindex starting =====
===== DaCapo luindex PASSED in 6203 msec =====
===== DaCapo pmd starting =====
===== DaCapo pmd PASSED in 24445 msec =====
===== DaCapo xalan starting =====
===== DaCapo xalan PASSED in 14990 msec =====

The results are in msec, so a lower number is better (i.e. faster processing).

For each run, I create an overall score, averaging out the results of each test, giving each one equal weighting. So for the sample results above, I'd average out 2630, 61779, 2226, 5079, 24531, 3396, 6203, 24445 and 14990, giving a result of 16364.33. To ensure I have solid, repeatable results - I run the benchmark 3 times and create a final score by averaging the results of each. If I notice a significant statistical difference from run to run, then I'll invalidate the results and analyze the system to ensure that nothing else is running or that there isn't a problem with the benchmark. For instance, when running this test on one of our medium-sized Hosting servers, the following results were returned:

Run 1 (avg): 14767.00
Run 2 (avg): 14751.70
Run 3 (avg): 14774.30
Final Averaged Result: 14764.33
(lower is better)

(Update 20080422: Just got some new servers [i.e. expensive toys] into our Hosting environment. Here are the new benchmark results -- FAST!)

Run 1 (avg): 13797.10
Run 2 (avg): 14305.40
Run 3 (avg): 13774.10
Final Averaged Result: 13958.89
(lower is better)

In order to test multiple servers that run the same OS, you just need to tar up the DaCapo installation directory, transfer to the new server, extract and then run the tests -- simple!

If you'd like to compare different platforms (say Solaris vs. Linux), just install the DaCapo Benchmark on each server, then download the appropriate Java JVMs for each platform and configure DaCapo to use them. Run the tests and compare the results.

I hope this helps.

--Chris

Other related Posts: