Rate this Entry

Benchmarking (Part 1): Java JVM Performance with VolanoMark

Posted January 9th, 2008 at 00:50 by chrisplough
Updated April 22nd, 2008 at 07:38 by chrisplough (Updating benchmark results for new hardware)

Hello!

I wanted to give a quick tutorial on how to benchmark JVM performance with VolanoMark. This is a great benchmark for determining the relative performance of different Java JVMs on various platforms. Also, since this is a highly multi-threaded test, it gives a good indication of relative OTM performance. Best of all, it fits into the MavenWire Benchmark Toolkit, because it works well. As an added bonus for the community - it's free!

First, some links you'll need:

VolanoMark
http://www.volano.com/benchmarks.html

BEA JRockit JVM
http://commerce.bea.com/products/web...t_prod_fam.jsp

Sun Java JVM
http://java.sun.com/javase/downloads/

HP Java JVM
http://www.hp.com/java/

IBM Java JVM
http://www.ibm.com/developerworks/java/

To get started, download and install VolanoMark, then download the JVMs that you'd like to test. Keep in mind that OTM v5.0 and up uses Java 1.4.x. For our tests here, I'm going to benchmark JRockit 1.4.2_12.

Go ahead and install VolanoMark to /opt/benchmarks/volanomark:

Code:

[benchmark@perftest01 volanomark]# export CLASSPATH=.
[benchmark@perftest01 volanomark]# java vmark2_5_0_9 -o /opt/benchmarks/volanomark
InstallShield Java (TM) Edition
Extracting installation code..............................done
99%..               
Extracted 34 files into 10 directories.
Installation is complete.

Now install your JDKs, per the directions given for each. After installing, I like to copy each one under the benchmarks directory, so that I have everything in one place and can easily tar it up and move it to other servers. For instance, I'd copy JRockit into /opt/benchmarks/jrockit1.4.2_12.

Next, edit /opt/benchmarks/volanomark/startup.sh. Under the platform that matches your system, add a section that lists the parameters for your installed JDK(s). For instance, on Linux, I'll edit it to look as follows. The bold sections have been added and I've omitted the section between the "..." marks for brevity:

Code:

#***********************************************************************
# Red Hat Enterprise Linux
# Usage: linux loop|net otm-jrockit-142_12
#                        bea81|blackdown13|blackdown14|ibm13|ibm14|
#                       sun13|sun14
#***********************************************************************
linux () {
    # Set a 2MB maximum stack size to avoid this bug:
    # http://developer.java.sun.com/developer/bugParade/bugs/4466587.html
    ulimit -Ss 2048
    if [ "$1" = "loop" ]
    then
        ulimit -Sn 1024
    else
        ulimit -Sn 10240
    fi
    for file in lib/*.jar; do classpath="$classpath:$file"; done
    case $2 in
        otm-jrockit-142_12)
            java=/opt/jrockit142_12/bin/java
            options="-jrockit -Xms256m -Xmx256m -Xgcprio:throughput -Xverbose:memory"
            ;;
        bea81)
...
... 
       *)
            echo "Vendor choice is otm-jrockit-142_12, bea81, blackdown13, blackdown14, ibm13, ibm14, sun13, or sun14."
            exit
            ;;
    esac
}

Now that the installation and configuration are complete, let's run some tests!

Open up two terminal sessions into your server. In the first one, run the server portion of the benchmark:

Code:

[benchmark@perftest01 volanomark]# ./startup.sh server loop otm-jrockit-142_12

Now in the second terminal session, run the client portion of the benchmark:

Code:

[benchmark@perftest01 volanomark]# ./startup.sh client loop otm-jrockit-142_12

The benchmark will take anywhere from 5-10 minutes to run. Once it is complete, record the "Average throughput" for each subtest within the benchmark (4 in total). These results are recorded within logfiles in the installation directory, named test-1.log, test-2.log, test-3.log and test-4.log.

Example Results:

Code:

java.vendor        = BEA Systems, Inc.
java.vendor.url    = http://www.bea.com/
java.version       = 1.4.2_12
java.class.version = 48.0
java.compiler      = null
os.name            = Linux
os.version         = 2.6.9-67.0.1.ELsmp
os.arch            = i386

VolanoMark version = 2.5.0.9
Messages sent      = 20000
Messages received  = 380000
Total messages     = 400000
Elapsed time       = 3.006 seconds
Average throughput = 133067 messages per second

To ensure I have solid, repeatable results - I run the benchmark 3 times and average out all 12 "Average throughput" results to give a final performance number. If the calculation returns a decimal result, then I round down to the nearest whole number. For instance, when running this test on one of our older Hosting servers, the following results were returned:

Run 1: 133067, 134726, 125589, 120048
Run 2: 129786, 130421, 137127, 135089
Run 3: 125196, 133333, 137599, 134293
Averaged Result: 131356
(higher is better)

(Update 20080209: We just received some new servers into our Hosting environment and I couldn't help benchmarking them. WOW! These are the fastest servers I've seen so far for running Java and OTM!)

Run 1: 243309, 220629, 214247, 246457
Run 2: 242102, 226244, 215633, 235156
Run 3: 259909, 245248, 218103, 245145
Averaged Result: 234348
(higher is better)

(Update 20080422: Just got some new servers [i.e. expensive toys] into our Hosting environment. Here are the new benchmark results -- FAST!)

Run 1: 333323, 319922, 319922, 328282
Run 2: 332893, 311982, 301123, 321231
Run 3: 322934, 322342, 301828, 322234
Averaged Result: 319848.08
(higher is better)

In order to test multiple servers that run the same OS, you just need to tar up the VolanoMark installation directory, transfer to the new server, extract and then run the tests -- simple!

If you'd like to compare different platforms (say Solaris vs. Linux), just install VolanoMark on each server, then download the appropriate Java JVMs for each platform and configure VolanoMark to use them. Run the tests and compare the results. That's the beauty of Java benchmarks - cross-platform testing is a breeze!

I hope this helps.

--Chris

Other related Posts: