View RSS Feed


Benchmarking (Part 1): Java JVM Performance with VolanoMark

Rate this Entry
by on January 9th, 2008 at 00:50 (35281 Views)
    Bookmark and Share    

I wanted to give a quick tutorial on how to benchmark JVM performance with VolanoMark. This is a great benchmark for determining the relative performance of different Java JVMs on various platforms. Also, since this is a highly multi-threaded test, it gives a good indication of relative OTM performance. Best of all, it fits into the MavenWire Benchmark Toolkit, because it works well. As an added bonus for the community - it's free!

First, some links you'll need:


Sun Java JVM


To get started, download and install VolanoMark, then download the JVMs that you'd like to test. Keep in mind that OTM v5.0 and up uses Java 1.4.x. For our tests here, I'm going to benchmark JRockit 1.4.2_12.

Go ahead and install VolanoMark to /opt/benchmarks/volanomark:
[benchmark@perftest01 volanomark]# export CLASSPATH=.
[benchmark@perftest01 volanomark]# java vmark2_5_0_9 -o /opt/benchmarks/volanomark
InstallShield Java (TM) Edition
Extracting installation code..............................done
Extracted 34 files into 10 directories.
Installation is complete.
Now install your JDKs, per the directions given for each. After installing, I like to copy each one under the benchmarks directory, so that I have everything in one place and can easily tar it up and move it to other servers. For instance, I'd copy JRockit into /opt/benchmarks/jrockit1.4.2_12.

Next, edit /opt/benchmarks/volanomark/ Under the platform that matches your system, add a section that lists the parameters for your installed JDK(s). For instance, on Linux, I'll edit it to look as follows. The bold sections have been added and I've omitted the section between the "..." marks for brevity:
# Red Hat Enterprise Linux
# Usage: linux loop|net otm-jrockit-142_12
#                        bea81|blackdown13|blackdown14|ibm13|ibm14|
#                       sun13|sun14
linux () {
    # Set a 2MB maximum stack size to avoid this bug:
    ulimit -Ss 2048
    if [ "$1" = "loop" ]
        ulimit -Sn 1024
        ulimit -Sn 10240
    for file in lib/*.jar; do classpath="$classpath:$file"; done
    case $2 in
            options="-jrockit -Xms256m -Xmx256m -Xgcprio:throughput -Xverbose:memory"
            echo "Vendor choice is otm-jrockit-142_12, bea81, blackdown13, blackdown14, ibm13, ibm14, sun13, or sun14."
Now that the installation and configuration are complete, let's run some tests!

Open up two terminal sessions into your server. In the first one, run the server portion of the benchmark:
[benchmark@perftest01 volanomark]# ./ server loop otm-jrockit-142_12
Now in the second terminal session, run the client portion of the benchmark:
[benchmark@perftest01 volanomark]# ./ client loop otm-jrockit-142_12
The benchmark will take anywhere from 5-10 minutes to run. Once it is complete, record the "Average throughput" for each subtest within the benchmark (4 in total). These results are recorded within logfiles in the installation directory, named test-1.log, test-2.log, test-3.log and test-4.log.

Example Results:
java.vendor        = BEA Systems, Inc.
java.vendor.url    =
java.version       = 1.4.2_12
java.class.version = 48.0
java.compiler      = null            = Linux
os.version         = 2.6.9-67.0.1.ELsmp
os.arch            = i386

VolanoMark version =
Messages sent      = 20000
Messages received  = 380000
Total messages     = 400000
Elapsed time       = 3.006 seconds
Average throughput = 133067 messages per second
To ensure I have solid, repeatable results - I run the benchmark 3 times and average out all 12 "Average throughput" results to give a final performance number. If the calculation returns a decimal result, then I round down to the nearest whole number. For instance, when running this test on one of our older Hosting servers, the following results were returned:
Run 1: 133067, 134726, 125589, 120048
Run 2: 129786, 130421, 137127, 135089
Run 3: 125196, 133333, 137599, 134293
Averaged Result: 131356
(higher is better)
(Update 20080209: We just received some new servers into our Hosting environment and I couldn't help benchmarking them. WOW! These are the fastest servers I've seen so far for running Java and OTM!)
Run 1: 243309, 220629, 214247, 246457
Run 2: 242102, 226244, 215633, 235156
Run 3: 259909, 245248, 218103, 245145
Averaged Result: 234348
(higher is better)
(Update 20080422: Just got some new servers [i.e. expensive toys] into our Hosting environment. Here are the new benchmark results -- FAST!)
Run 1: 333323, 319922, 319922, 328282
Run 2: 332893, 311982, 301123, 321231
Run 3: 322934, 322342, 301828, 322234
Averaged Result: 319848.08
(higher is better)
In order to test multiple servers that run the same OS, you just need to tar up the VolanoMark installation directory, transfer to the new server, extract and then run the tests -- simple!

If you'd like to compare different platforms (say Solaris vs. Linux), just install VolanoMark on each server, then download the appropriate Java JVMs for each platform and configure VolanoMark to use them. Run the tests and compare the results. That's the beauty of Java benchmarks - cross-platform testing is a breeze!

I hope this helps.


Other related Posts:

Submit "Benchmarking (Part 1): Java JVM Performance with VolanoMark" to Digg Submit "Benchmarking (Part 1): Java JVM Performance with VolanoMark" to Submit "Benchmarking (Part 1): Java JVM Performance with VolanoMark" to StumbleUpon Submit "Benchmarking (Part 1): Java JVM Performance with VolanoMark" to Google

Updated April 22nd, 2008 at 06:38 by chrisplough (Updating benchmark results for new hardware)

otm , performance


  1. ianlo's Avatar
    Hi Chris,

    Great stuff! Now I can show off!


Total Trackbacks 0
Trackback URL:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56