Rate this Entry

Benchmarking (Part 4): Real-World Network Performance with Soap Stone

Posted April 21st, 2008 at 19:16 by chrisplough

Hello!

I wanted to give a quick tutorial on how to benchmark real-world network performance using Soap Stone. I like this benchmark more than standard network throughput tests (ttcp, NetIO, etc) because it uses the Java JVM (just as OTM does) and it also utilizes application protocols that are similar to OTMs -- so rather than a meaningless throughput number, the results reflect the throughput you'll see with OTM and other multi-tier Java applications. Specifically, I test 3 protocols: http, rmi and raw. As always, this fits well into the MavenWire Benchmark Toolkit, because it is FREE and works very well.

First, some links you'll need:

Soap Stone Benchmark
http://soap-stone.sourceforge.net/

BEA JRockit JVM
http://commerce.bea.com/products/web...t_prod_fam.jsp

Sun Java JVM
http://java.sun.com/javase/downloads/

HP Java JVM
http://www.hp.com/java/

IBM Java JVM
http://www.ibm.com/developerworks/java/

To get started, download and install Soap Stone, then download the JVMs that you'd like to test. Keep in mind that OTM v5.x uses Java 1.4.x. For our tests here, I'm going to benchmark JRockit 1.4.2_12.

Go ahead and install Soap Stone to /opt/benchmarks/soapstone. Now install your JDKs, per the directions given for each. After installing, I like to copy each one under the /opt/benchmarks directory, so that I have everything in one place and can easily tar it up and move it to other servers. For instance, I'd copy JRockit into /opt/benchmarks/jrockit1.4.2_12. Ensure that the JDK you'd like to test is in your path and then run the following to verify:

Code:

[benchmark@perftest02 soapstone]$ ./java -version
java version "1.4.2_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_12-b03)
BEA JRockit(R) (build R27.1.0-109-73164-1.4.2_12-20061129-1418-linux-ia32, compiled mode)
[benchmark@perftest02 soapstone]$

Next, install it on another server (it's a network test - after all!). You may need to ensure that firewall rules are open, ports are available, etc - depending on your network configuration. When in doubt, talk with your network admin!

Open up one terminal session into server1. Run the following command to run the server side of the benchmark:

Code:

[benchmark@perftest01 soapstone]# java -jar soap-stone-full-0.952.jar server -server `hostname`

Next, open up one terminal session into server2. Run the following command to run the client side of the benchmark (replace [PROTOCOL] with either http, rmi or raw):

Code:

[benchmark@perftest02 soapstone]# java -jar soap-stone-full-0.952.jar client -server [SERVER1_IP] -protocol [PROTOCOL]

So here's the breakdown of the different protocols and when you'd use them:

HTTP
- Simulates the Browser to OTM Web server traffic (or Integration Server to OTM Web server)
RMI
- Simulates the OTM Web server (Tomcat) to OTM App server (WebLogic / OAS / WebSphere) traffic.
RAW
- Best simulates the OTM App (WebLogic / OAS / WebSphere) to OTM DB server traffic.

So - depending on the servers I'm testing, I'll choose the appropriate protocol from the list above.

The benchmark will take only a minute or so to run.

Once it is complete, check the output for errors. Once you're sure that you're benchmark runs are clean, then you can reliably run the benchmark and simply pull the timing results from the results.txt file.

Example Results:
HTTP Test

Code:

[root@perftest01 soapstone]# java -jar soap-stone-full-0.952.jar client -server perftest02 -protocol http
soap-stone: Plugin 'soap' loaded from ./plugins/jwsdp-plugin.jar
soap-stone: Plugin 'http' loaded from ./plugins/http-plugin.jar
soap-stone: Plugin 'raw' loaded from ./plugins/raw-plugin.jar
soap-stone: Plugin 'rmi' loaded from ./plugins/rmi-plugin.jar
soap-stone Client: command-line: soap-stone client -server perftest02 -protocol http
soap-stone Client: connecting to control socket: /perftest02:8001
.+.........
soap-stone Client: size=1500 bytes buffers=1000 avg=45.69 Mbit/sec min=7.34 Mbit/sec max=52.5 Mbit/sec

prefixes=java.runtime.,java.version.,java.vm.,os.,soap-stone. suppress=java.vm.specification
done in 3620 ms

RMI Test

Code:

[root@perftest01 soapstone]# java -jar soap-stone-full-0.952.jar client -server perftest02 -protocol rmi
soap-stone: Plugin 'soap' loaded from ./plugins/jwsdp-plugin.jar
soap-stone: Plugin 'http' loaded from ./plugins/http-plugin.jar
soap-stone: Plugin 'raw' loaded from ./plugins/raw-plugin.jar
soap-stone: Plugin 'rmi' loaded from ./plugins/rmi-plugin.jar
soap-stone Client: command-line: soap-stone client -server perftest02 -protocol rmi
soap-stone Client: connecting to control socket: /perftest02:8001
.+.........
soap-stone Client: size=1500 bytes buffers=1000 avg=62.29 Mbit/sec min=4.81 Mbit/sec max=69.78 Mbit/sec

prefixes=java.runtime.,java.version.,java.vm.,os.,soap-stone. suppress=java.vm.specification
done in 2965 ms

RAW Test

Code:

[root@perftest01 soapstone]# java -jar soap-stone-full-0.952.jar client -server perftest02-protocol raw
soap-stone: Plugin 'soap' loaded from ./plugins/jwsdp-plugin.jar
soap-stone: Plugin 'http' loaded from ./plugins/http-plugin.jar
soap-stone: Plugin 'raw' loaded from ./plugins/raw-plugin.jar
soap-stone: Plugin 'rmi' loaded from ./plugins/rmi-plugin.jar
soap-stone Client: command-line: soap-stone client -server perftest02 -protocol raw
soap-stone Client: connecting to control socket: /perftest02:8001
++..........
soap-stone Client: size=1500 bytes buffers=10000 avg=899.7 Mbit/sec min=894.07 Mbit/sec max=901.11 Mbit/sec

prefixes=java.runtime.,java.version.,java.vm.,os.,soap-stone. suppress=java.vm.specification
done in 2489 ms
[root@perftest01 soapstone]#

The results are in Mbit/sec, so a higher number is better (i.e. more throughput).

To ensure I have solid, repeatable results - I run the benchmark 3 times and create a final score by averaging the results of each. If I notice a significant statistical difference from run to run, then I'll invalidate the results and analyze the system to ensure that nothing else is running or that there isn't a problem with the benchmark. For instance, when running this test between two of our medium-sized Hosting servers, the following results were returned:

HTTP Test
Run 1 (Mbits/sec): 50.28
Run 2 (Mbits/sec): 50.25
Run 3 (Mbits/sec): 50.22
Final Averaged Result (Mbits/sec): 50.25
(higher is better)

RMI Test
Run 1 (Mbits/sec): 62.52
Run 2 (Mbits/sec): 63.28
Run 3 (Mbits/sec): 64.17
Final Averaged Result (Mbits/sec): 63.32
(higher is better)

RAW Test
Run 1 (Mbits/sec): 893.38
Run 2 (Mbits/sec): 899.00
Run 3 (Mbits/sec): 896.18
Final Averaged Result (Mbits/sec): 896.19
(higher is better)

In order to test multiple servers that run the same OS, you just need to tar up the Soap Stone installation directory and JVM, transfer to the new server, extract and then run the tests -- simple!

If you'd like to compare different platforms (say Solaris vs. Linux), just install the Soap Stone on each server, then download the appropriate Java JVMs for each platform and configure Soap Stone to use them. Run the tests and compare the results.

I hope this helps.

--Chris

Other related Posts: