[bsteemson]

Hi,

We periodically experience very slow web server responses, this at times is due to slow application server but recently we seing this when the server appears ok. I seems to conincide with a lot of the following errors in the mod_jk.log on the web server. Seems to imply some network or load issue.

Any assistance would be greatly appreciated.

Rgds
Brian

[Tue Sep 02 13:25:30 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:27:29 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:27:59 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:27:59 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:27:59 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:27:59 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:30:52 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:33:10 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems
[Tue Sep 02 13:35:51 2008] [error] ajp_service::jk_ajp_common.c (1659): Client connection aborted or network problems

[nick.polites]

What kind of server specs do you have here? Be sure to include processors and memory. How is the server load? Any I/O wait? What is your JVM set to? Do you have any firewalls between your web and app tier? This error does appear like there is an obvious problem with the network connection but I would also check you app tier and the console.log, glog.app.log weblogic.log.

Also check your DNS setting to make sure nothing funky is going on with resolving your IP or DNS name.

[bsteemson]

Thanks for the reply. The load on our servers is high we do at times see issues on the app server with gc times etc. Our app and db servers are IBM LPAR P590s, app with 8 Gig Ram and 4 CPUs entitlement with 16 LCPUs. The DB server has 16 Gig of Ram. The web server is and IBM Blade running Linnux with 2 Gig ram and 4 cpus. We are looking into the networks settings.

[nick.polites]

I can tell you that 2 GB is not enough for the web server even at the minimum settings. You need to have at least 4. If you set the JVM to 1 GB it will still grow to at least 1.5 which doesn't give the OS a lot to work with and really shouldn’t be used for a production environment. You should ideally increase the JVM to at least 1.5GB or the optimal 2-2.4GB . Even at the 1.5GB it will grow to 1.8GB-2GB. How high is the load?

[bsteemson]

Thanks,

The webserver JVM is set to 1Gig. The load on this server is high, all on the intergration is posted via the webserver as well as the UI. What sorts of things should I be looking at to measure load? And what sort of limits are there?

[nick.polites]

Well the first thing is that you'll need to increase the ram and bump the JVM up to at least 1.5GB. The next step would be to figure out how many transmissions are being sent into your web server. It may be that you need a dedicated web server just for your integration orders. Also how many users are accessing the web server?

Again what is the load number? High to you may not mean high to me and vice versa...anything about a 5 I consider is a bottleneck and something terrible is going on if it is occurring for a prolonged time.

[bsteemson]

Thanks,


We'll look at additional memory. The load average on the web server doesn't look too bad

14:09:11 up 169 days, 17:54, 2 users, load average: 0.61, 0.35, 0.30

App and server is high

02:09PM up 73 days, 15:59, 6 users, load average: 6.36, 5.66, 5.24

and DB servers is very high

02:10PM up 73 days, 16:01, 2 users, load average: 13.29, 10.97, 8.75

We having something in the order of 23,000 transmissions consisting of anywhere in the order of 29,000 - 37,000 transactions depending on the transmission type.

We do have issues with app and db server, but we are experiencing a large number of web timeouts even on simple queries.

[nick.polites]

Yep your problem is on the DB tier. It is having a domino effect on the app and web tiers. I don't know how powerful the IBM processors are but I would also check the I/O wait on your DB.

Also just so you can compare the speed of your storage run the following command on a mount that has at least 2 GB of free space:

time dd if=/dev/zero of=BigFile bs=1024k count=2048

This takes about 3.2 seconds to run on our fastest Dell MD 3000 storage arrays.

[bsteemson]

Results from the db server

stlsydhmlodb:mlop> time dd if=/dev/zero of=BigFile bs=1024k count=2048
2048+0 records in.
2048+0 records out.
real 0m14.47s
user 0m0.01s
sys 0m12.44s


Also have had the following feedback from Chirs Plough, so we'll be looking at increasing the weblogic threads and increasing memory on the webserver

The amount of memory on the web server is definitely to low. The benefit of JRockit is that it performs better than any other JDK, but does so by using a lot of memory outside of the Java Heap to optimize the code. So a 1GB Java heap can in turn cause the total Java process to use between 1.5 and1.75GB of memory. Apache will utilize another 125-256MB, which then doesn't leave enough for the OS and other processes.
The WebLogic Execute threads are still at the default of 50 and this is usually too low for any system that has a signifiicant number of web users and/or integration. This is easily adjustable by changing the config.xml.template file's ThreadCount parameter.
Having different platforms for the OTM Web and App tiers is a large potential issue. OTM makes RMI calls between these two tiers and if the JVM versions and codebase aren't identical - many issues can crop up, including performance, functionality and errors. The fact that the JVMs are from different providers (JRockit vs. IBM JDK) further complicates the issue.
I am also seeing high load on the DB server and agree that it attributes to poor web performance, but feel that the first two issues (memory and WL execute threads) may be a more immediate bottleneck.

[nick.polites]

The results of the DD aren't that bad, but are in line with legacy storage arrays. I believe some of of 4 year old LSI units performed like this. I would look into the database calls and try capturing any active sql calls. There may be an agent which is looping or possibly bad formed sql queries which may be causing the huge loads.