[amatorre]

Running OTM 5.5.4 on Linux with Websphere.

The app server stopped responding this morning (database and web stayed up).

Looking at the websphere logs, it reported the following error several times:

"Insufficient space in Javaheap to satisfy allocation request"

Further digging shows numerous 200 MB files written to /otm/websphere/profiles/default folder, named as such:

heapdump.20090728.085542.10998.phd

After about 20 of these files were generated, disk space ran out. The app server was completely pegged out on the CPU, as well.

Does anyone know the cause of this kind of dump?

Thanks,

[chrisplough]

This issue is related to the IBM JDK that WebSphere uses. When working with OTM, we've found a lot of instability, mostly around memory management. Sometimes you run out of Java Native memory and other times you run out of Java Heap memory. From the error, it seems that your heap was not big enough to satisfy the memory requirements of OTM.

Which platform are you running on - AIX or Linux?
How big is your OTM WebSphere Java Heap size?

--Chris

[josephliang]

I also had a nightmare working OTM with IBM JDK.

If you just want to disable heapdump, you can refer to "Disabling Java signal handling" section of Troubleshooting Java code on AIX: Data collection for AIX core dumps

But if you want to find out why (the real nightmare), you can refer to

• Troubleshooting Java code on AIX: Data collection for AIX core dumps
• "IBM JDK Diagnosis documentation" - https://www.ibm.com/developerworks/java/jdk/diagnosis/
• Getting more memory in AIX for your Java applications - http://www.ibm.com/developerworks/es...aix4java1.html

Just FYI, the nightmare tortured us for more than 3 months, even with IBM engineers' help

[chrisplough]

Joseph,

I'm glad to have someone else who's lived through this to share their experiences. The truth is I've worked with most of the OTM clients who are running on AIX and/or WebSphere and it is always very difficult.

Even back at G-Log, I've worked with the IBM engineers for weeks on end without getting a full resolution. The best we've been able to accomplish is to stabilize the environments, and keep it from crashing too often.

While the IBM hardware is pretty darn good (I'm a fan of the POWER CPUs), the OS and IBM JDK are a letdown. It just doesn't mix well with memory intensive / complex apps like OTM. We've had much more success with JRockit and Linux -- which is why we've standardized our Hosting environments on that tech stack.

--Chris

[josephliang]

Forgot to mention that the customer moved to Linux+JRockit eventually (2 years after gone live)
Another score for Red vs. Blue

[amatorre]

Thanks, gents. Your input is very helpful.

The AS is on Linux.

[chrisplough]

No problem. You'll find that the limitation is the IBM JDK that is used with WebSphere. On Linux, you have the option of using JRockit if your AS is WebLogic (or even OAS), which is much faster and more stable.

--Chris