[Kristof Stevens]

Since beginning of this week UPM changed their platform from HP-UX to Linux in Production, but on the webservers we are facing following issue :
Has anybody see this message before because we were not facing this message during our testing :

in console.log :

INFO | jvm 1 | 2007/12/05 16:32:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 2883334

On UI :

java.lang.OutOfMemoryError: allocLargeArray - Object size: 5766688, Num elements: 2883334
at java.lang.StringCoding$CharsetSD.decode([BII)[C(Unknown Source)
at java.lang.StringCoding.decode(Ljava.lang.String;[BII)[C(Unknown Source)
at java.lang.String.([BIILjava.lang.StringV(Unknown Source)
at java.io.ByteArrayOutputStream.toString(ByteArrayOu tputStream.java:174)
at glog.server.producer.ProducerXML.toXML(ProducerXML .java:73)
at glog.webserver.util.Producer.createMasterElementRe mote(Producer.java:159)
at glog.webserver.util.Producer.loadDocument(Producer .java:59)
at glog.webserver.util.AbstractManagementServlet.getD ocument(AbstractManagementServlet.java:32
at glog.webserver.util.AbstractServletProducer.proces s(AbstractServletProducer.java:80)
at glog.webserver.util.BaseServlet.service(BaseServle t.java:591)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at glog.webserver.screenlayout.ClientSessionTracker.d oFilter(ClientSessionTracker.java:54)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at glog.webserver.util.SetCharacterEncodingFilter.doF ilter(SetCharacterEncodingFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:17
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:14
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyo teHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(Handler Request.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelS ocket.java:754)
at org.apache.jk.common.ChannelSocket.processConnecti on(ChannelSocket.java:684)
at org.apache.jk.common.ChannelSocket$SocketConnectio n.runIt(ChannelSocket.java:876)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run()V(Unknown Source)

java version "1.4.2_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
BEA JRockit(R) (build R26.4.0-63-63688-1.4.2_11-20060626-2259-linux-ia32, )


# Java Additional Parameters
wrapper.java.additional.1=-jrockit
wrapper.java.additional.2=-Xms2800m
wrapper.java.additional.3=-Xmx2800m
wrapper.java.additional.4=-Xgcprioausetime
wrapper.java.additional.5=-Xverbose:memory
wrapper.java.additional.6=-Xnoclassgc
wrapper.java.additional.7=-DGC3EncodedPassword=Q0hBTkdFTUU=
wrapper.java.additional.8=-Dglog.properties=glog.tomcat.properties
wrapper.java.additional.9=-Dglog.home=%GLOG_HOME%/glog/config
wrapper.java.additional.10=-Duser.home=%GLOG_HOME%/glog/config
wrapper.java.additional.11=-Djava.io.tmpdir=%GLOG_HOME%/temp
wrapper.java.additional.12=-Djava.awt.headless=true
wrapper.java.additional.13=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
wrapper.java.additional.14=-Dweblogic.ThreadPoolSize=15
wrapper.java.additional.15=-Djava.endorsed.dirs=%GLOG_HOME%/tomcat/common/endorsed
wrapper.java.additional.16=-Dcatalina.base=%GLOG_HOME%/tomcat
wrapper.java.additional.17=-Dcatalina.home=%GLOG_HOME%/tomcat
wrapper.java.additional.18=-Xmanagement

[nick.polites]

Try changing them from 2800 to 2400 and see if the system improves. Usually for tomcat you shouldn't need to increase the JVM past 1536.

[chrisplough]

I definitely agree with Nick. I had this issue with a large client in APAC. Setting the heap to 2800MB didn't leave enough extra memory in the 32-bit memory space to handle the rest of JRockit's memory requirements, particularly around thread stacks.

Let us know if 2400MB works!

--Chris

[Kristof Stevens]

It is definitely Tomcat. We took over this 2800Mb setting from HP-UX when we changed platforms from HP-UX to Linux...But anyway, I decreased memory now back to 2400 Mb.

Will let you know the results if the OOM is gone.

Thx,

Kristof

[Kristof Stevens]

Today we were facing the same issue again after reducing memory from 2800 Mb to 2400 Mb on Tomcat. We'll update memory back to 2Gb.

INFO | jvm 1 | 2007/12/05 16:20:47 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7458224, Num elements: 3729102
INFO | jvm 1 | 2007/12/05 16:32:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 4194320, Num elements: 2097150
INFO | jvm 1 | 2007/12/05 17:07:25 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:07:45 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:07:58 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:08:15 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 14704656, Num elements: 14704640
INFO | jvm 1 | 2007/12/05 17:09:43 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7434256, Num elements: 7434240
INFO | jvm 1 | 2007/12/05 17:10:00 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:10:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:10:22 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:10:38 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/05 17:11:06 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
INFO | jvm 1 | 2007/12/06 16:09:28 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
INFO | jvm 1 | 2007/12/06 16:24:46 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
INFO | jvm 1 | 2007/12/06 16:25:03 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
INFO | jvm 1 | 2007/12/06 16:25:13 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
INFO | jvm 1 | 2007/12/06 16:25:24 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
INFO | jvm 1 | 2007/12/06 16:25:37 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
INFO | jvm 1 | 2007/12/06 16:26:58 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7799168, Num elements: 3899576
INFO | jvm 1 | 2007/12/07 13:00:53 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7487504, Num elements: 7487488
INFO | jvm 1 | 2007/12/07 13:01:28 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7487504, Num elements: 7487488
INFO | jvm 1 | 2007/12/07 13:03:07 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7446544, Num elements: 7446528

[nick.polites]

If you are still getting the out of memory errors, then something else is causing the problem. Have you performed thread dumps to see what is actively running? Please post those and I'll see what I can find. Usually this is caused by someone using SQL backdoor and putting a large query into it, or a CSV export.

[Kristof Stevens]

Sometimes this message appears on the UI screen when a user is opening a shipment.
Never seen it in SQL Backdoor or during CSV export/import. Then it's already too late to make a threaddump on the webserver.
We are facing this issue since we changed our Prod platform from HP-UX to Red Hat Linux. Don't you think it will help if we reduce memory back from 2400Mb to 2000Mb?

Kristof

[miks]

Typically, with out of memory errors, the root cause of the problem come in lines 6 & 7 in the error. You should try to correlate those errors with a business user who is generating the error. In other words, reproduce the problem. Once you do, setup a web conf with support to show them what is causing it.

Some of problems are caused by the application attempting to allocate more memory than it needs. (IOW an OTM BUG ) You can see in the memory errors that it is doubling at some point, which is not a good sign.

You have plenty of memory to run the app.

lines 6 & 7:

at glog.server.producer.ProducerXML.toXML(ProducerXML .java:73)
at glog.webserver.util.Producer.createMasterElementRe mote(Producer.java:159)

[nick.polites]

Originally Posted by Kristof Stevens

Sometimes this message appears on the UI screen when a user is opening a shipment.
Never seen it in SQL Backdoor or during CSV export/import. Then it's already too late to make a threaddump on the webserver.
We are facing this issue since we changed our Prod platform from HP-UX to Red Hat Linux. Don't you think it will help if we reduce memory back from 2400Mb to 2000Mb?

Kristof

Kristof,

I wasn't referring to where the issue would appear but what the issue would have been caused by. Getting the error in the UI is only a symptom that the system is out of memory. What caused it is totally unrelated. The only true way to know what happened is with the thread dumps. You need to have at least 3 every 30 seconds but I would recommend having at least 6 to get a true representation of where the issue is.

Nick

[sknmail@rediffmail.com]

Hi Kristof,

Recently after migration from 4.5 to 5.5, our client too had a similar scenario where the App Server (Instead of Web Server) was going out of memory, and we had a situation to restart the Prod environment 2-3 times in day.
Well after a weeks effort of constant analysis, we could find that one of our agent which gets triggered when Invoice gets created through integration had a Direct Sql Update (DSU) insert statement. As per Oracle Development team and support, the recommendation was to uncheck the Refresh Cache checkbox for all Insert statements (Which by default enabled in 5.5 for all DSUs). And luckily after this change in the agent, its now two weeks now, we didn't face this situation again of Out of Memory. OOM reminded that we also codenamed this issue as OOM

I would also recommend to check by enabling Agent, AgentUtility and SQL enabled to check if there is pattern when OOM occurs, do you see that at a specific time this issue appears during a day, also removing of threaddumps as suggested by Nick would be of great help for Oracle Support... well I agree that the configuration on workflow may be different on your setup compared to ours, but still I thought to share my experience.

Please do checkout some more loggings in all the enabled logs present in OTM...

Regards,
Suresh