otmfaqForumsBlogsRegister
FAQMembers ListCalendarToday's PostsSearch


 Subscribe Blogs:RSS
 Subscribe Forums:RSS
OTMFAQ Home
OTMFAQ Blogs
OTMFAQ Forums
OTMFAQ Tutorials

OTM SIG
MavenWire


Performance, Scalability and HA Optimizing the performance of OTM / G-Log, configuring Scalability (SCA) and maintaining High Availability.

Tags: , , ,

Reply
 
Submit Tools LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old March 7th, 2008, 14:07
Junior Member
 
Join Date: Jul 2007
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0
JayVal is on a distinguished road
Oracle Application Server running out of memory

We have this OAS running Out of Memory issue in Production on a daily basis and we had to restart the Server everytime. OTM becomes slow to the point that the application was not responding.

A check across the system showed the OAS JVM was using 100% of 1 CPU
And the OAS heap was maxed out on memory:

[Full GC 1024447K->1024447K(1024448K), 4.0970930 secs]
[Full GC 1024447K->1024447K(1024448K), 4.0963020 secs]
[Full GC 1024447K->1024447K(1024448K), 4.0915010 secs]

Garbage Collection (GC) was taking place, but little or no un-used memory was being Unloaded.

Scanning the OAS console.log, a pattern was seen where memory began rising.

10:47 AM – OK at this point
[GC 665769K->456278K(1023424K), 0.0744990 secs]

As we approach 11:21 AM, OAS starts using more total memory and releases less memory via GC.
The GC times increase also indicating more work is needed to find memory to Unload.

11:21 AM
[GC 875287K->672253K(1024448K), 3.7772500 secs]

By 11:26 AM, OAS has begun maxing the heap before GC runs,
and OAS begins performing FULL GC”s.

[Full GC 1024447K->815040K(1024448K), 3.4803020 secs]

At this point, GC is still Unloading memory and the application is still responding.

As time goes on, OAS is able to Unload less and less memory.

As of 11:38 AM, OAS is getting little or no memory back via GC.
At this point, the application has already stopped responding.

Please help.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old March 7th, 2008, 14:46
chrisplough's Avatar
Site Moderator
 
Join Date: Jun 2006
Location: West Chester, PA
Posts: 790
Blog Entries: 7
Thanks: 51
Thanked 194 Times in 119 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10
chrisplough has a spectacular aura aboutchrisplough has a spectacular aura aboutchrisplough has a spectacular aura about
Send a message via AIM to chrisplough
Re: Oracle Application Server running out of memory

JayVal,

The biggest issue is that you don't have enough memory allocated to the OTM App server. Allocating 1GB is the bare minimum for OTM and doesn't leave enough room for memory-intensive OTM processes.

If you're running on Win32, then you'll be limited to 1.2 or 1.4GB, but if you're running on any other platform, you should bump up to 2GB. This is done within your $OTM_HOME/oas/bin/oc4j.sh script.

Keep in mind that you'll need enough physical memory on your server to accommodate a heap of this size. If you're only running OAS on this server, and the heap is 2GB, then you should have 4GB of physical memory to allow room for the heap, Java native memory and OS. If you're running more, then you'll need more memory.

Hope this helps!

--Chris
__________________
Chris Plough
MavenWire

www.MavenWire.com
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old March 7th, 2008, 17:33
Junior Member
 
Join Date: Jul 2007
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0
JayVal is on a distinguished road
Re: Oracle Application Server running out of memory

Thanks Chris.

Here is some information I got. Also I am giving here the contents of the oc4j.sh. I am not an Admin guy, so I may not be able to put the pieces together.

1) The Oracle App server and Apache Web Server runs on the same box.
2) The OS is Linux 32bit
3) The System is 64bit.

**************************************
oc4j.sh contents
**************************************
#!/bin/sh
#
# oc4j - shell for invoking OC4J basic operations.
#
# Usage: oc4j [Options]
#
# Options:
# -start : start OC4J
# -shutdown -port <ORMI port> -password <password>
# : stop OC4J
# -version : display the version
# -help : display this message
#
# Copyright (c) 2004, 2005, Oracle. All rights reserved.
#
################################################## #######
########## START CONFIGURATION SECTION ##################
################################################## #######
GLOG_HOME=/u02/otmPROD
JAVA_HOME=/u02/otmPROD/jdk
ORACLE_HOME=$GLOG_HOME/oas
J2EE_HOME=$ORACLE_HOME/j2ee/home
VERBOSE=on
MEM_ARGS="-XX:PermSize=256m -XX:MaxPermSize=256m -Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -verbose:gc"
GC3_OPTIONS="-Duser.home=/u02/otmPROD/glog/config -Dglog.properties=glog.properties"
OC4J_JVM_ARGS="$MEM_ARGS $GC3_OPTIONS"
echo $JVMARGS
# Set DASH variables
DASH_HOME=$GLOG_HOME/dash; export DASH_HOME
XPRESS=$DASH_HOME/bin; export XPRESS
LD_LIBRARY_PATH=$DASH_HOME/lib:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH
PATH=$DASH_HOME/lib:$PATH; export PATH
#Any persistent arguments to specify at the JVM level can be set here
#By default this will be read from the operating system environment
if [ "$OC4J_JVM_ARGS" ]
then
JVMARGS=$OC4J_JVM_ARGS
else
JVMARGS=
fi
CMDARGS=-userThreads
if [ "$VERBOSE" ]
then
VERBOSE=$VERBOSE
else
VERBOSE=off
fi
ORMI_URL=ormi://localhost
ORMI_USER=oc4jadmin
OC4J_JAR=$J2EE_HOME/oc4j.jar
ADMIN_JAR=$J2EE_HOME/admin.jar
SERVER_XML=$J2EE_HOME/config/server.xml
################################################## #######
########## END CONFIGURATION SECTION ##################
################################################## #######
check_oc4j()
{
EXIT=0
if [ "$JAVA_HOME" = "" ]
then
echo "Error: JAVA_HOME environment variable is not defined."
check_msg="correct JAVA_HOME environment variable."
EXIT=2
elif [ ! -x $JAVA_HOME/bin/java ]
then
echo "Error: Can not find java executable in $JAVA_HOME/bin."
check_msg="correct java executable."
EXIT=2
elif [ "$ORACLE_HOME" = "" ]
then
echo "Error: The ORACLE_HOME environment variable must be set before executing this script. Set this to the directory into which you unzipped oc4j_extended.zip."
check_msg="correct ORACLE_HOME environment variable."
EXIT=2
elif [ ! -r "$OC4J_JAR" ]
then
check_msg="readable $OC4J_JAR."
EXIT=2
elif [ ! -w "$SERVER_XML" ]
then
check_msg="writable $SERVER_XML."
EXIT=2
elif [ ! -r "$ADMIN_JAR" ]
then
check_msg="readable $ADMIN_JAR."
EXIT=2
fi
}
echo_check_msg()
{
echo "Error: The command can not be run without $1"
}
start_oc4j()
{
echo "Starting OC4J from $J2EE_HOME ..."
CMDARGS="-userThreads -config $SERVER_XML"
while [ $# -ge 1 ]
do
case $1 in
*)
echo ""
echo "Error: The optoin \"$1\" was not recognized."
EXIT=1
return
;;
esac
done
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
}
shutdown_oc4j()
{
while [ $# -ge 1 ]
do
case $1 in
-port)
shift
if [ "$1" = "" ]
then
echo ""
echo "Error: You must specify the ORMI port value."
EXIT=1
return
else
ORMI_PORT=$1
shift
fi
;;
-password)
shift
if [ "$1" = "" ]
then
echo ""
echo "Error: You must specify the password value."
EXIT=1
return
else
ORMI_PASSWORD=$1
shift
fi
;;
*)
echo ""
echo "Error: The option \"$1\" was not recognized."
EXIT=1
return
;;
esac
done
if [ "$ORMI_PORT" = "" ]
then
echo ""
echo "Error: You must specify the ORMI port using the -port switch."
echo " The port value can be found in $J2EE_HOME/config/rmi.xml."
EXIT=1
return
fi
if [ "$ORMI_PASSWORD" = "" ]
then
echo ""
echo "Error: You must specify the ORMI password using the -password switch."
EXIT=1
return
fi
echo "Shutdown OC4J instance..."
CMDARGS="$ORMI_URL:$ORMI_PORT $ORMI_USER $ORMI_PASSWORD -shutdown"
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS
}
version()
{
echo "Getting the version of OC4J instance..."
CMDARGS="-version"
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
}
help()
{
cat <<EOF
Usage: oc4j [Options]
Options:
-start : start OC4J
-shutdown -port <ORMI port> -password <password>
: stop OC4J
-version : display the version
-help : display this message
EOF
}
########################################
### Start main function section ###
########################################
check_oc4j
if [ ! "$EXIT" -eq 0 ]
then
echo_check_msg "$check_msg"
else
if [ $# -eq 0 ]
then
help
else
# the first argument
CMDARG="$1"
# decrement number of arguments
shift
# get the rest of the command line
case $CMDARG in
-start)
start_oc4j "$@"
;;
-shutdown)
shutdown_oc4j "$@"
;;
-version)
if [ $# -gt 0 ] ; then
echo ""
echo "Error: The option \"$CMDARG\" does not take any argument."
EXIT=1
else
version
fi
;;
-help)
help
EXIT=0
;;
*)
echo ""
echo "Error: The option \"$CMDARG\" was not recognized."
EXIT=1
;;
esac
if [ $EXIT -eq 1 ]
then
help
fi
fi
fi
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old March 7th, 2008, 18:20
chrisplough's Avatar
Site Moderator
 
Join Date: Jun 2006
Location: West Chester, PA
Posts: 790
Blog Entries: 7
Thanks: 51
Thanked 194 Times in 119 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10
chrisplough has a spectacular aura aboutchrisplough has a spectacular aura aboutchrisplough has a spectacular aura about
Send a message via AIM to chrisplough
Re: Oracle Application Server running out of memory

Thanks for the details - based on the script, the memory values have been edited correctly (now 2048MB). Just make sure that you've got enough memory on this server. If running web and app on the same box, you should have 8GB of physical memory.

--Chris
__________________
Chris Plough
MavenWire

www.MavenWire.com
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old March 10th, 2008, 14:45
Junior Member
 
Join Date: Jul 2007
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0
JayVal is on a distinguished road
Re: Oracle Application Server running out of memory

Thanks again.

While working on this, I found something. The App and Web Servers run on UNIX environment. I find that the $OTM_HOME is not created. I am wondering if the memory setting change done to oc4j.sh will be picked up by the Oracle application server.

This is just an observation. Correct me if I am wrong.

Thanks,
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old March 10th, 2008, 15:09
chrisplough's Avatar
Site Moderator
 
Join Date: Jun 2006
Location: West Chester, PA
Posts: 790
Blog Entries: 7
Thanks: 51
Thanked 194 Times in 119 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10
chrisplough has a spectacular aura aboutchrisplough has a spectacular aura aboutchrisplough has a spectacular aura about
Send a message via AIM to chrisplough
Re: Oracle Application Server running out of memory

You're welcome.

I use the notation $OTM_HOME, simply to reference the OTM install directory. This variable doesn't need to be defined within your environment for OTM to startup. So, you should be fine.

After restarting OTM, you can verify that the memory values have increased by watching the $OTM_HOME/logs/oas/console.log file. The Java GC statements should look like this:
Code:
[Full GC 1024447K->815040K(2048896K), 3.4803020 secs]
Where 2048896K is the amount of memory assigned to OAS.

--Chris
__________________
Chris Plough
MavenWire

www.MavenWire.com
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply



Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Need website for downloading OTM V5.0 application RemyaNair Installation and Upgrades 1 March 3rd, 2008 11:41
Application Server out of Memory - glog.webserver.mediator.MediatorDiagServlet james_foran Performance, Scalability and HA 2 February 29th, 2008 00:09
Mapping of application with database gokul Database Administration 4 November 14th, 2007 16:01
[SOLVED] OTM 5.5: CU3 running on Linux AS 3 Upd 4 with JRockit 27.3.1 ianlo Performance, Scalability and HA 1 August 30th, 2007 18:12
[SOLVED] Installing Oracle DB Server - Resource chrisplough Database Administration 0 July 24th, 2006 12:23



All times are GMT. The time now is 22:09.
Copyright © 2008, Open Book Solutions LLC. All rights reserved.

Sponsored by MavenWire - MavenWire.com


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37