| | Performance, Scalability and HA Optimizing the performance of OTM / G-Log, configuring Scalability (SCA) and maintaining High Availability. |  | 
March 7th, 2008, 15:07
| | Member | | Join Date: Jul 2007
Posts: 35
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0 | | | Oracle Application Server running out of memory We have this OAS running Out of Memory issue in Production on a daily basis and we had to restart the Server everytime. OTM becomes slow to the point that the application was not responding. A check across the system showed the OAS JVM was using 100% of 1 CPU And the OAS heap was maxed out on memory: [Full GC 1024447K->1024447K(1024448K), 4.0970930 secs] [Full GC 1024447K->1024447K(1024448K), 4.0963020 secs] [Full GC 1024447K->1024447K(1024448K), 4.0915010 secs] Garbage Collection (GC) was taking place, but little or no un-used memory was being Unloaded. Scanning the OAS console.log, a pattern was seen where memory began rising. 10:47 AM – OK at this point [GC 665769K->456278K(1023424K), 0.0744990 secs] As we approach 11:21 AM, OAS starts using more total memory and releases less memory via GC. The GC times increase also indicating more work is needed to find memory to Unload. 11:21 AM [GC 875287K->672253K(1024448K), 3.7772500 secs] By 11:26 AM, OAS has begun maxing the heap before GC runs, and OAS begins performing FULL GC”s. [Full GC 1024447K->815040K(1024448K), 3.4803020 secs] At this point, GC is still Unloading memory and the application is still responding. As time goes on, OAS is able to Unload less and less memory. As of 11:38 AM, OAS is getting little or no memory back via GC. At this point, the application has already stopped responding. Please help. | 
March 7th, 2008, 15:46
|  | Site Moderator | | Join Date: Jun 2006 Location: West Chester, PA
Posts: 847
Thanks: 53
Thanked 205 Times in 124 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10 | | | Re: Oracle Application Server running out of memory JayVal,
The biggest issue is that you don't have enough memory allocated to the OTM App server. Allocating 1GB is the bare minimum for OTM and doesn't leave enough room for memory-intensive OTM processes.
If you're running on Win32, then you'll be limited to 1.2 or 1.4GB, but if you're running on any other platform, you should bump up to 2GB. This is done within your $OTM_HOME/oas/bin/oc4j.sh script.
Keep in mind that you'll need enough physical memory on your server to accommodate a heap of this size. If you're only running OAS on this server, and the heap is 2GB, then you should have 4GB of physical memory to allow room for the heap, Java native memory and OS. If you're running more, then you'll need more memory.
Hope this helps!
--Chris | 
March 7th, 2008, 18:33
| | Member | | Join Date: Jul 2007
Posts: 35
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0 | | | Re: Oracle Application Server running out of memory Thanks Chris.
Here is some information I got. Also I am giving here the contents of the oc4j.sh. I am not an Admin guy, so I may not be able to put the pieces together.
1) The Oracle App server and Apache Web Server runs on the same box.
2) The OS is Linux 32bit
3) The System is 64bit.
**************************************
oc4j.sh contents
**************************************
#!/bin/sh
#
# oc4j - shell for invoking OC4J basic operations.
#
# Usage: oc4j [Options]
#
# Options:
# -start : start OC4J
# -shutdown -port <ORMI port> -password <password>
# : stop OC4J
# -version : display the version
# -help : display this message
#
# Copyright (c) 2004, 2005, Oracle. All rights reserved.
#
################################################## #######
########## START CONFIGURATION SECTION ##################
################################################## #######
GLOG_HOME=/u02/otmPROD
JAVA_HOME=/u02/otmPROD/jdk
ORACLE_HOME=$GLOG_HOME/oas
J2EE_HOME=$ORACLE_HOME/j2ee/home
VERBOSE=on MEM_ARGS="-XX:PermSize=256m -XX:MaxPermSize=256m -Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -verbose:gc"
GC3_OPTIONS="-Duser.home=/u02/otmPROD/glog/config -Dglog.properties=glog.properties"
OC4J_JVM_ARGS="$MEM_ARGS $GC3_OPTIONS"
echo $JVMARGS
# Set DASH variables
DASH_HOME=$GLOG_HOME/dash; export DASH_HOME
XPRESS=$DASH_HOME/bin; export XPRESS
LD_LIBRARY_PATH=$DASH_HOME/lib:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH
PATH=$DASH_HOME/lib:$PATH; export PATH
#Any persistent arguments to specify at the JVM level can be set here
#By default this will be read from the operating system environment
if [ "$OC4J_JVM_ARGS" ]
then
JVMARGS=$OC4J_JVM_ARGS
else
JVMARGS=
fi
CMDARGS=-userThreads
if [ "$VERBOSE" ]
then
VERBOSE=$VERBOSE
else
VERBOSE=off
fi
ORMI_URL=ormi://localhost
ORMI_USER=oc4jadmin
OC4J_JAR=$J2EE_HOME/oc4j.jar
ADMIN_JAR=$J2EE_HOME/admin.jar
SERVER_XML=$J2EE_HOME/config/server.xml
################################################## #######
########## END CONFIGURATION SECTION ##################
################################################## #######
check_oc4j()
{
EXIT=0
if [ "$JAVA_HOME" = "" ]
then
echo "Error: JAVA_HOME environment variable is not defined."
check_msg="correct JAVA_HOME environment variable."
EXIT=2
elif [ ! -x $JAVA_HOME/bin/java ]
then
echo "Error: Can not find java executable in $JAVA_HOME/bin."
check_msg="correct java executable."
EXIT=2
elif [ "$ORACLE_HOME" = "" ]
then
echo "Error: The ORACLE_HOME environment variable must be set before executing this script. Set this to the directory into which you unzipped oc4j_extended.zip."
check_msg="correct ORACLE_HOME environment variable."
EXIT=2
elif [ ! -r "$OC4J_JAR" ]
then
check_msg="readable $OC4J_JAR."
EXIT=2
elif [ ! -w "$SERVER_XML" ]
then
check_msg="writable $SERVER_XML."
EXIT=2
elif [ ! -r "$ADMIN_JAR" ]
then
check_msg="readable $ADMIN_JAR."
EXIT=2
fi
}
echo_check_msg()
{
echo "Error: The command can not be run without $1"
}
start_oc4j()
{
echo "Starting OC4J from $J2EE_HOME ..."
CMDARGS="-userThreads -config $SERVER_XML"
while [ $# -ge 1 ]
do
case $1 in
*)
echo ""
echo "Error: The optoin \"$1\" was not recognized."
EXIT=1
return
;;
esac
done
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
}
shutdown_oc4j()
{
while [ $# -ge 1 ]
do
case $1 in
-port)
shift
if [ "$1" = "" ]
then
echo ""
echo "Error: You must specify the ORMI port value."
EXIT=1
return
else
ORMI_PORT=$1
shift
fi
;;
-password)
shift
if [ "$1" = "" ]
then
echo ""
echo "Error: You must specify the password value."
EXIT=1
return
else
ORMI_PASSWORD=$1
shift
fi
;;
*)
echo ""
echo "Error: The option \"$1\" was not recognized."
EXIT=1
return
;;
esac
done
if [ "$ORMI_PORT" = "" ]
then
echo ""
echo "Error: You must specify the ORMI port using the -port switch."
echo " The port value can be found in $J2EE_HOME/config/rmi.xml."
EXIT=1
return
fi
if [ "$ORMI_PASSWORD" = "" ]
then
echo ""
echo "Error: You must specify the ORMI password using the -password switch."
EXIT=1
return
fi
echo "Shutdown OC4J instance..."
CMDARGS="$ORMI_URL:$ORMI_PORT $ORMI_USER $ORMI_PASSWORD -shutdown"
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS
}
version()
{
echo "Getting the version of OC4J instance..."
CMDARGS="-version"
if [ "$VERBOSE" = "on" ]
then
echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
fi
$JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
}
help()
{
cat <<EOF
Usage: oc4j [Options]
Options:
-start : start OC4J
-shutdown -port <ORMI port> -password <password>
: stop OC4J
-version : display the version
-help : display this message
EOF
}
########################################
### Start main function section ###
########################################
check_oc4j
if [ ! "$EXIT" -eq 0 ]
then
echo_check_msg "$check_msg"
else
if [ $# -eq 0 ]
then
help
else
# the first argument
CMDARG="$1"
# decrement number of arguments
shift
# get the rest of the command line
case $CMDARG in
-start)
start_oc4j "$@"
;;
-shutdown)
shutdown_oc4j "$@"
;;
-version)
if [ $# -gt 0 ] ; then
echo ""
echo "Error: The option \"$CMDARG\" does not take any argument."
EXIT=1
else
version
fi
;;
-help)
help
EXIT=0
;;
*)
echo ""
echo "Error: The option \"$CMDARG\" was not recognized."
EXIT=1
;;
esac
if [ $EXIT -eq 1 ]
then
help
fi
fi
fi | 
March 7th, 2008, 19:20
|  | Site Moderator | | Join Date: Jun 2006 Location: West Chester, PA
Posts: 847
Thanks: 53
Thanked 205 Times in 124 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10 | | | Re: Oracle Application Server running out of memory Thanks for the details - based on the script, the memory values have been edited correctly (now 2048MB). Just make sure that you've got enough memory on this server. If running web and app on the same box, you should have 8GB of physical memory.
--Chris | 
March 10th, 2008, 15:45
| | Member | | Join Date: Jul 2007
Posts: 35
Thanks: 0
Thanked 0 Times in 0 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 0 | | | Re: Oracle Application Server running out of memory Thanks again.
While working on this, I found something. The App and Web Servers run on UNIX environment. I find that the $OTM_HOME is not created. I am wondering if the memory setting change done to oc4j.sh will be picked up by the Oracle application server.
This is just an observation. Correct me if I am wrong.
Thanks, | 
March 10th, 2008, 16:09
|  | Site Moderator | | Join Date: Jun 2006 Location: West Chester, PA
Posts: 847
Thanks: 53
Thanked 205 Times in 124 Posts
Groans: 0
Groaned at 0 Times in 0 Posts
Rep Power: 10 | | | Re: Oracle Application Server running out of memory You're welcome.
I use the notation $OTM_HOME, simply to reference the OTM install directory. This variable doesn't need to be defined within your environment for OTM to startup. So, you should be fine.
After restarting OTM, you can verify that the memory values have increased by watching the $OTM_HOME/logs/oas/console.log file. The Java GC statements should look like this: Code: [Full GC 1024447K->815040K(2048896K), 3.4803020 secs] Wh ere 2048896K is the amount of memory assigned to OAS. --Chris |  | | Thread Tools | | | | Display Modes | Linear Mode |
Posting Rules
| You may not post new threads You may not post replies You may not post attachments You may not edit your posts HTML code is Off | | | |