Manoj,
Sorry to hear that you're having "fun" with OTM and WebSphere -- especially on AIX. Good luck!
The limit for WebSphere should be the same as for WebLogic - and is limited by the IBM JDK. Regardless of the platform, you'll have to use the IBM JDK, rather than JRockit. This will cause the following (rough) memory limits:
- AIX = ~3000MB
- Linux = ~2000MB
The biggest issue you're likely to hit is that the IBM JDK can be unstable with OTM, because of the large amount of native (outside the java heap) memory that we require. This usually causes the IBM JDK to crash - sometimes only once a month, sometimes several times a week - depending on utilization.
--Chris