[sknmail@rediffmail.com]

Hi All,

We recently migrated the 5.0 test environment to 5.5 (CU3+RU#2). Well after this migration we performed a bulk plan volume testing wherein using the same planning parameter where around 1950 orders were bulk planned with around 72 Bulk plans of 25 orders each..

Well the total time taken to plan these 1950 orders was around 4hours, while same test when performed in 5.0 environment.. before migration some time earlier.. we could achieve this planning in around 2 hours..

I understand that there are many configuration on the application like agents.. but still we have the same agents we had on 5.0, but on 5.5 there are many planning params or properties that has got added.. Now does anybody from their past experience has encountered on switching off any plan param or properties drastically increased the bulk plan performance..

Would be great if any body can give some pointer on this topic...

Regards,
Suresh

[Shells]

Hey There Suresh. When we went from 4.5 to 5.5 we found quite a degredation in our bulk plan performance as well. We didn't so much change properties or planning parameters to resolve, but other items and were able to knock the time down quite a bit. Below is our setup for this particular customer.

Our Scenerio in this case is 1000-1200 orders required to build in under an hour. I'm not sure what your criteria in planning is, but ours is based on 4 source plants with any possible destination based on a little over 40,000 rates. Based on the timeframe and criteria above, we have done the following:

We opened 4 bulk planning threads in the event diag
We created planning groups based on source loc - so 4 total. Each group contains around 250-300 orders.
We found that having 1 usa to usa tl/ltl/parcel itinerary itinerary was really hurting us from a performance standpoint. Instead we inactivated that itinerary and created 4 itineraries - each from a particular source to usa. This cut down on calculation time.
Other constraints on the itinaries which also helped, but not required were items such as - a min dropoff weight (we put 2000 lbs), max stops (we put 6) and a max distance between deliveries (we put 150 miles). Constrainting itineraries helps cut down on scenerios analyzed during the building process.
We also put a max weight on our ltl rate offerings (20,000 lbs) and a (20,001 lbs)min weight on our TL offerings.
Logging also attributes to slowing down the bulk plans so we minimized logging and make sure user/adhoc logs are only used when actually troubleshooting issues and not just randomly running for the sake of being on.
The final step we took was we have a FULL COMPUTE Analyze scheduled to run on the db every weekend. This is different that the built in Oracle DBMS tool which just takes samples. Oracle has suggested in the past to do FULL COMPUTE Analyes with the OTM application. We did see a difference when running our plans.
When we first went to 5.5, the avg bulk plan was taking between 35-55 minutes. After the tweaks above, we were able to cut it down to 10-15 minutes allowing us to reach our goal in running the 1200 orders in under an hour.

Anyway - hope you are able to get your plans running smooth

[sknmail@rediffmail.com]

Hi Shells,

Thanks very much for the information you provided. Well we were doing this on our migrated 5.5 test environment, however we could find that around 500 orders were getting planned in around 1 hour time period. Later on our system admin associates found that on the Database server that CPU hyper threading was not enabled. After they enabled hyperthreading we planned the same 500 orders in 28 minutes time period.
I should look into the other points you have mentioned and let me confirm on that too.

Thanks very much.
Regards,
Suresh

[ianlo]

Originally Posted by sknmail@rediffmail.com

Hi Ian,

Good to know about the production migration from 5.0 to 5.5. I would interested in knowing if you have performed any volume testing on migrated 5.5 (probably Test or Dev environment) for performing bulkplan in comparison to the performance you had in 5.0 environment. Can you compare directly the performance we had on 5.0 to 5.5 after migration.

Also I would like to know what was the weblogic (I hope your app is weblogic and not wepshere or OAS) memory configured in weblogic.conf and for tomcat in tomcat.conf in 5.0 and now in 5.5 setup after migration.

In our case a bulk plan of 25 Orders is taking around 5-6 min on an average, while 77 bulkplans of 25 orders each took around 4 hours to plan. We have the weblogic and tomcat memory configured to 1536 and 1500 respectively. We have App having 4 CPUs around 8G RAM and DB 2 CPUs and 8G RAM. Do you know any planning parameter or properties which can eat up the planning performance after migration which can be turned off if its related to new features 5.5? I understand there are many things needs to be considered on the configuration on OTM and setup when we decided on performance.

Appreciate your information.

Thanks and Regards,
Suresh

Hi Suresh,

Sorry for this late reply. But yes we did extensive performance and load testing prior to migration to 5.5. We did a direct comparision because we were able to re-create the database and reload the orders again each time. This provided us with a reference environment to ensure that we had no performance issues.

However, during go-live, even after *extensive* performance testing we still hit performance issues. Interestingly the issue was not on the app or web. Rather it was the Oracle DB.

We realised that any DB migration especially one with a large change such as 9i to 10g, you cannot guarantee the same performance.

Honestly, I don't know whats going on inside the Oracle DB but it did not take us long to realise that after the production migration, the bottleneck was the DB as we had performance monitoring on both the app, web and DB. The Oracle was maxing out all our CPU (we have 4 CPU, IBM pSeries 1.6 GHZ Power5, 8 GB RAM)

We ended up having to spend about 1 week re-tuning the Oracle DB parameters as well as our queries.

The important thing is that you should be runninig the gather_table_stats.sql scripts provieded by Oracle (in the glog/oracle/script8 directory) at least once weekly. With the updated stats over a few weeks, performance improved alot.

I know you suspect that it is the app server but I think the app server being *static* code, the only variant here will be the DB. Hence, my bet is on the DB.

What are your thoughts? Have you had success yet?

Thanks and Best Regards,
Ian

[cspadola]

Can someone tell me if there is a disadvantage to increasing the number of threads that the batch event uses in the Event Diagnosis? We have a scenario where most of our bulk planning (90% or so) all occurs at the same time each day (starting at 16:00). Currently, the number of threads is set to the default of 2. When a user runs a bulk plan earlier in the day - even with just 30 orders - it takes 1-2 mins to complete. This has become the accepted performance based on our itineraries and characteristics of our orders.

However, when 16:00 rolls around each day and almost all the users run their respective bulk plans simultaneously, there is a queue built up. Bulk planning for each user increases to 10-13 mins each.

Can we safely increase this thread to 4 or even more? Are there any planning parameters I can look at to make this perform better?

We are running OTMv55-CU03 on Linux servers.

[chrisplough]

Hello!

Yes, it is okay to increase this parameter, but I'd be careful about going larger than 4, as Bulk Plans are very intensive (both on app and DB) and can induce a lot of contention. I'd bump this up by 1 at a time, slowly over a few days, until you stop seeing performance benefits.

Thanks!
--Chris