“No manager available?” Say what, EBS?

Here's a quick note for those of you running Oracle E-Business Suite 11.5.10.2 in a non-PCP RAC environment. It may be an edge case, but perhaps it'll help someone else out there.

I'd received a report from a member of our development team that certain transaction managers in our E-Business Suite 11.5.10.2 test system were "broken"...but only intermittently. Don't you just love that? Actions in the Forms interface that would submit a request to a transaction manager as 'immediate' requests would return the message, "No concurrent manager is defined to process this request." This problem could persist for most of a day, only affect one developer, disappear for several days, then return and plague the entire team for a week. During that time, other concurrent managers were functioning as expected, and there weren't any obvious performance problems causing concurrent managers to become unresponsive. When tracing a Forms session shown to trigger the error, we saw the following query appear in the trace file right before the "No manager defined" error was thrown:

SELECT /*+ ORDERED USE_NL (fa fcp fr fcpp fcq) INDEX (fcq,FND_CONCURRENT_QUEUES_N1)
INDEX (fcpp,FND_CONC_PROCESSOR_PROGRAMS_U2) */ 
FCQ.CONCURRENT_QUEUE_ID 
FROM FND_APPLICATION FA, FND_CONCURRENT_PROGRAMS FCP, FND_CONC_PROCESSOR_PROGRAMS FCPP, FND_RESPONSIBILITY FR, FND_CONCURRENT_QUEUES FCQ, FND_CONCURRENT_PROCESSES FCPR 
WHERE FCQ.PROCESSOR_APPLICATION_ID = FCPP.PROCESSOR_APPLICATION_ID 
AND FCQ.CONCURRENT_PROCESSOR_ID = FCPP.CONCURRENT_PROCESSOR_ID 
AND FCPP.CONCURRENT_PROGRAM_ID = FCP.CONCURRENT_PROGRAM_ID 
AND FCPP.PROGRAM_APPLICATION_ID = FCP.APPLICATION_ID 
AND FCP.APPLICATION_ID = FA.APPLICATION_ID 
AND FA.APPLICATION_SHORT_NAME = 'xx'
AND FCP.CONCURRENT_PROGRAM_NAME = 'yyyyy' 
AND FR.RESPONSIBILITY_ID = nnnnn 
AND FR.APPLICATION_ID = nnn 
AND FR.DATA_GROUP_ID = FCQ.DATA_GROUP_ID 
AND FCQ.MANAGER_TYPE = '3' 
AND FCPR.CONCURRENT_QUEUE_ID = FCQ.CONCURRENT_QUEUE_ID
AND FCPR.QUEUE_APPLICATION_ID = FCQ.APPLICATION_ID 
AND FCPR.PROCESS_STATUS_CODE = 'A' 
AND FCPR.INSTANCE_NUMBER = USERENV('instance') ORDER BY DBMS_RANDOM.RANDOM

This query would return zero rows, and cause EBS to report "No manager available," even though we'd verified before running the test that the required transaction manager was running and successfully processing requests from other sources.

But wait. See that bit right there, where it checks USERENV('instance')? Yeah, it made me think, too.

So what was going on?

The previous query shows that the code submitting the concurrent request was not just checking for a running transaction manager, but for a manager that was running on the same database instance as the user's current session. Since we were running in a RAC environment, it was entirely possible for the transaction manager process and the user's Forms session to be connected to different database nodes. If the CM and Forms sessions were both connected to RAC node 1, or both to RAC node 2, everything worked fine. But if the CM process were running on node 1 and the Forms session connected to node 2, or vice versa, things blew up.

Why did the transaction managers care which RAC node we were using when we submitted a request? The "old way" of running EBS transaction managers in a RAC environment required the use of DBMS_PIPE, which meant that there needed to be a transaction manager assigned to each RAC node. This is the default behavior for EBS 11i. In Release 12, the default is to use Advanced Queueing (AQ) for transaction managers, so the need for a 1:1 mapping between managers and RAC nodes is eliminated. Up-to-date versions of the ATG Family Pack also allow the use of queues for transaction managers in 11i. This behavior is controlled by setting the EBS system profile option "Concurrent: TM Transport Type," which takes the value PIPE or QUEUE.

This is not new information. The instructions for setting the "Concurrent: TM Transport Type" profile option in an EBS RAC environment are very clearly stated...but only in the context of configuring Parallel Concurrent Processing (PCP). Since we were running on a single applications tier at the time, it was decided not to configure PCP until we had deployed a second apps node. We therefore did not address the TM Transport Type profile option. Once we changed the option from PIPE to QUEUE in our single-app-node, 2-db-node environment, the problems reported by the developers were resolved.

So there you go. "Concurrent: TM Transport Type=QUEUE" -- it's not just for PCP anymore. Hey, that would've made a catchy post title!

References

Note 458453.1: What is the difference between PIPE and QUEUE for profile Concurrent:TM Transport Type ? (pay particular attention to the required init.ora adjustments; they vary depending on RDBMS release)
Note 240818.1: Concurrent Processing: Transaction Manager Setup and Configuration Requirement in an 11i RAC Environment
Note 1057802.1: Best Practices for Performance for Concurrent Managers in E-Business Suite
Note 823586.1: Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 11i (or whichever note is appropriate for your database release)

2 Comments

  1. Paul C
    Posted 25 February 2011 at 7:59 | Permalink

    Thanks John. You saved the day again! Now what about issues with RCVTP internal errors due to unable to write to the log file? :D

  2. Posted 25 February 2011 at 8:17 | Permalink

    Hi Paul,

    Heh. Well, that problem was more embarrassing than interesting. While I'm not averse to publicly recording when I do something dumb (this blog has a WowThatWasDumb tag, after all), I prefer that it not be for something as common as badly-set file permissions. ;-)

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*