Here's a quick note for those of you running Oracle E-Business Suite 18.104.22.168 in a non-PCP RAC environment. It may be an edge case, but perhaps it'll help someone else out there.
I'd received a report from a member of our development team that certain transaction managers in our E-Business Suite 22.214.171.124 test system were "broken"...but only intermittently. Don't you just love that? Actions in the Forms interface that would submit a request to a transaction manager as 'immediate' requests would return the message, "No concurrent manager is defined to process this request." This problem could persist for most of a day, only affect one developer, disappear for several days, then return and plague the entire team for a week. During that time, other concurrent managers were functioning as expected, and there weren't any obvious performance problems causing concurrent managers to become unresponsive. When tracing a Forms session shown to trigger the error, we saw the following query appear in the trace file right before the "No manager defined" error was thrown:
SELECT /*+ ORDERED USE_NL (fa fcp fr fcpp fcq) INDEX (fcq,FND_CONCURRENT_QUEUES_N1) INDEX (fcpp,FND_CONC_PROCESSOR_PROGRAMS_U2) */ FCQ.CONCURRENT_QUEUE_ID FROM FND_APPLICATION FA, FND_CONCURRENT_PROGRAMS FCP, FND_CONC_PROCESSOR_PROGRAMS FCPP, FND_RESPONSIBILITY FR, FND_CONCURRENT_QUEUES FCQ, FND_CONCURRENT_PROCESSES FCPR WHERE FCQ.PROCESSOR_APPLICATION_ID = FCPP.PROCESSOR_APPLICATION_ID AND FCQ.CONCURRENT_PROCESSOR_ID = FCPP.CONCURRENT_PROCESSOR_ID AND FCPP.CONCURRENT_PROGRAM_ID = FCP.CONCURRENT_PROGRAM_ID AND FCPP.PROGRAM_APPLICATION_ID = FCP.APPLICATION_ID AND FCP.APPLICATION_ID = FA.APPLICATION_ID AND FA.APPLICATION_SHORT_NAME = 'xx' AND FCP.CONCURRENT_PROGRAM_NAME = 'yyyyy' AND FR.RESPONSIBILITY_ID = nnnnn AND FR.APPLICATION_ID = nnn AND FR.DATA_GROUP_ID = FCQ.DATA_GROUP_ID AND FCQ.MANAGER_TYPE = '3' AND FCPR.CONCURRENT_QUEUE_ID = FCQ.CONCURRENT_QUEUE_ID AND FCPR.QUEUE_APPLICATION_ID = FCQ.APPLICATION_ID AND FCPR.PROCESS_STATUS_CODE = 'A' AND FCPR.INSTANCE_NUMBER = USERENV('instance') ORDER BY DBMS_RANDOM.RANDOM
This query would return zero rows, and cause EBS to report "No manager available," even though we'd verified before running the test that the required transaction manager was running and successfully processing requests from other sources.
But wait. See that bit right there, where it checks
USERENV('instance')? Yeah, it made me think, too.
So what was going on?
The previous query shows that the code submitting the concurrent request was not just checking for a running transaction manager, but for a manager that was running on the same database instance as the user's current session. Since we were running in a RAC environment, it was entirely possible for the transaction manager process and the user's Forms session to be connected to different database nodes. If the CM and Forms sessions were both connected to RAC node 1, or both to RAC node 2, everything worked fine. But if the CM process were running on node 1 and the Forms session connected to node 2, or vice versa, things blew up.
Why did the transaction managers care which RAC node we were using when we submitted a request? The "old way" of running EBS transaction managers in a RAC environment required the use of DBMS_PIPE, which meant that there needed to be a transaction manager assigned to each RAC node. This is the default behavior for EBS 11i. In Release 12, the default is to use Advanced Queueing (AQ) for transaction managers, so the need for a 1:1 mapping between managers and RAC nodes is eliminated. Up-to-date versions of the ATG Family Pack also allow the use of queues for transaction managers in 11i. This behavior is controlled by setting the EBS system profile option "Concurrent: TM Transport Type," which takes the value PIPE or QUEUE.
This is not new information. The instructions for setting the "Concurrent: TM Transport Type" profile option in an EBS RAC environment are very clearly stated...but only in the context of configuring Parallel Concurrent Processing (PCP). Since we were running on a single applications tier at the time, it was decided not to configure PCP until we had deployed a second apps node. We therefore did not address the TM Transport Type profile option. Once we changed the option from PIPE to QUEUE in our single-app-node, 2-db-node environment, the problems reported by the developers were resolved.
So there you go. "Concurrent: TM Transport Type=QUEUE" -- it's not just for PCP anymore. Hey, that would've made a catchy post title!
Note 458453.1: What is the difference between PIPE and QUEUE for profile Concurrent:TM Transport Type ? (pay particular attention to the required init.ora adjustments; they vary depending on RDBMS release)
Note 240818.1: Concurrent Processing: Transaction Manager Setup and Configuration Requirement in an 11i RAC Environment
Note 1057802.1: Best Practices for Performance for Concurrent Managers in E-Business Suite
Note 823586.1: Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 11i (or whichever note is appropriate for your database release)