This post takes its inspiration from a recent OTN forum thread about problems with E-Business Suite R12 web services, but it's not an attempt to resolve the root cause of that thread -- lots of other people had already chimed in to help by the time I got there. One thing that was going on in the thread as part of the troubleshooting, however, was an attempt to start web services using the generic 10gAS opmnctl script, instead of the R12-specific adopmnctl.sh script. This might not seem like such a bad idea on the surface, since adopmnctl.sh is "just" a wrapper script that calls opmnctl, and the latter script is very familiar to admins who have worked with Oracle Application Server components. But take a look at the difference in the behavior of the two scripts, even when R12 web services are down:
[applvis@londo ~]$ adopmnctl.sh status
You are running adopmnctl.sh version 120.6
Checking status of OPMN managed processes...
Unable to connect to opmn.
Opmn may not be up.
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /u01/ebs/R12VIS/inst/apps/R12VIS_londo/logs/appl/admin/log/adopmnctl.txt for more information ...
[applvis@londo ~]$ opmnctl status
Error reading opmn.xml
Message:{/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2/opmn/conf/opmn.xml (No such file or directory)}
Filename:{/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2/opmn/conf/opmn.xml}
FileContent:{}
(Disclosure: I added opmnctl to my PATH for ease of reading/typing; it's not there by default in the applmgr environment)
What's causing the difference? The opmnctl script uses the value of the ORACLE_CONFIG_HOME environment variable to set the ONS (Oracle Notification Service) home, and if that environment variable isn't set correctly, then opmnctl will look in the wrong place for the opmn.xml file, resulting in the error displayed above. The default value of ORACLE_CONFIG_HOME in the apps tier environment is the 10.1.2 configuration directory, but in order to function properly, opmnctl needs to see the 10.1.3 ORACLE_CONFIG_HOME. Here's an illustration:
First, the default value of ORACLE_CONFIG_HOME:
[applvis@londo ~]$ echo $ORACLE_CONFIG_HOME /u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2
Then, in the opmnctl script, see that the value of oracle.ons.oraclehome is set based on the value of ORACLE_CONFIG_HOME. If you call it directly from the apps environment, that will make it point to the 10.1.2 config directory, as shown by these lines in the opmnctl script (I've highlighted the relevant parts):
if [ -n "$ORACLE_CONFIG_HOME" ] then MODIFIED_ORA_CONFIG_HOME="`echo $ORACLE_CONFIG_HOME | tr -d "[:space:]"`" else MODIFIED_ORA_CONFIG_HOME="" fi if [ -z "$MODIFIED_ORA_CONFIG_HOME" ] then $JAVA -classpath $ARGUS_JAR:$OPTIC_JAR \ -Doracle.ons.oraclehome=$ORACLE_HOME \ oracle.ias.opmn.argus.Argus -help else $JAVA -classpath $ARGUS_JAR:$OPTIC_JAR \ -Doracle.ons.oraclehome=$MODIFIED_ORA_CONFIG_HOME \ oracle.ias.opmn.argus.Argus -help fi
If you run adopmnctl.sh instead, ORACLE_CONFIG_HOME gets set to the 10.1.3 config directory, as it should be, and that is the value that gets passed to the opmnctl script:
[applvis@londo ~]$ grep ORAENV_FILE `which adopmnctl.sh` # Set ORAENV_FILE to 10.1.3 Oracle Home Environment File ORAENV_FILE="/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3/R12VIS_londo.env" # Check the existence of ORAENV_FILE if [ ! -f "$ORAENV_FILE" ]; . $ORAENV_FILE [applvis@londo ~]$ grep ORACLE_CONFIG_HOME= /u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3/R12VIS_londo.env ORACLE_CONFIG_HOME="/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3"
Please note that the takeaway from all of this is NOT "manually change the value of ORACLE_CONFIG_HOME to run opmnctl." The takeaway is that running adopmnctl.sh is not the equivalent to running opmnctl, and that you can expect strange behavior if you try start R12 web services without using adopmnctl.sh. Whenever possible, use the tools provided as they're intended to be used.
