1. A bazillion browser tabs
2. An SR or two
3. Piles of patch READMEs
4. Resolving prerequisites until your eyes cross
5. Memorizing individual patch numbers, whether you like it or not
6. The reward for all the hard research work: "15 jobs running, 235 ready to run, and 50682 waiting."
7. Sporadic ORA-12545 errors in the adworker logs

Yeah, okay, #7 caught me a bit by surprise, too. The really tricky bit was that the worker wouldn't usually be marked as "Failed" when it received an ORA-12545. Instead, it would stay in a "Running" state, and eventually the entire patch session would go idle, until I manually restarted the errored worker. There is no shortage of troubleshooting notes for ORA-12545, but the essential condition is that the client cannot resolve the hostname of the database server. Considering how sporadic the errors were, this was more than a bit strange. So we tested client connectivity using the various permutations of the EBS database hostnames, verified that there wasn't any weirdness in the RAC listener setup, and still the problems persisted.

At a high level, it's always disturbing to have sporadic network wonkiness. More practically, it wasn't very much fun to contemplate writing production patching instructions that read, "Babysit the adpatch session very closely. Manually restart any workers that fail with ORA-12545." (I could also note that it's not very much fun to have a multi-hour patch test run grind to a halt 10 minutes after leaving the office for the day, but that would be whining, so I won't do that). I was just starting to consider running SQLNet traces to try to capture more information about this 1-in-1000 chance error condition, when one of the client's DBAs mentioned that the test systems and the DNS server were in separate data centers, connected over a WAN.

Rather than suggesting that we march over to the network admins and claim that DNS requests were being dropped on the floor (what a great way to make new friends!), and lacking the time to put together an easily-reproducible test to back up such a claim, I recommended adding the hostnames and IPs of the database servers to the apps tier's /etc/hosts file. All subsequent patch runs were free of ORA-12545 errors, which left me free to concentrate on other, more interesting errors. I'm not a big fan of solution-by-assumption, but in this case I was able to carry the work forward, and the investigation into potential DNS problems can wait for a quieter time.

1. In the last week or so, I’ve been involved in a two discussions about the viability/supportability of Oracle E-Business Suite on VMware, one on OracleCommunity.net, and the other on LinkedIn.
2. The IOUG recently sent out a VMware-sponsored email entitled, “Why Oracle DBAs should care about virtualization.”
3. I am an unreformed, unrepentant nerd.

So, if you're up for some lazy Saturday evening musings about Oracle and VMware, keep reading. I’ll try not to ramble too much.

FUD or misinterpretation? The difference between “supported” and “certified”

The crux of the recent online discussions in which I participated was the age-old question: Will Oracle support customers running on VMware, or not? The confusion arises from a My Oracle Support note stating Oracle’s position on the topic: Support Position for Oracle Products Running on VMWare Virtualized Environments (Doc ID 249212.1). If read too quickly, or too conservatively, it’s possible to conclude that Oracle won’t provide full support for its products in a VMware environment. Coupled with the knowledge that Oracle offers its own virtualization product, Oracle VM, that it does fully certify and support, it’s easy to see where Oracle customers might think twice about considering VMware as a virtualization platform.

It’s important to remember, however, that when it comes to Oracle products, there’s a big difference between certification and support. Certification of Oracle software on specific hardware platforms, for example, is pretty much out of scope for Oracle Support, and VMware is, in effect, providing a virtual hardware platform upon which to run your systems. The My Oracle Support note referenced above actually spells out pretty clearly how Oracle will support you if you’re running on VMware. What the note effectively states is, “If the reported issue looks like a problem with our software, we will support you. If your issue looks like it’s related to VMware, we’re going to send you to VMware for resolution.” While it might seem like this is a less-than-usual level of support, it’s actually entirely fair. If your problem could be traced to the OS, after all, you could expect to be referred to the OS vendor. Similarly, if the conclusion were that you had hardware problems, you could expect to be sent to your hardware vendor.

Incidentally, this illustrates a core benefit of the Oracle-OEL/Solaris-Oracle-VM technology stack: if you have issues that require intervention from Oracle Support, whether they’re with software, hardware, or virtualization layer, there’s no concern that you’ll be left playing “vendor volleyball.” Your issues will be handled entirely by Oracle, and there won’t ever be any finger-pointing between the various product support teams, because everyone is living in peace and harmony. *ahem*

Bottom line: Currently, yes, you’ll be supported if you run your Oracle environment in VMware. Unless your problem turns out to be VMware-related, in which case you probably want VMware’s help anyway.

That’s great and all, but does it work, particularly in production?

DIsclosure time: I’ve had my doubts in the past about the viability of VMware for production E-Business Suite environments, as noted here in a comment on Steven Chan’s blog. I’m just one guy, though. VMware’s products have matured since then, and some of the problems I alluded to were not strictly VMware issues. Even back then (2005-2006), though, I was comfortable running Grid Control and some production Oracle Collaboration Suite application tiers in VMware. From what I’ve read, things have only gotten better with VMware vSphere.

It’s worth noting that VMware itself runs Oracle products, including E-Business Suite, in a VMware virtualization environment. They also have a list of customer references on their web site; if you elect to use VMware in your production Oracle environment, you won’t be in uncharted waters. If you’d like to read about the experiences of an Oracle customer running on VMware without the marketing filter of the vendor’s website, I recommend reading Jay Weinshenker’s blog. That’s twice I’ve linked to him in two blog posts, but I’m not digitally stalking him, I swear. He just writes interesting stuff. I’ll probably link to him one more time before I’m done here.

Why not just use Oracle VM?

I have far more experience with VMware than I do with Oracle VM, so it shouldn’t be surprising that I present as a VMware fanboy. Oracle has made a clear commitment to the virtualization space, however, and has put together an attractive package to support it. Oracle VM might be a good fit for you, particularly if you:

• are interested primarily in virtualizing Oracle products (note: Oracle VM can be used for non-Oracle virtualization needs as well)
• haven’t already made an investment in VMware in other parts of your IT infrastructure.
• prefer having just one vendor to flog when things go wrong.

References/Additional reading

As if I haven’t packed enough links into this post, here are a few more:

• Apparently Oracle and VMware were on others’ minds this weekend, because while I composed this post, I saw a tweet referencing a VMware article, “Yes, You Really *Can* Run Tier-1 Enterprise Applications on vSphere.
• Jay Weinshenker (told you I’d link him again) has written about reasons to run Oracle on VMware vSphere, and has discussed the Oracle license savings that can be achieved with a carefully laid-out VMware cluster.
• The Oracle VM page on OTN links to a lot of technical and marketing material about Oracle’s virtualization product.
• Steven Chan and his team have blogged several times about virtualization in the context of Oracle E-Business Suite, including posts on Support Policies for Virtualization Technologies and Oracle E-Business Suite and Using Oracle VM with Oracle E-Business Suite Virtualization Kit. The “Virtualization” listing of the blog’s Concepts section has more.

ORA-00942: table or view does not exist
ORA-06512: at "APPS.GL_SECURITY_PKG", line 1427

Line 1427 of GL_SECURITY_PKG starts a query against table owned by APPS to which the custom schema did not have access. While this section of the code had not changed between the original system and the upgraded one, one thing that had changed was the package header.

Here's the header from the original package:

SQL> select line
  2       , text
  3    from all_source
  4   where owner = 'APPS'
  5     and name = 'GL_SECURITY_PKG'
  6     and type = 'PACKAGE'
  7     and line <= 2
  8   order by line asc
  9  /

      LINE TEXT
---------- --------------------------------------------------------------------------------
	 1 PACKAGE gl_security_pkg AS
	 2 /* $Header: gluoases.pls 120.10.12000000.1 2007/01/16 22:31:44 appldev ship $ */

And on the upgraded system:

      LINE TEXT
---------- --------------------------------------------------------------------------------
	 1 PACKAGE gl_security_pkg AUTHID CURRENT_USER AS
	 2 /* $Header: gluoases.pls 120.10.12000000.1 2007/01/16 22:31:44 appldev ship $ */

Somehow, the GL_SECURITY_PKG was recompiled with invoker's rights (CURRENT_USER) instead of the default DEFINER rights. This explains why the custom schema was suddenly unable to access the APPS-owned tables without explicit permission to read the tables.

It may be that someone manually compiled this package with AUTHID CURRENT_USER, but it seems more likely that this was done during patching. The short-term "band-aid" fix here is to grant read permission to the custom schema on the tables referenced in APPS.GL_SECURITY_PKG. The longer-term solution, currently underway, is to engage with Oracle Support to determine how or why this package was recompiled with the AUTHID CURRENT_USER property, and learn if there is new guidance regarding the use of this package.

[applvis@londo ~]$ adopmnctl.sh status

You are running adopmnctl.sh version 120.6

Checking status of OPMN managed processes...
Unable to connect to opmn.
Opmn may not be up.

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /u01/ebs/R12VIS/inst/apps/R12VIS_londo/logs/appl/admin/log/adopmnctl.txt for more information ...  

[applvis@londo ~]$ opmnctl status
Error reading opmn.xml
Message:{/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2/opmn/conf/opmn.xml (No such file or directory)}
Filename:{/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2/opmn/conf/opmn.xml}
FileContent:{}

(Disclosure: I added opmnctl to my PATH for ease of reading/typing; it's not there by default in the applmgr environment)

What's causing the difference? The opmnctl script uses the value of the ORACLE_CONFIG_HOME environment variable to set the ONS (Oracle Notification Service) home, and if that environment variable isn't set correctly, then opmnctl will look in the wrong place for the opmn.xml file, resulting in the error displayed above. The default value of ORACLE_CONFIG_HOME in the apps tier environment is the 10.1.2 configuration directory, but in order to function properly, opmnctl needs to see the 10.1.3 ORACLE_CONFIG_HOME. Here's an illustration:

First, the default value of ORACLE_CONFIG_HOME:

[applvis@londo ~]$ echo $ORACLE_CONFIG_HOME
/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.2

Then, in the opmnctl script, see that the value of oracle.ons.oraclehome is set based on the value of ORACLE_CONFIG_HOME. If you call it directly from the apps environment, that will make it point to the 10.1.2 config directory, as shown by these lines in the opmnctl script (I've highlighted the relevant parts):

if [ -n "$ORACLE_CONFIG_HOME" ]
then
MODIFIED_ORA_CONFIG_HOME="`echo $ORACLE_CONFIG_HOME | tr -d "[:space:]"`"
else
MODIFIED_ORA_CONFIG_HOME=""
fi
if [ -z "$MODIFIED_ORA_CONFIG_HOME" ]
then
$JAVA -classpath $ARGUS_JAR:$OPTIC_JAR \
-Doracle.ons.oraclehome=$ORACLE_HOME \
oracle.ias.opmn.argus.Argus -help
else
$JAVA -classpath $ARGUS_JAR:$OPTIC_JAR \
-Doracle.ons.oraclehome=$MODIFIED_ORA_CONFIG_HOME \
oracle.ias.opmn.argus.Argus -help
fi

If you run adopmnctl.sh instead, ORACLE_CONFIG_HOME gets set to the 10.1.3 config directory, as it should be, and that is the value that gets passed to the opmnctl script:

[applvis@londo ~]$ grep ORAENV_FILE `which adopmnctl.sh`
# Set ORAENV_FILE to 10.1.3 Oracle Home Environment File
ORAENV_FILE="/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3/R12VIS_londo.env"
# Check the existence of ORAENV_FILE
if [ ! -f "$ORAENV_FILE" ];
. $ORAENV_FILE
[applvis@londo ~]$ grep ORACLE_CONFIG_HOME= /u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3/R12VIS_londo.env
ORACLE_CONFIG_HOME="/u01/ebs/R12VIS/inst/apps/R12VIS_londo/ora/10.1.3"

Please note that the takeaway from all of this is NOT "manually change the value of ORACLE_CONFIG_HOME to run opmnctl." The takeaway is that running adopmnctl.sh is not the equivalent to running opmnctl, and that you can expect strange behavior if you try start R12 web services without using adopmnctl.sh. Whenever possible, use the tools provided as they're intended to be used.

APPS@R12VIS(11.1.0.7)>set autotrace traceonly explain
APPS@R12VIS(11.1.0.7)>select count(*) from DUAL;

Execution Plan
----------------------------------------------------------
ERROR: an uncaught error in function display has happened; please contact Oracle support
       Please provide also a DMP file of the used plan table PLAN_TABLE
       ORA-00904: "OTHER_TAG": invalid identifier

Well, okay, I wasn't really querying the DUAL table, but it makes the example shorter. Since this was only happening when I was connected as the APPS user, I suspected a private PLAN_TABLE had been created for APPS, and sure enough:

APPS@r12vis(11.1.0.7)>select owner
  2  , object_name
  3  , object_type
  4  from dba_objects
  5  where object_name like 'PLAN_TABLE%';

OWNER		OBJECT_NAME		       OBJECT_TYPE
--------------- ------------------------------ --------------------
SYS		PLAN_TABLE$		       TABLE
PUBLIC		PLAN_TABLE		       SYNONYM
APPS		PLAN_TABLE		       TABLE

It's not too unusual to have a private PLAN_TABLE in a schema, even in a post-9i database. Oracle has definitely kept utlxplan.sql up to date in 11g. What was surprising to me was the structure of the table:

APPS@r12vis(11.1.0.7)>desc plan_table;
 Name								   Null?    Type
 ----------------------------------------------------------------- -------- --------------------------------------------
 STATEMENT_ID								    VARCHAR2(30)
 TIMESTAMP								    DATE
 REMARKS								    VARCHAR2(80)
 OPERATION								    VARCHAR2(30)
 OPTIONS								    VARCHAR2(30)
 OBJECT_NODE								    VARCHAR2(128)
 OBJECT_OWNER								    VARCHAR2(30)
 OBJECT_NAME								    VARCHAR2(30)
 OBJECT_INSTANCE							    NUMBER(38)
 OBJECT_TYPE								    VARCHAR2(30)
 SEARCH_COLUMNS 							    NUMBER(38)
 ID									    NUMBER(38)
 PARENT_ID								    NUMBER(38)
 POSITION								    NUMBER(38)
 OTHER									    LONG

There are a lot of fields missing from that table. A quick reference check reveals that this is not a 9i version of the PLAN_TABLE, not even an 8i version, but a vintage Oracle 7.3 PLAN_TABLE. This is actually pretty cool, since it implies that Oracle's been eating its own dog food with respect to upgrading its demo instance. The Vision database clearly has a long and storied history.

The solution to my original problem, of course, is to just drop the old plan table. If you'd rather keep a static PLAN_TABLE around for the APPS schema, it's still a good idea to drop it and recreate it with the 11g utlxplan.sql.

In nutshell, here is what the Oracle Applications patch merge utility, admrgpch, does:

1. Reads through the drivers of each patch to be merged
2. Checks the version of each file within the patch to determine if it should be included in the merged patch
3. Copies the appropriate files to the new merged patch destination directory
4. Generates a new patch driver that encompasses all of the actions for the individual patches. This driver will be u_merge_name.drv, where merge_name is supplied as an argument to admrgpch. If you elect not to specify a value for merge_name, the default is "merged," which is both boring and potentially confusing to you if you perform multiple merges over the lifetime of your EBS instance.

You can watch these steps unfold for yourself by using the verbose flag when running admrgpch. The default level, 1 (QUIET), is kind of dull, but level 3 (LOUD) can be instructive. In particular, you'll see the decisions made in the course of comparing file versions:

• "File does not exist in the merged patch, adding...." - new file found
• "Versions Same" - no need to add this file to the merge, as it was found in an earlier patch
• "Replacing old entry with new one." -- Current file has a newer revision
• Presumably, there's also a message for "this version is older than the one in the merge", but my test run didn't include that condition.

Merging patches saves time in two ways. First, since the result of the process is a single new patch, there is no need to run multiple patching sessions. Second, the merged patch will often do less work than applying patches in series, since multiples versions of files are culled, and only the highest version of a given file is applied. Consider the following example, from a merged set of arbitrarily selected patches. This file exists in multiple individual patches, with different versions.

zathras: jpiwowar$ find unmerged -name zxifsrvctypspkgb.pls | xargs grep '$Header' | cut -f4 -d\

120.246.12010000.19
120.246.12010000.9
120.246.12010000.27
120.246.12010000.19
120.246.12010000.45
120.246.12010000.28

If I were to apply each patch individually, adpatch could have to process that PL/SQL file up to 6 times. Perhaps that doesn't seem like a big deal, but there are sure to be similar situations for other files, and that can add up to a significant amount of extra work for large groups of patches. Of course, I could also have the good fortune to apply the patch with the highest version of the file first, and therefore only process that .pls file once, but how often does that ever happen? In contrast, by creating a merged patch, earlier versions of the file are discarded in favor of the highest version:

zathras:ZX jpiwowar$ find merged_patches -name zxifsrvctypspkgb.pls | xargs grep '$Header' | cut -f4 -d\

120.246.12010000.45

Incidentally, while it may look like the transcription of a stifled sneeze, or the name of an Icelandic volcano, that's actually a real filename.

A final, slightly repetitive point: The result of the patch merging process is a new, single patch. The driver file for the merged patch will indicate that the patch is merged, but the only time this information matters is when the AD_BUGS and AD_APPLIED_PATCHES tables are populated at the end of the adpatch session. When adpatch is running, there are no indications of the source of the individual files being processed. There aren't any intermediate steps wherein adpatch records the completion of individual patches, because there aren't individual patches anymore.

Even though the individual patches have lost their identities during the merge, it is still possible to retrieve information about the bug fixes themselves by joining ad_applied_patches, ad_comprising_patches, ad_patch_drivers, and ad_bugs. Here's a query that associates a merged patch with its individual bug fixes. If you're not in the habit of giving your merged patches unique names, then you'll need to modify the query to use ad_applied_patches.applied_patch_id rather than patch_name. When reviewing the results of the query, remember that patch numbers often, but not always, correspond to bug numbers.

select ap.patch_name
     , ad.orig_patch_name
     , ad.driver_file_name
     , ad.merged_driver_flag
     , ad.merge_date
     , bug.bug_number
     , cp.patch_abstract
  from ad_applied_patches ap
  join ad_patch_drivers ad using (applied_patch_id)
  join ad_comprising_patches cp using (patch_driver_id)
  join ad_bugs bug using (bug_id)
 where ap.patch_name = '&merge_name'

We are planning our upgrade to R12, and need to know how many new tables/forms/views/etc there are in the new version. Where are the documents that list these changes?

In these cases, forum posters are usually directed (quite appropriately) to review the upgrade manuals and Release Content documents for R12, which probably contain pieces of the answer to the question. To the best of my knowledge, however, there is no comprehensive, monolithic list of schema changes that occur in the course of an R12 upgrade. One significant reason is the resource burden involved in maintaining a such a list. The upgrade process has so many start and end points. Are you upgrading from 11.0.x? 11.5.8? 11.5.10? Which family packs and CUs are applied? Are you upgrading to to 12.0.6? 12.1.1? 12.1.2? Keeping track of all of those permutations, while perhaps possible, would accomplish little beyond crushing the spirit of a small army of interns.

This is not the question we're looking for; you can go about your business...

An even bigger reason to not be concerned about the number of schema changes in the R12 upgrade: it doesn't seem to be particularly useful information. I can't think of any "effort estimate" metric that would benefit from know knowing the number of new tables, forms, etc. that have been changed, added, or removed. Even knowing that specific tables and forms have disappeared or changed isn't very helpful. After all, you're going to be running thorough tests of any custom reports and forms, anyway, as well as testing to see what common user activities are impacted by the upgrade. Right? (Please say, "right!") We found out pretty quickly, for example, that there were some tables used by our custom reports that had been emptied; the data had been moved to another schema, and our custom reports returned no rows. Good times, and a simple "does this table still exist in R12?" test would not have caught that new feature.

After such a long setup, you must know where this is going. During a long ago (2007-2008) R12 upgrade project, I actually ran some queries comparing the contents of a few key tables (ALL_OBJECTS, FND_APPLICATION, etc) before and after the upgrade. I'll present the results of those queries below. Perhaps someone can find some use for them that I was unable to divine.

Data of dubious utility

Anyone who's read more than two posts here knows that I disclaim like there's no tomorrow. This post is no different:

• The data presented is for an 11.5.10.2 system upgraded to 12.0.3. Your mileage will vary, because no one's upgrading to 12.0.3 these days. At least, I hope not.
• This data is based on notes and SQLPLUS spool files that are over 2 years old by now, which is not exactly "fresh." Since I no longer have the source data, I can't even do a useful sanity check to be sure I extracted it all correctly. Where possible (i.e. salvageable from my logs), I have included the text of my queries, primarily so readers can understand how I generated the data. There is no doubt substantial room for improvement. Many of the queries are slapdash jobs, since I was just doing informal research. Nonetheless, I believe in showing my work, even if it's sometimes embarrassing.
• If you read Oracle documentation that contradicts anything that you see here, you should definitely trust those docs. Furthermore, if your own investigations contradict what you see here, you should trust your own work.

Categories

After all of that, if you're still on board, follow the links below to jump to the relevant sections. If you get any interesting nuggets out of these lists, then I'm glad I could help. Otherwise, you could just summarize the answer to the "how many changes" question as, "Whoa, lots."

Applications
Concurrent Programs
Forms
Profile Options
Database Objects

Applications

The following tables summarize the number of new and removed applications, as well as an attempt at listing the applications that are marked obsolete, but not actually removed. The major assumption is that the application_id does not change between versions, though I didn't do much to confirm that assumption. The absence of a "removed" table suggests that I didn't find any deleted application_ids, but I don't rule out the possibility that I was just lazy.

New applications

SQL>select application_short_name
 2       , application_name
 3    from r12_application_tl r12 join r12_application using (application_id)
 4   where application_id not in
 5                        (
 6                         select application_id
 7                           from r11_application r11
 8                        )
 9   order by 1
10  /

Renamed applications

SQL>select application_short_name
 2       , r12.application_name new_name
 3       , r11.application_name old_name
 4    from r12_application_tl r12 join r12_application using (application_id)
 5         join r11_application_tl r11 using (application_id)
 6   where not regexp_like(r12.application_name,'obsolete','i')
 7     and r11.application_name <> r12.application_name
 8  /

Obsolete applications

SQL>select application_short_name
 2       , application_name
 3    from r12_application_tl join r12_application using (application_id)
 4   where regexp_like(application_name,'obsolete','i')
 5   order by 1
 6  /

Concurrent Programs

In addition to new concurrent programs introduced in R12, a handful were removed, renamed, or marked obsolete. I also generated a longer list that broke down the counts by application, and another that listed each changed concurrent program individually. In the interest of saving space (and formatting pain), I've just offered the summary data here, followed by the query.

SQL>select count(r12c.concurrent_program_id) prog_count
  2  	 , 'New' prog_status
  3   from r12_application_tl r12a join r12_concurrent_programs r12c
  4        using (application_id)
  5  where not exists
  6  	   (
  7  	    select application_id
  8  	         , concurrent_program_id
  9  	      from r11_concurrent_programs r11c
 10  	     where r11c.application_id = application_id
 11  	       and r12c.concurrent_program_id = r11c.concurrent_program_id
 12  	    )
 13  group by 'New'
 14  union all
 15  select  count(r11c.concurrent_program_id) prog_count
 16  	  , 'Removed' Status
 17    from r11_application_tl r11a join r11_concurrent_programs r11c
 18  	    using (application_id)
 19   where not exists
 20  	    (
 21  	     select application_id
 22  		  , concurrent_program_id
 23  	       from r12_concurrent_programs r12c
 24  	      where r12c.application_id = application_id
 25  		and r11c.concurrent_program_id = r12c.concurrent_program_id
 26  	    )
 27  group by 'Removed'
 28  union all
 29  select  count(r12c.user_concurrent_program_name) prog_count
 30  	  , 'Obsolete' prog_status
 31    from r12_application_tl r12a
 32  	    join
 33  	    r12_concurrent_programs_tl r12c
 34  	    using (application_id)
 35  	    join r11_concurrent_programs_tl r11c
 36  	    using (application_id, concurrent_program_id)
 37   where regexp_like(r12c.user_concurrent_program_name,'obsolete','i')
 38  	and r12c.user_concurrent_program_name <> r11c.user_concurrent_program_name
 39  group by 'Obsolete'
 40  union all
 41  select  count(r12c.user_concurrent_program_name) prog_count
 42  	  , 'Renamed' prog_status
 43    from r12_application_tl r12a
 44  	    join
 45  	    r12_concurrent_programs_tl r12c
 46  	    using (application_id)
 47  	    join r11_concurrent_programs_tl r11c
 48  	    using (application_id, concurrent_program_id)
 49   where not regexp_like(r12c.user_concurrent_program_name,'obsolete','i')
 50  	and r12c.user_concurrent_program_name <> r11c.user_concurrent_program_name
 51  group by 'Renamed'
 52  order by 2
 53  /

Forms

Here's a count by application of new and removed forms. We didn't have much in the way of customized forms, so I didn't drill down to form names, but the queries would be easy enough to modify if needed.

SQL>select r12a.application_name
  2       , count(r12f.user_form_name) new_forms
  3    from r12_application_tl r12a join r12_form_tl r12f
  4         using (application_id)
  5   where not exists
  6         (
  7          select application_id
  8               , form_id
  9            from r11_form_tl r11f
 10           where r11f.application_id = application_id
 11             and r12f.form_id = r11f.form_id
 12         )
 13  group by r12a.application_name
 14  order by 2 desc
 15  /

SQL>select r11a.application_name
  2       , count(r11f.user_form_name) deleted_forms
  3    from r11_application_tl r11a join r11_form_tl r11f
  4         using (application_id)
  5   where not exists
  6         (
  7          select application_id
  8               , form_id
  9            from r12_form_tl r12f
 10           where r12f.application_id = application_id
 11             and r11f.form_id = r12f.form_id
 12         )
 13  group by r11a.application_name
 14  order by 2 desc
 15  /

Profile Options

Summary data for new and removed profile options, with additional information here. Some profile options marked obsolete as well, but if there were any renamed, either I didn't run the query or the results weren't very interesting:

SQL>select count(distinct application_id) affected_apps
  2  	  , count(1) new_options
  3    from r12_profile_options r12
  4   where not exists
  5  	    (
  6  	     select 1
  7  	       from r11_profile_options r11
  8  	      where r12.application_id = r11.application_id
  9  		and r12.profile_option_id = r11.profile_option_id
 10  	    )
 11  /

SQL>select count(distinct application_id) affected_apps
  2  	  , count(1) removed_options
  3    from r11_profile_options r11
  4   where not exists
  5  	    (
  6  	     select 1
  7  	       from r12_profile_options r12
  8  	      where r12.application_id = r11.application_id
  9  		and r12.profile_option_id = r11.profile_option_id
 10  	    )
 11  /

SQL>select application_short_name
  2       , user_profile_option_name obsolete_option
  3    from r12_profile_options_tl
  4         join r12_profile_options
  5         using (profile_option_name)
  6         join r12_application
  7         using (application_id)
  8   where regexp_like(user_profile_option_name,'obsolete','i')
  9   order by 1
 10  /

Database Objects

How many new database objects? A ton. How many deleted database objects? A lot, but less than a ton. Breakdown by application would probably be instructive, since many of those tables and packages are likely in support of new applications.

SQL>select r12.object_type
  2  	  , count(object_name) new_objects
  3    from r12_objects r12
  4   where not exists
  5  	    (
  6  	     select 1
  7  	       from r11_objects r11
  8  	      where r12.owner = r11.owner
  9  		and r12.object_name = r11.object_name
 10  		and r12.object_type = r11.object_type
 11  	     )
 12  group by r12.object_type
 13  order by 1
 14  /

SQL>select r11.object_type
  2  	  , count(object_name) removed_objects
  3    from r11_objects r11
  4   where not exists
  5  	    (
  6  	     select 1
  7  	       from r12_objects r12
  8  	      where r12.owner = r11.owner
  9  		and r12.object_name = r11.object_name
 10  		and r12.object_type = r11.object_type
 11  	     )
 12  group by r11.object_type
 13  order by 1
 14  /

• 11gR2 High Availability Best Practices (Caleb Small)
• 11g New Features You Won't Hear About From Oracle (Dan Morgan)
• Edition Based Redefinition Presentation Slides (Dan Morgan)

I'm not going to go into a full recap of the presentations, but they were all full of really cool information. This post is an attempt to collect some of my mental notes, mostly cast in the context of one of my favorite topics, Oracle Applications.

11gR2 HA Best Practices

Caleb's presentation was very thorough and well-constructed. Dan gave him grief for boring the audience, but I think there was just so much new content to absorb that people were too busy processing to ask many questions on the fly. Here are some one-liners from my notebook (anything that looks like an opinion is my commentary/interpretation, not Caleb's):

• Lots more "moving parts" in 11gR2 Grid Infrastructure, clear "separation of duties" across three privileged OS accounts.
• Service startup order is a little different now
• Cluster status utilities show a lot more information, but need to learn to not rely upon crsstat as much
• Proper networking configuration of 11gR2 GI not for the faint of heart.
• Increased memory requirements will make this tougher to virtualize; I'm going to need a bigger laptop.
• ACFS looks interesting; I wonder if it will be a valid option for an (shared application tier filesystem) for Oracle Applications. (Turns out the answer is "not currently planned," based on this exchange I had w/ Steven Chan on his blog later in the week).

11g New Features

There's a lot of really neat stuff going on in this presentation. I'd like to call out small nugget that, while far from the most important, is still pretty interesting on the surface: "deferred segment creation." When a table is created, no extents are actually allocated until rows are inserted. Seems like an odd feature, but one touted benefit is for large ERP systems like SAP and Oracle Applications, where lots of tables are created that may never be used, depending on what products are implemented. Those thousands of initial extents can certainly add up to real storage, and a more cluttered data dictionary. I can't speak to SAP implementations, but I don't see it as a huge win for EBS customers, given that:

1. This feature is available only when tables are created, which means the benefit will only really be available when Oracle starts shipping Oracle Applications install media with an 11gR2 database. Anyone upgrading to the 11gR2 database will still be stuck with those empty extents.
2. Given the overall footprint of an EBS database, the storage savings isn't such a big deal. For example, here's the potential savings from eliminating "empty" tables from an R12 Vision database:

     SYSTEM@R12VIS(11.1.0.7)>select sum(bytes)/1024/1024 potential_savings
   2  from dba_segments s
   3  where exists (select table_name
   4                  from dba_tables
   5                 where num_rows = 0
   6                   and table_name = s.segment_name
   7               )
   8  /
   
   POTENTIAL_SAVINGS
   -----------------
   3850.36719
   

   3.5(ish) GB out of 200GB is okay, I guess, but not a huge deal for a system that's only going to keep growing. FWIW, I'm going to wave my hands and pretend that the fact that a Vision database has way more populated tables than a "fresh-install" EBS database is balanced by the fact that my quick query doesn't account for the possibility that table stats are stale and some of those tables are actually populated.

Of course, it's possible that I'm missing the point. It wouldn't be the first time! Maybe it really comes down more to a less-cluttered data dictionary. I mean, it can't be about tablespace fragmentation, since we're not supposed to care about that anymore, right?

Edition Based Redefinition (EBR)

This seemed like an interesting feature when I first heard about it last autumn, but I'll confess that I didn't quite comprehend the power of EBR until seeing Dan's demo (parts 1, 2, and 3 are on his Morgan's Library site, with part 4 still in the works). Setting aside the obvious benefits for home-grown applications, the potential benefits in an Oracle Applications environment are huge. Consider:

1. There's already an option to create a staged Applications System to shorten patch downtime windows, allowing administrators to run the "copy" and "generate" portions of large EBS patches prior to applying the patch to production. With EBR, it could be possible to stage the "database" portion of a patch as well, and switch to a new default edition at patch time. You'd probably still want to do the database staging at a quiet time in the database, of course, but daring souls could accomplish "almost-no-downtime" patching if EBR were worked into the Oracle Applications patching framework. Wicked.
2. EBR might even make it possible to truly have EBS patches that could be rolled back. The current patching process already backs up files that are replaced. Thoughtful application of cross-edition triggers might make it possible to revert to a previous edition without loss of data if a patch needs to be backed out. Granted, the process would have to be demonstrated to be pretty bullet-proof before I'd try it in production, but it could save restoring test and dev systems from backup in the event that a patch doesn't work out as expected.

Just as I might be missing the point about deferred segment creation's advantages in EBS, I might be guilty of over-extended enthusiasm with respect to edition based redefinition. Or maybe I've decided to turn this into a science fiction blog. It's sure to be far more complicated to implement EBR in an Oracle Applications context than I'm implying above, and this is only speculation on my part, not anything that's actually promised by Oracle. Still, a nerd can dream...

Thanks again to Caleb and Dan for the great presentations, and for your continued support in getting the user group launched!

Please note that I'm by no means an OLAP expert; if you have your own observations or experiences to share, including corrections to any errors I might have made, please leave a comment. The last thing I want to do is spread misinformation! And, as always, remember: test systems exist for a reason, and instructions from Oracle Support should trump anything you read in this blog entry

"No objects to export" error when exporting AWs

The export process is explained thoroughly in Note 352306.1. You may encounter the following error, however, when attempting to export an empty workspace:

BEGIN dbms_aw.execute('export all to eif file ''EXPORT_DIR/AWTEST.eif'''); END;
*
ERROR at line 1:
ORA-33390: There are no objects to export.
ORA-06512: at "SYS.DBMS_AW", line 93
ORA-06512: at "SYS.DBMS_AW", line 122
ORA-06512: at line 1

An export file will not be created, since there's no data in the workspace. Nonetheless, you will still need to recreate the AW in the 64-bit database, which leads us to the next section...

Before deleting AWs

In addition to gathering the OLAP workspace's name, schema, and tablespace, make sure that you make a note of how the AW is partitioned. This will allow you to more faithfully reconstruct the AW in the 64-bit database. Again, the basics can be found in Note 352306.1, except for a discussion of workspace partitioning. According to the documentation for DBMS_AW.AW_CREATE, by default, analytic workspaces are created with 8 partitions. Querying dba_segments seemed to tell a different story:

SYSTEM@mactest(10.2.0.4)>select segment_name
2    , segment_type
3    , count(*)
4   from dba_segments
5   where segment_name= 'AW$TESTDEFAULT'
6   group by segment_name
7   , segment_type;

SEGMENT_NAME                   SEGMENT_TYPE           COUNT(*)
------------------------------ -------------------- ----------
AW$TESTDEFAULT                 TABLE SUBPARTITION           16

This initially confused me, until I found that the table created for the default workspace is actually comprised of two partitions, each comprised of 8 subpartitions. Apparently, "partition" means different things to different people:

SYSTEM@mactest(10.2.0.4)>select table_name
2  , partition_name
3  , subpartition_count sub
4  from all_tab_partitions
5  where table_name = 'AW$TESTDEFAULT';

TABLE_NAME           PARTITION_NAME        SUB
-------------------- -------------- ----------
AW$TESTDEFAULT       PTN1                    8
AW$TESTDEFAULT       PTNN                    8

So, before you delete the AWs in the 32-bit database, be sure to consult the data dictionary. In most cases, you'll probably see segment count of 16 in dba_segments (implying a default partitioning scheme). But there are exceptions...

There's always one goofball

One of those exceptions came when my query to get a count of AW segments returned a 1. Naturally, I was expecting an even number, so this came as a surprise. At first, I thought this might be a special case when specifying partnum=>1 when creating the workspace:

SYSTEM@mactest(10.2.0.4)>exec dbms_aw.aw_create('JPTEST.TEST1PART','USERS',1);
PL/SQL procedure successfully completed.

SYSTEM@mactest(10.2.0.4)>select segment_name
2  , segment_type
3  ,count(*)
4  from dba_segments
5  where segment_name = 'AW$TEST1PART'
6  group by segment_name
7  , segment_type
8  /

SEGMENT_NAME                   SEGMENT_TYPE           COUNT(*)
------------------------------ -------------------- ----------
AW$TEST1PART                   TABLE SUBPARTITION            2

Then it occurred to me that zero is also a number...

SYSTEM@mactest(10.2.0.4)>exec dbms_aw.aw_create('JPTEST.TEST0PART','USERS',0);
PL/SQL procedure successfully completed.

SYSTEM@mactest(10.2.0.4)>select segment_name
2  , segment_type
3  ,count(*)
4  from dba_segments
5  where segment_name = 'AW$TEST0PART'
6  group by segment_type
7  , segment_name
8  /

SEGMENT_NAME                   SEGMENT_TYPE           COUNT(*)
------------------------------ -------------------- ----------
AW$TEST0PART                   TABLE                         1

Please recall: Note 352306.1 recommends using the Analytic Workspace Manager (AWM) tool to recreate the AWs in the 64-bit database. If you want to create the AWs manually, I suggest engaging with Oracle Support to get their approval. The preceding examples are provided only for illustration of what's going on when the AW is created.

Importing AWs

I don't really have much to add here, other than:

1. If you had any empty AWs in the 32-bit system, you won't have anything to import for those workspaces, though hopefully you recreated them in the 64-bit system
2. Isn't this process the sort of thing that cries out to be scripted? Would I really resort to such cheap, obvious devices to foreshadow my next post? I would indeed!

To launch the E-Business Suite R12 Forms interface directly, use the URL http://yourhost:port/forms/frmservlet. This will present a small login window where you can provide your EBS credentials. After that, you will need to select your responsibility from a list of values, and the appropriate forms menu will appear. A few additional things to consider:

• This method of accessing Forms is primarily intended to verify that the Forms server is functioning properly. Oracle does not recommend (or support) direct access to Forms as a normal mode of operation. Use sparingly, as a last resort.
• You need to supply login credentials for a user that has permission to authenticate locally to Oracle Applications. This is not normally a big deal, since it's the default configuration, but if your EBS instance is integrated with Single Sign-On, be aware that you can't connect to Forms directly with an SSO-only account.
• If your HTML interface is broken, as mine was, selecting an option from the Forms-based menu that would normally launch the HTML interface is likely to produce disappointing results.

Despite Oracle's warning to not use the Forms interface this way without specific guidance from Support, it seemed far better than writing an update statement against the FND_PROFILE_OPTION_VALUES table at 1AM.