Why merge EBS patches?

We already know this stuff, right?

“Everyone knows” that merging E-Business Suite patches is a good idea. It should be the number one, first-pass answer to the question of how to speed up patching, before staged APPL_TOP, distributed AD, database tuning, or even adding more workers. As part of my continuing series, “Captain Obvious over-explains the basics,” I've provided an anecdote to illustrate why merging patches is a good idea.

First things first

A few short notes about when you should and should not merge E-Business Suite patches, and a disclaimer:

  • Don't merge AD patches with non-AD patches. AD patches sometimes change the same utilities that you're using to apply patches, and should be addressed separately from other product patches.
  • Don't merge a patch if Oracle says that you shouldn't. It's rare that this will happen, but you still need to study your READMEs carefully.
  • Don't merge patches across codelines (11.5.x, 12.0, 12.1).
  • Other than that, go nuts. Seriously. There's no reason you can't merge, for example, HR, Inventory, and Order Management patches. You'd be surprised how many of the same files and patch actions are delivered with patches for separate, functionally distinct products.
  • Your results may not be as dramatic as what I'm describing in this post. Patch payloads differ, systems differ.

Save time before the work even starts

The larger the set of patches you have to apply, the more potential benefit you'll derive from merging the patches. A good example of a large patching exercise is the application of Extended Support baseline patches for EBS 11.5.10.2. I ended up with 123 patches to apply across multiple products, from rollups to family packs, with all of their small-to-mid-size pre-requisites. That’s a lot of work. So much work, in fact, that running the merged patch in noapply mode took 23 minutes:

[applmgr@testsrv log]$ grep ‘AutoPatch started’ u_baseline_final.log
AutoPatch started at: Thu Mar 03 2011 15:27:45
[applmgr@testsrv log]$ ls -l u_baseline_final.log
-rw-r—r— 1 applmgr oinstall 10390236 Mar 3 15:50 u_baseline_final.log

That's serious "think time" for what amounts to a long sequence of file version checks. But if you if you think that’s bad, watch what happened when I tried to run those patches in noapply mode individually. FWIW, the alias testpatch_this runs adpatch with a defaults file and apply=no, so I could get a timing run free of waits for user intervention.

[applmgr@testsrv baseline_src]$ time for i in `ls`; do cd $i; testpatch_this; cd ..; done
<lots of patch files later…>
real 135m5.485s
user 19m31.403s
sys 5m41.994s

Fewer decisions = less work to do

So, where did the time savings come from? The admrgpch utility does a pretty simple job. It:

  1. Unzips all the patches to be merged
  2. Parses the individual patch driver files and to identify the actions required by the patch
  3. Merges actions from the current patch driver file with those of previously-parsed patches based on file version
  4. Copies the required files to the new merged patch directory
  5. Assembles the necessary patch actions into a new merged patch driver file, based on on the contents of the individual driver files

Step 3 is where we saved nearly 2 hours. Here's a summary of the results of all of the merging and version-checking:

SQL> select count(1) comparisions
  2  from patch_comps
  3  /

COMPARISIONS
------------
      190038

SQL> select merge_action          
  2       , count(1) as file_count
  3    from patch_comps
  4   group by merge_action
  5   order by merge_action 
  6  /

MERGE_ACTI FILE_COUNT
---------- ----------
addnew		96925
addnover	11243
newer		14481
older		15257
same		50056
skipnover	 2076

6 rows selected.

There were just over 190000 potential files in our merged patch, accounting for at least one action each -- sometimes more, since there are forms to be generated, PL/SQL to be compiled, and executables to link after the files are copied into place. Along the way, admrgpch discarded 65000 files with the same or older version, and replaced 14000 with more recent versions. There's some repetition in the 'newer' and 'addnover' categories, but the rough result in this case is that close to 80000 of the possible files in this collection of patches have been discarded before adpatch is even run. No matter how fast your system is, if you hand it a pile of work that's 40% lighter than it could be, you can probably expect to finish a bit sooner.

But wait, there's more! (sort of)

In this case, merging patches has already saved us some significant time just in terms of cutting down the evaluation work that needs to be done at patch time. It's also interesting to look a bit more closely at the individual patch actions that are being culled. After all, if all we're doing is preventing several thousand files from being copied to the OA_HTML directory, well...it's nice, but not exactly compelling. If we can save time in the relinking and forms and reports generation phases, on the other hand, that's more real work that can be avoided. There's a small caveat here, of course: some of this "saved work" might be avoided by adpatch anyway, if the forms/reports to be generated are older than what's already on the system. Still, why relink a bunch of executables 20 times in one patch session, when once will do?

SQL> select patch_action
  2       , repeats
  3       ,  count(1) as file_count
  4       , repeats*count(1) as duplicates
  5        from
  6	   ( select filename
  7		      , patch_action
  8		      , count(1) - 1 as repeats
  9	       from patch_actions
 10	      where patch_action in ('genform','genrep', 'link')
 11	      group by filename, patch_action
 12	   )
 13   group by patch_action, repeats
 14   order by patch_action desc, repeats desc
PATCH_ACTION				    REPEATS FILE_COUNT DUPLICATES
---------------------------------------- ---------- ---------- ----------
link						 25	     2	       50
						 24	     9	      216
						 23	    33	      759
						 22	   313	     6886
						 21	     6	      126
						 19	     1	       19
						 15	     1	       15
						 13	     3	       39
						 12	     1	       12
						 11	     2	       22
						  4	     2		8
						  3	     8	       24
						  2	     2		4
						  1	     2		2
						  0	     4		0
****************************************		       ----------
Total Dups							     8182

genrep						 18	     1	       18
						  8	     1		8
						  4	     1		4
						  3	     3		9
						  2	    19	       38
						  1	   421	      421
						  0	   379		0
****************************************		       ----------
Total Dups							      498

genform 					 21	    13	      273
						 20	    12	      240
						 19	    53	     1007
						 17	     4	       68
						 15	     6	       90
						 14	     1	       14
						 10	   119	     1190
						  9	     1		9
						  7	     1		7
						  6	     5	       30
						  5	    21	      105
						  4	   107	      428
						  3	    77	      231
						  2	   394	      788
						  1	   742	      742
						  0	  1245		0
****************************************		       ----------
Total Dups							     5222

7 Comments

  1. Posted 21 March 2011 at 6:19 | Permalink

    Don't let clients read that - that's hours of valuable billable time LOST!

    You run your patches with just 2 workers right?

    :P

    Ok, sarcasm aside, good post showing the clear benefits from merging patches.

  2. Posted 21 March 2011 at 8:18 | Permalink

    Hi Jay,

    Sometimes, if I'm really pressed for time, I might jack that up to 4 workers. Crazy, I know. ;)

    Regards,

    John P.

  3. srikanth
    Posted 14 July 2011 at 2:50 | Permalink

    for the extended support for 11.5.10.2(metalink883202.1),,which patches can i merge?can i merge patches of differant modules?could u plz let me know

  4. Posted 14 July 2011 at 8:50 | Permalink

    Hi Srikanth,

    You can definitely merge patches of different modules, though merging AD patches with patches from other modules is not permitted.

    Regards,

    John P.

  5. srikanth
    Posted 15 July 2011 at 4:24 | Permalink

    thanks

  6. Paul Richards
    Posted 24 February 2013 at 10:47 | Permalink

    Got bit once by patch merging, so wary of doing it again. Merged two 11.5.9 patches, the merging of which left two scripts that should run in a precise order in the same group of scripts, every one of five test runs they ran in the right order. Come production run, they ran in the wrong order. Few days later - too many days for regression - the problem arising appears. Exact hit on Metalink for problem, exact cause of merging these two patches.

    Prefer saving time by having response files, built during testing, which also removes that weak link of manual typing.

  7. Posted 27 February 2013 at 11:27 | Permalink

    Hi Paul,
    I can certainly understand your point of view. Nothing hurts worse than getting burned in production by something that didn't show up in test. I will note, however, that the AD utilities, as well as Oracle's internal process for bundling patches and defining patch stages, have come a long way since 11.5.9. :) I've done a lot of patching in the last several years, and merges have saved me tons of downtime (error-free), particularly when it comes to upgrades and large maintenance packs. Even with carefully scripted response files, serially applying a stack of small patches incurs more overhead than applying a merged bundle.

    Nonetheless, as DBAs we are responsible for maintaining our systems in a way that allows us to be confident that they are performing as they should. Do what works best for you, in your environment. Thanks for reading and taking the time to comment!

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*