Author: Apps DBA

Clusterware Compatibility (Destructive) Testing

Destructive tests include forced failures by software and
hardware while the system is running with either minimal or high workload.
Oracle software – one or more of Oracle background processes is killed manually.
OS software – one or more of the cluster daemons is killed manually or the
system is forced to reboot. Hardware: Manual removal of network or disk
connectivity or power supply.
There are two major categories of cluster compatibility
tests:
 Clusterware (Destructive):
Starting with Oracle Database 10g, the certification and
validation process has been enhanced to include hardware destructive tests
executed under high system load.
   Cluster File System:
  Starting with Oracle
Database 11g, the certification and validation process has been further
enhanced to include a set of destructive and high availability tests, designed
to verify the use of cluster file system to support the various Oracle
Clusterware and Real Application Clusters components.

Useful adop options and syntax

1.Using analytics parameter in adop apply phase




$ adop phase=apply analytics=yes


Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):


ADZDCMPED.sql – This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.


ADZDSHOWED.sql – This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.


ADZDSHOWOBJS.sql – This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out


ADZDSHOWSM.sql – This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out


Note: The analytics parameter should only be used when required, because of the extra processing needed.






2. flags=autoskip 

 e.g adop phase=apply patches=12345678  flags=autoskip 

 This option is an alternative for  “Continue as if it were successful”?  in adpatch, very useful in cases where patch failed to compile forms/reports and exiting.
 We need not restart the adop session just to compile a single forms (.fmb,.fmx) file.
 Make sure to review the autoskip.log logfile and fix the issues in autoskip log whenever you use autoskip flag in adop cycle.


3.  skipsyncerror=(yes|no)  [default: no]


 Specifies whether to ignore errors that may occur during incremental file system synchronization.  This might happen if you applied
 a patch in the previous patching cycle that had errors but decided to continue with the cutover.  When the patch is synchronized on
 the next patching cycle, the apply errors may occur again, but can be ignored.


4. wait_on_failed_job=(yes|no)  [default: no]


 Controls whether adop apply command exits when all workers have failed.  Instead of exiting, you can force adop to wait, and use the “adctrl” to retry failed jobs.

e.g adop phase=apply patches=7777 wait_on_failed_job=yes



5. Abort


The Online Patching Cycle can be aborted at any time prior to Cutover


e.g adop phase=abort


This is needed  if unrecoverable error happened or the user decides that patch is not needed.
If adop phase=apply failed, user should try abandon=yes first.
The abort command drops the database patch edition and  returns the system to normal runtime state.  Immediately following abort, you must also run a full cleanup and
fs_clone operation to fully remove effects of the failed online patching cycle.



            

What Happens in ADOP apply Phase

Apply Phase is executed in PATCH Filesystem
During Apply phase of an ADOP cycle, adop execute patch drivers to update Patch Edition
We can have multiple apply phases in an adop cycle, Multiple patches including customizations can be installed in this phase.


The production application is online and accessible to users. RUN filesystem is not affected by the changes.
Patches are applied to the copy (Patch Edition)
 Changes are made in the isolation of an Edition 
The running application is unaffected by these changes




workers 
specifies number of parallel workers, automatically calculated if not specified  
abandon 
abandons a patching session (if set to “yes”) and must have an opposite specification to that of parameter “restart”.
restart
 restarts a patching session (if set to “yes”) and must have an opposite specification to that of parameter “abandon”.

Example: $ adop phase=apply workers=4 abandon=yes restart=no patches=1234567,8909228






For more adop options and their usage Click here

Autopatch error: The worker should not have status ‘Running’ or ‘Restarted’ at this point

If you are restarting a failed patch session in oracle applications, sometimes you may encounter the error


AutoPatch error:
The worker should not have status ‘Running’ or ‘Restarted’ at this point.


Telling workers to quit…


All workers have quit.


Connecting to APPS……Connected successfully.


AutoPatch error:


Error running SQL and EXEC commands in parallel


Cause:


1. adpatch or adop process was killed from OS level while patch is being applied


2. Database shutdown or terminated






Solution:


Using adctrl utility, use option 4 to change the worker status to Failed




Review the messages above, then press [Return] to continue.


                    AD Controller Menu
     —————————————————


     1.    Show worker status


     2.    Tell worker to restart a failed job


     3.    Tell worker to quit


     4.    Tell manager that a worker failed its job


     5.    Tell manager that a worker acknowledges quit


     6.    Restart a worker on the current machine


     7.    Exit




Enter your choice [1] : 4


Enter the worker number(s)/range(s) or ‘all’ for all workers,
or press [Return] to go back to the menu : all


Status changed to ‘Failed’ for worker 1.
Status changed to ‘Failed’ for worker 2.
Status changed to ‘Failed’ for worker 3.
Status changed to ‘Failed’ for worker 4.
Review the messages above, then press [Return] to continue.






select option “1. Show Worker Status” , The worker status will be “Failed” 


Now select option 2. “2. Tell worker to restart a failed job” 




Enter the worker number(s)/range(s) or ‘all’ for all workers,
or press [Return] to go back to the menu : all


Status changed to ‘Fixed, restart’ for worker 1.
Status changed to ‘Fixed, restart’ for worker 2.
Status changed to ‘Fixed, restart’ for worker 3.
Status changed to ‘Fixed, restart’ for worker 4.




Restart adpatch 


In 11i, R12.1.X version of oracle applications


when applying patch using adpatch , Select option Yes When it prompts for


‘Do you wish to Continue with Previous adpatch Session’


In R12.2.x, Use restart and abandon parameters




adop phase=apply patches=123456 restart=yes abandon=no 


To restart the patch session from where it failed.

ADOP Patching Fails with aipspv(): Error setting up PDI list for upgrade

While Applying patch on a 12.2.5 environment using online patch cycle in adop, Autopatch Failed with error


AutoPatch error:
Invalid password for schema “CUST1” (oracle_id = 20087)


AutoPatch error:
Failed getting PDI list for product ‘xx1’


AutoPatch error:
aipspv(): Error setting up PDI list for upgrade


Root Cause
=========


Password for Application schema is not changed using FNDCPASS


Solution
========


1.Change the password of custom application with FNDCPASS 


Syntax for FNDCPASS


$ FNDCPASS apps/<PASSWORD> 0 Y system/<PASSWORD> ORACLE CUST1 <NEWPASSWORD>
Log filename : L********.log




Report filename : O*****.out


2. Verify the password change is successful


$ cat L*******.log
+—————————————————————————+
Application Object Library: Version : 12.2


Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.


FNDCPASS:
+—————————————————————————+


Current system time is 10-MAR-2016 23:21:30


+—————————————————————————+


Working…
Password is changed successfully for user BOLINF.
FNDCPASS completed successfully.


+—————————————————————————+
Concurrent request completed successfully
Current system time is 10-MAR-2016 23:21:30


+—————————————————————————+






Please Note, In 12.2.x Environments FNDCPASS must be executed on RUN Filesystem.


3. Restart the online patching cycle 


While restarting adop apply phase after a previously failed session, use options abandon and restart




adop phase=apply patches=123456 restart=yes abandon=no 


[Use this command if you wish to continue the previous patch session from the step where it failed]


adop phase=apply patches=123456 restart=no abandon=yes




[Use this command if you wish to start a fresh patch session]








If changing password using FNDCPASS does not work, analyse further details about database user and application user.


Check if the database user and application user are defined correctly using the sql queries below


SQL> select ORACLE_ID from FND_ORACLE_USERID where oracle_username like ‘%CUST1%’;


 ORACLE_ID
———-
     2008


SQL> select application_id from fnd_application where application_short_name = ‘XX1’;


APPLICATION_ID
————–
         2003


SQL> select * from fnd_product_installations where oracle_id=2008;




If the user details are not correct from the output of above sql, Please correct them.

ORA-12504 While running Autoconfig on Database Server

After Oracle Database upgrade from 11.2.0.3 to 11.2.0.4, Autoconfig errored on Database server with the below error


[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>


  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      afdbprf.sh              INSTE8_PRF         1


  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /test1/oracle/product/11204/appsutil/install/test1_host1
      adcrobj.sh              INSTE8_APPLY       1




AutoConfig is exiting with status 2


====Running the failed script has given the ORA-12504 error



$ afdbprf.sh


afdbprf.sh started at Thu Mar  3 03:20:47 JST 2016




The environment settings are as follows …


       ORACLE_HOME : /test1/oracle/product/11204
        ORACLE_SID : TEST1
              PATH : /test1/oracle/product/11204/perl/bin:/test1/oracle/product/11204/perl/bin:/test1/oracle/product/11204/bin:/test1/oracle/product/11204/OPatch:/usr/bin:/usr/sbin:/test1/oracle/product/11204/jdk/bin:/usr/ccs/bin:/bin:/usr/bin/X11:/usr/local/bin::.
      Library Path : /test1/oracle/product/11204/lib:/usr/X11R6/lib:/usr/openwin/lib:/test1/oracle/product/11204/lib:/usr/dt/lib:/test1/oracle/product/11204/ctx/lib


Executable : /test1/oracle/product/11204/bin/sqlplus




Enter the APPS username: APPS


Enter the APPS password:


SQL*Plus: Release 11.2.0.4.0 Production on Thu Mar 3 03:20:55 2016


Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Enter value for 1: Enter value for 2: Enter value for 3: ERROR:
ORA-12504: TNS:listener was not given the SERVICE_NAME in CONNECT_DATA




ERRORCODE = 1 ERRORCODE_END
$






Solution
=========


s_instLocalListener is not correct in database server context file


1. Take Backup of Context file


2. Modify the value of s_instLocalListener from test1 to test1_LOCAL


3. Run Autoconfig

java.lang.OutOfMemoryError: Java heap space error in Output PostProcessor

Possible causes for Java Heap Space error in OPP


1. There is limited heap space in the JVM to process the report
2. The Java application running is out of memory
3. The scalable feature isn’t enabled
4. An older version of the JRE is being used in the configuration, 
5. There is insufficient space in the designated temp directory for processing the report, and reports are embedded with images which require a lot of space that unavailable.




The log will contain a java stack error similar to the following:
java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:2869)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:3000)
at oracle.xdo.parser.v2.XMLDocument.createTextNode(XMLDocument.java:735)
at oracle.xdo.parser.v2.DocumentBuilder.characters(DocumentBuilder.java:553)
at oracle.xdo.parser.v2.NonValidatingParser.reportCharacters(NonValidatingParser.java:1680)
at oracle.xdo.parser.v2.NonValidatingParser.parseText(NonValidatingParser.java:1391)
at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1268)






How to resolve Java Heap Space error in Output Post Processor?




Solution 1
==========




1. Determine what the heap size per OPP process is currently:
select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = ‘FNDCPOPP’);


2. The default should be:
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m


3. Increase the Heap Space per Process to 1024:
update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
‘J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m’
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = ‘FNDCPOPP’);


4. Bring the managers down.
5. Please run the Concurrent Manager Recovery feature to address any Concurrent Manager / Concurrent Processing issues within the Oracle Application Manager.  NOTE: Regular use of the CP Analyzer in Note 1411723.1 can provide an Immediate Analysis and Output of a Concurrent Environment.
6. Bring the managers up again.


OR


Solution 2
==========
1. Log into applications with the System Administrator responsibility.
2. Navigate to Concurrent -> Program -> Define
3. Query the XML Publisher Template Re-Generator program
4. Set the following value for the Executable Options : -Xmx1024m
5. Save changes.
6. Retest the program.


OR




Solution 3
==========


Configure the XML Publisher Administrator Configuration settings.
1. As XML Publisher Administrator navigate to Administration->Configuration.
2. Under Temporary Directory pick a temporary file location on your concurrent processing node. This should be at least 5GB or 20x larger than largest XML data file you generate
3.  Under FO Processing, set:
o Use XML Publisher’s XSLT processor set to True
o Enable scalable feature of XSLT processor set to False
o Enable XSLT runtime optimization set to True 



What Happens During ADOP Finalize Phase

The Finalize phase of adop cycle,takes care of below online activities


1.Perform the final operations that can be executed while the Application is online
2.Compile invalid objects
3.Generate derived objects
4.Pre-compute DDL to be run at Cutover


After Finalize phase, we can pause the adop online patching cycle until the appropriate downtime window is available.


Syntax for running finalize phase 


$ adop phase=finalize
$ adop phase=finalize finalize_mode=quick == This is Default Mode
$ adop phase=finalize finalize_mode=full    


Finalize can be run in 2 modes: “QUICK” or “FULL”
FULL gathers database dictionary statistics (not transaction tables statistics)
QUICK skips gathering database dictionary statistics.