Category: Patching

adop phase=fs_clone fails during Validation of /etc/hosts

 Issue:  Running fs_clone using command adop phase=fs_clone fails during txkADOPValidations.pl script


Error Details:

Error message from ADOPValidations log file

[ERROR]: Required content is missing from /etc/hosts file.


Cause:  

There are duplicate entries for localhost in /etc/hosts file in one of the slave nodes.


Solution:

Correct the entries in /etc/hosts and re-run fs_clone


Follow steps mentioned in MOS document:


adop phase=fs_clone Ends with “[ERROR]: Required content is missing from /etc/hosts file” (Doc ID 2815387.1)

How to Run hrglobal driver in Oracle EBS 12.2.X Version?

Steps to apply hrglobal driver in Oracle EBS 12.2.x version

1. Start an Oracle online EBS Patching cycle.

adop phase=prepare

2.Source the patch filesystem environment file

ADOP utillity is intelligent enough to switch to Patch Filesystem depending on the ADOP phase being run.For any Other manual operations, we need to set the environment to point to Patch Filesystem explicitly.

. ./EBSapps.env PATCH

3. Run DataInstall

java oracle.apps.per.DataInstall apps appspassword thin test.domain.com:1521:DEV

Select the Required localisations and save the changes.

Example of Data Install Changes Summary is shown below.

          DataInstall – Actions confirmation

Do you really wish to exit and save your changes?

      [Y]      – Yes, save then exit
      [N]      – No, don’t save but exit
      [Return] – To return to the DataInstall Main Menu

Enter your choice (for example Y) : Y

          DataInstall – Actions summary
          —————————–

The following actions will be performed:

Localisation         Product(s)               Leg. Data? Action
——————– ———————— ———- ————-
Global               Human Resources          Installed  Install
United Arab Emirates Human Resources          Installed  Install
United Arab Emirates Payroll                  Installed  Install

Localisation   College Data? Action
————– ————- ————-
United Kingdom
United States

Option                         Data?         Action
—————————— ————- ————-
JIT/Geocode

Legislation                            Action
——————————         ————-
ALL  All Legislations

4. Apply hrglobal.drv using below syntax.

adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv workers=8

Complete the ADOP patching cycle by running finalize,cutover and cleanup phases

5. adop phase=finalize

6. adop phase=cutover

7. adop phase=cleanup

8. Run fs_clone to synchronize run and patch filesystems
adop phase=fs_clone

Oracle EBS 12.2.x: ADOP error during Cutover phase

Background of the issue: I was applying hrblobal driver in a multi-node Production environment. Our Production application EBS has one Primary node and one DMZ Node.


After applying hrglobal driver using the online patching cycle, Cutover phase thrown below error message


==

Applying patch(es): <6390830> on node EBSDMZ failed.
If you choose to proceed with cutover, node EBSDMZ will be marked as abandoned.
Do you want adop to continue processing on completed nodes [y/n]? n
    As per user choice, adop will not proceed further with processing.


==


If we Provide input as ‘Y’ above, EBSDMZ node will be abandoned and we need to register the DMZ node again.




Solution:


1.Apply Patch on DMZ node using options=nodb and restart the cutover phase


On DMZ Node
=====
Source the patch filesystem
adop phase=apply patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv allnodes=NO action=nodb options=forceapply


2. Restart Cutover phase on Master node


Oracle Applications Patching : ADOP Prepare phase failed due to missing information in FND_NODES.

Applying Patches to Oracle EBS and Oracle Database is one of the main activities for an Oracle database administration.We Often get errors during Oracle patching, which could be due to environment issue or a configuration issue or an Oracle Bug.


Checking Patch logfiles and Oracle Patching worker logfiles is very important to identify and troubleshoot the patching error.


Recently, while i am applying patch on a Production environment, Prepare phase in Oracle EBS online patching failed due to below  error


Error Details:Information missing in FND_NODES table for one or more application tier nodes.





=================================================
ADOP (C.Delta.9)
Session ID: 19
Node: prodapp01
Phase: prepare
Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/adop.log
===========================================================================


Validating configuration on node: [prodapp01].
    Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/rwerpprodapp01
[ERROR]: Information missing in FND_NODES table for one or more application tier nodes. For details, refer to log file on the relevant node.
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
[WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
[WARNING]: Found invalid cross references in FS config files.
    [UNEXPECTED]Error occurred running “perl /applmgr/Oracle/PRODAPP/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/applmgr/Oracle/PRODAPP/fs2/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -patchctxfile=/applmgr/Oracle/PRODAPP/fs1/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -phase=prepare -logloc=/applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/prodapp01 -promptmsg=hide”
    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on prodapp01




[STATEMENT] Please run adopscanlog utility, using the command


“adopscanlog -latest=yes”


to get the list of the log files along with snippet of the error message corresponding to each log file.




adop exiting with status = 1 (Fail)
=================================================




Analysis
The environment on which issue is reported has two middle Tier nodes. Primary node is the Master node and DMZ tier is the slave node.
Due to autoconfig failure on Oracle EBS Slave node, Support_Web  column is not updated in FND_NODES after autoconfig execution.
SUPPORT_WEB column for DMZ node is null in FND_NODES



Solution


1. Bring down Middle Tier Services
2. Run autoconfig on all the Middle Tier nodes
3. Startup the Middle Tier Services
4. Verify that FND_NODES table has correct information
5. Restart adop Prepare Phase

ADOP Useful Options :skipsyncerror

There are Various options available with Oracle Applications Online Patching Utility (ADOP). Here , we will see the benefits of skipsyncerror option.


Scenario: 


Your Previous  adop session failed while applying patches and you did not find any solution to fix the errors, Oracle Support Provided a new patch to fix the issue. You will get errors while synchronization and need to ignore those errors.


Usage: adop phase=prepare skipsyncerror=yes




skipsyncerror Details


This option is used along with prepare phase.


Purpose:


This feature enables the user to specify that any synchronization errors in the prepare phase are expected to be fixed automatically in the synchronization that takes place with subsequent patches.


Values: yes/no


Default value is ‘no’. Set the value to ‘yes’ in order to work around synchronization failures that may occur when patches that failed to apply correctly in a previous patching cycle are synchronized during the prepare phase.



ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed


Complete error message from adop logfile
==================================


[aptest12@localhost <RUN> ]$ adop phase=prepare


Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:


Validating credentials.


Initializing.
    Run Edition context  : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
    Patch file system free space: 1130.34 GB


Validating system setup.
    [ERROR]     Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in (‘TEMPLATE’,’METADATA’,’config.txt’) and
CTX_TYPE=’A’ and (status is null or upper(status) in (‘S’,’F’))
and EXTRACTVALUE(XMLType(TEXT),’//file_edition_type’) = ‘run’
and EXTRACTVALUE(XMLType(TEXT),’//oa_service_group_status[@oa_var=”s_web_admin_status”]’)=’enabled’
and EXTRACTVALUE(XMLType(TEXT),’//oa_service_list/oa_service[@type=”admin_server”]/oa_service_status’)=’enabled’


    [ERROR]     Error Message:
    [ERROR]     ORA-31011: XML parsing failed
    [ERROR]     ORA-19202: Error occurred in XML processing
    [ERROR]     LPX-00229: input source is empty
    [ERROR]     ORA-06512: at “SYS.XMLTYPE”, line 272
    [ERROR]     ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in ‘    select node_name from FND_OAM_CONTEXT_FILES
    [ERROR]         where NAME not in (‘TEMPLATE’,’METADATA’,’config.txt’) and
    [ERROR]             CTX_TYPE=’A’ and (status is null or upper(status) in (‘S’,’F’))
    [ERROR]             and EXTRACTVALUE(<*>XMLType(TEXT),’//file_edition_type’) = ‘run’
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),’//oa_service_group_status[@oa_var=”s_web_admin_status”]’)=’enabled’
    [ERROR]             and EXTRACTVALUE(XMLType(TEXT),’//oa_service_list/oa_service[@type=”admin_server”]/oa_service_status’)=’enabled’
    [ERROR]     ‘)
    [UNEXPECTED]Error occurred identifying Admin server node




[STATEMENT] Please run adopscanlog utility, using the command


“adopscanlog -latest=yes”


to get the list of the log files along with snippet of the error message corresponding to each log file.




Cause:


fnd_oam_context_files table has incorrect information  or the table is corrupted




Solution:




1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.




SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;


Table created.


2. Truncate Table fnd_oam_context_files


SQL>  truncate table fnd_oam_context_files;


Table truncated.




3. Bring down MT Services


4. Run Autoconfig to populate fnd_oam_context_files  with right information.


Adpatch Worker Fails with “ORA-01031: insufficient privileges”

While Applying Patch on Customer environment, adpatch failed with error


Worker log has the below error.
=====================
GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role


AD Worker error:
The following ORACLE error:


ORA-01031: insufficient privileges




occurred while executing the SQL statement:


GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role


Error occurred in file


/test12/applmgr/1200/ad/12.0.0/patch/115/sql/ademusr.sql


with arguments ‘&systempwd &un_fnd &pw_fnd &un_apps &pw_apps’.


==============



Solution
=========


1.Login to oracle database server as oracle user


2. Connect to sqlplus as sysdba and run the failed sql statement
sqlplus / as sysdba


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role
  2  ;


Grant succeeded.


SQL>




3. Use adctrl to skip the failed worker (Option 8) and continue with Patching

What Happens during adop cutover Phase?

Cutover Phase Requires a brief downtime.


Following activities occur during Cutover:


+Users are logged off the system
+Finalize is run as part of cutover -if not run earlier in adop cycle
+Cutover phase cancels ADZPPATCH Concurrent Program
+Stops all Services on RUN and PATCH editions , switches filesystem and starts services on new RUN FS
+Flipping snapshots in run and patch editions.
+Users are brought back online on the patched system 


Environment is changed after cutover. Need to Re-source the env, for performing any actions after cutover.


Syntax:


$ adop phase=cutover


Multi-node Considerations
========================
By default cutover is invoked automatically on remote nodes.
If you want cutover individually:      
1. First node the cutover should be:           
$ adop phase=cutover allnodes=no       
2. For all other nodes you should do:        
$ adop phase=cutover allnodes=no action=nodb           
 * nodb because db work is already done




Cutover Parameters:
===================


mtrestart :Specifies whether or not to start Mid-Tier services at end of cutover, it allows you to perform some additional maintenance operations in the system before bringing it up.        – Takes values “yes” or “no”.        – Default: yes
allnodes : Specifies whether or not to run on all the nodes.        – Takes values “yes” or “no”.        – Default: yes
action: Specifies whether or not to take database action        – Takes values “db” or “nodb”        – Default: db