Troubleshooting - Apps DBA Diaries

Oracle Recovery Manager Troubleshooting ::ORA-27211: Failed to load Media Management Library

This is an error during Database Clone using RMAN.

I was trying to restore database using RMAN Backup

RMAN-00571: ===========================================================

RMAN-00569: ===============
ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command
RMAN-05501: aborting duplication of target database

RMAN-03009: failure of allocate command on ORA_AUX_SBT_TAPE_1 channel
ORA-19554: error allocating device, device type: SBT_TAPE, device name:

ORA-27211: Failed to load Media Management Library

Additional information:

Cause: Backup Configuration was modified.

Solution:
Use the below command to Resolve the error and continue with Database Restore

rman auxiliary /
run
{
allocate auxiliary channel c1 type disk;
allocate auxiliary channel c2 type disk;
duplicate database to ‘DEV’ backup location ‘/Daily_Backup/Backup_16042018’;
}
exit;
EOF

How to Resolve ORA-12514 error While accessing Oracle database link?

Our client had requirement to pull data from MSSQL server to Oracle Database. I installed Oracle Gateway Server for MSSQL and configured database link to access the MSSQL Database.
When trying to access a heterogenous database link between Oracle Database 12c and MSSQL database, some users were getting the below error
ORA-12154: TNS:could not resolve the connect identifier specified.

Analysis

Database link was created on the Oracle Database environment to access data from MSSQL Server.
Database link was created using below command

create public database link DBLINK_INTG connect to “MYSQL” identified by “Password” using ‘dg4msql’;

Error

SQL>select sysdate from dual@DBLINK_INTG;
ORA-12154: TNS:could not resolve the connect identifier specified
12154. 00000 – “TNS:could not resolve the connect identifier specified”
*Cause: A connection to a database or other service was requested using
a connect identifier, and the connect identifier specified could not
be resolved into a connect descriptor using one of the naming methods
configured. For example, if the type of connect identifier used was a
net service name then the net service name could not be found in a
naming method repository, or the repository could not be
located or reached.
*Action:
– If you are using local naming (TNSNAMES.ORA file):
– Make sure that “TNSNAMES” is listed as one of the values of the
NAMES.DIRECTORY_PATH parameter in the Oracle Net profile
(SQLNET.ORA)
– Verify that a TNSNAMES.ORA file exists and is in the proper
directory and is accessible.
– Check that the net service name used as the connect identifier
exists in the TNSNAMES.ORA file.
– Make sure there are no syntax errors anywhere in the TNSNAMES.ORA
file. Look for unmatched parentheses or stray characters. Errors
in a TNSNAMES.ORA file may make it unusable.
– If you are using directory naming:
– Verify that “LDAP” is listed as one of the values of the
NAMES.DIRETORY_PATH parameter in the Oracle Net profile
(SQLNET.ORA).
– Verify that the LDAP directory server is up and that it is
accessible.
– Verify that the net service name or database name used as the
connect identifier is configured in the directory.
– Verify that the default context being used is correct by
specifying a fully qualified net service name or a full LDAP DN
as the connect identifier
– If you are using easy connect naming:
– Verify that “EZCONNECT” is listed as one of the values of the
NAMES.DIRETORY_PATH parameter in the Oracle Net profile
(SQLNET.ORA).
– Make sure the host, port and service name specified
are correct.

Solution

Recreate Heterogenous Database link using below command

create public database link DBLINK_INTG connect to “MYSQL” identified by “Password” using ‘(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=Test1.domain.com)(PORT=1524))(CONNECT_DATA=(SID=dg4msql))(HS=OK))’

Oracle Applications Patching : ADOP Prepare phase failed due to missing information in FND_NODES.

Applying Patches to Oracle EBS and Oracle Database is one of the main activities for an Oracle database administration.We Often get errors during Oracle patching, which could be due to environment issue or a configuration issue or an Oracle Bug.

Checking Patch logfiles and Oracle Patching worker logfiles is very important to identify and troubleshoot the patching error.

Recently, while i am applying patch on a Production environment, Prepare phase in Oracle EBS online patching failed due to below error

Error Details:Information missing in FND_NODES table for one or more application tier nodes.

=================================================
ADOP (C.Delta.9)
Session ID: 19
Node: prodapp01
Phase: prepare
Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/adop.log
===========================================================================

Validating configuration on node: [prodapp01].
Log: /applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/rwerpprodapp01
[ERROR]: Information missing in FND_NODES table for one or more application tier nodes. For details, refer to log file on the relevant node.
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
[WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
[WARNING]: Found invalid cross references in FS config files.
[UNEXPECTED]Error occurred running “perl /applmgr/Oracle/PRODAPP/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl -contextfile=/applmgr/Oracle/PRODAPP/fs2/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -patchctxfile=/applmgr/Oracle/PRODAPP/fs1/inst/apps/PRODAPP_prodapp01/appl/admin/PRODAPP_prodapp01.xml -phase=prepare -logloc=/applmgr/Oracle/PRODAPP/fs_ne/EBSapps/log/adop/19/20171230_095636/prepare/validate/prodapp01 -promptmsg=hide”
[UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on prodapp01

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 1 (Fail)
=================================================

Analysis
The environment on which issue is reported has two middle Tier nodes. Primary node is the Master node and DMZ tier is the slave node.
Due to autoconfig failure on Oracle EBS Slave node, Support_Web column is not updated in FND_NODES after autoconfig execution.
SUPPORT_WEB column for DMZ node is null in FND_NODES

Solution

1. Bring down Middle Tier Services
2. Run autoconfig on all the Middle Tier nodes
3. Startup the Middle Tier Services
4. Verify that FND_NODES table has correct information
5. Restart adop Prepare Phase

How to Restore context file in oracle applications?

Scenario: Your Applications context file is either accidentally deleted or Corrupted. You do not have any latest backup of context file available.

Below are the steps to recreate context file

The Applications context file can be retrieved by running the adclonectx.pl script.

perl /clone/bin/adclonectx.pl retrieve

Above script prompts for Database connection Info, apps username/password , Available Context files and the location to which context files can be restored.When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost and retrieve it to the default location as in $CONTEXT_FILE parameter.

The above command can be used only when INST_TOP the is still intact. Incase if INST_TOP is also lost, the Applications tier context file may be retrieved as follows:

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the option of retrieving the Applications tier context file that has been lost.Choose the location to any writeable location and copy the restored context file to the value specified in $CONTEXT_FILE parameter on Application Tier.

To retrieve the Database tier context file,

Execute the following command on the Database tier:

perl /appsutil/clone/bin/adclonectx.pl retrieve

When prompted for the context file to be retrieved, select the Database tier context file and retrieve it to the default location ($CONTEXT_FILE on DB node).

Weblogic Managed Servers Crashed due to weblogic.socket.MaxMessageSizeExceededException

ISSUE
=====

Weblogic Managed Servers are crashing frequently .

Admin Server Logfile has the below error message.

<IOException occurred on socket: Socket[addr=test1.localhost.com/180.81.41.219,port=1345,localport=47093]
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: ‘10000080’ bytes exceeds the configured maximum of: ‘10000000’ bytes for protocol: ‘t3’.
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: ‘10000080’ bytes exceeds the configured maximum of: ‘10000000’ bytes for protocol: ‘t3’
at weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMuxableSocket.java:230)
at weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:351)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:989)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:920)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Due to above error, Admin Server is going into WARNING Status and Managed Servers are disconnected from ADMIN Server.

SOLUTION
========

1. Increase Maxmessage size from WLS console

Login to weblogic console url as weblogic admin user

WebLogic Console -> Servers ->Admin Server -> Protocols -> General -> Max Message
Size

Increase Max Message Size to 20000000 (Any value higher than Incoming message size from error message)

2.Restart MT Services

adpreclone.pl error while running copyBinary.sh on a 12.2 Environment

In a 12.2.5 environment adpreclone.pl errors out while running copyBinary.sh adpreclone.pl on appsTier

$perl adpreclone.pl appsTier

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.20

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server….

Running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

Running:
perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress

Setting the wls environment

Beginning application tier Stage – Thu Aug 18 02:43:26 2016

/test12/applmgr/fs1/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /test12/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/test12/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/test12/applmgr/fs1/EBSapps/comn/java/classes:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/test12/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/test12/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/test12/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/test12/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/test12/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar oracle.apps.ad.clone.StageAppsTier -e /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml -stage /test12/applmgr/fs1/EBSapps/comn/clone -tmp /tmp -method CUSTOM -showProgress -nopromptmsg

Log file located at /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/StageAppsTier_08180243.log

20% completed
ERROR while running Stage…
Thu Aug 18 02:50:43 2016

ERROR while running perl /test12/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/test12/applmgr/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/test12/applmgr/fs1/EBSapps/comn/clone component=appsTier method= appctx=/test12/applmgr/fs1/inst/apps/test12_vmohswrly128/appl/admin/test12_vmohswrly128.xml showProgress …
Thu Aug 18 02:50:43 2016
[aptest12@vmohswrly128 <RUN> ]

adpreclone log files shows the below error message

Error Message:

START: Creating FMW archive.
Running /test12/applmgr/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /test12/applmgr/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /test12/applmgr/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /test12/applmgr/fs1/FMW_Home -ldl /test12/applmgr/fs1/inst/apps/test12_vmohswrly128/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 298222 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Solution:

1. set the environment variable T2P_JAVA_OPTIONS as given below

$export T2P_JAVA_OPTIONS=”-Djava.io.tmpdir=/test12/applmgr/temp”

2. Make sure enough free space is available in the temp directory provided above

3.Rerun adpreclone.pl on appsTier

ADOP Prepare Phase Fails with ORA-31011: XML parsing failed

While applying Patch on a test instance (R12.2.5), Prepare phase failed with error ORA-31011: XML parsing failed

Complete error message from adop logfile
==================================

[aptest12@localhost <RUN> ]$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
Run Edition context : /test12/applmgr/fs2/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
Patch edition context: /test12/applmgr/fs1/inst/apps/test12_localhost/appl/admin/TEST12_localhost.xml
Patch file system free space: 1130.34 GB

Validating system setup.
[ERROR] Failed to execute SQL statement:
select node_name from FND_OAM_CONTEXT_FILES
where NAME not in (‘TEMPLATE’,’METADATA’,’config.txt’) and
CTX_TYPE=’A’ and (status is null or upper(status) in (‘S’,’F’))
and EXTRACTVALUE(XMLType(TEXT),’//file_edition_type’) = ‘run’
and EXTRACTVALUE(XMLType(TEXT),’//oa_service_group_status[@oa_var=”s_web_admin_status”]’)=’enabled’
and EXTRACTVALUE(XMLType(TEXT),’//oa_service_list/oa_service[@type=”admin_server”]/oa_service_status’)=’enabled’

[ERROR] Error Message:
[ERROR] ORA-31011: XML parsing failed
[ERROR] ORA-19202: Error occurred in XML processing
[ERROR] LPX-00229: input source is empty
[ERROR] ORA-06512: at “SYS.XMLTYPE”, line 272
[ERROR] ORA-06512: at line 1 (DBD ERROR: error possibly near <*> indicator at char 208 in ‘ select node_name from FND_OAM_CONTEXT_FILES
[ERROR] where NAME not in (‘TEMPLATE’,’METADATA’,’config.txt’) and
[ERROR] CTX_TYPE=’A’ and (status is null or upper(status) in (‘S’,’F’))
[ERROR] and EXTRACTVALUE(<*>XMLType(TEXT),’//file_edition_type’) = ‘run’
[ERROR] and EXTRACTVALUE(XMLType(TEXT),’//oa_service_group_status[@oa_var=”s_web_admin_status”]’)=’enabled’
[ERROR] and EXTRACTVALUE(XMLType(TEXT),’//oa_service_list/oa_service[@type=”admin_server”]/oa_service_status’)=’enabled’
[ERROR] ‘)
[UNEXPECTED]Error occurred identifying Admin server node

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

to get the list of the log files along with snippet of the error message corresponding to each log file.

Cause:

fnd_oam_context_files table has incorrect information or the table is corrupted

Solution:

1. Login to sqlplus as applsys and take backup of fnd_oam_context_files table.

SQL> conn applsys
Enter password:
Connected.
SQL> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

Table created.

2. Truncate Table fnd_oam_context_files

SQL> truncate table fnd_oam_context_files;

Table truncated.

3. Bring down MT Services

4. Run Autoconfig to populate fnd_oam_context_files with right information.

Adpatch Worker Fails with “ORA-01031: insufficient privileges”

While Applying Patch on Customer environment, adpatch failed with error

Worker log has the below error.
=====================
GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

AD Worker error:
The following ORACLE error:

ORA-01031: insufficient privileges

occurred while executing the SQL statement:

GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role

Error occurred in file

/test12/applmgr/1200/ad/12.0.0/patch/115/sql/ademusr.sql

with arguments ‘&systempwd &un_fnd &pw_fnd &un_apps &pw_apps’.

==============

Solution
=========

1.Login to oracle database server as oracle user

2. Connect to sqlplus as sysdba and run the failed sql statement
sqlplus / as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> GRANT select on DBA_USERS_WITH_DEFPWD to em_oam_monitor_role
2 ;

Grant succeeded.

SQL>

3. Use adctrl to skip the failed worker (Option 8) and continue with Patching