Apps DBA - Apps DBA Diaries

Concurrent request Error: The executable file for this concurrent program cannot be executed.

Issue: Concurrent request completed with below Error.

This request finished with an error and produced the following completion message: The executable file /test1/apps/fs2/EBSapps/appl/fnd/12.0.0/bin/XXCUST_PROG for this concurrent program cannot be executed.

ERROR:

The executable file /test1/apps/fs2/EBSapps/appl/fnd/12.0.0/bin/XXCUST_PROG for this concurrent program cannot be executed.

Contact your system administrator or support representative.

Cause:

Verify that the execution path to the executable file

Run the below command

$ls -lrt /test1/apps/fs2/EBSapps/appl/fnd/12.0.0/bin/XXCUST_PROG

Output of the above command shows executable is pointing to a wrong symbolic link

Solution:

1.Create a soft link with the program name as below:

$cd /test1/apps/fs2/EBSapps/appl/fnd/12.0.0/bin/

$ln -s -f $FND_TOP/bin/fndcpesr XXCUST_PROG

2. Re-run the concurrent program and test the issue.

How to update override email address using backend sql query in EBS 11i/12.x?

Query to check Override address in EBS workflow notification Mailer:

SQL>

select fscpv.parameter_value

from fnd_svc_comp_params_tl fscpt

,fnd_svc_comp_param_vals fscpv

where fscpt.display_name = ‘Test Address’

and fscpt.parameter_id = fscpv.parameter_id;

PARAMETER_VALUE

——————————————————————————–

NONE

SQL Query to update Override email address from backend in EBS :

SQL> update fnd_svc_comp_param_vals

set parameter_value = ‘TEST_EMAIL@DOMAIN.COM’

where parameter_id =

( select parameter_id

from fnd_svc_comp_params_tl

where display_name = ‘Test Address’);

1 row updated.

SQL> commit;

Commit complete.

Verify that override email address is updated:

SQL> select fscpv.parameter_value

from fnd_svc_comp_params_tl fscpt

,fnd_svc_comp_param_vals fscpv

where fscpt.display_name = ‘Test Address’

and fscpt.parameter_id = fscpv.parameter_id;

PARAMETER_VALUE

——————————————————————————–

TEST_EMAIL@DOMAIN.COM

SQL>

EBSO 12.2.x : Query to check status of ADOP session

Check Status of adop session using below sql query:

set pagesize 200;
set linesize 160;
col PREPARE_STATUS format a15
col APPLY_STATUS format a15
col CUTOVER_STATUS format a15
col ABORT_STATUS format a15
col STATUS format a15
select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS
from AD_ADOP_SESSIONS
order by ADOP_SESSION_ID;

Decoding Cutover status:

cutover_status=’Y’ ‘COMPLETED’
cutover_status not in (‘N’,’Y’,’X’) and status=’F’ ‘FAILED’
cutover_status=’0′ ‘CUTOVER STARTED’
cutover_status=’1′ ‘SERVICES SHUTDOWN COMPLETED’
cutover_status=’3′ ‘DB CUTOVER COMPLETED’
cutover_status=’D’ ‘FLIP SNAPSHOTS COMPLETED’
cutover_status=’4′ ‘FS CUTOVER COMPLETED’
cutover_status=’5′ ‘ADMIN STARTUP COMPLETED’
cutover_status=’6′ ‘SERVICES STARTUP COMPLETED’
cutover_status=’N’ ‘NOT STARTED’
cutover_status=’X’ ‘NOT APPLICABLE’

EBS 12.2.X Patching: ADOP Cutover failed while running txkADOPCutOverPhaseCtrlScript.pl

Issue:

During an adop online patching cycle, cutover phase failed while running the script txkADOPCutOverPhaseCtrlScript.pl

[PROCEDURE] Starting Middle Tier Services
[UNEXPECTED]Error occurred running “perl /test/app/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/test/app/fs1/inst/apps/EBSTEST_testebsmt111/appl/admin/EBSTEST_testebsmt111.xml -patchcontextfile=/test/app/fs1/2/inst/apps/EBSTEST_testebsmt111/appl/admin/EBSTEST_testebsmt111.xml -promptmsg=hide -console=off -sessionid=14 -timestamp=20201108_194155 -outdir=/test/app/fs_ne/EBSapps/log/adop/14/20201108_194155/cutover/testebsmt111 -action=forcestartup”
[UNEXPECTED]Cutover phase has failed.

Cause:

Review of all adop and cutover log files shows that Cutover phase failed after filesystem switch and ADMIN server startup,oacore services could not be started after cutover due to port conflict.

Resolution:

1. Check Status of adop session using below sql query:
set pagesize 200;
set linesize 160;
col PREPARE_STATUS format a15
col APPLY_STATUS format a15
col CUTOVER_STATUS format a15
col ABORT_STATUS format a15
col STATUS format a15
select NODE_NAME,ADOP_SESSION_ID, PREPARE_STATUS , APPLY_STATUS ,CUTOVER_STATUS , CLEANUP_STATUS , ABORT_STATUS , STATUS
from AD_ADOP_SESSIONS
order by ADOP_SESSION_ID;
For my issue, cutover status was 5, which means ADMIN startup is completed.

2. Restart the cutover phase with mtrestart=no option

adop phase=cutover mtrestart=no

3. Fix the issues related to services startup on RUN filesystem (which is switched after cutover).

4. Startup the services

5. Perform cleanup phase and complete the adop cycle
adop phase=cleanup

Useful srvctl commands

Useful srvctl commands for ORACLE RAC MAINTENANCE

srvctl -help or srvctl -v

srvctl commands to startup database:

srvctl start database -d db_name [-o start_options] [-c connect_str|-q]

srvctl start database -d db_name [-o open]

srvctl start database -d db_name -o nomount

srvctl start database -d db_name -o mount

srvctl commands to shutdown database:

srvctl stop database -d db_name [-o stop_options] [-c connect_str|-q]

srvctl stop database -d db_name [-o normal]

srvctl stop database -d db_name -o transactional

srvctl stop database -d db_name -o immediate

srvctl stop database -d db_name -o abort

srvctl config database

srvctl config database -d db_name [-a] [-t]

srvctl start listener -n node_name [-l listener_names]

srvctl stop listener -n node_name [-l listener_names]

srvctl status listener [-n node_name] [-l listener_names]

srvctl config listener -n node_name

adpreclone.pl Error on Applications Tier

Issue:

adpreclone.pl appsTier on EBS 12.2.x environment fails with error

ERROR: The context variable s_apps_jdbc_connect_descriptor is null

Cause:

Value of parameter “s_apps_jdbc_connect_descriptor” is null in Applications Context file.

Solution:

1. Take Backup of context file

2. Update the value of parameter “s_apps_jdbc_connect_descriptor” with JDBC connect string to connect to the database.

3. Run Autoconfig on Applications node.

4. Restart adpreclone.pl appsTier

Autoconfig Error : ORA-01422: exact fetch returns more than requested number of rows

Issue: Autoconfig on EBS 12.2 environment fails while running script txkGenADOPWrapper.pl

Below is the error message from logfile

=======================

Script Name : txkGenADOPWrapper.pl

Script Version : 120.0.12020000.3

ERROR DESCRIPTION:

(*******FATAL ERROR*******

PROGRAM : (/test/app/fs1/inst/apps/test_mtnode1/admin/install/txkGenADOPWrapper.pl)

TIME : Wed Nov 25 04:22:34 2020

FUNCTION: TXK::SQLPLUS::_doExecute [ Level 3 ]

MESSAGES:

SQLPLUS error: buffer=

SQL*Plus: Release 10.1.0.5.0 – Production on Wed Nov 25 04:22:33 2020

SQL> SQL> Connected.

SQL> SELECT ad_zd_adop.get_node_type(‘mtnode1’) FROM DUAL

ERROR at line 1:

ORA-01422: exact fetch returns more than requested number of rows

ORA-06512: at “APPS.AD_ZD_ADOP”, line 3010

=======================

Cause:

FND_NODES table has incorrect node information.

Found an extra row in FND_NODES table where node_name is null after clone.

Solution:

1. Clean up FND_NODES table

sqlplus apps/<password>

sql>exec FND_CONC_CLONE.SETUP_CLEAN;

2.Run Autoconfig on Database node(s).

cd $ORACLE_HOME/appsutil/bin

adconfig.sh

[Provide Database Context file and apps Password]

3. Run Autoconfig on apps tier

Source the RUN filesystem environment file.

cd $AD_TOP/bin

adconfig.sh

[Provide Applictions Context file and apps Password]

What is Split Brain Syndrome in Oracle RAC?

Split brain syndrome occurs when the Oracle RAC nodes are unable to communicate with each other via private interconnect, but the communication between client and RAC node is maintained. This can cause data Integrity issues when the same block is read or updated by two nodes and changes done from one node are overwritten by the other node because the block being changed is not locked.

When a node fails, the failed node is prevented from accessing all the shared disk devices and groups. This methodology is called I/O Fencing, Disk Fencing or Failure Fencing.

The node which first detects that one of the node is not accessible will evict that node from the RAC cluster group.This problem is solved by configuring the heartbeat connections through the same communication channels that are used to access the clients.