How to check if a database is RAC enabled

 To check if an Oracle database is RAC (Real Application Clusters) enabled, you can use several methods. Here are the most common ways to verify if your Oracle database is running in a RAC environment:

1. Check Using srvctl (Server Control Utility)

If Oracle Grid Infrastructure is installed, the srvctl utility can provide information about the RAC configuration.

srvctl config database -d <dbname>
  • If the database is RAC-enabled, this command will show the instances that are part of the RAC environment.

  • Example output for a RAC-enabled database:

    Database name: <dbname> Database is running. Instance(s): <instance1> <instance2>

If it shows multiple instances, then the database is RAC-enabled.

2. Check with ps or top Command (Unix/Linux)

You can check if multiple instances are running on different nodes by using the ps or top command.

ps -ef | grep pmon
  • In a RAC environment, you will typically see one pmon (process monitor) for each instance of the database.

  • For example, if the database is running in RAC, you might see something like:


    oracle 1234 1 0 09:00 ? 00:00:00 ora_pmon_<inst1> oracle 5678 1 0 09:01 ? 00:00:00 ora_pmon_<inst2>

Each pmon_<inst> corresponds to an individual instance in the RAC cluster.

3. Check the v$database View

You can query the v$database view to check if the database is configured in a RAC environment.


SELECT name, open_mode, cluster_database FROM v$database;
  • If the cluster_database column returns TRUE, the database is RAC-enabled.

  • Example output:


    NAME OPEN_MODE CLUSTER_DATABASE --------- -------------- ----------------- ORCL READ WRITE TRUE

If CLUSTER_DATABASE is TRUE, this indicates the database is part of a RAC configuration.

4. Check the crsctl Command (Cluster Resource Service)

If Oracle Grid Infrastructure is used for RAC, you can also check the status of the database service using the crsctl command.


crsctl status resource -t
  • This will show the resources managed by Oracle Clusterware, including databases, listeners, and services.
  • For a RAC database, you’ll see resources like ora.<dbname>.<inst_name>, where <dbname> is your database name, and <inst_name> refers to the individual instance names in the RAC configuration.

5. Check the listener.ora File

If the database is using multiple instances in a RAC setup, the listener.ora file on each node should contain entries for all instances.


cat $ORACLE_HOME/network/admin/listener.ora

You should see multiple listener configurations for different instances in the RAC, typically defined with the SID_LIST directive.

6. Check the dbs Directory for Instance-Specific Files

If you are on the server where the Oracle RAC instances are running, you can look for individual initialization parameter files (spfile or init.ora) for each RAC instance. These files are typically located in the $ORACLE_HOME/dbs/ directory.

For example:

  • spfile<dbname>_<inst_name>.ora
  • init<dbname>_<inst_name>.ora

Each RAC instance will have its own configuration file.

7. Check the Clusterware Logs

The clusterware logs can also provide information about the RAC configuration. You can check the Oracle Grid Infrastructure logs or the Oracle alert logs for entries related to RAC.

bash
$GRID_HOME/log/<hostname>/alert*.log

These logs will contain details about RAC node registrations, instance startup, and other cluster-related events.

  • If multiple database instances are configured on different servers and share the same storage, it’s likely a RAC environment.
  • The primary indicators include the cluster_database column in v$database, multiple pmon processes, and usage of tools like srvctl and crsctl to manage resources.

Useful srvctl commands

Useful srvctl commands for ORACLE RAC MAINTENANCE

srvctl -help or srvctl -v


srvctl commands to startup database:

srvctl start database -d db_name [-o start_options] [-c connect_str|-q]
srvctl start database -d db_name [-o open]
srvctl start database -d db_name -o nomount
srvctl start database -d db_name -o mount

srvctl commands to shutdown database:

srvctl stop database -d db_name [-o stop_options] [-c connect_str|-q]
srvctl stop database -d db_name [-o normal]
srvctl stop database -d db_name -o transactional
srvctl stop database -d db_name -o immediate
srvctl stop database -d db_name -o abort


srvctl config database
srvctl config database -d db_name [-a] [-t]


srvctl start listener -n node_name [-l listener_names]
srvctl stop listener -n node_name [-l listener_names]


srvctl status listener [-n node_name] [-l listener_names] 
srvctl config listener -n node_name

What is Split Brain Syndrome in Oracle RAC?

Split brain syndrome occurs when the Oracle RAC nodes are unable to communicate with each other via private interconnect, but the communication between client and RAC node is maintained. This can cause data Integrity issues  when the same block is read or updated by two nodes and changes done from one node are overwritten by the other node because the block being changed is not locked.

When a node fails, the failed node is prevented from accessing all the shared disk devices and groups. This methodology is called I/O Fencing, Disk Fencing or Failure Fencing.

The node which first detects that one of the node is not accessible will evict that node from the RAC cluster group.This problem is solved by configuring the heartbeat connections through the same communication channels that are used to access the clients.

What causes Node eviction in Oracle RAC?

What is node eviction?

Node eviction in RAC is done when a heartbeat indicates that a node is not responding, 

and the evicted node is re-started to make it a part of cluster.


Causes for RAC node eviction:


Node eviction on Oracle RAC environment can be due to any of the below reasons. 

– A failure of any of the major hardware components (CPU, RAM, network interconnect).

– A server that is experiencing RAM swapping.

– When communications to the voting disk is interrupted, causing the disconnected node to be evicted and re-boot.

– Database or ASM hang condition.


Below is the list of important log files to review in case of a node eviction

– Clusterware alert log

– Database alert log

– CSSD agent logs

– CSSD monitor logs

– System Message logs (/var/log/messages)

Connecting to a Database from SQLDeveloper fails with IO Error: The Network Adapter Could Not Establish The Connection

Issue: New Users were trying to access a 3-node RAC Oracle Database system using SQLDeveloper


Below is the connection string provided to user for Oracle Database Connection. RAC Virtual IP is provided to connect to Database.




TESTDB01=
       (DESCRIPTION=
               (ADDRESS=(PROTOCOL=tcp)(HOST=test001-vip1.domain.com)(PORT=1521))
               (ADDRESS=(PROTOCOL=tcp)(HOST=test002-vip1.domain.com)(PORT=1521))
               (ADDRESS=(PROTOCOL=tcp)(HOST=test003-vip1.domain.com)(PORT=1521))
           (CONNECT_DATA=
               (SERVICE_NAME=TESTDB01)
               (INSTANCE_NAME=TESTDB01)
           )
       )




User tried to connect using one of the VIP and got the below error in SQLDeveloper.
Status : Failure -Test failed: IO Error: The Network Adapter Could Not Establish The Connection.


Analysis:


Verified the Oracle Database Services and Listener Services are running fine.
Other Users were able to connect to SQLDeveloper using the same Connection String.


Solution:


User does not have network communication open to port 1521 on Oracle RAC Virtual IP.


ping  test001-vip1.domain.com
ping  test002-vip1.domain.com
ping  test003-vip1.domain.com


telnet test001-vip1.domain.com 1521
telnet test002-vip1.domain.com 1521
telnet test003-vip1.domain.com 1521


Ping to VIP address works fine but Telnet to port 1521 on the VIP address is giving “Connect Failed” Error.


When Network communication is opened on port 1521 on all the Oracle RAC Virtual IP servers/Addresses, User can connect to Oracle Database using SQLDeveloper.

How to enable ArchiveLog mode in Oracle Database RAC Environment?

Steps to enable Archivelog mode in RAC environment:


The following steps need to be taken to enable archive logging in a RAC database environment:


1. Shutdown immediate all database instances
$ srvctl stop database -d <db_unique_name>


2. Startup database in mount mode
$ srvctl start database -d <db_unique_name> -o mount


3. Enable archive logging
$ sqlplus / as sysdba
sql> alter database archivelog;
sql> exit;


4. Stop database
$ srvctl stop database -d <db_unique_name>


5.Start all database instances
$ srvctl start database -d <db_unique_name>


6.Verify archiving is enabled/disabled 
sql> archive log list;