Adding and Deleting a Node from Oracle RAC 11.2.0.3

Adding and Deleting a Node from Oracle RAC 11.2.0.3

In the article you will have a look at the steps related to extending an existing Oracle cluster by adding a node to the cluster and the opposite steps to remove a node from the cluster. Oracle 11.2.0.3 is used for verifying the reconfiguration steps in the article. Two nodes Oracle cluster consisting of nodes (raclinux1, raclinux2) with specifications described here and third node raclinux3 are used as a basis for the reconfiguration test described in the article. GI is installed using non-GNS setup and the database is admin policy managed. You can find information related to the upgrade to 11.2.0.3 here. The article will cover the following topics:

Add a node
- Add GI home to the new node
- Add RDBMS home to the new node
- Add instance to the new node
Remove a node
- Remove an instance from the node to be removed
- Remove RDBMS home from the node to be removed
- Remove GI home from the node to be removed

Add a node

Before you add a node make sure that the node is properly configured to support Oracle GI and RDBMS installation and database creation. For a detail list of the prerequisites for installing Oracle GI and RDBMS click here. As RAC3 VM running raclinux3 node was cloned from RAC1 running raclinux1 the prerequisites are met. I will illustrate the Cluster Verify Utility cluvfy commands to verify the prerequisites. The existing cluster nodes are raclinux1 and raclinux2. Node raclinux3 will be added.

Verify the prerequisites with cluvfy

cluvfy stage -post hwos -n raclinux3 -verbose

cluvfy comp peer -refnode raclinux1 -n raclinux3 -orainv oinstall -osdba asmdba -verbose

cluvfy stage -pre nodeadd -n raclinux3 -verbose

cluvfy stage -pre crsinst -n raclinux3 –verbose

Fix any problems related to failed check(s) and make sure that the prerequisites are met before proceeding further. For information related to the detail output of the commands look at Annex 1

1.1 Add GI home to the new node using addNode

In order to add GI to the new node raclinux3 addNode.sh will be used from an existing node raclinux2. Run, as grid user, addNode.sh from $GI_HOME/oui/bin assuming that you have met the prerequisites. If you want to override some errors set IGNORE_PREADDNODE_CHECKS variable to Y.

export IGNORE_PREADDNODE_CHECKS=Y

[grid@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Performing pre-checks for node addition

Checking node reachability…

Node reachability check passed from node “raclinux2″

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking CRS integrity…

Clusterware version consistency passed

CRS integrity check passed

Checking shared resources…

Checking CRS home location…

“/u01/app/11.2.0.3/grid” is shared

Shared resources check for node addition passed

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity for interface “eth1″

Node connectivity passed for interface “eth1″

TCP connectivity check passed for subnet “10.10.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed for subnet “10.10.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “raclinux3:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux2:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux3:/tmp”

Free disk space check passed for “raclinux2:/tmp”

Check for multiple users with UID value 1100 passed

User existence check passed for “grid”

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check passed for “shmmax”

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “make”

Package existence check passed for “binutils”

Package existence check passed for “gcc(x86_64)”

Package existence check passed for “libaio(x86_64)”

Package existence check passed for “glibc(x86_64)”

Package existence check passed for “compat-libstdc++-33(x86_64)”

Package existence check passed for “elfutils-libelf(x86_64)”

Package existence check passed for “elfutils-libelf-devel”

Package existence check passed for “glibc-common”

Package existence check passed for “glibc-devel(x86_64)”

Package existence check passed for “glibc-headers”

Package existence check passed for “gcc-c++(x86_64)”

Package existence check passed for “libaio-devel(x86_64)”

Package existence check passed for “libgcc(x86_64)”

Package existence check passed for “libstdc++(x86_64)”

Package existence check passed for “libstdc++-devel(x86_64)”

Package existence check passed for “sysstat”

Package existence check passed for “ksh”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user’s primary group passed

Checking OCR integrity…

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration…

Oracle Cluster Voting Disk configuration check passed

Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon’s boot time configuration check for slewing option passed

NTP common Time Server Check started…

PRVF-5408 : NTP Time Server “78.47.24.68″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “192.43.244.18″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “129.69.1.153″ is common only to the following nodes “raclinux2″

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

User “grid” is not part of “root” group. Check passed

Checking consistency of file “/etc/resolv.conf” across nodes

File “/etc/resolv.conf” does not have both domain and search entries defined

domain entry in file “/etc/resolv.conf” is consistent across nodes

search entry in file “/etc/resolv.conf” is consistent across nodes

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000″ ms on following nodes: raclinux3

File “/etc/resolv.conf” is not consistent across nodes

Checking VIP configuration.

Checking VIP Subnet configuration.

Check for VIP Subnet configuration passed.

Checking VIP reachability

Check for VIP reachability passed.

Pre-check for node addition was unsuccessful on all the nodes.

[grid@raclinux2 bin]$

export IGNORE_PREADDNODE_CHECKS=Y

[grid@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9951 MB Passed

Oracle Universal Installer, Version 11.2.0.3.0 Production

Performing tests to see whether nodes raclinux1,raclinux3 are available

……………………………………………………… 100% Done.

…

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /u01/app/11.2.0.3/grid

New Nodes

Space Requirements

New Nodes

raclinux3

/u01: Required 4.30GB : Available 269.12GB

Installed Products

Product Names

Oracle Grid Infrastructure 11.2.0.3.0

Sun JDK 1.5.0.30.03

Installer SDK Component 11.2.0.3.0

Oracle One-Off Patch Installer 11.2.0.1.7

Oracle Universal Installer 11.2.0.3.0

Oracle USM Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Enterprise Manager Common Core Files 10.2.0.4.4

Oracle DBCA Deconfiguration 11.2.0.3.0

Oracle RAC Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Server) 11.2.0.3.0

Installation Plugin Files 11.2.0.3.0

Universal Storage Manager Files 11.2.0.3.0

Oracle Text Required Support Files 11.2.0.3.0

Automatic Storage Management Assistant 11.2.0.3.0

Oracle Database 11g Multimedia Files 11.2.0.3.0

Oracle Multimedia Java Advanced Imaging 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Oracle Multimedia Locator RDBMS Files 11.2.0.3.0

Oracle Core Required Support Files 11.2.0.3.0

Bali Share 1.1.18.0.0

Oracle Database Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Client) 11.2.0.3.0

Expat libraries 2.0.1.0.1

Oracle Containers for Java 11.2.0.3.0

Perl Modules 5.10.0.0.1

Secure Socket Layer 11.2.0.3.0

Oracle JDBC/OCI Instant Client 11.2.0.3.0

Oracle Multimedia Client Option 11.2.0.3.0

LDAP Required Support Files 11.2.0.3.0

Character Set Migration Utility 11.2.0.3.0

Perl Interpreter 5.10.0.0.2

PL/SQL Embedded Gateway 11.2.0.3.0

OLAP SQL Scripts 11.2.0.3.0

Database SQL Scripts 11.2.0.3.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

SSL Required Support Files for InstantClient 11.2.0.3.0

SQL*Plus Files for Instant Client 11.2.0.3.0

Oracle Net Required Support Files 11.2.0.3.0

Oracle Database User Interface 2.2.13.0.0

RDBMS Required Support Files for Instant Client 11.2.0.3.0

RDBMS Required Support Files Runtime 11.2.0.3.0

XML Parser for Java 11.2.0.3.0

Oracle Security Developer Tools 11.2.0.3.0

Oracle Wallet Manager 11.2.0.3.0

Enterprise Manager plugin Common Files 11.2.0.3.0

Platform Required Support Files 11.2.0.3.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

RDBMS Required Support Files 11.2.0.3.0

Oracle Ice Browser 5.2.3.6.0

Oracle Help For Java 4.2.9.0.0

Enterprise Manager Common Files 10.2.0.4.3

Deinstallation Tool 11.2.0.3.0

Oracle Java Client 11.2.0.3.0

Cluster Verification Utility Files 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.3.0

Oracle LDAP administration 11.2.0.3.0

Cluster Verification Utility Common Files 11.2.0.3.0

Oracle Clusterware RDBMS Files 11.2.0.3.0

Oracle Locale Builder 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Buildtools Common Files 11.2.0.3.0

Oracle RAC Required Support Files-HAS 11.2.0.3.0

SQL*Plus Required Support Files 11.2.0.3.0

XDK Required Support Files 11.2.0.3.0

Agent Required Support Files 10.2.0.4.3

Parser Generator Required Support Files 11.2.0.3.0

Precompiler Required Support Files 11.2.0.3.0

Installation Common Files 11.2.0.3.0

Required Support Files 11.2.0.3.0

Oracle JDBC/THIN Interfaces 11.2.0.3.0

Oracle Multimedia Locator 11.2.0.3.0

Oracle Multimedia 11.2.0.3.0

HAS Common Files 11.2.0.3.0

Assistant Common Files 11.2.0.3.0

PL/SQL 11.2.0.3.0

HAS Files for DB 11.2.0.3.0

Oracle Recovery Manager 11.2.0.3.0

Oracle Database Utilities 11.2.0.3.0

Oracle Notification Service 11.2.0.3.0

SQL*Plus 11.2.0.3.0

Oracle Netca Client 11.2.0.3.0

Oracle Net 11.2.0.3.0

Oracle JVM 11.2.0.3.0

Oracle Internet Directory Client 11.2.0.3.0

Oracle Net Listener 11.2.0.3.0

Cluster Ready Services Files 11.2.0.3.0

Oracle Database 11g 11.2.0.3.0

—————————————————————————–

Instantiating scripts for add node (Monday, November 14, 2011 10:56:28 PM EET)

. 1% Done.

Instantiation of add node scripts complete

Copying to remote nodes (Monday, November 14, 2011 10:56:40 PM EET)

………………………………………………………………………………….. 96% Done.

Home copied to new nodes

Saving inventory on nodes (Monday, November 14, 2011 11:15:12 PM EET)

. 100% Done.

Save inventory complete

WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.

To register the new inventory please run the script at ‘/u01/app/oraInventory/orainstRoot.sh’ with root privileges on nodes ‘raclinux3′.

If you do not register the inventory, you may not be able to update or patch the products you installed.

The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

/u01/app/oraInventory/orainstRoot.sh #On nodes raclinux3

/u01/app/11.2.0.3/grid/root.sh #On nodes raclinux3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0.3/grid was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[grid@raclinux2 bin]$

After addNode.sh completes open a new session on the node to be added raclinux3 and run the suggested scripts as root user.

[root@raclinux3 grid]# /u01/app/oraInventory/orainstRoot.sh

Creating the Oracle inventory pointer file (/etc/oraInst.loc)

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

You have new mail in /var/spool/mail/root

[root@raclinux3 grid]#

[root@raclinux3 grid]# /u01/app/11.2.0.3/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

Creating trace directory

OLR initialization – successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node raclinux1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@raclinux3 grid]#

1.2 Add a RDBMS home to the new node using addNode

After GI home has been added to node raclinux3 successfully from any existing node, for example raclinux2, run addNode.sh as oracle user.

[oracle@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Performing pre-checks for node addition

Checking node reachability…

Node reachability check passed from node “raclinux2″

Checking user equivalence…

User equivalence check passed for user “oracle”

WARNING:

Node “raclinux3″ already appears to be part of cluster

Pre-check for node addition was successful.

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9543 MB Passed

Oracle Universal Installer, Version 11.2.0.3.0 Production

Performing tests to see whether nodes raclinux1,raclinux3 are available

……………………………………………………… 100% Done.

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /u01/app/oracle/product/11.2.0/db_3

New Nodes

Space Requirements

New Nodes

raclinux3

/u01: Required 4.53GB : Available 265.96GB

Installed Products

Product Names

Oracle Database 11g 11.2.0.3.0

Sun JDK 1.5.0.30.03

Installer SDK Component 11.2.0.3.0

Oracle One-Off Patch Installer 11.2.0.1.7

Oracle Universal Installer 11.2.0.3.0

Oracle USM Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Oracle DBCA Deconfiguration 11.2.0.3.0

Oracle RAC Deconfiguration 11.2.0.3.0

Oracle Database Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Client 10.3.2.1.0

Oracle Configuration Manager 10.3.5.0.1

Oracle ODBC Driverfor Instant Client 11.2.0.3.0

LDAP Required Support Files 11.2.0.3.0

SSL Required Support Files for InstantClient 11.2.0.3.0

Bali Share 1.1.18.0.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

Oracle Real Application Testing 11.2.0.3.0

Oracle Database Vault J2EE Application 11.2.0.3.0

Oracle Label Security 11.2.0.3.0

Oracle Data Mining RDBMS Files 11.2.0.3.0

Oracle OLAP RDBMS Files 11.2.0.3.0

Oracle OLAP API 11.2.0.3.0

Platform Required Support Files 11.2.0.3.0

Oracle Database Vault option 11.2.0.3.0

Oracle RAC Required Support Files-HAS 11.2.0.3.0

SQL*Plus Required Support Files 11.2.0.3.0

Oracle Display Fonts 9.0.2.0.0

Oracle Ice Browser 5.2.3.6.0

Oracle JDBC Server Support Package 11.2.0.3.0

Oracle SQL Developer 11.2.0.3.0

Oracle Application Express 11.2.0.3.0

XDK Required Support Files 11.2.0.3.0

RDBMS Required Support Files for Instant Client 11.2.0.3.0

SQLJ Runtime 11.2.0.3.0

Database Workspace Manager 11.2.0.3.0

RDBMS Required Support Files Runtime 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Exadata Storage Server 11.2.0.1.0

Provisioning Advisor Framework 10.2.0.4.3

Enterprise Manager Database Plugin — Repository Support 11.2.0.3.0

Enterprise Manager Repository Core Files 10.2.0.4.4

Enterprise Manager Database Plugin — Agent Support 11.2.0.3.0

Enterprise Manager Grid Control Core Files 10.2.0.4.4

Enterprise Manager Common Core Files 10.2.0.4.4

Enterprise Manager Agent Core Files 10.2.0.4.4

RDBMS Required Support Files 11.2.0.3.0

regexp 2.1.9.0.0

Agent Required Support Files 10.2.0.4.3

Oracle 11g Warehouse Builder Required Files 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.3.0

Oracle Text Required Support Files 11.2.0.3.0

Parser Generator Required Support Files 11.2.0.3.0

Oracle Database 11g Multimedia Files 11.2.0.3.0

Oracle Multimedia Java Advanced Imaging 11.2.0.3.0

Oracle Multimedia Annotator 11.2.0.3.0

Oracle JDBC/OCI Instant Client 11.2.0.3.0

Oracle Multimedia Locator RDBMS Files 11.2.0.3.0

Precompiler Required Support Files 11.2.0.3.0

Oracle Core Required Support Files 11.2.0.3.0

Sample Schema Data 11.2.0.3.0

Oracle Starter Database 11.2.0.3.0

Oracle Message Gateway Common Files 11.2.0.3.0

Oracle XML Query 11.2.0.3.0

XML Parser for Oracle JVM 11.2.0.3.0

Oracle Help For Java 4.2.9.0.0

Installation Plugin Files 11.2.0.3.0

Enterprise Manager Common Files 10.2.0.4.3

Expat libraries 2.0.1.0.1

Deinstallation Tool 11.2.0.3.0

Oracle Quality of Service Management (Client) 11.2.0.3.0

Perl Modules 5.10.0.0.1

JAccelerator (COMPANION) 11.2.0.3.0

Oracle Containers for Java 11.2.0.3.0

Perl Interpreter 5.10.0.0.2

Oracle Net Required Support Files 11.2.0.3.0

Secure Socket Layer 11.2.0.3.0

Oracle Universal Connection Pool 11.2.0.3.0

Oracle JDBC/THIN Interfaces 11.2.0.3.0

Oracle Multimedia Client Option 11.2.0.3.0

Oracle Java Client 11.2.0.3.0

Character Set Migration Utility 11.2.0.3.0

Oracle Code Editor 1.2.1.0.0I

PL/SQL Embedded Gateway 11.2.0.3.0

OLAP SQL Scripts 11.2.0.3.0

Database SQL Scripts 11.2.0.3.0

Oracle Locale Builder 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

SQL*Plus Files for Instant Client 11.2.0.3.0

Required Support Files 11.2.0.3.0

Oracle Database User Interface 2.2.13.0.0

Oracle ODBC Driver 11.2.0.3.0

Oracle Notification Service 11.2.0.3.0

XML Parser for Java 11.2.0.3.0

Oracle Security Developer Tools 11.2.0.3.0

Oracle Wallet Manager 11.2.0.3.0

Cluster Verification Utility Common Files 11.2.0.3.0

Oracle Clusterware RDBMS Files 11.2.0.3.0

Oracle UIX 2.2.24.6.0

Enterprise Manager plugin Common Files 11.2.0.3.0

HAS Common Files 11.2.0.3.0

Precompiler Common Files 11.2.0.3.0

Installation Common Files 11.2.0.3.0

Oracle Help for the Web 2.0.14.0.0

Oracle LDAP administration 11.2.0.3.0

Buildtools Common Files 11.2.0.3.0

Assistant Common Files 11.2.0.3.0

Oracle Recovery Manager 11.2.0.3.0

PL/SQL 11.2.0.3.0

Generic Connectivity Common Files 11.2.0.3.0

Oracle Database Gateway for ODBC 11.2.0.3.0

Oracle Programmer 11.2.0.3.0

Oracle Database Utilities 11.2.0.3.0

Enterprise Manager Agent 10.2.0.4.3

SQL*Plus 11.2.0.3.0

Oracle Netca Client 11.2.0.3.0

Oracle Multimedia Locator 11.2.0.3.0

Oracle Call Interface (OCI) 11.2.0.3.0

Oracle Multimedia 11.2.0.3.0

Oracle Net 11.2.0.3.0

Oracle XML Development Kit 11.2.0.3.0

Database Configuration and Upgrade Assistants 11.2.0.3.0

Oracle JVM 11.2.0.3.0

Oracle Advanced Security 11.2.0.3.0

Oracle Internet Directory Client 11.2.0.3.0

Oracle Enterprise Manager Console DB 11.2.0.3.0

HAS Files for DB 11.2.0.3.0

Oracle Net Listener 11.2.0.3.0

Oracle Text 11.2.0.3.0

Oracle Net Services 11.2.0.3.0

Oracle Database 11g 11.2.0.3.0

Oracle OLAP 11.2.0.3.0

Oracle Spatial 11.2.0.3.0

Oracle Partitioning 11.2.0.3.0

Enterprise Edition Options 11.2.0.3.0

—————————————————————————–

Instantiating scripts for add node (Tuesday, November 15, 2011 12:36:09 AM EET)

. 1% Done.

Instantiation of add node scripts complete

Copying to remote nodes (Tuesday, November 15, 2011 12:36:23 AM EET)

………………………………………………………………………………….. 96% Done.

Home copied to new nodes

Saving inventory on nodes (Tuesday, November 15, 2011 12:58:10 AM EET)

. 100% Done.

Save inventory complete

WARNING:

The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

/u01/app/oracle/product/11.2.0/db_3/root.sh #On nodes raclinux3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_3 was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[oracle@raclinux2 bin]$

Run as root on the new node raclinux3

[root@raclinux3 bin]# /u01/app/oracle/product/11.2.0/db_3/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_3

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Finished product-specific root actions.

[root@raclinux3 bin]#

Add an instance on raclinux3 using dbca

Login as oracle on a cluster node where there is a running instance, for example raclinux2, and run dbca.

Select Oracle RAC database and press Next.

Select Instance management and press Next.

Select Add Instance.

Select a database and enter username and credentials.

Press Next.

Enter Instance name, select node and press Next.

Press Next.

Review and press OK.

Wait until dbca completes.
Verification.

Verify that instance has been added successfully.

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

———– ————————————————————

1 raclinux1.gj.com:RACDB1

2 raclinux2.gj.com:RACDB2

3 raclinux3.gj.com:RACDB3

SQL>

[grid@raclinux3 grid]$ olsnodes -n -t

raclinux1 1 Pinned

raclinux2 2 Pinned

raclinux3 3 Unpinned

[grid@raclinux3 grid]$

Run the following command as grid user.

cluvfy stage -post nodeadd -n raclinux3 –verbose

The output of the command is in Annex 1.

Remove a node

In order to remove a node raclinux3 from the cluster the following ordered activities should take place:

Remove instance RACDB3 on raclinux3
Remove RDBMS home from raclinux3
Remove GI home from aclinux3

2.1 Remove instance RACDB3 on raclinux3

Select Oracle RAC database and press Next.

Select Instance Management.

Select Delete an Instance.

Select a database and enter username and credentials.

Select the instance

Press OK.

Wait until instance removal takes place.

Verify that the instance is removed from a node that is to remain.

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

———– ————————————————————

1 raclinux1.gj.com:RACDB1

2 raclinux2.gj.com:RACDB2

SQL>

Remove RDBMS home from raclinux3

On a node to be removed log in as oracle user and run the following command from $ORACLE_HOME/oui/bin.

[oracle@raclinux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3 “CLUSTER_NODES={raclinux3}” -local

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9832 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[oracle@raclinux3 bin]$

Run from deinstall with option local from $ORACLE_HOME/deinstall from the node to be removed (raclinux3) in order to remove the RDBMS binaries.

[oracle@raclinux3 deinstall]$ ./deinstall -local

Checking for required files and bootstrapping …

Please wait …

Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/oracle/product/11.2.0/db_3

Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database

Oracle Base selected for deinstall is: /u01/app/oracle

Checking for existence of central inventory location /u01/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.3/grid

The following nodes are part of this cluster: raclinux3

Checking for sufficient temp space availability on node(s) : ‘raclinux3′

## [END] Install check configuration ##

Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2011-11-15_02-05-17-AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2011-11-15_02-05-24-AM.log

Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_check2011-11-15_02-05-28-AM.log

Enterprise Manager Configuration Assistant END

Oracle Configuration Manager check START

OCM check log file location : /u01/app/oraInventory/logs//ocm_check8370.log

Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /u01/app/11.2.0.3/grid

The cluster node(s) on which the Oracle home deinstallation will be performed are:raclinux3

Since -local option has been specified, the Oracle home will be deinstalled only on the local node, ‘raclinux3′, and the global configuration will be removed.

Oracle Home selected for deinstall is: /u01/app/oracle/product/11.2.0/db_3

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

The option -local will not modify any database configuration for this Oracle home.

No Enterprise Manager configuration to be updated for any database(s)

No Enterprise Manager ASM targets to update

No Enterprise Manager listener targets to migrate

Checking the config status for CCR

Oracle Home exists with CCR directory, but CCR is not configured

CCR check is finished

Do you want to continue (y – yes, n – no)? [n]: y

A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2011-11-15_02-04-55-AM.out’

Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2011-11-15_02-04-55-AM.err’

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_clean2011-11-15_02-05-28-AM.log

Updating Enterprise Manager ASM targets (if any)

Updating Enterprise Manager listener targets (if any)

Enterprise Manager Configuration Assistant END

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2011-11-15_02-06-10-AM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2011-11-15_02-06-10-AM.log

De-configuring Local Net Service Names configuration file…

Local Net Service Names configuration file de-configured successfully.

De-configuring backup files…

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START

OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean8370.log

Oracle Configuration Manager clean END

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/oracle/product/11.2.0/db_3′ from the central inventory on the local node : Done

Delete directory ‘/u01/app/oracle/product/11.2.0/db_3′ on the local node : Done

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on local node. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2011-11-15_02-04-50AM’ on node ‘raclinux3′

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Cleaning the config for CCR

As CCR is not configured, so skipping the cleaning of CCR configuration

CCR clean is finished

Successfully detached Oracle home ‘/u01/app/oracle/product/11.2.0/db_3′ from the central inventory on the local node.

Successfully deleted directory ‘/u01/app/oracle/product/11.2.0/db_3′ on the local node.

Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[oracle@raclinux3 deinstall]$

From any node that will remain, for example raclinux2, run the following command from $ORACLE_HOME/oui/bin while logged in as oracle user.

[oracle@raclinux2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3 “CLUSTER_NODES={raclinux1, raclinux2}”

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9579 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[oracle@raclinux2 bin]$

Remove GI home from raclinux3

2.3.1 While logged in as root unpin the node to be removed from any node.

[root@raclinux3 bin]# ./crsctl unpin css -n raclinux3

CRS-4667: Node raclinux3 successfully unpinned.

[root@raclinux3 bin]#

2.3.2 As root from the node to be removed raclinux3 go to $GI_HOME/crs/install and execute the following command ./rootcrs.pl -deconfig -deinstall –force.

[root@raclinux3 install]# ./rootcrs.pl -deconfig -deinstall -force

Using configuration parameter file: ./crsconfig_params

Network exists: 1/192.168.20.0/255.255.255.0/eth0, type static

VIP exists: /raclinux1-vip/192.168.20.51/192.168.20.0/255.255.255.0/eth0, hosting node raclinux1

VIP exists: /raclinux2-vip/192.168.20.52/192.168.20.0/255.255.255.0/eth0, hosting node raclinux2

VIP exists: /raclinux3-vip/192.168.20.53/192.168.20.0/255.255.255.0/eth0, hosting node raclinux3

GSD exists

ONS exists: Local port 6100, remote port 6200, EM port 2016

CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.registry.acfs’ on ‘raclinux3′ succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘raclinux3′

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.DATADG.dg’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.DATADG.dg’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.DATA.dg’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.asm’ on ‘raclinux3′ succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘raclinux3′ has completed

CRS-2677: Stop of ‘ora.crsd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.crf’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.asm’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.evmd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.mdnsd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.crf’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.asm’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.cssd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.gipcd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.gpnpd’ on ‘raclinux3′ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘raclinux3′ has completed

CRS-4133: Oracle High Availability Services has been stopped.

Successfully deconfigured Oracle clusterware stack on this node

[root@raclinux3 install]#

2.3.3 Execute as root from a node that is to remain the following command.

[root@raclinux2 bin]# ./crsctl delete node -n raclinux3

CRS-4661: Node raclinux3 successfully deleted.

[root@raclinux2 bin]#

2.3.4 From the node to be deleted log in as grid and from $GI_HOME/oui/bin execute the following command.

[grid@raclinux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0.3/grid “CLUSTER_NODES={raclinux3}” CRS=TRUE -local

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[grid@raclinux3 bin]$

2.3.5 From the node to be deleted log in as grid and from $GI_HOME/deinstall the following deinstall command with a local option and follow the requests.

./deinstall -local

[grid@raclinux3 deinstall]$ ./deinstall -local

Checking for required files and bootstrapping …

Please wait …

Location of logs /tmp/deinstall2011-11-15_02-38-50AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/11.2.0.3/grid

Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster

Oracle Base selected for deinstall is: /u01/app/grid

Checking for existence of central inventory location /u01/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home

The following nodes are part of this cluster: raclinux3

Checking for sufficient temp space availability on node(s) : ‘raclinux3′

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2011-11-15_02-38-50AM/logs//crsdc.log

Enter an address or the name of the virtual IP used on node “raclinux3″[raclinux3-vip]

192.168.20.53

The following information can be collected by running “/sbin/ifconfig -a” on node “raclinux3″

Enter the IP netmask of Virtual IP “192.168.20.53″ on node “raclinux3″[255.255.255.0]

Enter the network interface name on which the virtual IP address “192.168.20.53″ is active

Enter an address or the name of the virtual IP[]

192.168.20.53

The following information can be collected by running “/sbin/ifconfig -a” on node “raclinux3″

Enter the IP netmask of the virtual IP “192.168.20.53″[]

255.255.255.0

Enter the network interface name on which the virtual IP address “192.168.20.53″ is active

eth0

Enter an address or the name of the virtual IP[]

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/netdc_check2011-11-15_02-40-55-AM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/asmcadc_check2011-11-15_02-41-00-AM.log

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is:

The cluster node(s) on which the Oracle home deinstallation will be performed are:raclinux3

Since -local option has been specified, the Oracle home will be deinstalled only on the local node, ‘raclinux3′, and the global configuration will be removed.

Oracle Home selected for deinstall is: /u01/app/11.2.0.3/grid

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

Following RAC listener(s) will be de-configured: LISTENER

Option -local will not modify any ASM configuration.

Do you want to continue (y – yes, n – no)? [n]: y

A log of this session will be written to: ‘/tmp/deinstall2011-11-15_02-38-50AM/logs/deinstall_deconfig2011-11-15_02-39-29-AM.out’

Any error messages from this session will be written to: ‘/tmp/deinstall2011-11-15_02-38-50AM/logs/deinstall_deconfig2011-11-15_02-39-29-AM.err’

######################## CLEAN OPERATION START ########################

ASM de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/asmcadc_clean2011-11-15_02-41-03-AM.log

ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/netdc_clean2011-11-15_02-41-03-AM.log

De-configuring RAC listener(s): LISTENER

De-configuring listener: LISTENER

Stopping listener on node “raclinux3″: LISTENER

Warning: Failed to stop listener. Listener may not be running.

Listener de-configured successfully.

De-configuring Naming Methods configuration file…

Naming Methods configuration file de-configured successfully.

De-configuring backup files…

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

—————————————->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node “raclinux3″.

/tmp/deinstall2011-11-15_02-38-50AM/perl/bin/perl -I/tmp/deinstall2011-11-15_02-38-50AM/perl/lib -I/tmp/deinstall2011-11-15_02-38-50AM/crs/install /tmp/deinstall2011-11-15_02-38-50AM/crs/install/rootcrs.pl -force -deconfig -paramfile ”/tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp”

Press Enter after you finish running the above commands

<—————————————-

Remove the directory: /tmp/deinstall2011-11-15_02-38-50AM on node:

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/11.2.0.3/grid’ from the central inventory on the local node : Done

Delete directory ‘/u01/app/11.2.0.3/grid’ on the local node : Done

Delete directory ‘/u01/app/oraInventory’ on the local node : Done

Delete directory ‘/u01/app/grid’ on the local node : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2011-11-15_02-38-50AM’ on node ‘raclinux3′

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Following RAC listener(s) were de-configured successfully: LISTENER

Oracle Clusterware is stopped and successfully de-configured on node “raclinux3″

Oracle Clusterware is stopped and de-configured successfully.

Successfully detached Oracle home ‘/u01/app/11.2.0.3/grid’ from the central inventory on the local node.

Successfully deleted directory ‘/u01/app/11.2.0.3/grid’ on the local node.

Successfully deleted directory ‘/u01/app/oraInventory’ on the local node.

Successfully deleted directory ‘/u01/app/grid’ on the local node.

Oracle Universal Installer cleanup was successful.

Run ‘rm -rf /etc/oraInst.loc’ as root on node(s) ‘raclinux3′ at the end of the session.

Run ‘rm -rf /opt/ORCLfmap’ as root on node(s) ‘raclinux3′ at the end of the session.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[grid@raclinux3 deinstall]$

[root@raclinux3 grid]# /tmp/deinstall2011-11-15_02-38-50AM/perl/bin/perl -I/tmp/deinstall2011-11-15_02-38-50AM/perl/lib -I/tmp/deinstall2011-11-15_02-38-50AM/crs/install /tmp/deinstall2011-11-15_02-38-50AM/crs/install/rootcrs.pl -force -deconfig -paramfile ”/tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp”

Using configuration parameter file: /tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp

****Unable to retrieve Oracle Clusterware home.

Start Oracle Clusterware stack and try again.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Modify failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Delete failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

################################################################

# You must kill processes or reboot the system to properly #

# cleanup the processes started by Oracle clusterware #

################################################################

ACFS-9313: No ADVM/ACFS installation detected.

Either /etc/oracle/olr.loc does not exist or is not readable

Make sure the file exists and it has read and execute access

Either /etc/oracle/olr.loc does not exist or is not readable

Make sure the file exists and it has read and execute access

Failure in execution (rc=-1, 256, No such file or directory) for command /etc/init.d/ohasd deinstall

error: package cvuqdisk is not installed

Successfully deconfigured Oracle clusterware stack on this node

[root@raclinux3 grid]#

2.3.6 From any remaining node, for example raclinux2, while logged in as grid user from $GI_HOME/oui/bin execute the following command.

[grid@raclinux2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0.3/grid “CLUSTER_NODES={raclinux1, raclinux2}” CRS=TRUE

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9581 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[grid@raclinux2 bin]$

2.3.7 Verify that a node has been removed.

[grid@raclinux2 bin]$ olsnodes -n -t

raclinux1 1 Pinned

raclinux2 2 Pinned

[grid@raclinux2 bin]$

cluvfy stage -post nodedel -n raclinux3

[grid@raclinux2 bin]$ cluvfy stage -post nodedel -n raclinux3

Performing post-checks for node removal

Checking CRS integrity…

Clusterware version consistency passed

CRS integrity check passed

Node removal check passed

Post-check for node removal was successful.

[grid@raclinux2 bin]$

Annex 1

[grid@raclinux2 ~]$ cluvfy stage -post hwos -n raclinux3 -verbose

Performing post-checks for hardware and operating system setup

Checking node reachability…

Check: Node reachability from node “raclinux2″

Destination Node Reachable?

———————————— ————————

raclinux3 yes

Result: Node reachability check passed from node “raclinux2″

Checking user equivalence…

Check: User equivalence for user “grid”

Node Name Status

———————————— ————————

raclinux3 passed

Result: User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Node Name Status

———————————— ————————

raclinux3 passed

Verification of the hosts config file successful

Interface information for node “raclinux3″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.23 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth1 10.10.20.23 10.10.20.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth2 192.168.156.103 192.168.156.0 0.0.0.0 UNKNOWN 08:00:27:8C:A0:9B 1500

eth3 192.168.2.24 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:E5:70:A8 1500

Check: Node connectivity of subnet “192.168.20.0″

Result: Node connectivity passed for subnet “192.168.20.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.20.23 passed

Result: TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity of subnet “10.10.20.0″

Result: Node connectivity passed for subnet “10.10.20.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “10.10.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:10.10.20.23 passed

Result: TCP connectivity check passed for subnet “10.10.20.0″

Check: Node connectivity of subnet “192.168.156.0″

Result: Node connectivity passed for subnet “192.168.156.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.156.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.156.103 passed

Result: TCP connectivity check passed for subnet “192.168.156.0″

Check: Node connectivity of subnet “192.168.2.0″

Result: Node connectivity passed for subnet “192.168.2.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.2.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.2.24 passed

Result: TCP connectivity check passed for subnet “192.168.2.0″

Interfaces found on subnet “192.168.20.0″ that are likely candidates for a private interconnect are:

raclinux3 eth0:192.168.20.23

Interfaces found on subnet “10.10.20.0″ that are likely candidates for a private interconnect are:

raclinux3 eth1:10.10.20.23

Interfaces found on subnet “192.168.156.0″ that are likely candidates for a private interconnect are:

raclinux3 eth2:192.168.156.103

Interfaces found on subnet “192.168.2.0″ that are likely candidates for a private interconnect are:

raclinux3 eth3:192.168.2.24

WARNING:

Could not find a suitable set of interfaces for VIPs

Result: Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “192.168.156.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.156.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “192.168.2.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.2.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking for multiple users with UID value 0

Result: Check for multiple users with UID value 0 passed

Check: Time zone consistency

Result: Time zone consistency check passed

Checking shared storage accessibility…

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sda raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdb raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdc raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdd raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sde raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdf raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdg raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdh raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdi raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdj raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdk raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdl raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdm raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdn raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdo raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdp raclinux3

Shared storage check was successful on nodes “raclinux3″

Post-check for hardware and operating system setup was successful.

[grid@raclinux2 ~]$

[grid@raclinux2 ~]$ cluvfy comp peer -refnode raclinux1 -n raclinux3 -orainv oinstall -osdba asmdba -verbose

Verifying peer compatibility

Checking peer compatibility…

Compatibility check: Physical memory [reference node: raclinux1]