Guenadi N Jilevski's Oracle BLOG

Fix ‘failed.javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated’ in OEM DC Console

gjilevski — Tue, 10 Jan 2012 11:15:52 +0000

Fix ‘failed.javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated’ in OEM DC Console

OEM Database control (DC) was showing ‘Connection as user grid failed. javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated’ on any occasion I was asked to specify the host credentials in OEM DC – part of Oracle 11.2 installation on OEL 5.4 x86_64.

It turned out that SSL certificate or emkey was corrupted and DB Console GUI was not unable to connect to its agent and verify the credentials (MOS 1157069.1).

Fixing the problem required re-securing the console as shown below.

Stop the console

[oracle@raclinux1 ~]$ emctl stop dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.

https://raclinux1.gj.com:1158/em/console/aboutApplication

Stopping Oracle Enterprise Manager 11g Database Control …

… Stopped.

[oracle@raclinux1 ~]$
Unsecure the console

[oracle@raclinux1 config]$ emctl unsecure dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.

https://raclinux1.gj.com:1158/em/console/aboutApplication

Configuring DBConsole for HTTP… Done.

DBCONSOLE already stopped… Done.

Agent is already stopped… Done.

Unsecuring dbconsole… Started.

DBConsole is now unsecured… Done.

Unsecuring dbconsole… Sucessful.

[oracle@raclinux1 config]$
Secure the console

[oracle@raclinux1 config]$ emctl secure dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.

http://raclinux1.gj.com:1158/em/console/aboutApplication

Please enter repository password:

DBCONSOLE already stopped… Done.

Agent is already stopped… Done.

Securing dbconsole… Started.

Please enter repository schema password:

Securing dbconsole… Sucessful.

[oracle@raclinux1 config]$
Start the console

[oracle@raclinux1 ~]$ emctl start dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.

https://raclinux1.gj.com:1158/em/console/aboutApplication

Starting Oracle Enterprise Manager 11g Database Control …………………….. started.

——————————————————————

Logs are generated in directory /u01/app/oracle/product/11.2.0/db_3/raclinux1_RACDB/sysman/log

[oracle@raclinux1 ~]$

Filed under: oracle

Build Active-Passive HA configuration for single instance database with Oracle GI 11.2.0.3

gjilevski — Mon, 09 Jan 2012 20:41:21 +0000

Build Active-Passive HA configuration for single instance database with Oracle GI 11.2.0.3

In this article you will have a look at the available options to protect a single instance database using Active/Passive configuration implemented using Oracle GI 11.2.0.3. Staring with Oracle 10.2, Oracle Clusterware provided ability to protect any third party application in a cluster by deploying Active/Passive configurations whereby only one node is active at a time and the remaining nodes are available for the active node to failover in case of the currently active node failure. For information related to building High Availability for third party applications with Oracle GI 11.2.0.3 click here. In the article you will find out how to use the GI 11.2.0.3 to set up an Active/Passive configuration for a single instance database. The following options will be covered:

Active/Passive configuration for single instance database with a floating VIP
Active/Passive configuration for single instance database without a floating VIP (SCAN only)
Oracle RAC One Node

Historically, cluster solutions such as HACMP, Veritas cluster, Sun cluster etc… were used to implement Active/Passive HA and failover configurations for single instance database where at any point in time only one node is running a database, listener and floating application VIP and upon node failure the applications failover to another cluster node. For accessibility to the database a floating application VIP was used to maintain the same IP regardless of the node of the cluster where Oracle database is running. Starting with 11.2 Oracle introduced the SCAN VIP that can be used to access the cluster and simplifies the configuration eliminating the Application VIP and listener failover.

Starting with Oracle 11.2 Oracle RAC One Node concept was introduced that is a standard out-off-the box single instance HA solution that can provide an Active/Passive configuration in an Oracle cluster. Oracle RAC One Node,
an additionally licensed option to the Oracle Database Enterprise Edition, is a fully supported and integrated Oracle Database failover solution running RAC enabled single instance database on one cluster node. According to MOS Note 790189.1,
Oracle RAC One Node is the preferred and recommended solution for failing over an Oracle 11g Release 2 Database going forward. Oracle continues to support alternative ways of failing over an Oracle Database but will not support action scripts and will not provide further white papers for on the matter of HA and failover configurations of a single instance Oracle databases going forward.

For information about Oracle RAC One Node installation and management click here.

The article will emphasize on building HA Active/Passive solution for database failover. In a nut shell, Oracle GI 11.2.0.3 manages resources. Database, listener and an application VIP need to be registered as cluster resources with respective profile attributes determining the restart and failover policies.

The environment used in the article is based on three (3) node OEL 5 cluster using Oracle 11.2.0.3 as described here. The Active/Passive configuration will have one active and two passive standby servers.

Active/Passive configuration for single instance database without a floating VIP (SCAN only)

In this section you will look at how to register a single instance database as a resource under GI 11.2.0.3 and implement an Active/Passive configuration. A SCAN will be used to access the database and there will be no floating application VIP for access to the database. The database will be the only resource that will need to be register for management under GI 11.2.0.3.

The database needs to be on a shared storage. For simplicity I installed a single instance database on shared ASM disk group DATA and DATADG using dbca.
Create directories for ADRCI and audit on each cluster node where the database is supposed to run or failover.

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/alert

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/cdump

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/hm

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/incident

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/incpkg

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/ir

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/lck

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata_dgif

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata_pv

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/stage

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/sweep

mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/trace

mkdir -p /u01/app/oracle/admin/DB11GR2/adump
Modify /etc/oratab on each cluster node where that database is supposed to run and include the following line.

DB11GR2:/u01/app/oracle/product/11.2.0/db_3:N
Create a pfile on each cluster node in $OH/dbs pointing to the SPFILE on ASM.

SQL> show parameter spfile

NAME TYPE VALUE

———————————— ———– ——————————

spfile string +DATA/db11gr2/spfiledb11gr2.ora

[oracle@raclinux2 dbs]$ cat initDB11GR2.ora

SPFILE=’+DATA/DB11GR2/spfileDB11GR2.ora’

[oracle@raclinux2 dbs]$

[oracle@raclinux2 dbs]$ pwd

/u01/app/oracle/product/11.2.0/db_3/dbs

[oracle@raclinux2 dbs]$ scp initDB11GR2.ora raclinux1:/u01/app/oracle/product/11.2.0/db_3/dbs/

initDB11GR2.ora 100% 41 0.0KB/s 00:00

[oracle@raclinux2 dbs]$ scp initDB11GR2.ora raclinux3:/u01/app/oracle/product/11.2.0/db_3/dbs/

initDB11GR2.ora 100% 41 0.0KB/s 00:00

[oracle@raclinux2 dbs]$
Create actions scripts on each node of the cluster in directory /u01/app/11.2.0.3/grid/crs/public. The main script is db11gr2_actionScript.pl calling actionstart.pl, actionStop.pl and actionClean.pl.

[root@raclinux1 public]# cat db11gr2_actionScript.pl

#!/usr/bin/perl

if ($#ARGV != 0) {

# This is an unreacheable condition

# Oracle Clusterware would always invoke with a valid argument

exit;

}

$command = $ARGV[0];

# Start command

if ($command eq “start”) {

system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionStart.pl\”");

# steps to start the resource

exit 0; #if the start is successfull

# exit 1 if the start has failed

}

# Stop command

if ($command eq “stop”) {

system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionStop.pl\”");

# steps to stop the resource

exit 0; # if the stop is successfull

# exit 1 if the stop has failed

}

# Check command

if ($command eq “check”) {

$found = qx(ps -ef | grep smon | grep DB11GR2 | grep -v grep | wc -l);

#$found = ‘ps -ef | grep smon | grep DB11GR2 | wc -l ‘;

chomp($found);

#print($found);

if ($found eq “0″ ) {exit 1;} else {exit 0; }

# steps to check the status of the resource

# exit 0 if the resource is online

# exit 1 otherwise

}

# Clean command

if ($command eq “clean”) {

system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionClean.pl\”");

# steps to force stop the resource

exit 0; # if the force stop is successfull

# exit 1 if the force stop has failed

}

[root@raclinux1 public]#

[root@raclinux1 public]# cat actionStart.pl

#!/usr/bin/perl

# Stop command

system(“export ORACLE_SID=DB11GR2

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

/u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

connect / as sysdba

startup

exit

EOF”);

exit 0; # if the stop is successfull

[root@raclinux1 public]#

[root@raclinux1 public]# cat actionStop.pl

#!/usr/bin/perl

# Stop command

system(“export ORACLE_SID=DB11GR2

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

/u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

connect / as sysdba

shutdown immediate

exit

EOF”);

exit 0; # if the stop is successfull

[root@raclinux1 public]# cat actionClean.pl

#!/usr/bin/perl

# Stop command

system(“export ORACLE_SID=DB11GR2

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

/u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

connect / as sysdba

shutdown immediate

exit

EOF”);

exit 0; # if the stop is successfull

[root@raclinux1 public]#
Test the actionscript to make sure that they work

[root@raclinux2 public]# ./db11gr2_actionScript.pl check

[root@raclinux2 public]# echo $?

0

[root@raclinux2 public]#

[root@raclinux2 public]# ./db11gr2_actionScript.pl stop

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 8 18:23:00 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

SQL> Connected.

SQL> Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

[root@raclinux2 public]# ./db11gr2_actionScript.pl check

[root@raclinux2 public]# echo $?

1

[root@raclinux2 public]#

[root@raclinux2 public]# ./db11gr2_actionScript.pl start

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 8 18:23:51 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

SQL> Connected to an idle instance.

SQL> ORACLE instance started.

Total System Global Area 626327552 bytes

Fixed Size 2230952 bytes

Variable Size 452986200 bytes

Database Buffers 167772160 bytes

Redo Buffers 3338240 bytes

Database mounted.

Database opened.

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

[root@raclinux2 public]# ./db11gr2_actionScript.pl check

[root@raclinux2 public]# echo $?

0

[root@raclinux2 public]#

Create the resource as root. Make sure that you specify correct full path for the action script. Attributes worth noting a bolded. The resource is planned to run on the list of nodes specified by HOSTING_MEMBERS. Placement policy is restricted. The resource will failover to another node if restarted more than once as RESTART_ATTEMPTS=1. The resource will run on only one cluster node at any given point in time CARDINALITY=1.

[root@raclinux2 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3,DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=, STOP_DEPENDENCIES=,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

[root@raclinux2 bin]#
Set the resource permissions

[root@raclinux2 bin]# ./crsctl getperm resource db11gr2

Name: db11gr2

owner:root:rwx,pgrp:root:r-x,other::r–

[root@raclinux2 bin]#

[root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:oracle:rwx

[root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:grid:rwx

[root@raclinux2 bin]#

[root@raclinux2 bin]# ./crsctl getperm resource db11gr2

Name: db11gr2

owner:root:rwx,pgrp:root:r-x,other::r–,user:oracle:rwx,user:grid:rwx

[root@raclinux2 bin]#
Test and start the resource.

[root@raclinux1 bin]# ./crsctl start res db11gr2

CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux1′

CRS-2676: Start of ‘db11gr2′ on ‘raclinux1′ succeeded

[root@raclinux1 bin]#

[root@raclinux3 bin]# ./crsctl relocate res db11gr2

CRS-2673: Attempting to stop ‘db11gr2′ on ‘raclinux1′

CRS-2677: Stop of ‘db11gr2′ on ‘raclinux1′ succeeded

CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux2′

CRS-2676: Start of ‘db11gr2′ on ‘raclinux2′ succeeded

You have new mail in /var/spool/mail/root

[root@raclinux3 bin]#

[root@raclinux3 bin]# ./crsctl relocate res db11gr2 -s raclinux2 -n raclinux3

CRS-2673: Attempting to stop ‘db11gr2′ on ‘raclinux2′

CRS-2677: Stop of ‘db11gr2′ on ‘raclinux2′ succeeded

CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux3′

CRS-2676: Start of ‘db11gr2′ on ‘raclinux3′ succeeded

[root@raclinux3 bin]#
Make sure that remote_listener register with the scan listeners.

SQL> alter system set remote_listener=’rac-scan:1521′ scope=both;

System altered.

SQL>
Change the tnsnames.ora alias to use scan listener.

From

DB11GR2 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = DB11GR2)

)

)

To

DB11GR2 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = DB11GR2)

)

)
Test node failure

Before Node crash

[root@raclinux1 bin]# ./crsctl stat res db11gr2 -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Cluster Resources

——————————————————————————–

db11gr2

1 ONLINE ONLINE raclinux3

[root@raclinux1 bin]#

After Node Crash

[root@raclinux1 bin]# ./crsctl stat res db11gr2 -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Cluster Resources

——————————————————————————–

db11gr2

1 ONLINE ONLINE raclinux1

[root@raclinux1 bin]#
Test process death. Kill pmon and observe the RESTAR_COUNT, FAILURE_COUNT and the STATE. Oracle GI restarts ones and on the second failure fails over to available cluster node. Number of restart prior to failover can be modified by changing RESTART_ATTEMPTS value.

[root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

NAME=db11gr2

TYPE=cluster_resource

LAST_SERVER=raclinux1

STATE=ONLINE on raclinux1

TARGET=ONLINE

CARDINALITY_ID=1

CREATION_SEED=400

RESTART_COUNT=0

FAILURE_COUNT=0

FAILURE_HISTORY=

ID=db11gr2 1 1

INCARNATION=1

LAST_RESTART=01/08/2012 23:58:36

LAST_STATE_CHANGE=01/08/2012 23:58:36

STATE_DETAILS=

INTERNAL_STATE=STABLE

[root@raclinux1 bin]#

kill -9 pid

[root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

NAME=db11gr2

TYPE=cluster_resource

LAST_SERVER=raclinux1

STATE=ONLINE on raclinux1

TARGET=ONLINE

CARDINALITY_ID=1

CREATION_SEED=400

RESTART_COUNT=1

FAILURE_COUNT=0

FAILURE_HISTORY=

ID=db11gr2 1 1

INCARNATION=2

LAST_RESTART=01/09/2012 00:03:37

LAST_STATE_CHANGE=01/09/2012 00:03:37

STATE_DETAILS=

INTERNAL_STATE=STABLE

[root@raclinux1 bin]#

kill -9

[root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

NAME=db11gr2

TYPE=cluster_resource

LAST_SERVER=raclinux2

STATE=ONLINE on raclinux2

TARGET=ONLINE

CARDINALITY_ID=1

CREATION_SEED=400

RESTART_COUNT=0

FAILURE_COUNT=1

FAILURE_HISTORY=1326060967:raclinux1

ID=db11gr2 1 1

INCARNATION=3

LAST_RESTART=01/09/2012 00:17:16

LAST_STATE_CHANGE=01/09/2012 00:17:16

STATE_DETAILS=

INTERNAL_STATE=STABLE

[root@raclinux1 bin]#

kill -9

[root@raclinux2 bin]# ./crsctl stat res db11gr2 -v

NAME=db11gr2

TYPE=cluster_resource

LAST_SERVER=raclinux2

STATE=ONLINE on raclinux2

TARGET=ONLINE

CARDINALITY_ID=1

CREATION_SEED=400

RESTART_COUNT=1

FAILURE_COUNT=1

FAILURE_HISTORY=1326060967:raclinux1

ID=db11gr2 1 1

INCARNATION=4

LAST_RESTART=01/09/2012 00:49:30

LAST_STATE_CHANGE=01/09/2012 00:49:30

STATE_DETAILS=

INTERNAL_STATE=STABLE

[root@raclinux2 bin]#

kill -9

[root@raclinux2 bin]# ./crsctl stat res db11gr2 -v

NAME=db11gr2

TYPE=cluster_resource

LAST_SERVER=raclinux1

STATE=ONLINE on raclinux1

TARGET=ONLINE

CARDINALITY_ID=1

CREATION_SEED=400

RESTART_COUNT=0

FAILURE_COUNT=1

FAILURE_HISTORY=1326063300:raclinux2

ID=db11gr2 1 1

INCARNATION=5

LAST_RESTART=01/09/2012 00:55:30

LAST_STATE_CHANGE=01/09/2012 00:55:30

STATE_DETAILS=

INTERNAL_STATE=STABLE

[root@raclinux2 bin]#
Check Oracle Net connectivity with the db11gr2 connect sting to verify database location transparency.

Active/Passive configuration for single instance database with a floating VIP

In this section you will look at the pre Oracle 11.2, when SCAN was introduced, approach to make a database highly available in an Active/Passive configuration. The following resources will be created and registered for management under Oracle GI. Note the order of creation and registration of resources will determine their dependencies.

Create and register an application VIP resource
Create and register a listener resource
Create and register a database resource.

Database resource depends on Listener resource. Listener resource depends on Application VIP resource. The dependencies will be specified with the resource profile attributes. The specified configuration was the only available and supported prior to Oracle 11.2 and is still supported with the caveats mentioned in Note 780189.1.

Create and register an application VIP resource.

An application VIP resource needed to be created in order to have a fixed floating IP for the clients to access the database. The clients will use this application VIP regardless of the cluster node hosting the database. For details to setting the application VIP refer to previous post here.

appvipcfg create -network=1 -ip=192.168.20.111 -vipname=MyTestVIP -user=grid

[root@raclinux1 bin]# ./appvipcfg create -network=1 -ip=192.168.20.111 -vipname=MyTestVIP -user=root

Production Copyright 2007, 2008, Oracle.All rights reserved

2011-11-12 20:09:34: Creating Resource Type

2011-11-12 20:09:34: Executing /u01/app/11.2.0/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0/grid/crs/template/appvip.type

2011-11-12 20:09:34: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0/grid/crs/template/appvip.type

2011-11-12 20:09:34: Create the Resource

2011-11-12 20:09:34: Executing /u01/app/11.2.0/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr “USR_ORA_VIP=192.168.20.111,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL=’owner:root:rwx,pgrp:root:r-x,other::r–,user:root:r-x’,HOSTING_MEMBERS=raclinux1.gj.com,APPSVIP_FAILBACK=”

2011-11-12 20:09:34: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr “USR_ORA_VIP=192.168.20.111,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL=’owner:root:rwx,pgrp:root:r-x,other::r–,user:root:r-x’,HOSTING_MEMBERS=raclinux1.gj.com,APPSVIP_FAILBACK=”

[root@raclinux1 bin]#

Add the following to /etc/hosts on each cluster node.

192.168.20.111 raclinux-vip.gj.com raclinux-vip
Create and register a listener resource
1. Create a local listener in RDBMS $OH. Choose port 1522.
  
  [oracle@raclinux1 admin]$ cat listener.ora
# listener.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora

# Generated by Oracle configuration tools.

LISTENER_HA =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522))

)

(DESCRIPTION =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))

)

)

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_HA = ON

ADR_BASE_LISTENER_HA = /u01/app/oracle

[oracle@raclinux1 admin]$ cat endpoints_listener.ora

LISTENER_HA_RACLINUX1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.20.111)(PORT=1522)(IP=FIRST)))) # line added by Agent

[oracle@raclinux1 admin]$
1. Propagate the listener configuration on all cluster nodes.
2. Create an action scrip in $GI_HOME/crs/public on each cluster node
  
  [oracle@raclinux1 public]$ cat LISTENER_HA_actionScript.pl
  
  #!/usr/bin/perl
  
  if ($#ARGV != 0) {
  
  # This is an unreacheable condition
  
  # Oracle Clusterware would always invoke with a valid argument
  
  exit;
  
  }
  
  $command = $ARGV[0];
  
  # Start command
  
  if ($command eq “start”) {
  
  system(“su – oracle -c \”lsnrctl start listener_ha\”");
  
  # steps to start the resource
  
  exit 0; #if the start is successfull
  
  # exit 1 if the start has failed
  
  }
  
  # Stop command
  
  if ($command eq “stop”) {
  
  system(“su – oracle -c \”lsnrctl stop listener_ha\”");
  
  # steps to stop the resource
  
  exit 0; #if the stop is successfull
  
  # exit 1 if the stop has failed
  
  }
  
  # Check command
  
  if ($command eq “check”) {
  
  $found = qx(ps -ef | grep listener_ha | grep -v grep | wc -l);
  
  #$found = ‘ps -ef | grep smon | grep DB11GR2 | wc -l ‘;
  
  chomp($found);
  
  #print($found);
  
  if ($found eq “0″ ) {exit 1;} else {exit 0; }
  
  # steps to check the status of the resource
  
  # exit 0 if the resource is online
  
  # exit 1 otherwise
  
  }
  
  # Clean command
  
  if ($command eq “clean”) {
  
  system(“su – oracle -c \”lsnrctl stop listener_ha\”");
  
  # steps to force stop the resource
  
  exit 0; #if the force stop is successfull
  
  # exit 1 if the force stop has failed
  
  }
  
  [oracle@raclinux1 public]$
3. Test the action script and make sure that performs as expected prior to integration with Oracle GI.
  
  [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check
  
  [root@raclinux1 public]# echo $?
  
  0
  
  [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl stop
  
  LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 20:52:03
  
  Copyright (c) 1991, 2011, Oracle. All rights reserved.
  
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))
  
  The command completed successfully
  
  [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check
  
  [root@raclinux1 public]# echo $?
  
  1
  
  [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl start
  
  LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 20:52:17
  
  Copyright (c) 1991, 2011, Oracle. All rights reserved.
  
  Starting /u01/app/oracle/product/11.2.0/db_3/bin/tnslsnr: please wait…
  
  TNSLSNR for Linux: Version 11.2.0.3.0 – Production
  
  System parameter file is /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora
  
  Log messages written to /u01/app/oracle/diag/tnslsnr/raclinux1/listener_ha/alert/log.xml
  
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.111)(PORT=1522)))
  
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))
  
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))
  
  STATUS of the LISTENER
  
  ————————
  
  Alias listener_ha
  
  Version TNSLSNR for Linux: Version 11.2.0.3.0 – Production
  
  Start Date 09-JAN-2012 20:52:17
  
  Uptime 0 days 0 hr. 0 min. 0 sec
  
  Trace Level off
  
  Security ON: Local OS Authentication
  
  SNMP OFF
  
  Listener Parameter File /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora
  
  Listener Log File /u01/app/oracle/diag/tnslsnr/raclinux1/listener_ha/alert/log.xml
  
  Listening Endpoints Summary…
  
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.111)(PORT=1522)))
  
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))
  
  The listener supports no services
  
  The command completed successfully
  
  [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check
  
  [root@raclinux1 public]# echo $?
  
  0
  
  [root@raclinux1 public]#
4. Create the listener resource. Pay attention to the attributes in bold face. Note that the listener depends on Application VIP created earlier and run on the same node.
  
  [root@raclinux3 bin]# ./crsctl add resource Listener -type cluster_resource -attr ” ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/Listener_actionScript.pl, DESCRIPTION=Listener, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(intermediate:MyTestVIP) pullup(intermediate:MyTestVIP), STOP_DEPENDENCIES=hard(intermediate:MyTestVIP),CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″
  
  [root@raclinux3 bin]#
  
  [root@raclinux3 bin]#
  
  [root@raclinux1 bin]# ./crsctl start res Listener
  
  CRS-2679: Attempting to clean ‘Listener’ on ‘raclinux1′
  
  CRS-2681: Clean of ‘Listener’ on ‘raclinux1′ succeeded
  
  CRS-2672: Attempting to start ‘Listener’ on ‘raclinux1′
  
  CRS-2676: Start of ‘Listener’ on ‘raclinux1′ succeeded
  
  [root@raclinux1 bin]# ./crsctl status res Listener
  
  NAME=Listener
  
  TYPE=cluster_resource
  
  TARGET=ONLINE
  
  STATE=ONLINE on raclinux1
  
  [root@raclinux1 bin]#
Create and register a database resource
1. Create a database as in the previous section.
2. Repeat the steps 1 to 6 inclusive from the previous section for Active/Passive with SCAN.
3. Create an action script. I will reuse the db11gr2_actionScript.pl from the previous section. Prepare the database as in the previous section.
4. Teet the action scripts.
5. Create the database resource and start the resource.
  
  Pay attention to the dependencies! The database needs to start on the same cluster node where the listener is started. The listener starts on the node where the application VIP starts.
  
  [root@raclinux3 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=dbha, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=’hard(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg) pullup(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg)’,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″
  
  [root@raclinux3 bin]#
  
  [root@raclinux1 bin]#
  
  [root@raclinux1 bin]#./crsctl modify resource db11gr2 -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=dbha, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=’hard(ora.DATA.dg,ora.DATADG.dg,Listener) pullup(ora.DATA.dg,ora.DATADG.dg,Listener)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg)’,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″
  
  [root@raclinux1 bin]#
  
  [root@raclinux1 bin]# ./crsctl start res db11gr2
  
  CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux1′
  
  CRS-2676: Start of ‘db11gr2′ on ‘raclinux1′ succeeded
  
  [root@raclinux1 bin]#
6. Setup the listener registration due to non default port( I am using port 1522).
  
  SQL> show parameter listener
  
  NAME TYPE VALUE
  
  ———————————— ———– ——————————
  
  listener_networks string
  
  local_listener string
  
  remote_listener string rac-scan:1521
  
  SQL>
  
  SQL> alter system set local_listener=’192.168.20.111:1522′ scope=both;
  
  System altered.
  
  SQL>
7. Setup Oracle Net
  
  DB11 =
  
  (DESCRIPTION =
  
  (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522))
  
  (CONNECT_DATA =
  
  (SERVER = DEDICATED)
  
  (SERVICE_NAME = DB11GR2)
  
  )
  
  )
  
  Test
  
  [oracle@raclinux3 ~]$ lsnrctl service listener_ha
  
  LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 14:48:22
  
  Copyright (c) 1991, 2011, Oracle. All rights reserved.
  
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))
  
  Services Summary…
  
  Service “DB11GR2″ has 1 instance(s).
  
  Instance “DB11GR2″, status READY, has 1 handler(s) for this service…
  
  Handler(s):
  
  “DEDICATED” established:0 refused:0 state:ready
  
  LOCAL SERVER
  
  Service “DB11GR2XDB” has 1 instance(s).
  
  Instance “DB11GR2″, status READY, has 1 handler(s) for this service…
  
  Handler(s):
  
  ”D000″ established:0 refused:0 current:0 max:1022 state:ready
  
  DISPATCHER
  
  (ADDRESS=(PROTOCOL=tcp)(HOST=raclinux3.gj.com)(PORT=18626))
  
  The command completed successfully
  
  [oracle@raclinux3 ~]$
  
  [oracle@raclinux1 admin]$ tnsping db11
  
  TNS Ping Utility for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 14:54:48
  
  Copyright (c) 1997, 2011, Oracle. All rights reserved.
  
  Used parameter files:
  
  Used TNSNAMES adapter to resolve the alias
  
  Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DB11GR2)))
  
  OK (110 msec)
  
  [oracle@raclinux1 admin]$ sqlplus system/sys1@db11
  
  SQL*Plus: Release 11.2.0.3.0 Production on Mon Jan 9 14:55:01 2012
  
  Copyright (c) 1982, 2011, Oracle. All rights reserved.
  
  Connected to:
  
  Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
  
  With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
  
  Data Mining and Real Application Testing options
  
  SQL>
8. Test server failure
  
  [root@raclinux3 bin]# ./crsctl stat res -t
  
  ——————————————————————————–
  
  NAME TARGET STATE SERVER STATE_DETAILS
  
  ——————————————————————————–
  
  Local Resources
  
  ——————————————————————————–
  
  ora.DATA.dg
  
  ONLINE ONLINE raclinux1
  
  ONLINE ONLINE raclinux2
  
  ONLINE ONLINE raclinux3
  
  ……………………………………………
  
  ——————————————————————————–
  
  Cluster Resources
  
  ——————————————————————————–
  
  Listener
  
  1 ONLINE ONLINE raclinux3
  
  MyTest
  
  1 ONLINE ONLINE raclinux3
  
  MyTestVIP
  
  1 ONLINE ONLINE raclinux3
  
  MyTestVIP1
  
  1 ONLINE ONLINE raclinux1
  
  db11gr2
  
  1 ONLINE ONLINE raclinux3
  
  ora.LISTENER_SCAN1.lsnr
  
  1 ONLINE ONLINE raclinux1
  
  …………………………………………………………….
  
  [root@raclinux3 bin]#
  
  Kill raclinux3
  
  [root@raclinux3 bin]# ./crsctl stat res -t
  
  ……………………………………………….
  
  ——————————————————————————–
  
  Cluster Resources
  
  ——————————————————————————–
  
  Listener
  
  1 ONLINE ONLINE raclinux1
  
  MyTest
  
  1 ONLINE ONLINE raclinux1
  
  MyTestVIP
  
  1 ONLINE ONLINE raclinux1
  
  MyTestVIP1
  
  1 ONLINE ONLINE raclinux1
  
  db11gr2
  
  1 ONLINE ONLINE raclinux1
  
  ……………………………………………
9. Test process crash. Similarly to the previous section Oracle GI restarts the resource only once. On the second restart the resource fails over. Number of restart prior to failover can be modified by RESTART_ATTEMPTS.

Active/Passive configuration using Oracle RAC One Node

For information about installing and managing Oracle RAC One Node click here.

Oracle RAC One Node is RAC-enabled single instance database that runs on only one node in the cluster. In case of a failure of the node where RAC One Node is running the instance fails over to another node of the cluster. Upon Oracle RAC One Node installation the user specifies:

List of nodes for the HOSTING_MEMBERS to limit the nodes where RAC One node will run in an admin-managed database.
A server pool for the policy managed RAC One Node Database.

Added advantage for Oracle RAC One Node is the tight integration with srvctl utility and ability to perform an online relocation to another cluster node. For the previous two cases you cannot use srvctl utility to manage Active/Passive configurations and do not have ability for an online relocation to another cluster node. Oracle RAC One Node is an additional cost option to the Oracle Enterprise Edition and presently does not support Data Guard. In note 790189.1 Oracle explicitly states that Oracle RAC One Node will be a direction that will be followed for failing over an Oracle 11.2 database and newer.

Sources

Using Oracle Clusterware to Protect Oracle Database 10g with Oracle Enterprise Manager Grid Control Integration
Oracle Clusterware and Application Failover Management [ID 790189.1]

Filed under: oracle

Clone GI and RDBMS homes in Oracle RAC 11.2.0.3 with clone.pl

gjilevski — Thu, 17 Nov 2011 17:58:12 +0000

Clone GI and RDBMS homes in Oracle RAC 11.2.0.3 with clone.pl

In this article you will look at how to use clone.pl script to clone GI and RDBMS home of an existing Oracle RAC database installation to add a new node and extend the existing cluster. For additional approaches to add or delete a node to an Oracle RAC click here. The clone.pl can also be used to clone existing GI and RDBMS homes of one cluster to create a new Oracle clusters on another set of servers. You will have a look at how to manually add a RDBMS instance to the new node without using the dbca. The article refers to an existing Oracle cluster consisting of nodes raclinux1 and raclinux2 as described here and later upgraded to 11.2.0.3 as described here.

You will look at the following topics related to extending a node using clone.pl

Clone GI home to the new node
Clone RDBMS home to the new node
Manually add an instance to the new node

Cloning existing home to either extend a cluster to a new node or to create a new cluster has an advantage of considerably reducing deployment time as cloning copies an original source GI and DRBMS home along with all patches and creates an identical image on the target node(s). Apart from distributing all patches from source to target location the cloning mechanism maintains the oracle inventory on the target location resulting in having cloned Oracle GI and RDBMS homes that can be maintained and patched as any standard Oracle home installation using opatch, OUI or OEM.

Clone GI home to the new node

The section outlines the steps to clone an existing Oracle GI installation to a new node that will become a member of the cluster.
1. Assuming that GI is installed on the source stop the GI using the following command.
  
  [root@raclinux2 bin]# ./crsctl stop crs
2. Create a stage directory and a tar ball of the source
  
  [root@raclinux2 grid]# mkdir /u01/stageGI
  
  [root@raclinux2 grid]# cp -prf /u01/app/11.2.0.3/grid /u01/stageGI
  
  [root@raclinux2 grid]# pwd
  
  /u01/stageGI/grid
  
  [root@raclinux2 grid]#
  
  [root@raclinux2 grid]# tar -cvf /tmp/tar11203.tar .
  
  [root@raclinux2 grid]#
3. Start GI on the source no
  
  [root@raclinux2 bin]# ./crsctl start crs
4. Create a software location on the new node raclinux3 and extract the tar ball.
  
  As root execute on the new node raclinux3
  
  mkdir –p /u01/app/11.2.0.3/grid
  
  mkdir –p /u01/app/grid
  
  mkdir –p /u01/app/oracle
  
  chown grid:oinstall /u01/app/11.2.0.3/grid
  
  chown grid:oinstall /u01/app/grid
  
  chown oracle:oinstall /u01/app/oracle
  
  chown –R grid:oinstall /u01
  
  mkdir –p /u01/app/oracle
  
  chmod –R 775 /u01/
  
  As grid user execute on the new node raclinux3
  
  cd /u01/app/11.2.0.3/grid
  
  [grid@raclinux3 grid]$ tar -xvf /tmp/tar11203.tar
5. Clean the node specific configuration details and set proper permissions and ownership.
  
  As root execute the following
  
  cd /u01/app/11.2.0.3/grid
  
  rm -rf raclinux2
  
  rm -rf log/raclinux2
  
  rm -rf gpnp/raclinux2
  
  rm -rf crs/init
  
  rm -rf cdata
  
  rm -rf crf
  
  find gpnp -type f -exec rm -f {} \;
  
  rm -rf network/admin/*.ora
  
  find . -name ’*.ouibak’ -exec rm {} \;
  
  find . -name ‘*.ouibak.1′ -exec rm {} \;
  
  rm -rf root.sh*
  
  cd cfgtoollogs
  
  find . -type f -exec rm -f {} \;
  
  chown -R grid:oinstall /u01/app/11.2.0.3/grid
  
  chown –R grid:oinstall /u01
  
  chmod –R 775 /u01/
  
  As grid user execute
  
  [grid@raclinux3 cfgtoollogs]$ chmod u+s /u01/app/11.2.0.3/grid/bin/oracle
  
  [grid@raclinux3 cfgtoollogs]$ chmod g+s /u01/app/11.2.0.3/grid/bin/oracle
6. Run clone.pl from $GI_HOME/clone/bin on the new node raclinux3 as grid user
  
  Before running clone.pl prepare the following information:

ORACLE_BASE=
/u01/app/grid
ORACLE_HOME=
/u01/app/11.2.0.3/grid
ORACLE_HOME_NAME=
Ora11g_gridinfrahome2 – use OUI from any existing cluster node

Run clone.pl as specified below

[grid@raclinux3 bin]$ perl clone.pl ORACLE_HOME=/u01/app/11.2.0.3/grid ORACLE_HOME_NAME=Ora11g_gridinfrahome2 ORACLE_BASE=/u01/app/grid SHOW_ROOTSH_CONFIRMATION=false

./runInstaller -clone -waitForCompletion ”ORACLE_HOME=/u01/app/11.2.0.3/grid” “ORACLE_HOME_NAME=Ora11g_gridinfrahome2″ “ORACLE_BASE=/u01/app/grid” “SHOW_ROOTSH_CONFIRMATION=false” -silent -noConfig -nowait

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-11-15_07-36-57PM. Please wait …Oracle Universal Installer, Version 11.2.0.3.0 Production

You can find the log of this install session at:

/u01/app/oraInventory/logs/cloneActions2011-11-15_07-36-57PM.log

………………………………………………………………………………………. 100% Done.

Installation in progress (Tuesday, November 15, 2011 7:38:24 PM EET)

………………………………………………………………………………….. 95% Done.

Install successful

Linking in progress (Tuesday, November 15, 2011 7:38:33 PM EET)

Link successful

Setup in progress (Tuesday, November 15, 2011 7:39:36 PM EET)

Setup successful

End of install phases.(Tuesday, November 15, 2011 7:39:59 PM EET)

The cloning of Ora11g_gridinfrahome2 was successful.

Please check ‘/u01/app/oraInventory/logs/cloneActions2011-11-15_07-36-57PM.log’ for more details.

[grid@raclinux3 bin]$

Verify the log file to ensure that the cloning process is successful.

Verify from any existing cluster node, for example raclinux2 using cluvfy that the prerequisites are met

[grid@raclinux2 gpnp]$ cluvfy stage -pre nodeadd -n raclinux3 -vip raclinux3-vip

Performing pre-checks for node addition

Checking node reachability…

Node reachability check passed from node “raclinux2″

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking CRS integrity…

Clusterware version consistency passed

CRS integrity check passed

Checking shared resources…

Checking CRS home location…

“/u01/app/11.2.0.3/grid” is not shared

Shared resources check for node addition passed

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity for interface “eth1″

Node connectivity passed for interface “eth1″

TCP connectivity check passed for subnet “10.10.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed for subnet “10.10.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “raclinux3:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux2:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux3:/tmp”

Free disk space check passed for “raclinux2:/tmp”

Check for multiple users with UID value 1100 passed

User existence check passed for “grid”

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check passed for “shmmax”

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “make”

Package existence check passed for “binutils”

Package existence check passed for “gcc(x86_64)”

Package existence check passed for “libaio(x86_64)”

Package existence check passed for “glibc(x86_64)”

Package existence check passed for “compat-libstdc++-33(x86_64)”

Package existence check passed for “elfutils-libelf(x86_64)”

Package existence check passed for “elfutils-libelf-devel”

Package existence check passed for “glibc-common”

Package existence check passed for “glibc-devel(x86_64)”

Package existence check passed for “glibc-headers”

Package existence check passed for “gcc-c++(x86_64)”

Package existence check passed for “libaio-devel(x86_64)”

Package existence check passed for “libgcc(x86_64)”

Package existence check passed for “libstdc++(x86_64)”

Package existence check passed for “libstdc++-devel(x86_64)”

Package existence check passed for “sysstat”

Package existence check passed for “ksh”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user’s primary group passed

Checking OCR integrity…

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration…

Oracle Cluster Voting Disk configuration check passed

Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon’s boot time configuration check for slewing option passed

NTP common Time Server Check started…

PRVF-5408 : NTP Time Server “80.190.97.205″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “192.43.244.18″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “129.69.1.153″ is common only to the following nodes “raclinux2″

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

User “grid” is not part of “root” group. Check passed

Checking consistency of file “/etc/resolv.conf” across nodes

File “/etc/resolv.conf” does not have both domain and search entries defined

domain entry in file “/etc/resolv.conf” is consistent across nodes

search entry in file “/etc/resolv.conf” is consistent across nodes

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000″ ms on following nodes: raclinux3

File “/etc/resolv.conf” is not consistent across nodes

Checking VIP configuration.

Checking VIP Subnet configuration.

Check for VIP Subnet configuration passed.

Checking VIP reachability

Check for VIP reachability passed.

Pre-check for node addition was unsuccessful on all the nodes.

[grid@raclinux2 gpnp]$
From any existing cluster node, for example raclinux2 while logged in as grid user run the following command from $ORACLE_HOME_HOME/oui/bin

[grid@raclinux2 bin]$ ./addNode.sh -silent -noCopy ORACLE_HOME=/u01/app/11.2.0.3/grid “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}” “CLUSTER_NEW_VIPS={raclinux3-vip}” CRS_ADDNODE=true CRS_DHCP_ENABLED=false

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9616 MB Passed

Oracle Universal Installer, Version 11.2.0.3.0 Production

Copyright (C) 1999, 2011, Oracle. All rights reserved.

Performing tests to see whether nodes raclinux1,raclinux3 are available

……………………………………………………… 100% Done.

.

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /u01/app/11.2.0.3/grid

New Nodes

Space Requirements

New Nodes

raclinux3

/u01: Required 4.78GB : Available 265.43GB

Installed Products

Product Names

Oracle Grid Infrastructure 11.2.0.3.0

Sun JDK 1.5.0.30.03

Installer SDK Component 11.2.0.3.0

Oracle One-Off Patch Installer 11.2.0.1.7

Oracle Universal Installer 11.2.0.3.0

Oracle USM Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Enterprise Manager Common Core Files 10.2.0.4.4

Oracle DBCA Deconfiguration 11.2.0.3.0

Oracle RAC Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Server) 11.2.0.3.0

Installation Plugin Files 11.2.0.3.0

Universal Storage Manager Files 11.2.0.3.0

Oracle Text Required Support Files 11.2.0.3.0

Automatic Storage Management Assistant 11.2.0.3.0

Oracle Database 11g Multimedia Files 11.2.0.3.0

Oracle Multimedia Java Advanced Imaging 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Oracle Multimedia Locator RDBMS Files 11.2.0.3.0

Oracle Core Required Support Files 11.2.0.3.0

Bali Share 1.1.18.0.0

Oracle Database Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Client) 11.2.0.3.0

Expat libraries 2.0.1.0.1

Oracle Containers for Java 11.2.0.3.0

Perl Modules 5.10.0.0.1

Secure Socket Layer 11.2.0.3.0

Oracle JDBC/OCI Instant Client 11.2.0.3.0

Oracle Multimedia Client Option 11.2.0.3.0

LDAP Required Support Files 11.2.0.3.0

Character Set Migration Utility 11.2.0.3.0

Perl Interpreter 5.10.0.0.2

PL/SQL Embedded Gateway 11.2.0.3.0

OLAP SQL Scripts 11.2.0.3.0

Database SQL Scripts 11.2.0.3.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

SSL Required Support Files for InstantClient 11.2.0.3.0

SQL*Plus Files for Instant Client 11.2.0.3.0

Oracle Net Required Support Files 11.2.0.3.0

Oracle Database User Interface 2.2.13.0.0

RDBMS Required Support Files for Instant Client 11.2.0.3.0

RDBMS Required Support Files Runtime 11.2.0.3.0

XML Parser for Java 11.2.0.3.0

Oracle Security Developer Tools 11.2.0.3.0

Oracle Wallet Manager 11.2.0.3.0

Enterprise Manager plugin Common Files 11.2.0.3.0

Platform Required Support Files 11.2.0.3.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

RDBMS Required Support Files 11.2.0.3.0

Oracle Ice Browser 5.2.3.6.0

Oracle Help For Java 4.2.9.0.0

Enterprise Manager Common Files 10.2.0.4.3

Deinstallation Tool 11.2.0.3.0

Oracle Java Client 11.2.0.3.0

Cluster Verification Utility Files 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.3.0

Oracle LDAP administration 11.2.0.3.0

Cluster Verification Utility Common Files 11.2.0.3.0

Oracle Clusterware RDBMS Files 11.2.0.3.0

Oracle Locale Builder 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Buildtools Common Files 11.2.0.3.0

Oracle RAC Required Support Files-HAS 11.2.0.3.0

SQL*Plus Required Support Files 11.2.0.3.0

XDK Required Support Files 11.2.0.3.0

Agent Required Support Files 10.2.0.4.3

Parser Generator Required Support Files 11.2.0.3.0

Precompiler Required Support Files 11.2.0.3.0

Installation Common Files 11.2.0.3.0

Required Support Files 11.2.0.3.0

Oracle JDBC/THIN Interfaces 11.2.0.3.0

Oracle Multimedia Locator 11.2.0.3.0

Oracle Multimedia 11.2.0.3.0

HAS Common Files 11.2.0.3.0

Assistant Common Files 11.2.0.3.0

PL/SQL 11.2.0.3.0

HAS Files for DB 11.2.0.3.0

Oracle Recovery Manager 11.2.0.3.0

Oracle Database Utilities 11.2.0.3.0

Oracle Notification Service 11.2.0.3.0

SQL*Plus 11.2.0.3.0

Oracle Netca Client 11.2.0.3.0

Oracle Net 11.2.0.3.0

Oracle JVM 11.2.0.3.0

Oracle Internet Directory Client 11.2.0.3.0

Oracle Net Listener 11.2.0.3.0

Cluster Ready Services Files 11.2.0.3.0

Oracle Database 11g 11.2.0.3.0

—————————————————————————–

Instantiating scripts for add node (Tuesday, November 15, 2011 8:22:13 PM EET)

. 1% Done.

Instantiation of add node scripts complete

Saving inventory on nodes (Tuesday, November 15, 2011 8:22:18 PM EET)

. 100% Done.

Save inventory complete

WARNING:

The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

/u01/app/11.2.0.3/grid/root.sh #On nodes raclinux3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0.3/grid was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[grid@raclinux2 bin]$

DO NOT RUN root.sh after successful completion!!!
Copy cluster configuration and gpnp profile and certificates to the new node raclinux3

[grid@raclinux2 install]$ scp :/u01/app/11.2.0.3/grid/crs/install/crsconfig_params raclinux3:/u01/app/11.2.0.3/grid/crs/install/crsconfig_params

crsconfig_params 100% 3965 3.9KB/s 00:00

[grid@raclinux2 install]$ scp :/u01/app/11.2.0.3/grid/crs/install/crsconfig_addparams raclinux3:/u01/app/11.2.0.3/grid/crs/install/crsconfig_addparams

crsconfig_addparams 100% 751 0.7KB/s 00:00

[grid@raclinux2 install]$

Copy the content of /u01/app/11.2.0.3/grid/gpnp on any existing cluster node, for example raclinux2 to /u01/app/11.2.0.3/grid/gpnp on raclinux3.
Run root.sh from raclinux3 as root user

If the re-link request do not happen the reconfiguration might be successful.

[root@raclinux3 grid]# ./root.sh

Check /u01/app/11.2.0.3/grid/install/root_raclinux3.gj.com_2011-11-15_20-19-20.log for the output of root script

[root@raclinux3 grid]# cat /u01/app/11.2.0.3/grid/install/root_raclinux3.gj.com_2011-11-15_20-19-20.log

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0.3/grid

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

The oracle binary is currently linked with RAC disabled.

Please execute the following steps to relink oracle binary

and rerun the command with RAC enabled:

setenv ORACLE_HOME

cd /rdbms/lib

make -f ins_rdbms.mk rac_on ioracle

/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

[root@raclinux3 grid]#

Re-link the GI binaries and repeat the root.sh

As root execute on the new node raclinux3

chown grid:oinstall /u01/app/11.2.0.3/grid

chmod –R 775 /u01/app/11.2.0.3/grid

Re-link the GI binaries.

[grid@raclinux3 lib]$ make -f ins_rdbms.mk rac_on ioracle

rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so

cp /u01/app/11.2.0.3/grid/lib//libskgxpg.so /u01/app/11.2.0.3/grid/lib/libskgxp11.so

- Use stub SKGXN library

cp /u01/app/11.2.0.3/grid/lib/libskgxns.so /u01/app/11.2.0.3/grid/lib/libskgxn2.so

/usr/bin/ar d /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a ksnkcs.o

/usr/bin/ar cr /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a /u01/app/11.2.0.3/grid/rdbms/lib/kcsm.o

chmod 755 /u01/app/11.2.0.3/grid/bin

– Linking Oracle

rm -f /u01/app/11.2.0.3/grid/rdbms/lib/oracle

gcc -o /u01/app/11.2.0.3/grid/rdbms/lib/oracle -m64 -L/u01/app/11.2.0.3/grid/rdbms/lib/ -L/u01/app/11.2.0.3/grid/lib/ -L/u01/app/11.2.0.3/grid/lib/stubs/ -Wl,-E /u01/app/11.2.0.3/grid/rdbms/lib/opimai.o /u01/app/11.2.0.3/grid/rdbms/lib/ssoraed.o /u01/app/11.2.0.3/grid/rdbms/lib/ttcsoi.o -Wl,–whole-archive -lperfsrv11 -Wl,–no-whole-archive /u01/app/11.2.0.3/grid/lib/nautab.o /u01/app/11.2.0.3/grid/lib/naeet.o /u01/app/11.2.0.3/grid/lib/naect.o /u01/app/11.2.0.3/grid/lib/naedhs.o /u01/app/11.2.0.3/grid/rdbms/lib/config.o -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo “-loraolap11″ ; fi` -lslax11 -lpls11 -lrt -lplp11 -lserver11 -lclient11 -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/11.2.0.3/grid/lib/libavserver11.a ] ; then echo “-lavserver11″ ; else echo “-lavstub11″; fi` `if [ -f /u01/app/11.2.0.3/grid/lib/libavclient11.a ] ; then echo “-lavclient11″ ; fi` -lknlopt -lslax11 -lpls11 -lrt -lplp11 -ljavavm11 -lserver11 -lwwg `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep “kxmnsd.o” > /dev/null 2>&1 ; then echo ” ” ; else echo “-lordsdo11″; fi` -L/u01/app/11.2.0.3/grid/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged -lippsmerged -lippcore -lippcpemerged -lippcpmerged -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -Wl,-rpath,/u01/app/11.2.0.3/grid/lib -lm `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -ldl -lm -L/u01/app/11.2.0.3/grid/lib

test ! -f /u01/app/11.2.0.3/grid/bin/oracle ||\

mv -f /u01/app/11.2.0.3/grid/bin/oracle /u01/app/11.2.0.3/grid/bin/oracleO

mv /u01/app/11.2.0.3/grid/rdbms/lib/oracle /u01/app/11.2.0.3/grid/bin/oracle

chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle

[grid@raclinux3 lib]$

Rerun root.sh

[root@raclinux3 grid]# ./root.sh

Check /u01/app/11.2.0.3/grid/install/root_raclinux3.gj.com_2011-11-15_23-34-00.log for the output of root script

[root@raclinux3 grid]# cat /u01/app/11.2.0.3/grid/install/root_raclinux3.gj.com_2011-11-15_23-34-00.log

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0.3/grid

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node raclinux1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Preparing packages for installation…

cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@raclinux3 grid]#
Verify that Oracle Clusterware (OC) started.

Use crsctl check cluster –all

[root@raclinux3 bin]# ./crsctl check cluster -all

**************************************************************

raclinux1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

raclinux2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

raclinux3:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

[root@raclinux3 bin]#

Use the following command to verify that the node was successfully added. Skip the failed check due to a SCAN registered with a single entry in /etc/hosts.

[grid@raclinux2 gpnp]$ cluvfy stage -post nodeadd -n raclinux3 -verbose

Performing post-checks for node addition

Checking node reachability…

Check: Node reachability from node “raclinux2″

Destination Node Reachable?

———————————— ————————

raclinux3 yes

Result: Node reachability check passed from node “raclinux2″

Checking user equivalence…

Check: User equivalence for user “grid”

Node Name Status

———————————— ————————

raclinux3 passed

Result: User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Node Name Status

———————————— ————————

raclinux3 passed

raclinux2 passed

raclinux1 passed

Verification of the hosts config file successful

Interface information for node “raclinux3″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.23 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth0 192.168.20.53 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth1 10.10.20.23 10.10.20.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth1 169.254.105.52 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth2 192.168.156.103 192.168.156.0 0.0.0.0 UNKNOWN 08:00:27:8C:A0:9B 1500

eth3 192.168.2.24 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:E5:70:A8 1500

Interface information for node “raclinux2″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.22 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.111 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.100 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.112 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.52 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth1 10.10.20.22 10.10.20.0 0.0.0.0 10.0.5.2 08:00:27:41:52:72 1500

eth1 169.254.206.240 169.254.0.0 0.0.0.0 10.0.5.2 08:00:27:41:52:72 1500

eth2 192.168.156.102 192.168.156.0 0.0.0.0 10.0.5.2 08:00:27:13:BC:77 1500

eth3 10.0.5.15 10.0.5.0 0.0.0.0 10.0.5.2 08:00:27:93:4A:17 1500

Interface information for node “raclinux1″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.21 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:80:E3:C1 1500

eth0 192.168.20.51 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:80:E3:C1 1500

eth1 10.10.20.21 10.10.20.0 0.0.0.0 10.0.5.2 08:00:27:FD:AA:42 1500

eth1 169.254.89.140 169.254.0.0 0.0.0.0 10.0.5.2 08:00:27:FD:AA:42 1500

eth2 192.168.156.101 192.168.156.0 0.0.0.0 10.0.5.2 08:00:27:B0:B4:C7 1500

eth3 10.0.5.15 10.0.5.0 0.0.0.0 10.0.5.2 08:00:27:8D:38:97 1500

Check: Node connectivity for interface “eth0″

Source Destination Connected?

—————————— —————————— —————-

raclinux3[192.168.20.23] raclinux3[192.168.20.53] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.22] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.111] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.100] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.112] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.52] yes

raclinux3[192.168.20.23] raclinux1[192.168.20.21] yes

raclinux3[192.168.20.23] raclinux1[192.168.20.51] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.22] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.111] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.100] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.112] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.52] yes

raclinux3[192.168.20.53] raclinux1[192.168.20.21] yes

raclinux3[192.168.20.53] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.111] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.100] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.22] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.22] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.100] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.111] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.111] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.100] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.100] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.100] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.100] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.112] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.112] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.112] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.52] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.52] raclinux1[192.168.20.51] yes

raclinux1[192.168.20.21] raclinux1[192.168.20.51] yes

Result: Node connectivity passed for interface “eth0″

Check: TCP connectivity of subnet “192.168.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.20.23 passed

raclinux2:192.168.20.22 raclinux3:192.168.20.53 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.111 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.100 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.112 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.52 passed

raclinux2:192.168.20.22 raclinux1:192.168.20.21 passed

raclinux2:192.168.20.22 raclinux1:192.168.20.51 passed

Result: TCP connectivity check passed for subnet “192.168.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking cluster integrity…

Node Name

————————————

raclinux1

raclinux2

raclinux3

Cluster integrity check passed

Checking CRS integrity…

Clusterware version consistency passed

The Oracle Clusterware is healthy on node “raclinux3″

The Oracle Clusterware is healthy on node “raclinux2″

The Oracle Clusterware is healthy on node “raclinux1″

CRS integrity check passed

Checking shared resources…

Checking CRS home location…

“/u01/app/11.2.0.3/grid” is not shared

Result: Shared resources check for node addition passed

Checking node connectivity…

Checking hosts config file…

Node Name Status

———————————— ————————

raclinux3 passed

raclinux2 passed

raclinux1 passed

Verification of the hosts config file successful

Interface information for node “raclinux3″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.23 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth0 192.168.20.53 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth1 10.10.20.23 10.10.20.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth1 169.254.105.52 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth2 192.168.156.103 192.168.156.0 0.0.0.0 UNKNOWN 08:00:27:8C:A0:9B 1500

eth3 192.168.2.24 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:E5:70:A8 1500

Interface information for node “raclinux2″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.22 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.111 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.100 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.112 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth0 192.168.20.52 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:F7:87:C6 1500

eth1 10.10.20.22 10.10.20.0 0.0.0.0 10.0.5.2 08:00:27:41:52:72 1500

eth1 169.254.206.240 169.254.0.0 0.0.0.0 10.0.5.2 08:00:27:41:52:72 1500

eth2 192.168.156.102 192.168.156.0 0.0.0.0 10.0.5.2 08:00:27:13:BC:77 1500

eth3 10.0.5.15 10.0.5.0 0.0.0.0 10.0.5.2 08:00:27:93:4A:17 1500

Interface information for node “raclinux1″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.21 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:80:E3:C1 1500

eth0 192.168.20.51 192.168.20.0 0.0.0.0 10.0.5.2 08:00:27:80:E3:C1 1500

eth1 10.10.20.21 10.10.20.0 0.0.0.0 10.0.5.2 08:00:27:FD:AA:42 1500

eth1 169.254.89.140 169.254.0.0 0.0.0.0 10.0.5.2 08:00:27:FD:AA:42 1500

eth2 192.168.156.101 192.168.156.0 0.0.0.0 10.0.5.2 08:00:27:B0:B4:C7 1500

eth3 10.0.5.15 10.0.5.0 0.0.0.0 10.0.5.2 08:00:27:8D:38:97 1500

Check: Node connectivity for interface “eth0″

Source Destination Connected?

—————————— —————————— —————-

raclinux3[192.168.20.23] raclinux3[192.168.20.53] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.22] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.111] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.100] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.112] yes

raclinux3[192.168.20.23] raclinux2[192.168.20.52] yes

raclinux3[192.168.20.23] raclinux1[192.168.20.21] yes

raclinux3[192.168.20.23] raclinux1[192.168.20.51] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.22] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.111] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.100] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.112] yes

raclinux3[192.168.20.53] raclinux2[192.168.20.52] yes

raclinux3[192.168.20.53] raclinux1[192.168.20.21] yes

raclinux3[192.168.20.53] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.111] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.100] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.22] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.22] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.22] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.100] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.111] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.111] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.111] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.100] raclinux2[192.168.20.112] yes

raclinux2[192.168.20.100] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.100] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.100] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.112] raclinux2[192.168.20.52] yes

raclinux2[192.168.20.112] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.112] raclinux1[192.168.20.51] yes

raclinux2[192.168.20.52] raclinux1[192.168.20.21] yes

raclinux2[192.168.20.52] raclinux1[192.168.20.51] yes

raclinux1[192.168.20.21] raclinux1[192.168.20.51] yes

Result: Node connectivity passed for interface “eth0″

Check: TCP connectivity of subnet “192.168.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.20.23 passed

raclinux2:192.168.20.22 raclinux3:192.168.20.53 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.111 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.100 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.112 passed

raclinux2:192.168.20.22 raclinux2:192.168.20.52 passed

raclinux2:192.168.20.22 raclinux1:192.168.20.21 passed

raclinux2:192.168.20.22 raclinux1:192.168.20.51 passed

Result: TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity for interface “eth1″

Source Destination Connected?

—————————— —————————— —————-

raclinux3[10.10.20.23] raclinux2[10.10.20.22] yes

raclinux3[10.10.20.23] raclinux1[10.10.20.21] yes

raclinux2[10.10.20.22] raclinux1[10.10.20.21] yes

Result: Node connectivity passed for interface “eth1″

Check: TCP connectivity of subnet “10.10.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:10.10.20.22 raclinux3:10.10.20.23 passed

raclinux2:10.10.20.22 raclinux1:10.10.20.21 passed

Result: TCP connectivity check passed for subnet “10.10.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed for subnet “10.10.20.0″.

Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking node application existence…

Checking existence of VIP node application (required)

Node Name Required Running? Comment

———— ———————— ———————— ———-

raclinux3 yes yes passed

raclinux2 yes yes passed

raclinux1 yes yes passed

VIP node application check passed

Checking existence of NETWORK node application (required)

Node Name Required Running? Comment

———— ———————— ———————— ———-

raclinux3 yes yes passed

raclinux2 yes yes passed

raclinux1 yes yes passed

NETWORK node application check passed

Checking existence of GSD node application (optional)

Node Name Required Running? Comment

———— ———————— ———————— ———-

raclinux3 no no exists

raclinux2 no no exists

raclinux1 no no exists

GSD node application is offline on nodes “raclinux3,raclinux2,raclinux1″

Checking existence of ONS node application (optional)

Node Name Required Running? Comment

———— ———————— ———————— ———-

raclinux3 no yes passed

raclinux2 no yes passed

raclinux1 no yes passed

ONS node application check passed

Checking Single Client Access Name (SCAN)…

SCAN Name Node Running? ListenerName Port Running?

—————- ———— ———— ———— ———— ————

rac-scan raclinux2 true LISTENER_SCAN1 1521 true

Checking TCP connectivity to SCAN Listeners…

Node ListenerName TCP connectivity?

———— ———————— ————————

raclinux2 LISTENER_SCAN1 yes

TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for “rac-scan”…

ERROR:

PRVG-1101 : SCAN name “rac-scan” failed to resolve

SCAN Name IP Address Status Comment

———— ———————— ———————— ———-

rac-scan 192.168.20.100 failed NIS Entry

ERROR:

PRVF-4657 : Name resolution setup check for “rac-scan” (IP address: 192.168.20.100) failed

ERROR:

PRVF-4664 : Found inconsistent name resolution entries for SCAN name “rac-scan”

Verification of SCAN VIP and Listener setup failed

Checking to make sure user “grid” is not in “root” group

Node Name Status Comment

———— ———————— ————————

raclinux3 passed does not exist

Result: User “grid” is not part of “root” group. Check passed

Checking if Clusterware is installed on all nodes…

Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes…

Check: CTSS Resource running on all nodes

Node Name Status

———————————— ————————

raclinux3 passed

Result: CTSS resource check passed

Querying CTSS for time offset on all nodes…

Result: Query of CTSS for time offset passed

Check CTSS state started…

Check: CTSS state

Node Name State

———————————— ————————

raclinux3 Observer

CTSS is in Observer state. Switching over to clock synchronization checks using NTP

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

The NTP configuration file “/etc/ntp.conf” is available on all nodes

NTP Configuration file check passed

Checking daemon liveness…

Check: Liveness for “ntpd”

Node Name Running?

———————————— ————————

raclinux3 yes

Result: Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

Checking NTP daemon command line for slewing option “-x”

Check: NTP daemon command line

Node Name Slewing Option Set?

———————————— ————————

raclinux3 yes

Result:

NTP daemon slewing option check passed

Checking NTP daemon’s boot time configuration, in file “/etc/sysconfig/ntpd”, for slewing option “-x”

Check: NTP daemon’s boot time configuration

Node Name Slewing Option Set?

———————————— ————————

raclinux3 yes

Result:

NTP daemon’s boot time configuration check for slewing option passed

Checking whether NTP daemon or service is using UDP port 123 on all nodes

Check for NTP daemon or service using UDP port 123

Node Name Port Open?

———————————— ————————

raclinux3 yes

NTP common Time Server Check started…

NTP Time Server “.LOCL.” is common to all nodes on which the NTP daemon is running

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Checking on nodes “[raclinux3]“…

Check: Clock time offset from NTP Time Server

Time Server: .LOCL.

Time Offset Limit: 1000.0 msecs

Node Name Time Offset Status

———— ———————— ————————

raclinux3 0.0 passed

Time Server “.LOCL.” has time offsets that are within permissible limits for nodes “[raclinux3]“.

Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Oracle Cluster Time Synchronization Services check passed

Post-check for node addition was unsuccessful on all the nodes.

[grid@raclinux2 gpnp]$
Conclusion

For some reason I was asked to re-link the binaries despite that a cluster enabled home was used as a source. At the end, after root.sh completed OC was up and running.

Clone RDBMS home to the new node

Cloning RDBMS home is conceptually similar to cloning GI and involves the following steps described below. The new node will be raclinux3 and the target directory will be /u01/app/oracle/product/11.2.0/db_3
1. Prepare a stage directory and a tar ball
  
  [root@raclinux2 bin]# mkdir -p /u01/stageRDBMS
  
  [root@raclinux2 bin]# chmod -R 775 /u01/stageRDBMS
  
  [root@raclinux2 bin]# chown -R oracle:oinstall /u01/stageRDBMS
  
  [root@raclinux2 bin]#
  
  [root@raclinux2 stageRDBMS]# cp -prf /u01/app/oracle/product/11.2.0/db_3 /u01/stageRDBMS
  
  [root@raclinux2 stageRDBMS]# cd /u01/app/oracle/product/11.2.0/db_3
  
  [root@raclinux2 db_3]#
  
  [root@raclinux2 db_3]# tar -cvf /tmp/tarRDBMS.tar .
  
  [root@raclinux2 db_3]#
2. Prepare the destination directory, extract the tar on the destination directory and set ownership and permissions
[oracle@raclinux3 db_3]$ tar -xvf /tmp/tar11203.tar

[root@raclinux3 db_3]# chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/db_3

[root@raclinux3 db_3]# chmod u+s /u01/app/oracle/product/11.2.0/db_3/bin/oracle

[root@raclinux3 db_3]# chmod g+s /u01/app/oracle/product/11.2.0/db_3/bin/oracle

[root@raclinux3 db_3]#
1. Prepare a script, for example named start_clone.sh calling clone.pl
  
  Login as oracle user to raclinux3 and create a start_clone.sh script with the following content.
  
  ORACLE_BASE=/u01/app/oracle
  
  ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3
  
  cd $ORACLE_HOME/clone/bin
  
  THISNODE=’hostname -s’
  
  E01=ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3
  
  E02=ORACLE_HOME_NAME=OraDb11g_home1
  
  E03=ORACLE_BASE=/u01/app/oracle
  
  C01=”CLUSTER_NODES={raclinux1,raclinux2,raclinux3}”
  
  ##C01=”-O’CLUSTER_NODES={raclinux1, raclinux2, raclinux3}’”
  
  C02=”LOCAL_NODE=raclinux3″
  
  perl $ORACLE_HOME/clone/bin/clone.pl $E01 $E02 $E03 $C01 $C02
2. While logged in a oracle user to raclinux3 execute the start_clone.sh
  
  [oracle@raclinux3 bin]$ ./start_clone.sh
  
  ./runInstaller -clone -waitForCompletion ”ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3″ “ORACLE_HOME_NAME=OraDb11g_home1″ “ORACLE_BASE=/u01/app/oracle” “CLUSTER_NODES={raclinux1,raclinux2,raclinux3}” “LOCAL_NODE=raclinux3″ -silent -noConfig -nowait
  
  Starting Oracle Universal Installer…
  
  Checking swap space: must be greater than 500 MB. Actual 9818 MB Passed
  
  Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-11-16_03-07-49AM. Please wait …Oracle Universal Installer, Version 11.2.0.3.0 Production
  
  Copyright (C) 1999, 2011, Oracle. All rights reserved.
  
  You can find the log of this install session at:
  
  /u01/app/oraInventory/logs/cloneActions2011-11-16_03-07-49AM.log
  
  …..
  
  Performing tests to see whether nodes raclinux1,raclinux2 are available
  
  ……………………………………………………… 100% Done.
  
  Installation in progress (Wednesday, November 16, 2011 3:11:03 AM EET)
  
  …………………………………………………………………….. 80% Done.
  
  Install successful
  
  Linking in progress (Wednesday, November 16, 2011 3:11:17 AM EET)
  
  Link successful
  
  Setup in progress (Wednesday, November 16, 2011 3:13:51 AM EET)
  
  Setup successful
  
  End of install phases.(Wednesday, November 16, 2011 3:14:40 AM EET)
  
  WARNING:
  
  The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.
  
  /u01/app/oracle/product/11.2.0/db_3/root.sh #On nodes raclinux3
  
  To execute the configuration scripts:
  
  1. Open a terminal window
  
  2. Log in as “root”
  
  3. Run the scripts in each cluster node
  
  The cloning of OraDb11g_home1 was successful.
  
  Please check ‘/u01/app/oraInventory/logs/cloneActions2011-11-16_03-07-49AM.log’ for more details.
  
  [oracle@raclinux3 bin]$
3. Login as root on raclinux3 and execute root.sh
  
  [root@raclinux3 bin]# /u01/app/oracle/product/11.2.0/db_3/root.sh
  
  Check /u01/app/oracle/product/11.2.0/db_3/install/root_raclinux3.gj.com_2011-11-16_03-21-22.log for the output of root script
  
  [root@raclinux3 bin]#
  
  [root@raclinux3 bin]# cat /u01/app/oracle/product/11.2.0/db_3/install/root_raclinux3.gj.com_2011-11-16_03-21-22.log
  
  Performing root user operation for Oracle 11g
  
  The following environment variables are set as:
  
  ORACLE_OWNER= oracle
  
  ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_3
  
  Entries will be added to the /etc/oratab file as needed by
  
  Database Configuration Assistant when a database is created
  
  Finished running generic part of root script.
  
  Now product-specific root actions will be performed.
  
  Finished product-specific root actions.
  
  [root@raclinux3 bin]#
Manually add an instance to the new node

While dbca and OEM can be used to add an instance to raclinux3 the sections will explain how to manually add the instance.

All steps are to be executed as oracle user form any node of the cluster, for example raclinux2.
1. Add an instance RACDB3 for the RACDB database for node raclinux3
  
  [oracle@raclinux1 ~]$ srvctl add instance -d racdb -i RACDB3 -n raclinux3
  
  [oracle@raclinux1 ~]$
2. Set config parameters
  
  SQL> show parameter create
  
  NAME TYPE VALUE
  
  ———————————— ———– ——————————
  
  create_bitmap_area_size integer 8388608
  
  create_stored_outlines string
  
  db_create_file_dest string +DATA
  
  db_create_online_log_dest_1 string
  
  db_create_online_log_dest_2 string
  
  db_create_online_log_dest_3 string
  
  db_create_online_log_dest_4 string
  
  db_create_online_log_dest_5 string
  
  SQL>
  
  SQL> alter system set thread=3 scope=both sid=’RACDB3′;
  
  System altered.
  
  SQL>
  
  SQL> alter system set instance_number=3 scope=spfile sid=’RACDB3′;
  
  System altered.
  
  SQL>
  
  SQL> alter system set UNDO_TABLESPACE=UNDOTBS3 scope=both sid=’RACDB3′;
  
  System altered.
  
  SQL>
3. Create and enable redo log thread for the RACDB3 instance
  
  SQL> alter database add logfile thread 3 group 8;
  
  Database altered.
  
  SQL> alter database add logfile thread 3 group 9;
  
  Database altered.
  
  SQL> alter database add logfile thread 3 group 10;
  
  Database altered.
  
  SQL>
  
  SQL> alter database enable thread 3;
  
  Database altered.
  
  SQL>
4. Start instance RACDB3 on raclinux3
  
  [oracle@raclinux1 ~]$ srvctl start instance -d racdb -i RACDB3
  
  [oracle@raclinux1 ~]$
5. Verify add instance
  
  [oracle@raclinux3 ~]$ srvctl status database -d racdb
  
  Instance RACDB1 is running on node raclinux1
  
  Instance RACDB2 is running on node raclinux2
  
  Instance RACDB3 is running on node raclinux3
  
  [oracle@raclinux3 ~]$
  
  [oracle@raclinux3 ~]$ srvctl config database -d racdb
  
  Database unique name: RACDB
  
  Database name: RACDB
  
  Oracle home: /u01/app/oracle/product/11.2.0/db_3
  
  Oracle user: oracle
  
  Spfile: +DATA/racdb/spfileracdb.ora
  
  Domain:
  
  Start options: open
  
  Stop options: immediate
  
  Database role: PRIMARY
  
  Management policy: AUTOMATIC
  
  Server pools: RACDB
  
  Database instances: RACDB1,RACDB2,RACDB3
  
  Disk Groups: DATA,DATADG
  
  Mount point paths:
  
  Services:
  
  Type: RAC
  
  Database is administrator managed
  
  [oracle@raclinux3 ~]$
  
  SQL> select * from v$active_instances;
  
  INST_NUMBER INST_NAME
  
  ———– ————————————————————
  
  1 raclinux1.gj.com:RACDB1
  
  2 raclinux2.gj.com:RACDB2
  
  3 raclinux3.gj.com:RACDB3
  
  SQL>
Summary

In the article you looked at some practical use of clone.pl to clone GI and RDBMMS home in a RAC environment to extend the cluster on a new node. You had a glimpse at how to manually add a instance to the new RAC cluster node.

Filed under: oracle

Adding and Deleting a Node from Oracle RAC 11.2.0.3

gjilevski — Wed, 16 Nov 2011 22:58:23 +0000

Adding and Deleting a Node from Oracle RAC 11.2.0.3

In the article you will have a look at the steps related to extending an existing Oracle cluster by adding a node to the cluster and the opposite steps to remove a node from the cluster. Oracle 11.2.0.3 is used for verifying the reconfiguration steps in the article. Two nodes Oracle cluster consisting of nodes (raclinux1, raclinux2) with specifications described here and third node raclinux3 are used as a basis for the reconfiguration test described in the article. GI is installed using non-GNS setup and the database is admin policy managed. You can find information related to the upgrade to 11.2.0.3 here. The article will cover the following topics:

Add a node
- Add GI home to the new node
- Add RDBMS home to the new node
- Add instance to the new node
Remove a node
- Remove an instance from the node to be removed
- Remove RDBMS home from the node to be removed
- Remove GI home from the node to be removed

Add a node

Before you add a node make sure that the node is properly configured to support Oracle GI and RDBMS installation and database creation. For a detail list of the prerequisites for installing Oracle GI and RDBMS click here. As RAC3 VM running raclinux3 node was cloned from RAC1 running raclinux1 the prerequisites are met. I will illustrate the Cluster Verify Utility cluvfy commands to verify the prerequisites. The existing cluster nodes are raclinux1 and raclinux2. Node raclinux3 will be added.

Verify the prerequisites with cluvfy

cluvfy stage -post hwos -n raclinux3 -verbose

cluvfy comp peer -refnode raclinux1 -n raclinux3 -orainv oinstall -osdba asmdba -verbose

cluvfy stage -pre nodeadd -n raclinux3 -verbose

cluvfy stage -pre crsinst -n raclinux3 –verbose

Fix any problems related to failed check(s) and make sure that the prerequisites are met before proceeding further. For information related to the detail output of the commands look at Annex 1

1.1 Add GI home to the new node using addNode

In order to add GI to the new node raclinux3 addNode.sh will be used from an existing node raclinux2. Run, as grid user, addNode.sh from $GI_HOME/oui/bin assuming that you have met the prerequisites. If you want to override some errors set IGNORE_PREADDNODE_CHECKS variable to Y.

export IGNORE_PREADDNODE_CHECKS=Y

[grid@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Performing pre-checks for node addition

Checking node reachability…

Node reachability check passed from node “raclinux2″

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking CRS integrity…

Clusterware version consistency passed

CRS integrity check passed

Checking shared resources…

Checking CRS home location…

“/u01/app/11.2.0.3/grid” is shared

Shared resources check for node addition passed

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Check: Node connectivity for interface “eth0″

Node connectivity passed for interface “eth0″

TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity for interface “eth1″

Node connectivity passed for interface “eth1″

TCP connectivity check passed for subnet “10.10.20.0″

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “192.168.20.0″.

Subnet mask consistency check passed for subnet “10.10.20.0″.

Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “raclinux3:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux2:/u01/app/11.2.0.3/grid”

Free disk space check passed for “raclinux3:/tmp”

Free disk space check passed for “raclinux2:/tmp”

Check for multiple users with UID value 1100 passed

User existence check passed for “grid”

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check passed for “shmmax”

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “make”

Package existence check passed for “binutils”

Package existence check passed for “gcc(x86_64)”

Package existence check passed for “libaio(x86_64)”

Package existence check passed for “glibc(x86_64)”

Package existence check passed for “compat-libstdc++-33(x86_64)”

Package existence check passed for “elfutils-libelf(x86_64)”

Package existence check passed for “elfutils-libelf-devel”

Package existence check passed for “glibc-common”

Package existence check passed for “glibc-devel(x86_64)”

Package existence check passed for “glibc-headers”

Package existence check passed for “gcc-c++(x86_64)”

Package existence check passed for “libaio-devel(x86_64)”

Package existence check passed for “libgcc(x86_64)”

Package existence check passed for “libstdc++(x86_64)”

Package existence check passed for “libstdc++-devel(x86_64)”

Package existence check passed for “sysstat”

Package existence check passed for “ksh”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user’s primary group passed

Checking OCR integrity…

OCR integrity check passed

Checking Oracle Cluster Voting Disk configuration…

Oracle Cluster Voting Disk configuration check passed

Time zone consistency check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon’s boot time configuration check for slewing option passed

NTP common Time Server Check started…

PRVF-5408 : NTP Time Server “78.47.24.68″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “192.43.244.18″ is common only to the following nodes “raclinux2″

PRVF-5408 : NTP Time Server “129.69.1.153″ is common only to the following nodes “raclinux2″

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

User “grid” is not part of “root” group. Check passed

Checking consistency of file “/etc/resolv.conf” across nodes

File “/etc/resolv.conf” does not have both domain and search entries defined

domain entry in file “/etc/resolv.conf” is consistent across nodes

search entry in file “/etc/resolv.conf” is consistent across nodes

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000″ ms on following nodes: raclinux3

File “/etc/resolv.conf” is not consistent across nodes

Checking VIP configuration.

Checking VIP Subnet configuration.

Check for VIP Subnet configuration passed.

Checking VIP reachability

Check for VIP reachability passed.

Pre-check for node addition was unsuccessful on all the nodes.

[grid@raclinux2 bin]$

export IGNORE_PREADDNODE_CHECKS=Y

[grid@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9951 MB Passed

Oracle Universal Installer, Version 11.2.0.3.0 Production

Performing tests to see whether nodes raclinux1,raclinux3 are available

……………………………………………………… 100% Done.

…

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /u01/app/11.2.0.3/grid

New Nodes

Space Requirements

New Nodes

raclinux3

/u01: Required 4.30GB : Available 269.12GB

Installed Products

Product Names

Oracle Grid Infrastructure 11.2.0.3.0

Sun JDK 1.5.0.30.03

Installer SDK Component 11.2.0.3.0

Oracle One-Off Patch Installer 11.2.0.1.7

Oracle Universal Installer 11.2.0.3.0

Oracle USM Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Enterprise Manager Common Core Files 10.2.0.4.4

Oracle DBCA Deconfiguration 11.2.0.3.0

Oracle RAC Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Server) 11.2.0.3.0

Installation Plugin Files 11.2.0.3.0

Universal Storage Manager Files 11.2.0.3.0

Oracle Text Required Support Files 11.2.0.3.0

Automatic Storage Management Assistant 11.2.0.3.0

Oracle Database 11g Multimedia Files 11.2.0.3.0

Oracle Multimedia Java Advanced Imaging 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Oracle Multimedia Locator RDBMS Files 11.2.0.3.0

Oracle Core Required Support Files 11.2.0.3.0

Bali Share 1.1.18.0.0

Oracle Database Deconfiguration 11.2.0.3.0

Oracle Quality of Service Management (Client) 11.2.0.3.0

Expat libraries 2.0.1.0.1

Oracle Containers for Java 11.2.0.3.0

Perl Modules 5.10.0.0.1

Secure Socket Layer 11.2.0.3.0

Oracle JDBC/OCI Instant Client 11.2.0.3.0

Oracle Multimedia Client Option 11.2.0.3.0

LDAP Required Support Files 11.2.0.3.0

Character Set Migration Utility 11.2.0.3.0

Perl Interpreter 5.10.0.0.2

PL/SQL Embedded Gateway 11.2.0.3.0

OLAP SQL Scripts 11.2.0.3.0

Database SQL Scripts 11.2.0.3.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

SSL Required Support Files for InstantClient 11.2.0.3.0

SQL*Plus Files for Instant Client 11.2.0.3.0

Oracle Net Required Support Files 11.2.0.3.0

Oracle Database User Interface 2.2.13.0.0

RDBMS Required Support Files for Instant Client 11.2.0.3.0

RDBMS Required Support Files Runtime 11.2.0.3.0

XML Parser for Java 11.2.0.3.0

Oracle Security Developer Tools 11.2.0.3.0

Oracle Wallet Manager 11.2.0.3.0

Enterprise Manager plugin Common Files 11.2.0.3.0

Platform Required Support Files 11.2.0.3.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

RDBMS Required Support Files 11.2.0.3.0

Oracle Ice Browser 5.2.3.6.0

Oracle Help For Java 4.2.9.0.0

Enterprise Manager Common Files 10.2.0.4.3

Deinstallation Tool 11.2.0.3.0

Oracle Java Client 11.2.0.3.0

Cluster Verification Utility Files 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.3.0

Oracle LDAP administration 11.2.0.3.0

Cluster Verification Utility Common Files 11.2.0.3.0

Oracle Clusterware RDBMS Files 11.2.0.3.0

Oracle Locale Builder 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Buildtools Common Files 11.2.0.3.0

Oracle RAC Required Support Files-HAS 11.2.0.3.0

SQL*Plus Required Support Files 11.2.0.3.0

XDK Required Support Files 11.2.0.3.0

Agent Required Support Files 10.2.0.4.3

Parser Generator Required Support Files 11.2.0.3.0

Precompiler Required Support Files 11.2.0.3.0

Installation Common Files 11.2.0.3.0

Required Support Files 11.2.0.3.0

Oracle JDBC/THIN Interfaces 11.2.0.3.0

Oracle Multimedia Locator 11.2.0.3.0

Oracle Multimedia 11.2.0.3.0

HAS Common Files 11.2.0.3.0

Assistant Common Files 11.2.0.3.0

PL/SQL 11.2.0.3.0

HAS Files for DB 11.2.0.3.0

Oracle Recovery Manager 11.2.0.3.0

Oracle Database Utilities 11.2.0.3.0

Oracle Notification Service 11.2.0.3.0

SQL*Plus 11.2.0.3.0

Oracle Netca Client 11.2.0.3.0

Oracle Net 11.2.0.3.0

Oracle JVM 11.2.0.3.0

Oracle Internet Directory Client 11.2.0.3.0

Oracle Net Listener 11.2.0.3.0

Cluster Ready Services Files 11.2.0.3.0

Oracle Database 11g 11.2.0.3.0

—————————————————————————–

Instantiating scripts for add node (Monday, November 14, 2011 10:56:28 PM EET)

. 1% Done.

Instantiation of add node scripts complete

Copying to remote nodes (Monday, November 14, 2011 10:56:40 PM EET)

………………………………………………………………………………….. 96% Done.

Home copied to new nodes

Saving inventory on nodes (Monday, November 14, 2011 11:15:12 PM EET)

. 100% Done.

Save inventory complete

WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.

To register the new inventory please run the script at ‘/u01/app/oraInventory/orainstRoot.sh’ with root privileges on nodes ‘raclinux3′.

If you do not register the inventory, you may not be able to update or patch the products you installed.

The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

/u01/app/oraInventory/orainstRoot.sh #On nodes raclinux3

/u01/app/11.2.0.3/grid/root.sh #On nodes raclinux3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0.3/grid was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[grid@raclinux2 bin]$

After addNode.sh completes open a new session on the node to be added raclinux3 and run the suggested scripts as root user.

[root@raclinux3 grid]# /u01/app/oraInventory/orainstRoot.sh

Creating the Oracle inventory pointer file (/etc/oraInst.loc)

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

You have new mail in /var/spool/mail/root

[root@raclinux3 grid]#

[root@raclinux3 grid]# /u01/app/11.2.0.3/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params

Creating trace directory

OLR initialization – successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node raclinux1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@raclinux3 grid]#

1.2 Add a RDBMS home to the new node using addNode

After GI home has been added to node raclinux3 successfully from any existing node, for example raclinux2, run addNode.sh as oracle user.

[oracle@raclinux2 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={raclinux3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={raclinux3-vip}”

Performing pre-checks for node addition

Checking node reachability…

Node reachability check passed from node “raclinux2″

Checking user equivalence…

User equivalence check passed for user “oracle”

WARNING:

Node “raclinux3″ already appears to be part of cluster

Pre-check for node addition was successful.

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9543 MB Passed

Oracle Universal Installer, Version 11.2.0.3.0 Production

Performing tests to see whether nodes raclinux1,raclinux3 are available

……………………………………………………… 100% Done.

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /u01/app/oracle/product/11.2.0/db_3

New Nodes

Space Requirements

New Nodes

raclinux3

/u01: Required 4.53GB : Available 265.96GB

Installed Products

Product Names

Oracle Database 11g 11.2.0.3.0

Sun JDK 1.5.0.30.03

Installer SDK Component 11.2.0.3.0

Oracle One-Off Patch Installer 11.2.0.1.7

Oracle Universal Installer 11.2.0.3.0

Oracle USM Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Oracle DBCA Deconfiguration 11.2.0.3.0

Oracle RAC Deconfiguration 11.2.0.3.0

Oracle Database Deconfiguration 11.2.0.3.0

Oracle Configuration Manager Client 10.3.2.1.0

Oracle Configuration Manager 10.3.5.0.1

Oracle ODBC Driverfor Instant Client 11.2.0.3.0

LDAP Required Support Files 11.2.0.3.0

SSL Required Support Files for InstantClient 11.2.0.3.0

Bali Share 1.1.18.0.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

Oracle Real Application Testing 11.2.0.3.0

Oracle Database Vault J2EE Application 11.2.0.3.0

Oracle Label Security 11.2.0.3.0

Oracle Data Mining RDBMS Files 11.2.0.3.0

Oracle OLAP RDBMS Files 11.2.0.3.0

Oracle OLAP API 11.2.0.3.0

Platform Required Support Files 11.2.0.3.0

Oracle Database Vault option 11.2.0.3.0

Oracle RAC Required Support Files-HAS 11.2.0.3.0

SQL*Plus Required Support Files 11.2.0.3.0

Oracle Display Fonts 9.0.2.0.0

Oracle Ice Browser 5.2.3.6.0

Oracle JDBC Server Support Package 11.2.0.3.0

Oracle SQL Developer 11.2.0.3.0

Oracle Application Express 11.2.0.3.0

XDK Required Support Files 11.2.0.3.0

RDBMS Required Support Files for Instant Client 11.2.0.3.0

SQLJ Runtime 11.2.0.3.0

Database Workspace Manager 11.2.0.3.0

RDBMS Required Support Files Runtime 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

Exadata Storage Server 11.2.0.1.0

Provisioning Advisor Framework 10.2.0.4.3

Enterprise Manager Database Plugin — Repository Support 11.2.0.3.0

Enterprise Manager Repository Core Files 10.2.0.4.4

Enterprise Manager Database Plugin — Agent Support 11.2.0.3.0

Enterprise Manager Grid Control Core Files 10.2.0.4.4

Enterprise Manager Common Core Files 10.2.0.4.4

Enterprise Manager Agent Core Files 10.2.0.4.4

RDBMS Required Support Files 11.2.0.3.0

regexp 2.1.9.0.0

Agent Required Support Files 10.2.0.4.3

Oracle 11g Warehouse Builder Required Files 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.3.0

Oracle Text Required Support Files 11.2.0.3.0

Parser Generator Required Support Files 11.2.0.3.0

Oracle Database 11g Multimedia Files 11.2.0.3.0

Oracle Multimedia Java Advanced Imaging 11.2.0.3.0

Oracle Multimedia Annotator 11.2.0.3.0

Oracle JDBC/OCI Instant Client 11.2.0.3.0

Oracle Multimedia Locator RDBMS Files 11.2.0.3.0

Precompiler Required Support Files 11.2.0.3.0

Oracle Core Required Support Files 11.2.0.3.0

Sample Schema Data 11.2.0.3.0

Oracle Starter Database 11.2.0.3.0

Oracle Message Gateway Common Files 11.2.0.3.0

Oracle XML Query 11.2.0.3.0

XML Parser for Oracle JVM 11.2.0.3.0

Oracle Help For Java 4.2.9.0.0

Installation Plugin Files 11.2.0.3.0

Enterprise Manager Common Files 10.2.0.4.3

Expat libraries 2.0.1.0.1

Deinstallation Tool 11.2.0.3.0

Oracle Quality of Service Management (Client) 11.2.0.3.0

Perl Modules 5.10.0.0.1

JAccelerator (COMPANION) 11.2.0.3.0

Oracle Containers for Java 11.2.0.3.0

Perl Interpreter 5.10.0.0.2

Oracle Net Required Support Files 11.2.0.3.0

Secure Socket Layer 11.2.0.3.0

Oracle Universal Connection Pool 11.2.0.3.0

Oracle JDBC/THIN Interfaces 11.2.0.3.0

Oracle Multimedia Client Option 11.2.0.3.0

Oracle Java Client 11.2.0.3.0

Character Set Migration Utility 11.2.0.3.0

Oracle Code Editor 1.2.1.0.0I

PL/SQL Embedded Gateway 11.2.0.3.0

OLAP SQL Scripts 11.2.0.3.0

Database SQL Scripts 11.2.0.3.0

Oracle Locale Builder 11.2.0.3.0

Oracle Globalization Support 11.2.0.3.0

SQL*Plus Files for Instant Client 11.2.0.3.0

Required Support Files 11.2.0.3.0

Oracle Database User Interface 2.2.13.0.0

Oracle ODBC Driver 11.2.0.3.0

Oracle Notification Service 11.2.0.3.0

XML Parser for Java 11.2.0.3.0

Oracle Security Developer Tools 11.2.0.3.0

Oracle Wallet Manager 11.2.0.3.0

Cluster Verification Utility Common Files 11.2.0.3.0

Oracle Clusterware RDBMS Files 11.2.0.3.0

Oracle UIX 2.2.24.6.0

Enterprise Manager plugin Common Files 11.2.0.3.0

HAS Common Files 11.2.0.3.0

Precompiler Common Files 11.2.0.3.0

Installation Common Files 11.2.0.3.0

Oracle Help for the Web 2.0.14.0.0

Oracle LDAP administration 11.2.0.3.0

Buildtools Common Files 11.2.0.3.0

Assistant Common Files 11.2.0.3.0

Oracle Recovery Manager 11.2.0.3.0

PL/SQL 11.2.0.3.0

Generic Connectivity Common Files 11.2.0.3.0

Oracle Database Gateway for ODBC 11.2.0.3.0

Oracle Programmer 11.2.0.3.0

Oracle Database Utilities 11.2.0.3.0

Enterprise Manager Agent 10.2.0.4.3

SQL*Plus 11.2.0.3.0

Oracle Netca Client 11.2.0.3.0

Oracle Multimedia Locator 11.2.0.3.0

Oracle Call Interface (OCI) 11.2.0.3.0

Oracle Multimedia 11.2.0.3.0

Oracle Net 11.2.0.3.0

Oracle XML Development Kit 11.2.0.3.0

Database Configuration and Upgrade Assistants 11.2.0.3.0

Oracle JVM 11.2.0.3.0

Oracle Advanced Security 11.2.0.3.0

Oracle Internet Directory Client 11.2.0.3.0

Oracle Enterprise Manager Console DB 11.2.0.3.0

HAS Files for DB 11.2.0.3.0

Oracle Net Listener 11.2.0.3.0

Oracle Text 11.2.0.3.0

Oracle Net Services 11.2.0.3.0

Oracle Database 11g 11.2.0.3.0

Oracle OLAP 11.2.0.3.0

Oracle Spatial 11.2.0.3.0

Oracle Partitioning 11.2.0.3.0

Enterprise Edition Options 11.2.0.3.0

—————————————————————————–

Instantiating scripts for add node (Tuesday, November 15, 2011 12:36:09 AM EET)

. 1% Done.

Instantiation of add node scripts complete

Copying to remote nodes (Tuesday, November 15, 2011 12:36:23 AM EET)

………………………………………………………………………………….. 96% Done.

Home copied to new nodes

Saving inventory on nodes (Tuesday, November 15, 2011 12:58:10 AM EET)

. 100% Done.

Save inventory complete

WARNING:

The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

/u01/app/oracle/product/11.2.0/db_3/root.sh #On nodes raclinux3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db_3 was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[oracle@raclinux2 bin]$

Run as root on the new node raclinux3

[root@raclinux3 bin]# /u01/app/oracle/product/11.2.0/db_3/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/11.2.0/db_3

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Finished product-specific root actions.

[root@raclinux3 bin]#

Add an instance on raclinux3 using dbca

Login as oracle on a cluster node where there is a running instance, for example raclinux2, and run dbca.

Select Oracle RAC database and press Next.

Select Instance management and press Next.

Select Add Instance.

Select a database and enter username and credentials.

Press Next.

Enter Instance name, select node and press Next.

Press Next.

Review and press OK.

Wait until dbca completes.
Verification.

Verify that instance has been added successfully.

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

———– ————————————————————

1 raclinux1.gj.com:RACDB1

2 raclinux2.gj.com:RACDB2

3 raclinux3.gj.com:RACDB3

SQL>

[grid@raclinux3 grid]$ olsnodes -n -t

raclinux1 1 Pinned

raclinux2 2 Pinned

raclinux3 3 Unpinned

[grid@raclinux3 grid]$

Run the following command as grid user.

cluvfy stage -post nodeadd -n raclinux3 –verbose

The output of the command is in Annex 1.

Remove a node

In order to remove a node raclinux3 from the cluster the following ordered activities should take place:

Remove instance RACDB3 on raclinux3
Remove RDBMS home from raclinux3
Remove GI home from aclinux3

2.1 Remove instance RACDB3 on raclinux3

Select Oracle RAC database and press Next.

Select Instance Management.

Select Delete an Instance.

Select a database and enter username and credentials.

Select the instance

Press OK.

Wait until instance removal takes place.

Verify that the instance is removed from a node that is to remain.

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

———– ————————————————————

1 raclinux1.gj.com:RACDB1

2 raclinux2.gj.com:RACDB2

SQL>

Remove RDBMS home from raclinux3

On a node to be removed log in as oracle user and run the following command from $ORACLE_HOME/oui/bin.

[oracle@raclinux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3 “CLUSTER_NODES={raclinux3}” -local

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9832 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[oracle@raclinux3 bin]$

Run from deinstall with option local from $ORACLE_HOME/deinstall from the node to be removed (raclinux3) in order to remove the RDBMS binaries.

[oracle@raclinux3 deinstall]$ ./deinstall -local

Checking for required files and bootstrapping …

Please wait …

Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/oracle/product/11.2.0/db_3

Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database

Oracle Base selected for deinstall is: /u01/app/oracle

Checking for existence of central inventory location /u01/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.3/grid

The following nodes are part of this cluster: raclinux3

Checking for sufficient temp space availability on node(s) : ‘raclinux3′

## [END] Install check configuration ##

Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2011-11-15_02-05-17-AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2011-11-15_02-05-24-AM.log

Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_check2011-11-15_02-05-28-AM.log

Enterprise Manager Configuration Assistant END

Oracle Configuration Manager check START

OCM check log file location : /u01/app/oraInventory/logs//ocm_check8370.log

Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /u01/app/11.2.0.3/grid

The cluster node(s) on which the Oracle home deinstallation will be performed are:raclinux3

Since -local option has been specified, the Oracle home will be deinstalled only on the local node, ‘raclinux3′, and the global configuration will be removed.

Oracle Home selected for deinstall is: /u01/app/oracle/product/11.2.0/db_3

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

The option -local will not modify any database configuration for this Oracle home.

No Enterprise Manager configuration to be updated for any database(s)

No Enterprise Manager ASM targets to update

No Enterprise Manager listener targets to migrate

Checking the config status for CCR

Oracle Home exists with CCR directory, but CCR is not configured

CCR check is finished

Do you want to continue (y – yes, n – no)? [n]: y

A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2011-11-15_02-04-55-AM.out’

Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2011-11-15_02-04-55-AM.err’

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_clean2011-11-15_02-05-28-AM.log

Updating Enterprise Manager ASM targets (if any)

Updating Enterprise Manager listener targets (if any)

Enterprise Manager Configuration Assistant END

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2011-11-15_02-06-10-AM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2011-11-15_02-06-10-AM.log

De-configuring Local Net Service Names configuration file…

Local Net Service Names configuration file de-configured successfully.

De-configuring backup files…

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START

OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean8370.log

Oracle Configuration Manager clean END

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/oracle/product/11.2.0/db_3′ from the central inventory on the local node : Done

Delete directory ‘/u01/app/oracle/product/11.2.0/db_3′ on the local node : Done

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on local node. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2011-11-15_02-04-50AM’ on node ‘raclinux3′

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Cleaning the config for CCR

As CCR is not configured, so skipping the cleaning of CCR configuration

CCR clean is finished

Successfully detached Oracle home ‘/u01/app/oracle/product/11.2.0/db_3′ from the central inventory on the local node.

Successfully deleted directory ‘/u01/app/oracle/product/11.2.0/db_3′ on the local node.

Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[oracle@raclinux3 deinstall]$

From any node that will remain, for example raclinux2, run the following command from $ORACLE_HOME/oui/bin while logged in as oracle user.

[oracle@raclinux2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3 “CLUSTER_NODES={raclinux1, raclinux2}”

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9579 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[oracle@raclinux2 bin]$

Remove GI home from raclinux3

2.3.1 While logged in as root unpin the node to be removed from any node.

[root@raclinux3 bin]# ./crsctl unpin css -n raclinux3

CRS-4667: Node raclinux3 successfully unpinned.

[root@raclinux3 bin]#

2.3.2 As root from the node to be removed raclinux3 go to $GI_HOME/crs/install and execute the following command ./rootcrs.pl -deconfig -deinstall –force.

[root@raclinux3 install]# ./rootcrs.pl -deconfig -deinstall -force

Using configuration parameter file: ./crsconfig_params

Network exists: 1/192.168.20.0/255.255.255.0/eth0, type static

VIP exists: /raclinux1-vip/192.168.20.51/192.168.20.0/255.255.255.0/eth0, hosting node raclinux1

VIP exists: /raclinux2-vip/192.168.20.52/192.168.20.0/255.255.255.0/eth0, hosting node raclinux2

VIP exists: /raclinux3-vip/192.168.20.53/192.168.20.0/255.255.255.0/eth0, hosting node raclinux3

GSD exists

ONS exists: Local port 6100, remote port 6200, EM port 2016

CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.registry.acfs’ on ‘raclinux3′ succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘raclinux3′

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.DATADG.dg’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.DATADG.dg’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.DATA.dg’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.asm’ on ‘raclinux3′ succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘raclinux3′ has completed

CRS-2677: Stop of ‘ora.crsd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.crf’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.asm’ on ‘raclinux3′

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.evmd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.mdnsd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.crf’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘raclinux3′ succeeded

CRS-2677: Stop of ‘ora.asm’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.cssd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.gipcd’ on ‘raclinux3′ succeeded

CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘raclinux3′

CRS-2677: Stop of ‘ora.gpnpd’ on ‘raclinux3′ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘raclinux3′ has completed

CRS-4133: Oracle High Availability Services has been stopped.

Successfully deconfigured Oracle clusterware stack on this node

[root@raclinux3 install]#

2.3.3 Execute as root from a node that is to remain the following command.

[root@raclinux2 bin]# ./crsctl delete node -n raclinux3

CRS-4661: Node raclinux3 successfully deleted.

[root@raclinux2 bin]#

2.3.4 From the node to be deleted log in as grid and from $GI_HOME/oui/bin execute the following command.

[grid@raclinux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0.3/grid “CLUSTER_NODES={raclinux3}” CRS=TRUE -local

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 10001 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[grid@raclinux3 bin]$

2.3.5 From the node to be deleted log in as grid and from $GI_HOME/deinstall the following deinstall command with a local option and follow the requests.

./deinstall -local

[grid@raclinux3 deinstall]$ ./deinstall -local

Checking for required files and bootstrapping …

Please wait …

Location of logs /tmp/deinstall2011-11-15_02-38-50AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/11.2.0.3/grid

Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster

Oracle Base selected for deinstall is: /u01/app/grid

Checking for existence of central inventory location /u01/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home

The following nodes are part of this cluster: raclinux3

Checking for sufficient temp space availability on node(s) : ‘raclinux3′

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2011-11-15_02-38-50AM/logs//crsdc.log

Enter an address or the name of the virtual IP used on node “raclinux3″[raclinux3-vip]

192.168.20.53

The following information can be collected by running “/sbin/ifconfig -a” on node “raclinux3″

Enter the IP netmask of Virtual IP “192.168.20.53″ on node “raclinux3″[255.255.255.0]

Enter the network interface name on which the virtual IP address “192.168.20.53″ is active

Enter an address or the name of the virtual IP[]

192.168.20.53

The following information can be collected by running “/sbin/ifconfig -a” on node “raclinux3″

Enter the IP netmask of the virtual IP “192.168.20.53″[]

255.255.255.0

Enter the network interface name on which the virtual IP address “192.168.20.53″ is active

eth0

Enter an address or the name of the virtual IP[]

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/netdc_check2011-11-15_02-40-55-AM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/asmcadc_check2011-11-15_02-41-00-AM.log

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is:

The cluster node(s) on which the Oracle home deinstallation will be performed are:raclinux3

Since -local option has been specified, the Oracle home will be deinstalled only on the local node, ‘raclinux3′, and the global configuration will be removed.

Oracle Home selected for deinstall is: /u01/app/11.2.0.3/grid

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

Following RAC listener(s) will be de-configured: LISTENER

Option -local will not modify any ASM configuration.

Do you want to continue (y – yes, n – no)? [n]: y

A log of this session will be written to: ‘/tmp/deinstall2011-11-15_02-38-50AM/logs/deinstall_deconfig2011-11-15_02-39-29-AM.out’

Any error messages from this session will be written to: ‘/tmp/deinstall2011-11-15_02-38-50AM/logs/deinstall_deconfig2011-11-15_02-39-29-AM.err’

######################## CLEAN OPERATION START ########################

ASM de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/asmcadc_clean2011-11-15_02-41-03-AM.log

ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2011-11-15_02-38-50AM/logs/netdc_clean2011-11-15_02-41-03-AM.log

De-configuring RAC listener(s): LISTENER

De-configuring listener: LISTENER

Stopping listener on node “raclinux3″: LISTENER

Warning: Failed to stop listener. Listener may not be running.

Listener de-configured successfully.

De-configuring Naming Methods configuration file…

Naming Methods configuration file de-configured successfully.

De-configuring backup files…

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

—————————————->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node “raclinux3″.

/tmp/deinstall2011-11-15_02-38-50AM/perl/bin/perl -I/tmp/deinstall2011-11-15_02-38-50AM/perl/lib -I/tmp/deinstall2011-11-15_02-38-50AM/crs/install /tmp/deinstall2011-11-15_02-38-50AM/crs/install/rootcrs.pl -force -deconfig -paramfile ”/tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp”

Press Enter after you finish running the above commands

<—————————————-

Remove the directory: /tmp/deinstall2011-11-15_02-38-50AM on node:

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/11.2.0.3/grid’ from the central inventory on the local node : Done

Delete directory ‘/u01/app/11.2.0.3/grid’ on the local node : Done

Delete directory ‘/u01/app/oraInventory’ on the local node : Done

Delete directory ‘/u01/app/grid’ on the local node : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2011-11-15_02-38-50AM’ on node ‘raclinux3′

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Following RAC listener(s) were de-configured successfully: LISTENER

Oracle Clusterware is stopped and successfully de-configured on node “raclinux3″

Oracle Clusterware is stopped and de-configured successfully.

Successfully detached Oracle home ‘/u01/app/11.2.0.3/grid’ from the central inventory on the local node.

Successfully deleted directory ‘/u01/app/11.2.0.3/grid’ on the local node.

Successfully deleted directory ‘/u01/app/oraInventory’ on the local node.

Successfully deleted directory ‘/u01/app/grid’ on the local node.

Oracle Universal Installer cleanup was successful.

Run ‘rm -rf /etc/oraInst.loc’ as root on node(s) ‘raclinux3′ at the end of the session.

Run ‘rm -rf /opt/ORCLfmap’ as root on node(s) ‘raclinux3′ at the end of the session.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[grid@raclinux3 deinstall]$

[root@raclinux3 grid]# /tmp/deinstall2011-11-15_02-38-50AM/perl/bin/perl -I/tmp/deinstall2011-11-15_02-38-50AM/perl/lib -I/tmp/deinstall2011-11-15_02-38-50AM/crs/install /tmp/deinstall2011-11-15_02-38-50AM/crs/install/rootcrs.pl -force -deconfig -paramfile ”/tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp”

Using configuration parameter file: /tmp/deinstall2011-11-15_02-38-50AM/response/deinstall_Ora11g_gridinfrahome2.rsp

****Unable to retrieve Oracle Clusterware home.

Start Oracle Clusterware stack and try again.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Modify failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Delete failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

################################################################

# You must kill processes or reboot the system to properly #

# cleanup the processes started by Oracle clusterware #

################################################################

ACFS-9313: No ADVM/ACFS installation detected.

Either /etc/oracle/olr.loc does not exist or is not readable

Make sure the file exists and it has read and execute access

Either /etc/oracle/olr.loc does not exist or is not readable

Make sure the file exists and it has read and execute access

Failure in execution (rc=-1, 256, No such file or directory) for command /etc/init.d/ohasd deinstall

error: package cvuqdisk is not installed

Successfully deconfigured Oracle clusterware stack on this node

[root@raclinux3 grid]#

2.3.6 From any remaining node, for example raclinux2, while logged in as grid user from $GI_HOME/oui/bin execute the following command.

[grid@raclinux2 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0.3/grid “CLUSTER_NODES={raclinux1, raclinux2}” CRS=TRUE

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB. Actual 9581 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

‘UpdateNodeList’ was successful.

[grid@raclinux2 bin]$

2.3.7 Verify that a node has been removed.

[grid@raclinux2 bin]$ olsnodes -n -t

raclinux1 1 Pinned

raclinux2 2 Pinned

[grid@raclinux2 bin]$

cluvfy stage -post nodedel -n raclinux3

[grid@raclinux2 bin]$ cluvfy stage -post nodedel -n raclinux3

Performing post-checks for node removal

Checking CRS integrity…

Clusterware version consistency passed

CRS integrity check passed

Node removal check passed

Post-check for node removal was successful.

[grid@raclinux2 bin]$

Annex 1

[grid@raclinux2 ~]$ cluvfy stage -post hwos -n raclinux3 -verbose

Performing post-checks for hardware and operating system setup

Checking node reachability…

Check: Node reachability from node “raclinux2″

Destination Node Reachable?

———————————— ————————

raclinux3 yes

Result: Node reachability check passed from node “raclinux2″

Checking user equivalence…

Check: User equivalence for user “grid”

Node Name Status

———————————— ————————

raclinux3 passed

Result: User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Node Name Status

———————————— ————————

raclinux3 passed

Verification of the hosts config file successful

Interface information for node “raclinux3″

Name IP Address Subnet Gateway Def. Gateway HW Address MTU

—— ————— ————— ————— ————— —————– ——

eth0 192.168.20.23 192.168.20.0 0.0.0.0 UNKNOWN 08:00:27:86:CA:20 1500

eth1 10.10.20.23 10.10.20.0 0.0.0.0 UNKNOWN 08:00:27:A4:7B:A4 1500

eth2 192.168.156.103 192.168.156.0 0.0.0.0 UNKNOWN 08:00:27:8C:A0:9B 1500

eth3 192.168.2.24 192.168.2.0 0.0.0.0 UNKNOWN 08:00:27:E5:70:A8 1500

Check: Node connectivity of subnet “192.168.20.0″

Result: Node connectivity passed for subnet “192.168.20.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.20.23 passed

Result: TCP connectivity check passed for subnet “192.168.20.0″

Check: Node connectivity of subnet “10.10.20.0″

Result: Node connectivity passed for subnet “10.10.20.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “10.10.20.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:10.10.20.23 passed

Result: TCP connectivity check passed for subnet “10.10.20.0″

Check: Node connectivity of subnet “192.168.156.0″

Result: Node connectivity passed for subnet “192.168.156.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.156.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.156.103 passed

Result: TCP connectivity check passed for subnet “192.168.156.0″

Check: Node connectivity of subnet “192.168.2.0″

Result: Node connectivity passed for subnet “192.168.2.0″ with node(s) raclinux3

Check: TCP connectivity of subnet “192.168.2.0″

Source Destination Connected?

—————————— —————————— —————-

raclinux2:192.168.20.22 raclinux3:192.168.2.24 passed

Result: TCP connectivity check passed for subnet “192.168.2.0″

Interfaces found on subnet “192.168.20.0″ that are likely candidates for a private interconnect are:

raclinux3 eth0:192.168.20.23

Interfaces found on subnet “10.10.20.0″ that are likely candidates for a private interconnect are:

raclinux3 eth1:10.10.20.23

Interfaces found on subnet “192.168.156.0″ that are likely candidates for a private interconnect are:

raclinux3 eth2:192.168.156.103

Interfaces found on subnet “192.168.2.0″ that are likely candidates for a private interconnect are:

raclinux3 eth3:192.168.2.24

WARNING:

Could not find a suitable set of interfaces for VIPs

Result: Node connectivity check passed

Checking multicast communication…

Checking subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “10.10.20.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “192.168.156.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.156.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Checking subnet “192.168.2.0″ for multicast communication with multicast group “230.0.1.0″…

Check of subnet “192.168.2.0″ for multicast communication with multicast group “230.0.1.0″ passed.

Check of multicast communication passed.

Checking for multiple users with UID value 0

Result: Check for multiple users with UID value 0 passed

Check: Time zone consistency

Result: Time zone consistency check passed

Checking shared storage accessibility…

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sda raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdb raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdc raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdd raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sde raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdf raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdg raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdh raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdi raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdj raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdk raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdl raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdm raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdn raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdo raclinux3

Disk Sharing Nodes (1 in count)

———————————— ————————

/dev/sdp raclinux3

Shared storage check was successful on nodes “raclinux3″

Post-check for hardware and operating system setup was successful.

[grid@raclinux2 ~]$

[grid@raclinux2 ~]$ cluvfy comp peer -refnode raclinux1 -n raclinux3 -orainv oinstall -osdba asmdba -verbose

Verifying peer compatibility

Checking peer compatibility…

Compatibility check: Physical memory [reference node: raclinux1]