How to Change Interconnect/Public Network (Interface or Subnet) in Oracle Clusterware 10gr2 and 11gr2

The purpose of this article is to describe how to change or update the cluster_interconnect and/or public interface information that is stored in OCR. It may be necessary to change or update interface names, or subnet associated with an interface if there is a network change affecting the servers, or if the original information that was input during the installation was incorrect.   It may also be the case that for some reason, the Oracle Interface Configuration Assistant  ('oifcfg')  did not succeed during the installation.
Network information(interface, subnet and role of each interface) for Oracle Clusterware is managed by 'oifcfg', but actual IP address for each interfaces are not - 'oifcfg' can not update IP address information. 'oifcfg getif' can be used to find out currently configured interfaces in OCR:
# $ORA_CRS_HOME/bin/oifcfg getif
eth0 10.2.156.0 global public
eth1 192.168.0.0 global cluster_interconnect
Above example is from a Linux system, but the syntax is the same for any platform.  On Unix/Linux systems, the interface names are generally assigned by the OS, and standard names vary by platform. For Windows systems, see additional notes below. Above example shows currently interface eth0 is used for public with subnet 10.2.156.0, and eth1 for cluster_interconnect/private with subnet  192.168.0.0.
If the subnet for either the 'public' or 'cluster_interconnect' interface is incorrect and needs to be changed, you should make the changes using oifcfg as the same user who did the original CRS installation, because the 'setif' option updates the OCR and sets permissions in the OCR for the interface based on who is running the command.  This is true on all platforms, including Windows.
1. Public Network Change
If the change is only public IP address and the new ones are still in the same subnet, nothing needs to be done on clusterware level (all changes needs to be done on OS level to reflect the change). If the change involves different subnet or interface, as there is not a 'modify' option - you will need to delete the interface and add it back with the correct information.  So, in the example here, the subnet is being changed from 10.2.156.0  to 10.2.166.0 via two separate commands - first a 'delif'  followed by a 'setif':
# $ORA_CRS_HOME/bin/oifcfg delif -global eth0
# $ORA_CRS_HOME/bin/oifcfg setif -global eth0/10.2.166.0:public
syntax: oifcfg setif <interface-name>/<subnet>:<cluster_interconnect|public>
2. Private Network Change
2a.For pre-11gR2, if you wish to change the cluster_interconnect information and/or private IP address, hosts file needs to be modified on each node to reflect the change while the Oracle Clusterware Stack is down on all nodes. After the stack has restarted, to change the cluster_interconnect used by RDBMS and ASM instances, run oifcfg. In this example:
# $ORA_CRS_HOME/bin/oifcfg delif -global eth1
# $ORA_CRS_HOME/bin/oifcfg setif -global eth1/192.168.1.0:cluster_interconnect
2b.For 11gR2 and higher do the following procedure:
2b1. Ensure CRS is running on ALL cluster nodes in the cluster.
    As grid user, add new interface:
    Find the interface which needs to be removed. For example:
    $ oifcfg getif
    eth1 100.17.10.0 global public
    eth0 192.168.0.0 global cluster_interconnect
    Here the eth0 interface will be replaced by bond0 interface.
    Add new interface bond0:
    # oifcfg setif -global <interface>/<subnet>:cluster_interconnect
    # oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
    This can be done with -global option even if the interface is not available yet, but this     can not be done with -node option if the interface is not available, it will lead to node     eviction.
2b2. Verify the change:
    # oifcfg getif
2b3. Shutdown CRS on all nodes and disable the CRS  as root user:
    # crsctl stop crs
    # crsctl disable crs
2b4. Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change.
    # ifconfig -a
    # ping <private hostname>
2b5. Enable CRS and restart CRS on all nodes as root user:
    # crsctl enable crs
    # crsctl start crs
2b6. Remove the old interface:
    # oifcfg delif -global eth0
2b7. Verify the correct interface subnet is in use by re-running oifcfg with the 'getif' option:
    # $ORA_CRS_HOME/bin/oifcfg getif
    eth0 10.2.166.0 global public
    eth1 192.168.1.0 global cluster_interconnect
Notes for Windows Systems
The syntax for changing the interfaces on Windows/RAC clusters is the same as on Unix or Linux, but the interface names will be slightly different.   On Windows systems, the default names assigned to the interfaces are generally names such as:
Local Area Connection
Local Area Connection 1
Local Area Connection 2
If you are using an interface name that has spaces in it, such as the examples above, you must enclose the name in quotes.   Also, be aware that it is case sensitive.

How to change VIP address in Oracle cluster 10g r2  and 11g r2

The VIP address is a static IP address with a virtual host name defined and resolved through either the DNS or your hosts file. During Oracle Clusterware installation, you are prompted to enter a virtual IP and virtual host name for each of the nodes in the cluster. Virtual IP addresses  are stored in OCR, and different components within the Oracle Clusterware HA framework depend on these VIPs. Generally, VIP addresses can be changed during a scheduled, complete cluster outage. However, in some cases, downtime is not required. For example, if the VIP for node 1 was entered incorrectly in DNS, then the modification would need to occur only on that node. From 10.2.0.3 onwards, the ASM/database instance dependency on the VIP resource is removed, so the VIP could be modified without having to take down the ASM/database instance, only client connections to this node be impacted during this time. In this article, I am going to show the general procedure for Oracle 10G R2 and 11G R2.

• Confirm the current IP address for the VIP by running the ifconfig -a command. On Windows, run ipconfig /all command. You could see the current VIP bound to one of network interface.

         # ifconfig -a
         $ srvctl config nodeapps -n <node> -a

• Stop all resources that are dependent on the VIP on that node (must be done for Oracle 10.2.0.2 upwards). First, stop the database instance, and then the ASM instance. When done, stop nodeapps.

          $ srvctl stop asm -n <node_name>
          $ srvctl stop instance -d database_name -i instance_name
          $ srvctl stop nodeapps -n node_name
          For 11g r2
         $ srvctl stop instance -d <database_name> -n <node_name>
         $ srvctl stop vip -n <node_name>   -f

• Verify that the VIP is no longer running by executing the ifconfig -a agian, and confirm that its interface is no longer listed in the output. if the interface still shows as online, this is an indication that a resource which dependent on the VIP is still running. The crs_stat -t command can help to show resources that are still online.

         $ crs_stat -t
         for 11g r2
         $ crsctl stat res -t
         Note that the -f option is required for 11G R2

• Make any changes necessary to all nodes' /etc/hosts file and make the necessary DNS changes, to associate the new IP address with the old host name.

• Modify the VIP resource

          as root user:
          # srvctl modify nodeapps -n <node>   -A  <new_vip_address or     new_vip_hostname>/netmask
          /[if1[if2....]]>
          eg:
          # srvctl modify nodeapps -n racnode1  -A  racnode1-nvip/255.255.255.0/eth2
          Verify the change
          $ srvctl config nodeapps -n <node_name> -a
          $ srvctl config nodeapps -a (11G R2)

• Restarting resource

          $ srvctl start nodeapps -n <node_name>
          $ srvctl start instance -d <database_name> -i <instance_name>
          $ srvctl start asm -n <node_name>

• Others

           Modify listener.ora and tnsnames.ora for release prior to 11g R2 to reflect the VIP     change if necessary.

Manually control Oracle Clusterware Stack (10G R2)

In this article I am going to show how to  use commands to control manage Oracle Clusterware (10G R2).
This article I uses and refers to some Oracle documents and Oracle resources (Oracle Metalink).
CRS is the primary program for managing High Availability oprations of applications within the cluster. Applications that CRS manages are called resources. By default, CRS can manage RAC resources such as database instance, ASM instances, listeners, instance VIPs, services, ONS, and GSD. CRS is also able to manage other types of application processes adn application VIPs.
CRS resources are managed according to their configuration parameters(resource profile) stored in OCR and an action script stored anywhere you want.
CRS provides the following commands to support the life cycle of resource:
CRS_PROFILE creates and edit a resource profile.
CRS_REGISTER adds the resource to the list of applications managed by CRS
CRS_START starts the resource
CRS_STAT informs you about the current status of a list of resources
CRS_RELOCATE moves the resource to another node of cluster.
CRS_UNREGISTER removes the resource from CRS
RAC RESOURCES
The CRS_START -t command shows you all the resources currently under Oracle Clusterware control. The resources start with the prefix .ora are the resources that implement RAC HA in cluster environment.
The state of resource can be ONLINE, OFFLINE, UNKNOWN. UNKNOWN results from a failed start/stop action, and can be reset by a CRS_STOP -f command.
You  can use the CRS_STAT -p resource_name command to show OCR contents for the named resource. Here is brief description of the most important attributes:
NAME is the name of the application resource.
TYPE always is APPLICATION for all CRS resources.
ACTION_SCRIPT is the name and location of the action script.
ACTIVE_PLACEMENT default to 0
AUTO_START is a flag indicating whether Oracle Clusterware should automatically start a resource after a cluster restart. When set to 0, Oracle Clusterware starts the resource only if it had been running before the restart. When set to 1, Oracle Clusterware always starts the resource after a start. When set to 2, Oracle Clusterware never starts the resource.
CHECK_INTERVAL is the time interval, in seconds, between repeated executions of the check command for the application.
FAILOVER_DELAY is the amount of time, in seconds, that Oracle Clusterware waits before trying to restart or fail over a resource.
PLACEMENT defines the placement policy( BALANCED, FAVORED, or RESTRICTED) that specifies how Oracle Clusterware chooses the cluster node on which to restart the resource:
HOSTING_MEMBERS  defines a list of ordered nodes that can host the resource
BALANCED: OC (Oracle Clusterware) favors starting or restarting the application on the node that is currently running the fewest resources.
FAVORED: OC refers to the list of nodes in the HOSTING_MEMBERS attribute of the application profile. If none of the nodes in the hosting node list are available, then OC places the application on any available node.
RESTRICTED: similar to the FAVORED policy, except that if none of the nodes on the hosting list are available, then OC does not start or restart the application
VOTING DISK
CSS is the service that determines which nodes in the cluster are available, and provides cluster group membership and locking services. CSS determines node availability via communication through a dedicated private network with a voting disk used as a secondary communication mechanism. This can be done by sending heartbeat messages through the network. The voting disk is a shared raw disk patition or file on a cluster file system that is available to all nodes in the cluster. The voting disk is used to communicate the node state information used to determine which nodes go offline. Without voting disk, it can not determine whether it is experiencing a network failure or other nodes are no longer available.
The CSS has two inportant parameters:
MISSCOUNT presents the maximum time, in seconds, that a network heartbeat across the interconnect can be missed before entering into a cluster reconfiguration for node eviction purposes. The default value for the MISSCOUNT parameter value is 30 seconds.
DISKTIMEOUT parameter represents the maximum time, in seconds, that disk heartbeat can be missed before entering into a cluster reconfiguration for node eviction purposes. Its default value is 200 seconds.
MULTIPLEXING VOTING DISK
Voting disk is a vital resource for your cluster availability. It is desirable to use multiple voting disk when using less reliable storage. OC requires you have at least three voting disks to avoid a single point of failure. The multiplexed voting disk should be located on physically idependent storage devices. You can have up to 32 voting disks, using the following formular to determine the number of voting disks: V = F*2 +1 where V is number of voting disks, and F id the number of disk failures you want to survive.
CHANGE VOTING DISK CONFIGURATION
To add a new voting disk:
    # crsctl add css votedisk < new voting disk path >
To remove a voting disk:
    # crsctl delete css votedisk <old voting disk path>
If your cluster is down, then you can use the -FORCE option:
    # crsctl add css votedisk < new voting disk path> -force
    # crsctl delete css votedisk <old voting disk path> -force
Note that you can not change your voting disk configuration online. To work around the problem, perform the configuration change with the -force option while clusterware is down on all nodes.
BACKUP and RECOVER YOUR VOTING DISKS
It is recommended to use symbolic links to specify your voting disk path because voting disk paths are stored in OCR, and it is not supported to edit the OCR file directly.
A new backup of one your available voting disks should be taken any time a new node is added, or an existing node is removed. Using the dd command (ocopy in Windows environment). A backup taken via the dd command can be a hot backup. Using the following commands to complete this task:
List the voting disks currently used by CSS
    $ crsctl query css votedisk
Backup a voting disk using command:
    $ dd if=<voting disk path> of=<backup path> bs=4k
The block size for the dd command should be 4K.
Recover voting disks by restoring the first one by using the dd command, and then multiplex it if necessaty. If no voting disk is available, reinstall Oracle Clusterware.
MANAGING OCR FILES and LOCATIONS
The ocrconfig tool is the main configuration tool for OCR. ocrcheck tool enables you to verify the OCR integrity of both OCR and its mirror. Use ocrdump tool to write OCR contents, or part of it, to a text or XML file.
    -export option generates logical backup
    - import option restores your OCR information taken by -export
    -upgrade and -downgrade upgrades or downgrades OCR
    -showbackup option to view the generated backups in default location. You can change backup location using the -backuplocation option.
    -replace ocr or -replace ocrmirror options to add, remove, or replace the primary OCR files.
    - repair option to change the OCR parameters
The default location of each automatically generated OCR backup file is the <CRS Home>/cdata/<cluster_name> directory.
OCR content is automatically backed up physically at every 4 hours, OCR keeps the last three copies, at the end of every day, OCR keeps the last two copies, at the end of every week, CRS keeps the last two copies.
BACK UP OCR MANUALLY
Daily backups of your automatic OCR backups to a different storage device.
Doing logical backup of your OCR before and after making sinificant changes using command:
    # ocrconfig -export file_name

crs-0223: resource 'ora.racN.resource" has placement in RAC 10gr2

This error usually happens when Oracle Clusterware and Oracle Database software are in version 10.2.0.1 or 10.2.0.2.  There are 2 ways to fix this issue:

1. Upgrade the current Oracle Clusterware and Oracle Database software to the lastest version (10.2.0.5 or 11G). This way takes time and has to consider some other aspects.
2. Temporarily fix by using manual commands.

In this article I will use the second method to fix this issue:

• Firstly, check current CRS as well as other Cluster services using the following command            

                        $ crs_stat -l

• List all services with status OFFLINE and UNKNOWN
• Shutdown these services and clear their status with command

                        $ crs_stop -f   "service_name"
replace "service_name" with name of service which has been listed with status OFFLINE and UNKNOWN

• restart all these services with command

                        $ crs_start "service_name"

• Check status of these services again

Welcome!

Hi Everyone!
This is my blog, it is designed for my interesting subject - Oracle DBA. Thank you for coming and viewing my blog. It's content will be here soon!
Tuan Nguyen