1. Check backup of OCR
ocrconfig -showbackupcrsctl query css votedisk
Location to check – /cdata/cluster_name
2.Take a backup of the gpnp profile on all nodes before starting.
cd /gpnp/dbnode1/profiles/peercp -p profile.xml profile.xml.bk
3.Determine if the CLUSTER_INTERCONNECT parameter is used in the Oracle Database and Oracle ASM instances using the following command
set linesize 300col NAME form a50col VALUE form a50SELECT inst_id, name,value FROM gv$parameter WHERE name =’cluster_interconnects’;INST_ID NAME VALUE———- ————————————————– ————————————————–1 cluster_interconnects 192.168.10.312 cluster_interconnects 192.168.10.32ASMALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.31′ SCOPE=SPFILE SID=’+ASM1′;ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.32′ SCOPE=SPFILE SID=’+ASM2′;DATABASEALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.31′ SCOPE=SPFILE SID=’DATABASE1′;ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.32′ SCOPE=SPFILE SID=’DATABASE2′;
4.Shut down all cluster-managed services on each database server as the oracle user using the following command:
srvctl stop home -o -s /tmp/dbnode1_dbhome -n dbnode1srvctl stop home -o -s /tmp/dbnode2_dbhome -n dbnode2
5. Modify the cluster interconnect interface to use the BONDIB0 interface on the first database server as follows – NEEDS TO BE DONE ON FIRST NODE ONLY
a. Log in as the oracle user.b. Set ORACLE_HOME to the Grid Infrastructure home.c. Set the base for the ORACLE_SID environment variable using the following command. TheORACLE_HOME environment variable must be set to the Grid Infrastructure home.5.1 oifcfg iflisteth0 10.240.64.0bondeth0 10.240.48.0bondib0 192.168.8.0bondib0 169.254.0.0
5.2 oifcfg getifbondeth0 10.240.48.0 global publicbondib0 192.168.8.0 global cluster_interconnect
Assign BONDIB0 as the global cluster interconnect interface using the following command:
5.3 oifcfg setif -global bond0/192.168.20.0:cluster_interconnect5.4 Check Ips againoifcfg getifOutputbondeth0 10.240.48.0 global publicBondib0 192.168.8.0 global cluster_interconnectBondib0 192.168.20.0 global cluster_interconnect
6.Shut down Oracle Clusterware and Oracle Clusterware CRS on each database server as follows
a./bin/crsctl stop crs -fb./bin/crsctl disable crs
7.Change the InfiniBand IP addresses on each Exadata Storage Server as follows:
7.1 login as root7.2 service celld stop7.3 service ocrvottargetd stop7.4 ipconf
[root@cellnode1 ~]# service celld stopStopping the RS, CELLSRV, and MS services…The SHUTDOWN of services was successful.[root@cellnode1 ~]# service ocrvottargetd stop[root@cellnode1 ~]# service ocrvottargetd status[root@cellnode1 ~]# ipconfLogging started to /var/log/cellos/ipconf.logInterface ib0 is Linked. hca: mlx4_0Interface ib1 is Linked. hca: mlx4_0Interface eth0 is Linked. driver/mac: igb/00:21:28:d7:cf:82Interface eth1 is … Unlinked. driver/mac: igb/00:21:28:d7:cf:83Interface eth2 is … Unlinked. driver/mac: igb/00:21:28:d7:cf:84Interface eth3 is … Unlinked. driver/mac: igb/00:21:28:d7:cf:85
Network interfacesName State IP address Netmask Gateway Net type Hostnameib0 Linkedib1 Linkedeth0 Linkedeth1 Unlinkedeth2 Unlinkedeth3 UnlinkedWarning. Some network interface(s) are disconnected. Check cables and swicthes and retryDo you want to retry (y/n) [y]: n
The current nameserver(s): 10.239.32.5 10.38.9.31 10.239.96.36 10.8.130.191Do you want to change it (y/n) [n]:The current timezone: Asia/KolkataDo you want to change it (y/n) [n]:The current NTP server(s): 10.46.192.98Do you want to change it (y/n) [n]:
Network interfacesName State IP address Netmask Gateway Net type Hostnameeth0 Linked 10.240.64.237 255.255.255.0 10.240.64.1 Management cellnode1.enterprisenet.orgeth1 Unlinkedeth2 Unlinkedeth3 Unlinkedbondib0 ib0,ib1 192.168.10.33 255.255.252.0 Private cellnode1-priv.enterprisenet.orgSelect interface name to configure or press Enter to continue: bondib0Selected interface. bondib0IP address or none [192.168.10.33]: 192.168.20.33Netmask [255.255.252.0]:Fully qualified hostname or none [cellnode1-priv.enterprisenet.org]:Continue configuring or re-configuring interfaces? (y/n) [y]: nSelect canonical hostname from the list below1: cellnode1.enterprisenet.org2: cellnode1-priv.enterprisenet.orgCanonical fully qualified domain name [1]:Select default gateway interface from the list below1: eth0Default gateway interface [1]:
Canonical hostname: cellnode1.enterprisenet.orgNameservers: 10.239.32.5 10.38.9.31 10.239.96.36 10.8.130.191Timezone: Asia/KolkataNTP servers: 10.46.192.98Default gateway device: eth0Network interfacesName State IP address Netmask Gateway Net type Hostnameeth0 Linked 10.240.64.237 255.255.255.0 10.240.64.1 Management cellnode1.enterprisenet.orgeth1 Unlinkedeth2 Unlinkedeth3 Unlinkedbondib0 ib0,ib1 192.168.20.33 255.255.252.0 Private cellnode1-priv.enterprisenet.orgIs this correct (y/n) [y]:Do you want to configure basic ILOM settings (y/n) [y]: nInfo. IP address of bondib0 changed to “192.168.20.33”Info. Run /opt/oracle.cellos/validations/init.d/saveconfigInfo. Custom changes have been detected in /etc/sysconfig/network-scripts/ifcfg-bondib0Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “IPADDR=192.168.10.33” will be overwritten with “IPADDR=192.168.20.33”Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “IPADDR=192.168.10.33” will be commentedWarning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “NETWORK=192.168.8.0” will be overwritten with “NETWORK=192.168.20.0”Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “NETWORK=192.168.8.0” will be commentedWarning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “BROADCAST=192.168.11.255” will be overwritten with “BROADCAST=192.168.23.255”Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “BROADCAST=192.168.11.255” will be commentedInfo. Original file will be saved in /opt/oracle.cellos/conf/network-scripts/backup_by_Exadata_ipconfInfo. Original file will be saved in /etc/ssh/sshd_config.backupbyExadataInfo. Generate /etc/ssh/sshd_config with ListenAddress(es) 10.240.64.237, 192.168.20.33Stopping sshd: [ OK ]Starting sshd: [ OK ]bash /opt/oracle.cellos/ib_set_node_desc.sh
Re-login using new IP address 10.240.64.237 if you were disconnected after following commandsecho -nifdown bondib0sleep 1/opt/oracle.cellos/reset_rule_table.sh 200sleep 1ifup bondib0sleep 3cellcli -e “alter cell startup services rs”cellcli -e “alter cell startup services ms”cellcli -e “alter cell interconnect1=bondib0”cellcli -e “alter cell shutdown services all force”service ocrvottargetd restartsleep 1tgtadm: can’t find the target
Starting the RS services…Getting the state of RS services… runningStarting MS services…The STARTUP of MS services was successful.A restart of all services is required to put new network configuration into effect. MS-CELLSRV communication may be hampered until restart.Cell cellnode1 successfully altered
Stopping the RS, CELLSRV, and MS services…The SHUTDOWN of services was successful.ipaddress1=192.168.20.33/22[root@cellnode1 ~]#7.5 as root user – reboot
7.6 Verify the newly-assigned InfiniBand address on Exadata Storage Server using the following commandcellcli -e list cell detail | grep ipaddress1
Current output :
[root@cellnode1 ~]# cellcli -e list cell detail | grep ipaddress1ipaddress1: 192.168.10.33/22
Output after change
[root@cellnode1 ~]# cellcli -e list cell detail | grep ipaddress1ipaddress1: 192.168.20.33/22
7.7. If there is an error indicating that one or more cell services are not running, then restart the cell services using the following command:cellcli -e alter cell restart services all
8.Change the InfiniBand IP addresses on each database server as follows – NEEDS TO BE PERFORMED ON ALL NODES ONE BY ONE
8.1 login as root8.2 cd /etc/sysconfig/network-scripts8.3 cp ifcfg-bondib0 orig_ifcfg-bondib08.4 Edit the ifcfg-bond0 file to update the IPADDR, NETMASK, NETWORK and BROADCASTfields.
[root@dbnode1 network-scripts]# cat ifcfg-bondib0#### DO NOT REMOVE THESE LINES ######## %GENERATED BY CELL% ####DEVICE=bondib0USERCTL=noBOOTPROTO=noneONBOOT=yesIPADDR=192.168.10.31NETMASK=255.255.252.0NETWORK=192.168.8.0BROADCAST=192.168.11.255BONDING_OPTS=”mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100″IPV6INIT=noMTU=65520[root@dbnode2 network-scripts]# cat ifcfg-bondib0#### DO NOT REMOVE THESE LINES ######## %GENERATED BY CELL% ####DEVICE=bondib0USERCTL=noBOOTPROTO=noneONBOOT=yesIPADDR=192.168.10.32NETMASK=255.255.252.0NETWORK=192.168.8.0BROADCAST=192.168.11.255BONDING_OPTS=”mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100″IPV6INIT=noMTU=65520
UPDATED VALUE for DBNODE1 and DBNODE2FOR DBNODE1?IPADDR=192.168.20.31NETMASK=255.255.252.0NETWORK=192.168.20.0BROADCAST=192.168.23.255FOR DBNODE2?IPADDR=192.168.20.32NETMASK=255.255.252.0NETWORK=192.168.20.0BROADCAST=192.168.23.2558.5 reboot8.6 Verify the InfiniBand IP address information using the following commandCurrent value – dbnode1bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00inet addr:192.168.10.32 Bcast:192.168.11.255 Mask:255.255.252.0inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:LinkUP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:41159776048 (38.3 GiB) TX bytes:34502152558 (32.1 GiB)
Current value – dbnode2bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00inet addr:192.168.10.31 Bcast:192.168.11.255 Mask:255.255.252.0inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:LinkUP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:34394734207 (32.0 GiB) TX bytes:41626158309 (38.7 GiB)
ifconfig -a
bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00inet addr:192.168.20.32 Bcast:192.168.23.255 Mask:255.255.252.0inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:LinkUP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:41159776048 (38.3 GiB) TX bytes:34502152558 (32.1 GiB)
Current value – dbnode2
bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00inet addr:192.168.20.31 Bcast:192.168.23.255 Mask:255.255.252.0inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:LinkUP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:34394734207 (32.0 GiB) TX bytes:41626158309 (38.7 GiB)
9.Modify Cellinit.ora (Do not edit the cellinit.ora or cellip.ora files when the database or Oracle ASM instance are running)
9.1 cd /etc/oracle/cell/network-config/Current value on dbnode1[root@dbnode1 network-config]# cat cellinit.oraipaddress1=192.168.10.31/22[root@dbnode2 network-config]# cat cellinit.oraipaddress1=192.168.10.32/22
9.1 cp cellinit.ora cellinit.new
9.2 Modify cellinit.new toipaddress1=192.168.20.31/22ipaddress1=192.168.20.32/22
9.3 rename the file backmv cellinit.new cellinit.ora
10.Modify cellip.ora
10.1 cd /etc/oracle/cell/network-config/Current Values :[root@dbnode1 network-config]# cat cellip.oracell=”192.168.10.33″cell=”192.168.10.34″cell=”192.168.10.35″
[root@dbnode2 network-config]# cat cellip.oracell=”192.168.10.33″cell=”192.168.10.34″cell=”192.168.10.35″
10.2 cp cellip.ora orig_cellip.ora10.3 modify cellip.ora to following on both nodescell=”192.168.20.33″cell=”192.168.20.34″cell=”192.168.20.35″
11.Update the /etc/hosts file on each database server and Exadata Storage Servers11.1 login as root11.2 Current value on both dbnode1 and dbnode2### Compute Node Private Interface details#####192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv#####192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv192.168.10.31 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.10.32 dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details#####192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv#####192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv#####192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv192.168.10.33 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.10.34 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.10.35 cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE1#### END Generated by Exadata ####### Compute Node Private Interface details192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details# 192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-privOn CELLNODE2#### END Generated by Exadata ####### Compute Node Private Interface details192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv# 192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-privUpdated valueOn CELLNODE3#### END Generated by Exadata ####### Compute Node Private Interface details192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv# 192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv11.3 cp /etc/hosts /etc/orig_hosts11.4 update toOn both dbnode1 and dbnode2#####192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv#####192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv#####192.168.10.31 dbnode1-priv.enterprisenet.org dbnode1-priv#####192.168.10.32 dbnode2-priv.enterprisenet.org dbnode2-priv192.168.20.31 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.20.32 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details#####192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv#####192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv#####192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv#####192.168.10.33 cellnode1-priv.enterprisenet.org cellnode1-priv#####192.168.10.34 cellnode2-priv.enterprisenet.org cellnode2-priv#####192.168.10.35 cellnode3-priv.enterprisenet.org cellnode3-priv192.168.20.33 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.20.34 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.20.35 cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE1#### END Generated by Exadata ####### Compute Node Private Interface details###192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv###192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv192.168.20.31 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.20.32 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details### 192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv##192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv##192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv# 192.168.20.33 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.20.34 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.20.35 cellnode3-priv.enterprisenet.org cellnode3-privOn CELLNODE2#### END Generated by Exadata ####### Compute Node Private Interface details###192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv###192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv192.168.20.31 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.20.32 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details##192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv### 192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv##192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv192.168.20.33 cellnode1-priv.enterprisenet.org cellnode1-priv# 192.168.20.34 cellnode2-priv.enterprisenet.org cellnode2-priv192.168.20.35 cellnode3-priv.enterprisenet.org cellnode3-privOn CELLNODE3#### END Generated by Exadata ####### Compute Node Private Interface details###192.168.10.1 dbnode1-priv.enterprisenet.org dbnode1-priv###192.168.10.2 dbnode2-priv.enterprisenet.org dbnode2-priv192.168.20.31 dbnode1-priv.enterprisenet.org dbnode1-priv192.168.20.32 dbnode2-priv.enterprisenet.org dbnode2-priv### CELL Node Private Interface details##192.168.10.3 cellnode1-priv.enterprisenet.org cellnode1-priv## 192.168.10.4 cellnode2-priv.enterprisenet.org cellnode2-priv###192.168.10.5 cellnode3-priv.enterprisenet.org cellnode3-priv192.168.20.33 cellnode1-priv.enterprisenet.org cellnode1-priv192.168.20.34 cellnode2-priv.enterprisenet.org cellnode2-priv#192.168.20.35 cellnode3-priv.enterprisenet.org cellnode3-priv
12. Start Oracle Clusterware on each server using the following command
/bin/crsctl start crs
13. Verify the cluster interconnect is using the RDS protocol on each database server by examining the Oracle ASM alert.log
cd /u01/app/oracle/diag/asm/+asm/+ASM1/trace for dbnode2cd /u01/app/oracle/diag/asm/+asm/+ASM2/trace for dbnode2view alert_+ASM1.logview alert_+ASM2.logSearch for “CELL interconnect IPC version: Oracle RDS/IP (generic)”
13.1 Check using skgxpinfo
Current ouput
oracle >skgxpinfo
rds
13.2 If the instance is not using the RDS protocol over InfiniBand, then relink the Oracle binary as follows(if it is using RDS already, it does not change while chaging IP):
13.2.1 :As the oracle user, shut down any processes using the Oracle binarycrscrtl stop crs -f13.2.2 As the root user/crs/install/rootcrs.pl -unlock13.2.3 as oracle usercd /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/libmake -f ins_rdbms.mk ipc_rds ioracle13.2.4 run the following command if relinking the grid infrastructure home. Do not perform this step if you are not relinking the grid infrastructure home.GRID_HOME/crs/install/rootcrs.pl -patch
14.Start all cluster-managed services using the SRVCTL utility as follows
14.1 Login as oracle14.2 srvctl start home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /tmp/dbnode1_dbhome -n dbnode114.3 srvctl status database -d DATABASE
15. Verifiy using on both ASM and DB instance
set linesize 300col NAME form a50col VALUE form a50SELECT inst_id, name,value FROM gv$parameter WHERE name =’cluster_interconnects’;
16. Delete the Old Private Network
oifcfg delif -global bondib0/192.168.8.0
16.1 Verify Old Interface is no longer present
oifcfg getif==> oifcfg getifbondeth0 10.240.48.0 global publicbondib0 192.168.30.0 global cluster_interconnect
17.1 Enable Oracle Clusterware CRS automatic restart on each database server as follows
u01/app/11.2.0/grid/bin/crsctl enable crs on both nodes
18. Restart the cluster one by one and verify
Advertisements
