Chancing All Passwords on Exadata

All the components of an Exadata system have default passwords. We will look at each component and how to change the default passwords for each.

Database Server
An Exadata X5-2 has eight database servers. Each server has the following ID with defaults passwords:
•    Root
•    Oracle
•    Grid

As a user, you can either go in individually, change the passwords on each server or use the utility DCLI that Oracle provides on an Exadata to change all the passwords in parallel on all servers. Oracle provides files that include various server configurations. For the database component, the dbs_group file is used to change the root, grid and Oracle passwords on all database servers.

#cd /opt/oracle.SupportTools/onecommand
[root@xex1dbadm01 onecommand]# cat dbs_group
xex1dbadm01
xex1dbadm02
xex1dbadm03
xex1dbadm04
xex1dbadm05
xex1dbadm06
xex1dbadm07
xex1dbadm08
ORAPASS=<oraclpassword>
ROOTPASS=<rootpassword>
GRIDPASS=<gridpassword>
dcli -l root -g dbs_group “echo ${ROOTPASS} | passwd –stdin root”
dcli -l root -g dbs_group “echo ${ORAPASS} | passwd –stdin oracle”
dcli -l root -g dbs_group “echo ${GRIDPASS} | passwd –stdin grid”

This will allow for parallel execution of change password for all the servers in the file dbs_group and the end result being new passwords on all your database servers.
Database Server Service Processor
Each Oracle Exadata Database server comes with an ILOM (integrated lights on management) interface, which is also known as a service processor. Each service processor comes with a default password that should be changed immediately.

$ cd /opt/oracle.SupportTools/onecommand
HOSTLIST=`cat /opt/oracle.SupportTools/onecommand/dbs_group`
for TSOH in $HOSTLIST
do
echo $TSOH
ipmitool -H $TSOH-ilom -U root -P <old password> set password 2 <New password>
done

Cell Server Password Change
A full Exadata X5-2 comes with 14 storage cells, and, as such, it is important to be able to use DCLI to change the password,
which allows for changing all the accounts on the cell server (i.e., root, celladmin and cellmonitor).

ROOTPASS=<rootpass>
CELLADMPASS=<celladminpassword>
CELLMONPASS=<cellmonpass>
dcli -l root -g ~/cell_group “echo ${CELLADMPASS} | passwd –stdin celladmin”
dcli -l root -g ~/cell_group “echo ${CELLMONPASS} | passwd –stdin cellmonitor”
dcli -l root -g ~/cell_group “echo ${ROOTPASS} | passwd –stdin root”

Storage Cell Service Processor
Each Exadata storage cell has a service processor similar to a database server, and a similar strategy can be used to the database server for changing ILOM passwords.

$ cd /opt/oracle.SupportTools/onecommand
HOSTLIST=`cat /opt/oracle.SupportTools/onecommand/cell_group`
for TSOH in $HOSTLIST
do
echo $TSOH
ipmitool -H $TSOH-ilom -U root -P <old password> set password 2 <New password>
done

InfiniBand Switches
A Full Rack Exadata has three InfiniBand switches, and, as with other components, it is important to change the passwords. Due to Oracle Bug 13494021,
you might have to perform some extra steps on each InfiniBand switch.

ssh root@<infiniband switch>
–only if you hit bug 13494021 you will do this
cd /conf
cp -p shadow shadow.backup
cd /etc
cp -p shadow /conf/shadow
ln -sf /etc/shadow.ilom shadow
ls -l shadow*
— End Bug Fix
#Passwd nm2user
#passwd ilom-admin
#passwd root
#passwd ilom-operator

Cisco Switch

An Exadata system also contains a Cisco brand switch. It is important to check what utility is available during install time.
It is preferable to have ssh enabled on the switch rather than telnet, which ships as default on the X5-2. Oracle My Oracle Support (MOS) Note 1415044.1 can be used to reconfigure the Cisco switch to ssh only. Once the configuration is complete, you can change the password from the default using the below commands.

ssh admin@<ciscoswitch>
Switch>enable
Password:
Switch#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#enable password <new password>
Switch(config)#enable secret <new password>
The enable secret you have chosen is the same as your enable password.
This is not recommended. Re-enter the enable secret.
Switch(config)#end
Switch#write memory
#exit

Advertisements

Installing Oracle R Distribution and Oracle R Enterprise on Exadata

ORE Installation Documentation and Requirements

Check the official Oracle R Enterprise documentation for a description of the requirements and installations steps.

http://docs.oracle.com/cd/E36939_01/doc/doc.13/e36763/intro.htm#OREAD110

Check the support matrix to ensure that the R distribution supports the version of ORE to be installed.

http://docs.oracle.com/cd/E36939_01/doc/doc.13/e36762/toc.htm#ORERN105

Install the version of ORE that is compatible with the Client / Studio that the customer is using. In this case the customer is using R 2.15.3.

Install R 2.15.3 Distribution.

http://docs.oracle.com/cd/E36939_01/doc/doc.13/e36762/toc.htm#ORERN104

http://docs.oracle.com/cd/E36939_01/doc/doc.13/e36763.pdf

Software Download

http://www.oracle.com/technetwork/database/options/advanced-analytics/r-enterprise/ore-downloads-1502823.html

Overview

  1. Install Oracle R Distribution (RPMs) on all Exadata Nodes. The Oracle R distribution is a prerequisite for installing Oracle R Enterprise.
  2. Execute the Oracle R Enterprise installation script on all Exadata Nodes. First time execution on the first node will create the database repository and packages. Subsequent execution on other nodes will copy the required libraries on that node.
  3. Install Supporting files on all Exadata Nodes.
  4. Create or grant access to the database user.

Obtaining RPMs

The method used for this installation is the RPM method. (Most Exadata systems will not have direct access to the public yum database).

There are several options for obtaining the required RPMS. The method used in this case was to download the rpms from the public YUM server using a VM that has internet access. The rpms were then transferred to Exadata.

Using the yum –downloadonly feature:

yum install yum-downloadonly

yum install <rpmfile> –y –downloadonly –downloaddir=/u01/downloads

or

yum reinstall <rpmfile> –y –downloadonly –downloaddir=/u01/downloads

Installing RPMs

Check exadata for rpms already installed (eg: rpm –qv libXau-devel )

The list below is the order in which the RPMs were installed based on their dependencies.

(The required RPMs and the versions will change over time. Check documentation).

Install the RPMs on all Exadata DB nodes:

rpm -Uvh libXau-devel-1.0.1-3.1.x86_64.rpm

rpm -Uvh libX11-devel-1.0.3-11.el5_7.1.x86_64.rpm mesa-libGL-devel-6.5.1-7.11.el5_9.x86_64.rpm xorg-x11-proto-devel-7.1-13.el5.x86_64.rpm libXdmcp-devel-1.0.1-2.1.x86_64.rpm

rpm -Uvh libtiff-3.8.2-18.el5_8.x86_64.rpm

rpm -Uvh cups-libs-1.3.7-30.el5_9.3.x86_64.rpm

rpm -Uvh cairo-1.2.4-5.el5.x86_64.rpm

rpm -Uvh atk-1.12.2-1.fc6.x86_64.rpm

rpm -Uvh hicolor-icon-theme-0.9-2.1.noarch.rpm

rpm -Uvh bitstream-vera-fonts-1.10-7.noarch.rpm

rpm -Uvh pango-1.14.9-8.0.1.el5_7.3.x86_64.rpm

rpm -Uvh gtk2-2.10.4-29.el5.x86_64.rpm

rpm -Uvh poppler-0.5.4-19.el5.x86_64.rpm

rpm -Uvh poppler-utils-0.5.4-19.el5.x86_64.rpm

rpm -Uvh paps-0.6.6-20.el5.x86_64.rpm

rpm -Uvh dbus-python-0.70-9.el5_4.x86_64.rpm

rpm -Uvh avahi-0.6.16-10.el5_6.x86_64.rpm

rpm -Uvh avahi-compat-libdns_sd-0.6.16-10.el5_6.x86_64.rpm

rpm -Uvh cups-1.3.7-30.el5_9.3.x86_64.rpm

warning: user lp does not exist – using root

warning: user lp does not exist – using root

rpm -Uvh libgfortran-4.1.2-52.el5_8.1.i386.rpm

rpm -Uvh tetex-fonts-3.0-33.15.el5_8.1.x86_64.rpm

rpm -Uvh tetex-dvips-3.0-33.15.el5_8.1.x86_64.rpm

rpm -Uvh libgfortran-4.1.2-54.el5.x86_64.rpm

rpm -Uvh libfontenc-1.0.2-2.2.el5.x86_64.rpm

rpm -Uvh libXfont-1.2.2-1.0.4.el5_7.x86_64.rpm

rpm -Uvh ttmkfdir-3.0.9-23.el5.x86_64.rpm

rpm -Uvh xorg-x11-font-utils-7.1-3.x86_64.rpm

rpm -Uvh libFS-1.0.0-3.1.x86_64.rpm

rpm -Uvh chkfontpath-1.10.1-1.1.x86_64.rpm xorg-x11-xfs-1.0.2-5.el5_6.1.x86_64.rpm

rpm -Uvh urw-fonts-2.3-6.1.1.noarch.rpm

rpm -Uvh ghostscript-fonts-5.50-13.1.1.noarch.rpm ghostscript-8.70-14.el5_8.1.x86_64.rpm

rpm -Uvh netpbm-10.35.58-10.el5.x86_64.rpm

rpm -Uvh netpbm-progs-10.35.58-10.el5.x86_64.rpm

rpm -Uvh desktop-file-utils-0.10-7.x86_64.rpm

rpm -Uvh dialog-1.0.20051107-1.2.2.x86_64.rpm

rpm -Uvh ed-0.2-39.el5_2.x86_64.rpm

rpm -Uvh tetex-3.0-33.15.el5_8.1.x86_64.rpm

rpm -Uvh tetex-latex-3.0-33.15.el5_8.1.x86_64.rpm

rpm -Uvh R-core-2.15.3-1.el5.x86_64.rpm

rpm -Uvh bzip2-devel-1.0.3-6.el5_5.x86_64.rpm

rpm -Uvh gmp-4.1.4-10.el5.x86_64.rpm

rpm -Uvh gcc-gfortran-4.1.2-54.el5.x86_64.rpm

rpm -Uvh pcre-devel-6.6-6.el5_6.1.x86_64.rpm

rpm -Uvh tcl-devel-8.4.13-6.el5.x86_64.rpm

rpm -Uvh tk-devel-8.4.13-5.el5_1.1.x86_64.rpm

rpm -Uvh zlib-devel-1.2.3-7.el5.x86_64.rpm

rpm -Uvh texinfo-4.8-14.el5.x86_64.rpm

rpm -Uvh texinfo-tex-4.8-14.el5.x86_64.rpm

rpm -Uvh R-devel-2.15.3-1.el5.x86_64.rpm

rpm -Uvh libRmath-2.15.3-1.el5.x86_64.rpm

rpm -Uvh libRmath-devel-2.15.3-1.el5.x86_64.rpm

rpm -Uvh R-2.15.3-1.el5.x86_64.rpm

At this point if we try to install the rpms below we will get that they are “Already installed”….

rpm -Uvh zlib-1.2.3-7.el5.i386.rpm

rpm -Uvh zlib-1.2.3-7.el5.x86_64.rpm

rpm -Uvh libgcc-4.1.2-54.el5.i386.rpm

rpm -Uvh libgcc-4.1.2-54.el5.i386.rpm

rpm -Uvh libstdc++-4.1.2-54.el5.x86_64.rpm

rpm -Uvh libstdc++-4.1.2-54.el5.i386.rpm

rpm -Uvh tcl-8.4.13-6.el5.x86_64.rpm

rpm -Uvh libstdc++-devel-4.1.2-54.el5.x86_64.rpm

rpm -Uvh libstdc++-devel-4.1.2-54.el5.i386.rpm

rpm -Uvh cpp-4.1.2-54.el5.x86_64.rpm

rpm -Uvh gcc-c++-4.1.2-54.el5.x86_64.rpm

rpm -Uvh gcc-4.1.2-54.el5.x86_64.rpm

rpm -Uvh mesa-libGL-6.5.1-7.11.el5_9.x86_64.rpm

Check the R software installed and the version

[root@dm01dbadm04 ~]# R

Oracle Distribution of R version 2.15.3 (–) — “Security Blanket”

Copyright (C) The R Foundation for Statistical Computing

ISBN 3-900051-07-0

Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type ‘license()’ or ‘licence()’ for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type ‘contributors()’ for more information and

‘citation()’ on how to cite R or R packages in publications.

Type ‘demo()’ for some demos, ‘help()’ for on-line help, or

‘help.start()’ for an HTML browser interface to help.

Type ‘q()’ to quit R.

You are using Oracle’s distribution of R. Please contact

Oracle Support for any problems you encounter with this

distribution.

> q()

Save workspace image? [y/n/c]: n

Install Oracle R Enterprise

  1. Download the ORE Server and Supporting zip packages for Linux 64-bit.

http://www.oracle.com/technetwork/database/options/advanced-analytics/r-enterprise/ore-downloads-1502823.html

  1. Transfer both the Server and the Supporting files to each Exadata node and unzip.
  2. Login as the “oracle” user and set the ORACLE_SID and Oracle environment variables.
  3. Execute the “install.sh” script within the “server” directory.

[oracle@dm01dbadm02 server]$ ./install.sh

Oracle R Enterprise 1.3.1 Server Installation.

Copyright (c) 2012, 2013 Oracle and/or its affiliates. All rights reserved.

Checking R ………………. Pass

Checking R libraries ……… Pass

Checking ORACLE_HOME ……… Pass

Checking ORACLE_SID ………. Pass

Checking sqlplus …………. Pass

Checking ORACLE instance ….. Pass

Checking ORE …………….. Pass

Current configuration

R_HOME               = /usr/lib64/R

R_LIBS_USER         = /u01/app/oracle/product/11.2.0.3/dbhome_1/R/library

ORACLE_HOME         = /u01/app/oracle/product/11.2.0.3/dbhome_1

ORACLE_SID           = phppovx2

Do you wish to install ORE? [yes]

Choosing RQSYS tablespaces

PERMANENT tablespace to use for RQSYS [SYSAUX]: RQSYS

ERROR: PERMANENT tablespace RQSYS not found

PERMANENT tablespace to use for RQSYS [SYSAUX]:

TEMPORARY tablespace to use for RQSYS [TEMP]:

Tablespaces summary

PERMANENT tablespace = SYSAUX

TEMPORARY tablespace = TEMP

Installing libraries ……… Pass

Installing RQSYS data …….. Pass

Installing RQSYS code …….. Pass

Installing ORE packages …… Pass

Creating ORE script ………. Pass

NOTE: ORE has been enabled for all database users. Next, install the

supporting packages.

You may create an ORE user with the demo_user.sh script, which

automatically grants the required privileges. A complete list of

privileges is available in the script rquser.sql.

To use ORE Embedded R Execution functionality, grant the user

the RQADMIN role.

Please, consult the documentation for more information.

Done

[oracle@dm01dbadm02 server]$

  1. Change directory to the “supporting” directory from the Supporting zip file and install the supporting files

ORE CMD INSTALL ROracle_1.1-9_R_x86_64-unknown-linux-gnu.tar.gz

ORE CMD INSTALL DBI_0.2-5_R_x86_64-unknown-linux-gnu.tar.gz

ORE CMD INSTALL png_0.1-4_R_x86_64-unknown-linux-gnu.tar.gz

Check ORE installation….

ORE -e “library(ORE)”

  1. Execute the ORE server install.sh script on all other RAC nodes :

[oracle@dm01dbadm01 server]$ ./install.sh

Oracle R Enterprise 1.3.1 Server Installation.

Copyright (c) 2012, 2013 Oracle and/or its affiliates. All rights reserved.

Checking R ………………. Pass

Checking R libraries ……… Pass

Checking ORACLE_HOME ……… Pass

Checking ORACLE_SID ………. Pass

Checking sqlplus …………. Pass

Checking ORACLE instance ….. Pass

Checking ORE …………….. Pass

Current configuration

R_HOME               = /usr/lib64/R

R_LIBS_USER         = /u01/app/oracle/product/11.2.0.3/dbhome_1/R/library

ORACLE_HOME         = /u01/app/oracle/product/11.2.0.3/dbhome_1

ORACLE_SID           = osspovx1

Installing libraries ……… Pass

Installing ORE packages …… Pass

Creating ORE script ………. Pass

NOTE: ORE has been enabled for all database users. Next, install the

supporting packages.

You may create an ORE user with the demo_user.sh script, which

automatically grants the required privileges. A complete list of

privileges is available in the script rquser.sql.

To use ORE Embedded R Execution functionality, grant the user

the RQADMIN role.

Please, consult the documentation for more information.

Done

The following libraries will get copied to the location shown.

ls –l $ORACLE_HOME/lib/ore.so

ls –l $ORACLE_HOME/lib/librqe.so

ls –l $ORACLE_HOME/R/library

Install the supporting files on all other Exadata DB nodes.

[oracle@dm01dbadm01 supporting]$ ORE CMD INSTALL ROracle_1.1-9_R_x86_64-unknown-linux-gnu.tar.gz

* installing to library ‘/u01/app/oracle/product/11.2.0.3/dbhome_1/R/library’

* installing *binary* package ‘ROracle’ …

* DONE (ROracle)

[oracle@dm01dbadm01 supporting]$ ORE CMD INSTALL DBI_0.2-5_R_x86_64-unknown-linux-gnu.tar.gz

* installing to library ‘/u01/app/oracle/product/11.2.0.3/dbhome_1/R/library’

* installing *binary* package ‘DBI’ …

* DONE (DBI)

[oracle@dm01dbadm01 supporting]$ ORE CMD INSTALL png_0.1-4_R_x86_64-unknown-linux-gnu.tar.gz

* installing to library ‘/u01/app/oracle/product/11.2.0.3/dbhome_1/R/library’

* installing *binary* package ‘png’ …

* DONE (png)

[oracle@dm01dbadm01 supporting]$

  1. Grant the following privileges to existing db users that will be used for ORE.

grant RQADMIN to OSSDM;

grant create mining model to OSSDM;

Execute demo_user.sh to create new database users for ORE.

  1. Validate ORE installation….

ORE -e “library(ORE)”

See the documentation for further validation checks:

http://docs.oracle.com/cd/E36939_01/doc/doc.13/e36763/postinstall.htm#OREAD190

Upgrading/Downgrading Exadata iLOM firmware

This post is not really Exadata specific, really it applies to any use of the Sun X4170 series of hardwre. And it definitely seems a long way from being a traditional DBA task. As I’ve had to perform this task a number of times recently I thought I’d document it here.

There are actually a number of ways of doing it, but finding and navigating the documentation can be challenging. The way I have performed and iLOM upgrade (or downgrade) has been to use the web interface. It can be done via the command line, but the web interface has worked well for me.

The software for the iLOM can be found in the software distribution, but you have to do some digging as it is contained within the dbw.tbz tar ball, which itself is found in the image_version.tbz file which is under the db_patch_version directory. What you need is something of the form:

1
2
3
[root@db01] tar xvf 11.2.2.4.2.111221.tbz opt/oracle.cellos/iso/cellbits/dbfw.tbz
 
[root@db01] tar xvf dbfw.tbz ActualFirmwareFiles/ILOM-3_0_9_19_e_r66106-Sun_Fire_X4170_X4270_X4275.pkg

This extracts just your iLOM pkg file.

So with your .pkg to hand, fire your web browser at the iLOM you want to upgrade and then choose Maintenance:

Obviously you click the upgrade Firmware Upgrade tab and then the Enter Upgrade Mode:

You now choose your iLOM pkg file and press the Upload button. This takes you to the following type of screen:

Where you can see you current version and what version you are going to upgrade/downgrade to. Once you hit the Start Upgrade button You are taken to a screen that shows the progress of the loading of the new firmware.

All very straightforward.

However, Oracle advise to follow MOS: 1327022.1 which has the following command line steps:

Check the pkg file is the version you want:

1
2
# cd /opt/oracle.cellos/iso/cellbits
# tar -tf dbfw.tbz

Now Check your current Firmware version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# /opt/oracle.cellos/CheckHWnFWProfile -d
[SystemModel]
 SUN FIRE X4170 SERVER
[BIOSVendor]
 American Megatrends Inc.
[BIOSVersion]
 07060232
[BIOSDate]
 07/25/2011
[ProcessorInformation]
 Cores:16
 Intel(R) Xeon(R) E5540 2.53GHz
[ILOMVersion]
 3.0.9.19.e r66106
[PhysicalMemoryGB]
 72
[DiskControllerPCIeSlotNumber]
 PCI-E Slot 0
[DiskControllerModel]
 LSI MegaRAID SAS 9261-8i
[DiskControllerFirmwareRevision]
 12.12.0-0048
[InfinibandHCAPCIeSlotNumber]
 PCI-E Slot 2
[InfinibandHCAPCIeSlotWidth]
 x8
[InfinibandHCADeviceID]
 26428
[InfinibandHCAChipRevision]
 A0
[InfinibandHCAImageType]
 ConnectX
[InfinibandHCAFirmwareVersion]
 2.7.0
[SASBackplaneFirmwareVersion]
 NotApplicable
[DiskControllerPCIeSlotWidth]
 x8
[InfinibandHCAHardwareRev]
 50
[LightsOutFirmwareVersion]
 Firmware Revision : 3.0
[Powersupplies]
 PS0 and PS1 ok
[Enclosure:Slot:DiskType:DiskMake:DiskModel:MinFwVersion]
 ALL:ALL SAS HITACHI H103014SCSUN146G A2A8

The above output is from a V2 running 11.2.2.4.2.

Finally you can upgrade the firmware with the following:

1
# /opt/oracle.cellos/CheckHWnFWProfile -U /opt/oracle.cellos/iso/cellbits

After this is complete, check your version again.

How to change IP and subnet mask of Exadata database nodes and Storage cells

1. Check backup of  OCR
ocrconfig -showbackup
crsctl query css votedisk
Location to check  – /cdata/cluster_name
2.Take a backup of the gpnp profile on all nodes before starting.
cd /gpnp/dbnode1/profiles/peer
cp -p profile.xml profile.xml.bk
3.Determine if the CLUSTER_INTERCONNECT parameter is used in the Oracle Database and Oracle  ASM instances using the following command
set linesize 300
col NAME form a50
col VALUE form a50
SELECT inst_id, name,value FROM gv$parameter WHERE name =’cluster_interconnects’;
   INST_ID NAME                                               VALUE
———- ————————————————– ————————————————–
         1 cluster_interconnects                              192.168.10.31
         2 cluster_interconnects                              192.168.10.32
ASM
ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.31′ SCOPE=SPFILE SID=’+ASM1′;
ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.32′ SCOPE=SPFILE SID=’+ASM2′;
DATABASE
ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.31′ SCOPE=SPFILE SID=’DATABASE1′;
ALTER SYSTEM SET CLUSTER_INTERCONNECTS=’192.168.20.32′ SCOPE=SPFILE SID=’DATABASE2′;
4.Shut down all cluster-managed services on each database server as the oracle user using the  following command:
srvctl stop home -o -s /tmp/dbnode1_dbhome -n dbnode1
srvctl stop home -o -s /tmp/dbnode2_dbhome -n dbnode2
5. Modify the cluster interconnect interface to use the BONDIB0 interface on the first database server as follows – NEEDS TO BE DONE ON  FIRST NODE ONLY
a. Log in as the oracle user.
b. Set ORACLE_HOME to the Grid Infrastructure home.
c. Set the base for the ORACLE_SID environment variable using the following command. The
ORACLE_HOME environment variable must be set to the Grid Infrastructure home.
5.1  oifcfg iflist
eth0  10.240.64.0
bondeth0  10.240.48.0
bondib0  192.168.8.0
bondib0  169.254.0.0
5.2  oifcfg getif
bondeth0  10.240.48.0  global  public
bondib0  192.168.8.0  global  cluster_interconnect
Assign BONDIB0 as the global cluster interconnect interface using the following command:
5.3 oifcfg setif -global bond0/192.168.20.0:cluster_interconnect
5.4 Check Ips again
oifcfg getif
Output
bondeth0 10.240.48.0 global public
Bondib0 192.168.8.0 global cluster_interconnect
Bondib0 192.168.20.0 global cluster_interconnect
6.Shut down Oracle Clusterware and Oracle Clusterware CRS on each database server as follows
a./bin/crsctl stop crs -f
b./bin/crsctl disable crs
7.Change the InfiniBand IP addresses on each Exadata Storage Server as follows:
7.1 login as root
7.2 service celld stop
7.3 service ocrvottargetd stop
7.4 ipconf
[root@cellnode1 ~]# service celld stop
Stopping the RS, CELLSRV, and MS services…
The SHUTDOWN of services was successful.
[root@cellnode1 ~]# service ocrvottargetd stop
[root@cellnode1 ~]# service ocrvottargetd status
[root@cellnode1 ~]# ipconf
Logging started to /var/log/cellos/ipconf.log
Interface ib0 is Linked.  hca: mlx4_0
Interface ib1 is Linked.  hca: mlx4_0
Interface eth0 is Linked.  driver/mac: igb/00:21:28:d7:cf:82
Interface eth1 is … Unlinked.  driver/mac: igb/00:21:28:d7:cf:83
Interface eth2 is … Unlinked.  driver/mac: igb/00:21:28:d7:cf:84
Interface eth3 is … Unlinked.  driver/mac: igb/00:21:28:d7:cf:85
Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
ib0      Linked
ib1      Linked
eth0     Linked
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
Warning. Some network interface(s) are disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n
The current nameserver(s): 10.239.32.5 10.38.9.31 10.239.96.36 10.8.130.191
Do you want to change it (y/n) [n]:
The current timezone: Asia/Kolkata
Do you want to change it (y/n) [n]:
The current NTP server(s): 10.46.192.98
Do you want to change it (y/n) [n]:
Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     10.240.64.237   255.255.255.0   10.240.64.1     Management   cellnode1.enterprisenet.org
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    192.168.10.33   255.255.252.0                   Private      cellnode1-priv.enterprisenet.org
Select interface name to configure or press Enter to continue: bondib0
Selected interface. bondib0
IP address or none [192.168.10.33]: 192.168.20.33
Netmask [255.255.252.0]:
Fully qualified hostname or none [cellnode1-priv.enterprisenet.org]:
Continue configuring or re-configuring interfaces? (y/n) [y]: n
Select canonical hostname from the list below
1: cellnode1.enterprisenet.org
2: cellnode1-priv.enterprisenet.org
Canonical fully qualified domain name [1]:
Select default gateway interface from the list below
1: eth0
Default gateway interface [1]:
Canonical hostname: cellnode1.enterprisenet.org
Nameservers: 10.239.32.5 10.38.9.31 10.239.96.36 10.8.130.191
Timezone: Asia/Kolkata
NTP servers: 10.46.192.98
Default gateway device: eth0
Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     10.240.64.237   255.255.255.0   10.240.64.1     Management   cellnode1.enterprisenet.org
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    192.168.20.33   255.255.252.0                   Private      cellnode1-priv.enterprisenet.org
Is this correct (y/n) [y]:
Do you want to configure basic ILOM settings (y/n) [y]: n
Info. IP address of bondib0 changed to “192.168.20.33”
Info. Run /opt/oracle.cellos/validations/init.d/saveconfig
Info. Custom changes have been detected in /etc/sysconfig/network-scripts/ifcfg-bondib0
Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “IPADDR=192.168.10.33” will be overwritten with “IPADDR=192.168.20.33”
Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “IPADDR=192.168.10.33” will be commented
Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “NETWORK=192.168.8.0” will be overwritten with “NETWORK=192.168.20.0”
Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “NETWORK=192.168.8.0” will be commented
Warning. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Existing line “BROADCAST=192.168.11.255” will be overwritten with “BROADCAST=192.168.23.255”
Info. Config /etc/sysconfig/network-scripts/ifcfg-bondib0. Original line “BROADCAST=192.168.11.255” will be commented
Info. Original file will be saved in /opt/oracle.cellos/conf/network-scripts/backup_by_Exadata_ipconf
Info. Original file will be saved in /etc/ssh/sshd_config.backupbyExadata
Info. Generate /etc/ssh/sshd_config with ListenAddress(es) 10.240.64.237, 192.168.20.33
Stopping sshd:                                             [  OK  ]
Starting sshd:                                             [  OK  ]
bash /opt/oracle.cellos/ib_set_node_desc.sh
Re-login using new IP address 10.240.64.237 if you were disconnected after following commands
echo -n
ifdown bondib0
sleep 1
/opt/oracle.cellos/reset_rule_table.sh 200
sleep 1
ifup bondib0
sleep 3
cellcli -e “alter cell startup services rs”
cellcli -e “alter cell startup services ms”
cellcli -e “alter cell interconnect1=bondib0”
cellcli -e “alter cell shutdown services all force”
service ocrvottargetd restart
sleep 1
tgtadm: can’t find the target
Starting the RS services…
Getting the state of RS services…  running
Starting MS services…
The STARTUP of MS services was successful.
A restart of all services is required to put new network configuration into effect. MS-CELLSRV communication may be hampered until restart.
Cell cellnode1 successfully altered
Stopping the RS, CELLSRV, and MS services…
The SHUTDOWN of services was successful.
ipaddress1=192.168.20.33/22
[root@cellnode1 ~]#
7.5  as root user   – reboot
7.6 Verify the newly-assigned InfiniBand address on Exadata Storage Server using the following command
cellcli -e list cell detail | grep ipaddress1
Current output :
[root@cellnode1 ~]# cellcli -e list cell detail | grep ipaddress1
         ipaddress1:             192.168.10.33/22
Output after change
[root@cellnode1 ~]# cellcli -e list cell detail | grep ipaddress1
         ipaddress1:             192.168.20.33/22
7.7. If there is an error indicating that one or more cell services are not running, then restart the cell  services using the following command:
cellcli -e alter cell restart services all
8.Change the InfiniBand IP addresses on each database server as follows  – NEEDS TO BE PERFORMED ON ALL NODES ONE BY ONE
8.1 login as root
8.2 cd /etc/sysconfig/network-scripts
8.3 cp ifcfg-bondib0 orig_ifcfg-bondib0  
8.4 Edit the ifcfg-bond0 file to update the IPADDR, NETMASK, NETWORK and BROADCAST
fields.
[root@dbnode1 network-scripts]# cat ifcfg-bondib0
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.31
NETMASK=255.255.252.0
NETWORK=192.168.8.0
BROADCAST=192.168.11.255
BONDING_OPTS=”mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100″
IPV6INIT=no
MTU=65520
[root@dbnode2 network-scripts]# cat ifcfg-bondib0
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=bondib0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.32
NETMASK=255.255.252.0
NETWORK=192.168.8.0
BROADCAST=192.168.11.255
BONDING_OPTS=”mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100″
IPV6INIT=no
MTU=65520
UPDATED VALUE for DBNODE1 and DBNODE2
FOR DBNODE1?
IPADDR=192.168.20.31
NETMASK=255.255.252.0
NETWORK=192.168.20.0
BROADCAST=192.168.23.255
FOR DBNODE2?
IPADDR=192.168.20.32
NETMASK=255.255.252.0
NETWORK=192.168.20.0
BROADCAST=192.168.23.255
8.5 reboot
8.6 Verify the InfiniBand IP address information using the following command
Current value – dbnode1
bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:192.168.10.32  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41159776048 (38.3 GiB)  TX bytes:34502152558 (32.1 GiB)
Current value – dbnode2
bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:192.168.10.31  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:34394734207 (32.0 GiB)  TX bytes:41626158309 (38.7 GiB)
ifconfig -a
bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:192.168.20.32  Bcast:192.168.23.255  Mask:255.255.252.0
          inet6 addr: fe80::221:2800:1cf:2b97/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:62232895 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67053494 errors:3 dropped:14 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41159776048 (38.3 GiB)  TX bytes:34502152558 (32.1 GiB)
Current value – dbnode2
bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:192.168.20.31  Bcast:192.168.23.255  Mask:255.255.252.0
          inet6 addr: fe80::221:2800:1cf:1e67/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:67191396 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63617557 errors:0 dropped:11 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:34394734207 (32.0 GiB)  TX bytes:41626158309 (38.7 GiB)
9.Modify Cellinit.ora  (Do not edit the cellinit.ora or cellip.ora files when the database or Oracle ASM instance are running)
9.1 cd /etc/oracle/cell/network-config/
Current value on dbnode1
[root@dbnode1 network-config]# cat cellinit.ora
ipaddress1=192.168.10.31/22
[root@dbnode2 network-config]# cat cellinit.ora
ipaddress1=192.168.10.32/22
9.1 cp cellinit.ora cellinit.new
9.2 Modify cellinit.new to
ipaddress1=192.168.20.31/22
ipaddress1=192.168.20.32/22
9.3 rename the file back
mv cellinit.new cellinit.ora
10.Modify cellip.ora
10.1 cd /etc/oracle/cell/network-config/
Current Values :
[root@dbnode1 network-config]# cat cellip.ora
cell=”192.168.10.33″
cell=”192.168.10.34″
cell=”192.168.10.35″
[root@dbnode2 network-config]# cat cellip.ora
cell=”192.168.10.33″
cell=”192.168.10.34″
cell=”192.168.10.35″
10.2 cp cellip.ora orig_cellip.ora
10.3 modify cellip.ora to following on both nodes
cell=”192.168.20.33″
cell=”192.168.20.34″
cell=”192.168.20.35″
11.Update the /etc/hosts file on each database server and Exadata Storage Servers
11.1 login as root
11.2 Current value on both dbnode1 and dbnode2
### Compute Node Private Interface details
#####192.168.10.1       dbnode1-priv.enterprisenet.org dbnode1-priv
#####192.168.10.2       dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.10.31   dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.10.32   dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
#####192.168.10.3       cellnode1-priv.enterprisenet.org cellnode1-priv
#####192.168.10.4       cellnode2-priv.enterprisenet.org cellnode2-priv
#####192.168.10.5       cellnode3-priv.enterprisenet.org cellnode3-priv
192.168.10.33   cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.10.34   cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.10.35   cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE1
#### END Generated by Exadata ####
### Compute Node Private Interface details
192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
# 192.168.10.3  cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.10.4    cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.10.5    cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE2
#### END Generated by Exadata ####
### Compute Node Private Interface details
192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
192.168.10.3    cellnode1-priv.enterprisenet.org cellnode1-priv
# 192.168.10.4  cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.10.5    cellnode3-priv.enterprisenet.org cellnode3-priv
Updated value
On CELLNODE3
#### END Generated by Exadata ####
### Compute Node Private Interface details
192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
192.168.10.3    cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.10.4    cellnode2-priv.enterprisenet.org cellnode2-priv
# 192.168.10.5  cellnode3-priv.enterprisenet.org cellnode3-priv
11.3 cp /etc/hosts /etc/orig_hosts
11.4 update to
On both dbnode1 and dbnode2
#####192.168.10.1       dbnode1-priv.enterprisenet.org dbnode1-priv
#####192.168.10.2       dbnode2-priv.enterprisenet.org dbnode2-priv
#####192.168.10.31   dbnode1-priv.enterprisenet.org dbnode1-priv
#####192.168.10.32   dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.20.31   dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.20.32   dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
#####192.168.10.3       cellnode1-priv.enterprisenet.org cellnode1-priv
#####192.168.10.4       cellnode2-priv.enterprisenet.org cellnode2-priv
#####192.168.10.5       cellnode3-priv.enterprisenet.org cellnode3-priv
#####192.168.10.33   cellnode1-priv.enterprisenet.org cellnode1-priv
#####192.168.10.34   cellnode2-priv.enterprisenet.org cellnode2-priv
#####192.168.10.35   cellnode3-priv.enterprisenet.org cellnode3-priv
192.168.20.33   cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.20.34   cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.20.35   cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE1
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
###192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.20.31    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.20.32    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
### 192.168.10.3  cellnode1-priv.enterprisenet.org cellnode1-priv
##192.168.10.4    cellnode2-priv.enterprisenet.org cellnode2-priv
##192.168.10.5    cellnode3-priv.enterprisenet.org cellnode3-priv
# 192.168.20.33  cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.20.34    cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.20.35    cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE2
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
###192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.20.31    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.20.32    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
##192.168.10.3    cellnode1-priv.enterprisenet.org cellnode1-priv
### 192.168.10.4  cellnode2-priv.enterprisenet.org cellnode2-priv
##192.168.10.5    cellnode3-priv.enterprisenet.org cellnode3-priv
192.168.20.33    cellnode1-priv.enterprisenet.org cellnode1-priv
# 192.168.20.34  cellnode2-priv.enterprisenet.org cellnode2-priv
192.168.20.35    cellnode3-priv.enterprisenet.org cellnode3-priv
On CELLNODE3
#### END Generated by Exadata ####
### Compute Node Private Interface details
###192.168.10.1    dbnode1-priv.enterprisenet.org dbnode1-priv
###192.168.10.2    dbnode2-priv.enterprisenet.org dbnode2-priv
192.168.20.31    dbnode1-priv.enterprisenet.org dbnode1-priv
192.168.20.32    dbnode2-priv.enterprisenet.org dbnode2-priv
### CELL Node Private Interface details
##192.168.10.3    cellnode1-priv.enterprisenet.org cellnode1-priv
## 192.168.10.4  cellnode2-priv.enterprisenet.org cellnode2-priv
###192.168.10.5    cellnode3-priv.enterprisenet.org cellnode3-priv
192.168.20.33      cellnode1-priv.enterprisenet.org cellnode1-priv
192.168.20.34       cellnode2-priv.enterprisenet.org cellnode2-priv
#192.168.20.35    cellnode3-priv.enterprisenet.org cellnode3-priv
12. Start Oracle Clusterware on each server using the following  command
/bin/crsctl start crs
13. Verify the cluster interconnect is using the RDS protocol on each database server by  examining the Oracle ASM alert.log
cd /u01/app/oracle/diag/asm/+asm/+ASM1/trace  for dbnode2
cd /u01/app/oracle/diag/asm/+asm/+ASM2/trace  for dbnode2
view alert_+ASM1.log
view alert_+ASM2.log
Search for “CELL interconnect IPC version: Oracle RDS/IP (generic)”
13.1 Check using skgxpinfo 
Current ouput
oracle >skgxpinfo
rds
13.2 If the instance is not using the RDS protocol over InfiniBand, then relink the Oracle binary as follows(if it is using RDS already, it does not change while chaging IP):
13.2.1 :As the oracle user, shut down any processes using the Oracle binary
crscrtl stop crs -f
13.2.2 As the root user
/crs/install/rootcrs.pl -unlock
13.2.3 as oracle user
cd /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/lib
make -f ins_rdbms.mk ipc_rds ioracle
13.2.4 run the following command if relinking the grid infrastructure home. Do not perform this step if you are not relinking the grid infrastructure home.
GRID_HOME/crs/install/rootcrs.pl -patch
14.Start all cluster-managed services using the SRVCTL utility as follows
14.1 Login as oracle
14.2 srvctl start home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /tmp/dbnode1_dbhome -n dbnode1
14.3 srvctl status database -d DATABASE
15. Verifiy using on both ASM and DB instance
set linesize 300
col NAME form a50
col VALUE form a50
SELECT inst_id, name,value FROM gv$parameter WHERE name =’cluster_interconnects’;
16. Delete the Old Private Network
oifcfg delif -global bondib0/192.168.8.0
16.1 Verify Old Interface is no longer present
oifcfg getif
==> oifcfg getif
bondeth0  10.240.48.0  global  public
bondib0  192.168.30.0  global  cluster_interconnect
17.1 Enable Oracle Clusterware CRS automatic restart on each database server as follows
u01/app/11.2.0/grid/bin/crsctl enable crs on both nodes
18. Restart the cluster one by one and verify

Changing Admin/Management IP on Exadata

The administration or management network is called the NET0 network and is used to provide shell access via SSH to the Exadata DMA. The NET0 network is enabled on interface eth0. To display the configuration, log in to a compute node and run the following ifconfig command:

[root@cm01dbm01 ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:21:28:B2:17:02
inet addr:172.16.1.10 Bcast:172.16.1.255 Mask:255.255.255.0
… Output omitted for brevity
[root@cm01dbm01 ~]#

This NET0/admin network is comprised of a single network interface and is not typically deployed with channel bonding, although in some rare cases clients choose to bond the eth0 interface with the NET3/eth3 interface to provide additional network interface redundancy. When configuring DNS for your Exadata Database Machine, the host name of each compute node is registered with the IP address of the NET0/admin interface. Validate this by using nslookup from any host on your network and compare the IP address returned from the DNS query with that of the previous ifconfig output:

Macintosh-8:~ jclarke$ nslookup cm01dbm01
Server: 11.11.1.250

Address: 11.11.1.250#53
Name: cm01dbm01.centroid.com
Address: 172.16.1.10
Macintosh-8:~ jclarke$

To change the IP address of your NET0/admin network, edit /etc/sysconfig/network-scripts/ifcfg-eth0. A sample of this configuration file based on the previous ifconfig output is provided:

root@cm01dbm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.16.1.10
NETMASK=255.255.255.0
NETWORK=10.16.1.0
BROADCAST=10.16.1.255
GATEWAY=10.16.1.1
HOTPLUG=no
IPV6INIT=no
HWADDR=00:21:28:b2:17:02

[root@cm01dbm01 ~]#

After changing ifcfg-eth0, determine if other network interfaces require changes as well as the ILOM interface, NET1-2 interfaces, and any other configuration files. Specifically, when changing the management network’s IP address, you will need to locate the entry in /etc/hosts and make the necessary modification:

[root@cm01dbm01 ~]# grep cm01dbm01 /etc/hosts
192.168.10.1 cm01dbm01-priv.centroid.com cm01dbm01-priv
10.16.1.10 cm01dbm01.centroid.com cm01dbm01
192.168.10.1 cm01dbm01-priv.centroid.com cm01dbm01-priv

[root@cm01dbm01 ~]#

Next, change the ListenAddress to match your new eth0 IP address in /etc/ssh/sshd_config. This will enable remote users to obtain shell access via SSH to the administrative network:

[root@cm01dbm01 ~]# grep Listen /etc/ssh/sshd_config
#ListenAddress 0.0.0.0
#ListenAddress ::
ListenAddress 10.16.1.10

ListenAddress 192.168.10.1

After these changes are made, you must modify your routing rules for your management network in /etc/sysconfig/network-scripts/route-eth0. For example:

[root@cm01dbm01] # cd /etc/sysconfig/network-scripts
[root@cm01dbm01] # network-scripts]# ls route-eth0
route-eth0
[root@cm01dbm01 network-scripts]# cat route-eth0
172.16.1.0/24 dev eth0 table 220

default via 172.16.1.1 dev eth0 table 220
[root@cm01dbm01] #

After the changes are complete, restart your network services by running one of the following commands:

root@cm01dbm01 ~]# service network restart
… or
root@cm01dbm01 ~]# ifdown eth0

root@cm01dbm01 ~]# ifup eth0