Home > Solaris > How to manually change the network configuration of a SPARC SuperCluster T5-8

How to manually change the network configuration of a SPARC SuperCluster T5-8

PURPOSE

After the initial system deployment, it may occasionally be necessary to change some or all of the network configuration on a SPARC SuperCluster. For example, if the rack is moved to another location where the management and/or client subnets are different. Or when remote gateways, DNS or NTP servers are changed.

Changes to the 1GbE management network must be performed on all components in the rack. Changes to the 10GbE client network must be performed on the T5-8 Solaris domains and database components in those domains. The process is manual, and is different for each component type.

DETAILS

Components on the 1GbE management network

These are the components connected to the 1GbE management network:

  • T5-8 ILOMs (2 or 4)
  • T5-8 Solaris domains (primary & guests) (up to 4 per T5-8) 7320 appliance head ILOMs (2)
  • 7320 appliance heads (2)
  • Exadata storage server ILOMs (3 or 6) *
  • Exadata storage servers (3 or 6) *
  • Cisco 4948 switch (1) *
  • NM2-36P IB switches (3) *
  • PDUs (2) *

The procedures for changing the components marked with * are located in separate documents (see references below).

Components on the 10GbE client networks

These are the components that have IP addresses configured on the 10GbE client networks connected to the T5-8s:

  • T5-8 Solaris domains (primary & guests)
    Oracle RAC VIPs *
  • Oracle DB scan addresses *
  • Oracle DB listeners *

1. T5-8 Solaris domains (management & client networks)

Execute the following commands to change IP addresses in the T5-8 Solaris domains. Note that the commands should be executed while connected via the ILOM and then the system consoles, so that your session is not disconnected by changing network addresses.

Note that you connect from the primary domain to a guest domains console by telnet’ing to it’s port e.g.

# telnet localhost 5003

You can identify the guest domains console ports using the ldm command e,g,

# ldm ls

1) ssh to each T5-8 ILOM, login as root…….
2) Start primary domain console and login as root

> start /SP/console

For each domain (primary and all guests)…….

3) Change all host IP addresses (management & client networks)

3.a) Solaris 11 domain

# ipadm show-addr

Identify management & client address objects that need to be changed. Delete & recreate each one:

# ipadm delete-addr <addr-name>
# ipadm create-addr -T static -a xx.xx.xx.xx/yy <addr-name>

e.g.

# ipadm delete-addr bondmgt0/v4
# ipadm create-addr -T static -a 172.16.112.11/24 bondmgt0/v4 # ipadm delete-addr sc_ipmp0/v4
# ipadm create-addr -T static -a 172.16.220.56/24 sc_ipmp0/v4

3.b) Solaris 10 Domain

# ifconfig -a

Identify management & client address interfaces that need to be changed. For each one:

# vi hostname.<interface_name>

If IP addresses are listed in this file, update accordingly e.g.

172.16.112.11 group bondmgt0 netmask + broadcast + up

4) DNS
4.a ) S11 domain
Change dns/client SMF service:

# svccfg -s dns/client setprop config/nameserver = net_address: <new-address-list> # svccfg -s dns/client setprop config/domain = astring: <new-domain>
# svccfg -s dns/client setprop config/search = astring: <new-search-list>

e.g.

# svccfg -s dns/client setprop config/nameserver = net_address: “(172.16.220.100 172.16.220.101)” # svccfg -s dns/client setprop config/domain = astring: a.myco.com
# svccfg -s dns/client setprop config/search = astring: ‘(“a.myco.com” “myco.com”)’

Restart SMF service:

# svcadm refresh dns/client # svcadm restart dns/client

4.b) S10 domain

Change dns service config file:

# vi /etc/resolv.conf

Change the lines starting “nameserver”, “domain” and “search” e.g.

nameserver 172.16.220.100 nameserver 172.16.220.101 domain a.myco.com
search a.myco.com myco.com

Restart SMF service:

# svcadm refresh dns/client # svcadm restart dns/client

5) NTP
Change ntp service config file:

# vi /etc/inet/ntp.conf

Change the line starting “server” e.g.

server 172.16.220.105

Restart SMF service:

# svcadm refresh ntp # svcadm restart ntp

Change management and client network entries e.g.

192.168.20.0 255.255.252.0 192.168.30.0 255.255.252.0 172.16.112.0 255.255.255.0 172.16.220.0 255.255.255.0

7) Static & default routes

# route -p show # netstat -rn

Identify static & default routes. Delete & recreate each one:

# route -p delete <destination> <gateway> # route -p add <destination> <gateway>

e.g.

# route -p delete default 220.53.28.1 # route -p add default 172.16.220.1

8) /etc/inet/hosts

# vi /etc/inet/hosts

Change all appropriate host & IP address entries

2. T5-8 ILOMs (management & client networks) ssh to each T5-8 ILOM, login as root…….

1) change ILOM NTP entry

> set /SP/clients/ntp/server/1 address=<new NTP server address> > set /SP/clock usentpserver=enabled

e.g.

> set /SP/clients/ntp/server/1 address=172.16.220.105 > set /SP/clock usentpserver=enabled

2) Change ILOM DNS entry

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=<new DNS server address> > set /SP/clients/dns searchpath=<new search list>

e.g.

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=172.16.220.100
> set /SP/clients/dns searchpath=”a.myco.com myco.com”

3) Change ILOM management IP address & gateway

> cd /SP/network
> set pendingipaddress = > set pendingipdiscovery > set pendingipgateway = > set pendingipnetmask = > set commitpending=true

172.16.112.16 = static 172.16.112.1 255.255.255.0

Note that your ssh session will hang & eventually disconnect when you commit these changes because the ILOM address (via which you’re connected) will be changed.

3. 7320 appliance hosts

Execute the following commands to change IP addresses in the 7320 appliance hosts. Note that the commands should be executed while connected via the ILOM and then the system console so that your session is not disconnected by changing network addresses.

ssh to each T5-8 ILOM, login as root……. 1) Start system console and login as root

> start /SP/console

2) Change management IP address

host:> cd /

host:> configuration net interfaces host:> show

Select the management network interface (usually igb0) e.g.

host:> select igb0
host:> set v4addrs=172.16.112.33/24 host:> commit

3) Change management gateway

host:> cd /

host:> configuration net routing host:> show

Select the route that represents the current default route (destination 0.0.0.0/0) e.g.

host:> select route-000
host:> set gateway=172.16.112.1 host:> commit

host:> configuration services dns host:> set servers=<new-address-list> host:> set domain=<new_domain>

e.g.

host:> cd /
host:> configuration services dns
host:> set servers=172.16.220.100,172.16.220.101 host:> set domain=a.myco.com
host:> commit

5) Change NTP servers

host:> cd /
host:> configuration services ntp

host:> set servers=172.16.220.105

host:> set servers=<new-server-list>

e.g.

host:> cd /
host:> configuration services ntp

host:> set servers=172.16.220.105

host:> commit

Note that the new DNS and NTP server addresses will only be accepted if they can be contacted by the storage appliance. So steps 4 & 5 above can only be completed after the SuperCluster is connected to the new network and the new DNS and NTP servers are contactable.

4. 7320 appliance ILOMs
ssh to each 7320 ILOM, login as root…….

1) change ILOM NTP entry

> set /SP/clients/ntp/server/1 address=<new NTP server address>

> set /SP/clock usentpserver=enabled

e.g.

> set /SP/clients/ntp/server/1 address=172.16.220.105

> set /SP/clock usentpserver=enabled

2) Change ILOM DNS entry

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=<new DNS server address>

> set /SP/clients/dns searchpath=<new search list>

e.g.

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=172.16.220.100
> set /SP/clients/dns searchpath=”a.myco.com myco.com”

3) Change ILOM management IP address & gateway

> cd /SP/network
> set pendingipaddress = 172.16.112.16

> set pendingipdiscovery = static

> set pendingipgateway = 172.16.112.1

> set pendingipnetmask = 255.255.255.0

> set commitpending = true

Note that your ssh session will hang & eventually disconnect when you commit these changes because the ILOM address (via which you’re connected) will be changed.

5. Oracle database addresses

Refer to Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1)

6, Exadata storage servers hosts & ILOMS (management network)

Refer to Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1)

7. Cisco 4948 management switch

First check with the network administrator if it is appropriate to change the Cisco switch configuration. Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

8. PDUs
Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

9. NM2-36P IB switches

Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: