DNS client configuration steps in Oracle Solaris 11

DNS client configuration in Solaris 11 is based on SMF service rather than file based. When you configure a system as DNS client, you will be performing the following two configurations.

I. DNS client setup

II. Configure Name service switch to use DNS

I. DNS client setup

1. Configure using SMF service network/dns/client

# svccfg -s network/dns/client
svc:/network/dns/client> setprop config/search = astring: (“test.com” “service.test.com”)
svc:/network/dns/client> setprop config/nameserver = net_address: (192.168.10.10 192.168.10.11)
svc:/network/dns/client> exit

2. Enable the DNS client service (when you configure it for the first time)

#svccfg enable -r dns/client

3. Restart/Refresh DNS client service (It is done when there is any update to the configuration)

#svcadm refresh dns/client

#svcadm restart dns/client

4. Verify /etc/resolv.conf if it is updated with the changes.

# more /etc/resolv.conf
#
# _AUTOGENERATED_FROM_SMF_V1_
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
# DO NOT EDIT THIS FILE. EDITS WILL BE LOST.
# See resolv.conf(4) for details.

search test.com service.test.com
nameserver 192.168.10.10
nameserver 192.168.10.11

II. Configuring Name service switch to use DNS

1. Configure using SMF service system/name-service/switch

# svccfg -s system/name-service/switch
svc:/system/name-service/switch> setprop config/host = astring: “files dns”
svc:/system/name-service/switch>exit

2. Restart/Refresh name-service/switch service

#svcadm refresh name-service/switch

#svcadm restart name-service/switch

3. Verfiy host entry in /etc/nsswitch.conf is updated with dns.

# more /etc/nsswitch.conf

#
# _AUTOGENERATED_FROM_SMF_V1_
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
# DO NOT EDIT THIS FILE. EDITS WILL BE LOST.
# See nsswitch.conf(4) for details.

passwd: files
group: files
hosts: files dns
ipnodes: files dns

Categories: Solaris Tags: ,

SuperCluster: Creating a DB listener on Infiniband

March 20, 2015 Leave a comment

You can use below instructions for creating a database listener on Infiniband (IB) on SuperCluster. Creating such a listener enables database clients to connect to it over IB (as opposed to over the Ethernet Client Network). This is typical for an application running on the SuperCluster in an Application Domain. Listeners over IB can be setup to use the TCP protocol, the SDP(Sockets Direct Protocol) protocol, or both.

On SuperCluster, connections to the database over IB are established on the Storage IB Network that is setup using the IB partition 8503. This IB partition is created as part of the initial installation and configuration of the SuperCluster. There is no need to create it as part of the IB listener setup.

The example bellow adds an IB listener to an existing 2-node RAC. The Storage IB subnet is 192.168.30.0/22.

Add the vips for Client Network and Storage IB Network to /etc/hosts. Do this on both nodes:

x.y.132.103   node1-vip
192.168.30.65 node1-vipIB
x.y.132.104   node2-vip
192.168.30.66 node2-vipIB

The addresses starting with x.y. are vips on the 10GbE Client Network and should already be listed in /etc/hosts. There is no need to add these addresses.

As root on node1 register the IB subnet in the Grid by running the following commands from the bin directory of the Grid home t(the usual Grid home is /u01/app/11.2.x.y/grid/bin). Also register the two vips:

root@node1# oifcfg iflist
bondmgt0  x.y.183.0
bondeth0  x.y.132.0
bondib0  192.168.28.0
root@node1# srvctl add network -k 2 -S 192.168.28.0/255.255.252.0/stor_ipmp1 -w static -v
root@node1# crsctl start res ora.net2.network
root@node1# srvctl add vip -n node1 -A node1-vipIB/255.255.252.0/stor_ipmp1 -k 2
root@node1# srvctl add vip -n node2 -A node2-vipIB/255.255.252.0/stor_ipmp1 -k 2
root@node1# srvctl start vip -i node1-vipIB
root@node1# srvctl start vip -i node2-vipIB

As grid one node1, create a LISTENER_IB listener by running the following commands from the bin directory of the Grid home (if there is no grid account, use the oracle account):

grid@node1$ srvctl add listener -l LISTENER_IB -k 2 -p 1522
grid@node1$ srvctl start listener -l LISTENER_IB
grid@node1$ srvctl status listener -l LISTENER_IB
Listener LISTENER_IB is enabled
Listener LISTENER_IB is running on node(s): node1,node2

As oracle one node1 check that the listener is properly registered by running the following commands from the bin directory of the Grid home:

oracle@node1$ lsnrctl status LISTENER_IB
  LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 30-JUN-2014 15:49:2
  Copyright (c) 1991, 2011, Oracle.  All rights reserved.
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))
  STATUS of the LISTENER
------------------------
Alias
Version
Start Date
Uptime
Trace Level
Security
SNMP
Listener Parameter File
Listener Log File
LISTENER_IB
TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
30-JUN-2014 15:48:38
0 days 0 hr. 0 min. 50 sec
off
ON: Local OS Authentication
OFF
/u01/app/11.2.x.y/grid/network/admin/listener.ora
/u01/app/11.2.x.y/grid/log/diag/tnslsnr/node1/listener_ib/alert/log.xml
  Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=192.168.30.65)(PORT=1522)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.30.65)(PORT=1522)))
  The listener supports no services

As oracle on node1, edit tnsnames.ora located in the network/admin directory of the Grid home:

If the IB listener is to be used only with a specific standalone database then the tnsnames.ora from the Oracle home must be modified instead.

Get the database name ‘DBNAME’ with srvctl from the bin directory in Grid home. This value is assigned to SERVICE_NAME in tnsnames.ora:

oracle@node1$ srvctl config database
  DBNAME

Use vi ‘set list’ command to view invisible characters. These can create problems during the next step (executing sql statement) and should be removed.

## BEGIN
DBNAME_IB =
(DESCRIPTION =
        (LOAD_BALANCE=on)
        (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vipIB)(PORT = 1522))
        (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vipIB)(PORT = 1522))
        (CONNECT_DATA =
                (SERVER = DEDICATED)
                (SERVICE_NAME = DBNAME)
))
LISTENER_IBREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node2-vipIB)(PORT = 1522))
))
LISTENER_IBLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node1-vipIB)(PORT = 1522))
                (ADDRESS = (PROTOCOL = SDP)(HOST =  node1-vipIB)(PORT = 1522))
))
LISTENER_IPLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
))

## END

As oracle on node2, add the following lines to tnsnames.ora in the network/admin directory of the Grid home:

DBNAME_IB =
(DESCRIPTION =
        (LOAD_BALANCE=on)
        (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vipIB)(PORT = 1522))
        (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vipIB)(PORT = 1522))
        (CONNECT_DATA =
                (SERVER = DEDICATED)
                (SERVICE_NAME = DBNAME)
))
LISTENER_IBREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node1-vipIB)(PORT = 1522))
))
LISTENER_IBLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node2-vipIB)(PORT = 1522))
                (ADDRESS = (PROTOCOL = SDP)(HOST =  node2-vipIB)(PORT = 1522))
))
LISTENER_IPLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
))
## END

As oracle on both nodes, register the new listener at the database level. Make sure to set ORACLE_SID to the proper value before running sqlplus:

oracle@node1$ export ORACLE_SID=DBNAME1
oracle@node1$ sqlplus / as sysdba
SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;
oracle@node2$ export ORACLE_SID=DBNAME2
oracle@node2$ sqlplus / as sysdba
SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL) (REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;

As oracle on node1, restart LISTENER_IB and check its status:

oracle@node1$ srvctl stop listener -l LISTENER_IB
oracle@node1$ srvctl start listener -l LISTENER_IB
oracle@node1$ export TNS_ADMIN=/u01/app/11.2.x.y/grid/network/admin
oracle@node1$ lsnrctl status LISTENER_IB
LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 04-JUL-2014 11:53:20
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))
STATUS of the LISTENER
------------------------
Alias
Version
Start Date
LISTENER_IB
TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
04-JUL-2014 11:52:32
0 days 0 hr. 0 min. 47 sec
off
ON: Local OS Authentication
OFF
/u01/app/11.2.0.3/grid/network/admin/listener.ora
/u01/app/11.2.0.3/grid/log/diag/tnslsnr/rmb-zpr-db-
Uptime
Trace Level
Security
SNMP
Listener Parameter File
Listener Log File
int1/listener_ib/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.30.86)(PORT=1522)))
Services Summary...
Service "DBNAME" has 2 instance(s).
  Instance "DBNAME1", status READY, has 1 handler(s) for this service...
  Instance "DBNAME2", status READY, has 1 handler(s) for this service...
The command completed successfully
Categories: Oracle, SDP, Solaris Tags: , , ,

How to manually change the network configuration of a SPARC SuperCluster T5-8

March 19, 2015 Leave a comment

PURPOSE

After the initial system deployment, it may occasionally be necessary to change some or all of the network configuration on a SPARC SuperCluster. For example, if the rack is moved to another location where the management and/or client subnets are different. Or when remote gateways, DNS or NTP servers are changed.

Changes to the 1GbE management network must be performed on all components in the rack. Changes to the 10GbE client network must be performed on the T5-8 Solaris domains and database components in those domains. The process is manual, and is different for each component type.

DETAILS

Components on the 1GbE management network

These are the components connected to the 1GbE management network:

  • T5-8 ILOMs (2 or 4)
  • T5-8 Solaris domains (primary & guests) (up to 4 per T5-8) 7320 appliance head ILOMs (2)
  • 7320 appliance heads (2)
  • Exadata storage server ILOMs (3 or 6) *
  • Exadata storage servers (3 or 6) *
  • Cisco 4948 switch (1) *
  • NM2-36P IB switches (3) *
  • PDUs (2) *

The procedures for changing the components marked with * are located in separate documents (see references below).

Components on the 10GbE client networks

These are the components that have IP addresses configured on the 10GbE client networks connected to the T5-8s:

  • T5-8 Solaris domains (primary & guests)
    Oracle RAC VIPs *
  • Oracle DB scan addresses *
  • Oracle DB listeners *

1. T5-8 Solaris domains (management & client networks)

Execute the following commands to change IP addresses in the T5-8 Solaris domains. Note that the commands should be executed while connected via the ILOM and then the system consoles, so that your session is not disconnected by changing network addresses.

Note that you connect from the primary domain to a guest domains console by telnet’ing to it’s port e.g.

# telnet localhost 5003

You can identify the guest domains console ports using the ldm command e,g,

# ldm ls

1) ssh to each T5-8 ILOM, login as root…….
2) Start primary domain console and login as root

> start /SP/console

For each domain (primary and all guests)…….

3) Change all host IP addresses (management & client networks)

3.a) Solaris 11 domain

# ipadm show-addr

Identify management & client address objects that need to be changed. Delete & recreate each one:

# ipadm delete-addr <addr-name>
# ipadm create-addr -T static -a xx.xx.xx.xx/yy <addr-name>

e.g.

# ipadm delete-addr bondmgt0/v4
# ipadm create-addr -T static -a 172.16.112.11/24 bondmgt0/v4 # ipadm delete-addr sc_ipmp0/v4
# ipadm create-addr -T static -a 172.16.220.56/24 sc_ipmp0/v4

3.b) Solaris 10 Domain

# ifconfig -a

Identify management & client address interfaces that need to be changed. For each one:

# vi hostname.<interface_name>

If IP addresses are listed in this file, update accordingly e.g.

172.16.112.11 group bondmgt0 netmask + broadcast + up

4) DNS
4.a ) S11 domain
Change dns/client SMF service:

# svccfg -s dns/client setprop config/nameserver = net_address: <new-address-list> # svccfg -s dns/client setprop config/domain = astring: <new-domain>
# svccfg -s dns/client setprop config/search = astring: <new-search-list>

e.g.

# svccfg -s dns/client setprop config/nameserver = net_address: “(172.16.220.100 172.16.220.101)” # svccfg -s dns/client setprop config/domain = astring: a.myco.com
# svccfg -s dns/client setprop config/search = astring: ‘(“a.myco.com” “myco.com”)’

Restart SMF service:

# svcadm refresh dns/client # svcadm restart dns/client

4.b) S10 domain

Change dns service config file:

# vi /etc/resolv.conf

Change the lines starting “nameserver”, “domain” and “search” e.g.

nameserver 172.16.220.100 nameserver 172.16.220.101 domain a.myco.com
search a.myco.com myco.com

Restart SMF service:

# svcadm refresh dns/client # svcadm restart dns/client

5) NTP
Change ntp service config file:

# vi /etc/inet/ntp.conf

Change the line starting “server” e.g.

server 172.16.220.105

Restart SMF service:

# svcadm refresh ntp # svcadm restart ntp

Change management and client network entries e.g.

192.168.20.0 255.255.252.0 192.168.30.0 255.255.252.0 172.16.112.0 255.255.255.0 172.16.220.0 255.255.255.0

7) Static & default routes

# route -p show # netstat -rn

Identify static & default routes. Delete & recreate each one:

# route -p delete <destination> <gateway> # route -p add <destination> <gateway>

e.g.

# route -p delete default 220.53.28.1 # route -p add default 172.16.220.1

8) /etc/inet/hosts

# vi /etc/inet/hosts

Change all appropriate host & IP address entries

2. T5-8 ILOMs (management & client networks) ssh to each T5-8 ILOM, login as root…….

1) change ILOM NTP entry

> set /SP/clients/ntp/server/1 address=<new NTP server address> > set /SP/clock usentpserver=enabled

e.g.

> set /SP/clients/ntp/server/1 address=172.16.220.105 > set /SP/clock usentpserver=enabled

2) Change ILOM DNS entry

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=<new DNS server address> > set /SP/clients/dns searchpath=<new search list>

e.g.

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=172.16.220.100
> set /SP/clients/dns searchpath=”a.myco.com myco.com”

3) Change ILOM management IP address & gateway

> cd /SP/network
> set pendingipaddress = > set pendingipdiscovery > set pendingipgateway = > set pendingipnetmask = > set commitpending=true

172.16.112.16 = static 172.16.112.1 255.255.255.0

Note that your ssh session will hang & eventually disconnect when you commit these changes because the ILOM address (via which you’re connected) will be changed.

3. 7320 appliance hosts

Execute the following commands to change IP addresses in the 7320 appliance hosts. Note that the commands should be executed while connected via the ILOM and then the system console so that your session is not disconnected by changing network addresses.

ssh to each T5-8 ILOM, login as root……. 1) Start system console and login as root

> start /SP/console

2) Change management IP address

host:> cd /

host:> configuration net interfaces host:> show

Select the management network interface (usually igb0) e.g.

host:> select igb0
host:> set v4addrs=172.16.112.33/24 host:> commit

3) Change management gateway

host:> cd /

host:> configuration net routing host:> show

Select the route that represents the current default route (destination 0.0.0.0/0) e.g.

host:> select route-000
host:> set gateway=172.16.112.1 host:> commit

host:> configuration services dns host:> set servers=<new-address-list> host:> set domain=<new_domain>

e.g.

host:> cd /
host:> configuration services dns
host:> set servers=172.16.220.100,172.16.220.101 host:> set domain=a.myco.com
host:> commit

5) Change NTP servers

host:> cd /
host:> configuration services ntp

host:> set servers=172.16.220.105

host:> set servers=<new-server-list>

e.g.

host:> cd /
host:> configuration services ntp

host:> set servers=172.16.220.105

host:> commit

Note that the new DNS and NTP server addresses will only be accepted if they can be contacted by the storage appliance. So steps 4 & 5 above can only be completed after the SuperCluster is connected to the new network and the new DNS and NTP servers are contactable.

4. 7320 appliance ILOMs
ssh to each 7320 ILOM, login as root…….

1) change ILOM NTP entry

> set /SP/clients/ntp/server/1 address=<new NTP server address>

> set /SP/clock usentpserver=enabled

e.g.

> set /SP/clients/ntp/server/1 address=172.16.220.105

> set /SP/clock usentpserver=enabled

2) Change ILOM DNS entry

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=<new DNS server address>

> set /SP/clients/dns searchpath=<new search list>

e.g.

> set /SP/clients/dns auto_dns=enabled
> set /SP/clients/dns nameserver=172.16.220.100
> set /SP/clients/dns searchpath=”a.myco.com myco.com”

3) Change ILOM management IP address & gateway

> cd /SP/network
> set pendingipaddress = 172.16.112.16

> set pendingipdiscovery = static

> set pendingipgateway = 172.16.112.1

> set pendingipnetmask = 255.255.255.0

> set commitpending = true

Note that your ssh session will hang & eventually disconnect when you commit these changes because the ILOM address (via which you’re connected) will be changed.

5. Oracle database addresses

Refer to Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1)

6, Exadata storage servers hosts & ILOMS (management network)

Refer to Changing IP addresses on Exadata Database Machine (Doc ID 1317159.1)

7. Cisco 4948 management switch

First check with the network administrator if it is appropriate to change the Cisco switch configuration. Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

8. PDUs
Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

9. NM2-36P IB switches

Refer to Oracle Exadata Database Machine Owner’s Guide, chapter 6

Configuring Interface Bonding (Wheezy)

This is a brief article detailing the steps to configure network interface bonding on Debian Wheezy (7.0 stable). The procedure is very different from RHEL/CentOS. I will be configuring active-backup (i.e. failover) mode bonding – there are other modes available, including round-robin load-balanced, LACP aggregation, etc. Read /usr/share/doc/ifenslave-2.6/README.Debian or http://www.kernel.org/doc/Documentation/networking/bonding.txt for further information.

First, verify via ifconfig that your two slave interfaces are available – I’ll be bonding eth0 and eth1 into a bond called bond0:

Install the ifenslave package:

Next, stop networking. As you’re stopping networking, ensure that you’re connected to your host via a console of some form:

Modify /etc/network/interfaces. Remove (or comment out) any existing configuration for your slave interfaces (eth0 and eth1), and configure your new bond0 interface appropriately:

bond_miimon is the MII link monitoring frequency in milliseconds, and bond_{down,up}delay are the time, in milliseconds, to wait before disabling or enabling an interface in the bond (to safeguard against flapping), and should be a multiple of the bond_miimon value. You can adjust these values to suit your needs. These bond_<parameter> directives correlate to the <parameter> directives passed to the bonding module itself.

Once configured, start networking:

There is no need to fiddle with module loading (editing /etc/modules, creating a file under /etc/modprobe.d, etc.) – the ifenslave-2.6 package deposits scripts to do this for us. Take a look at /etc/network/if-pre-up.d/ifenslave to see this being done.

You can see the other scripts installed by ifenslave-2.6 with a dpkg -L:

Running ifconfig -a should now show the correct network configuration:

You can also view the contents of /proc/net/bonding/bond0 to check the status of the bond:

Categories: linux Tags: , , ,

How to Update a File on All Zones at the Same Time

I recently modified /etc/resolv.conf on all of my global zones after building new nameservers. I wanted a quick way to copy this updated configuration to all child zones. A simple one-liner does the trick on each global zone (assuming all your zones are in the /var/zones zonepath):

Categories: Solaris Tags: ,
Follow

Get every new post delivered to your Inbox.

Join 533 other followers