Home > Oracle, SDP, Solaris > SuperCluster: Creating a DB listener on Infiniband

SuperCluster: Creating a DB listener on Infiniband

You can use below instructions for creating a database listener on Infiniband (IB) on SuperCluster. Creating such a listener enables database clients to connect to it over IB (as opposed to over the Ethernet Client Network). This is typical for an application running on the SuperCluster in an Application Domain. Listeners over IB can be setup to use the TCP protocol, the SDP(Sockets Direct Protocol) protocol, or both.

On SuperCluster, connections to the database over IB are established on the Storage IB Network that is setup using the IB partition 8503. This IB partition is created as part of the initial installation and configuration of the SuperCluster. There is no need to create it as part of the IB listener setup.

The example bellow adds an IB listener to an existing 2-node RAC. The Storage IB subnet is 192.168.30.0/22.

Add the vips for Client Network and Storage IB Network to /etc/hosts. Do this on both nodes:

x.y.132.103   node1-vip
192.168.30.65 node1-vipIB
x.y.132.104   node2-vip
192.168.30.66 node2-vipIB

The addresses starting with x.y. are vips on the 10GbE Client Network and should already be listed in /etc/hosts. There is no need to add these addresses.

As root on node1 register the IB subnet in the Grid by running the following commands from the bin directory of the Grid home t(the usual Grid home is /u01/app/11.2.x.y/grid/bin). Also register the two vips:

root@node1# oifcfg iflist
bondmgt0  x.y.183.0
bondeth0  x.y.132.0
bondib0  192.168.28.0
root@node1# srvctl add network -k 2 -S 192.168.28.0/255.255.252.0/stor_ipmp1 -w static -v
root@node1# crsctl start res ora.net2.network
root@node1# srvctl add vip -n node1 -A node1-vipIB/255.255.252.0/stor_ipmp1 -k 2
root@node1# srvctl add vip -n node2 -A node2-vipIB/255.255.252.0/stor_ipmp1 -k 2
root@node1# srvctl start vip -i node1-vipIB
root@node1# srvctl start vip -i node2-vipIB

As grid one node1, create a LISTENER_IB listener by running the following commands from the bin directory of the Grid home (if there is no grid account, use the oracle account):

grid@node1$ srvctl add listener -l LISTENER_IB -k 2 -p 1522
grid@node1$ srvctl start listener -l LISTENER_IB
grid@node1$ srvctl status listener -l LISTENER_IB
Listener LISTENER_IB is enabled
Listener LISTENER_IB is running on node(s): node1,node2

As oracle one node1 check that the listener is properly registered by running the following commands from the bin directory of the Grid home:

oracle@node1$ lsnrctl status LISTENER_IB
  LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 30-JUN-2014 15:49:2
  Copyright (c) 1991, 2011, Oracle.  All rights reserved.
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))
  STATUS of the LISTENER
------------------------
Alias
Version
Start Date
Uptime
Trace Level
Security
SNMP
Listener Parameter File
Listener Log File
LISTENER_IB
TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
30-JUN-2014 15:48:38
0 days 0 hr. 0 min. 50 sec
off
ON: Local OS Authentication
OFF
/u01/app/11.2.x.y/grid/network/admin/listener.ora
/u01/app/11.2.x.y/grid/log/diag/tnslsnr/node1/listener_ib/alert/log.xml
  Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=192.168.30.65)(PORT=1522)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.30.65)(PORT=1522)))
  The listener supports no services

As oracle on node1, edit tnsnames.ora located in the network/admin directory of the Grid home:

If the IB listener is to be used only with a specific standalone database then the tnsnames.ora from the Oracle home must be modified instead.

Get the database name ‘DBNAME’ with srvctl from the bin directory in Grid home. This value is assigned to SERVICE_NAME in tnsnames.ora:

oracle@node1$ srvctl config database
  DBNAME

Use vi ‘set list’ command to view invisible characters. These can create problems during the next step (executing sql statement) and should be removed.

## BEGIN
DBNAME_IB =
(DESCRIPTION =
        (LOAD_BALANCE=on)
        (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vipIB)(PORT = 1522))
        (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vipIB)(PORT = 1522))
        (CONNECT_DATA =
                (SERVER = DEDICATED)
                (SERVICE_NAME = DBNAME)
))
LISTENER_IBREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node2-vipIB)(PORT = 1522))
))
LISTENER_IBLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node1-vipIB)(PORT = 1522))
                (ADDRESS = (PROTOCOL = SDP)(HOST =  node1-vipIB)(PORT = 1522))
))
LISTENER_IPLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
))

## END

As oracle on node2, add the following lines to tnsnames.ora in the network/admin directory of the Grid home:

DBNAME_IB =
(DESCRIPTION =
        (LOAD_BALANCE=on)
        (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vipIB)(PORT = 1522))
        (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vipIB)(PORT = 1522))
        (CONNECT_DATA =
                (SERVER = DEDICATED)
                (SERVICE_NAME = DBNAME)
))
LISTENER_IBREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node1-vipIB)(PORT = 1522))
))
LISTENER_IBLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST =  node2-vipIB)(PORT = 1522))
                (ADDRESS = (PROTOCOL = SDP)(HOST =  node2-vipIB)(PORT = 1522))
))
LISTENER_IPLOCAL =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
))
## END

As oracle on both nodes, register the new listener at the database level. Make sure to set ORACLE_SID to the proper value before running sqlplus:

oracle@node1$ export ORACLE_SID=DBNAME1
oracle@node1$ sqlplus / as sysdba
SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;
oracle@node2$ export ORACLE_SID=DBNAME2
oracle@node2$ sqlplus / as sysdba
SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL) (REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;

As oracle on node1, restart LISTENER_IB and check its status:

oracle@node1$ srvctl stop listener -l LISTENER_IB
oracle@node1$ srvctl start listener -l LISTENER_IB
oracle@node1$ export TNS_ADMIN=/u01/app/11.2.x.y/grid/network/admin
oracle@node1$ lsnrctl status LISTENER_IB
LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 04-JUL-2014 11:53:20
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))
STATUS of the LISTENER
------------------------
Alias
Version
Start Date
LISTENER_IB
TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
04-JUL-2014 11:52:32
0 days 0 hr. 0 min. 47 sec
off
ON: Local OS Authentication
OFF
/u01/app/11.2.0.3/grid/network/admin/listener.ora
/u01/app/11.2.0.3/grid/log/diag/tnslsnr/rmb-zpr-db-
Uptime
Trace Level
Security
SNMP
Listener Parameter File
Listener Log File
int1/listener_ib/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.30.86)(PORT=1522)))
Services Summary...
Service "DBNAME" has 2 instance(s).
  Instance "DBNAME1", status READY, has 1 handler(s) for this service...
  Instance "DBNAME2", status READY, has 1 handler(s) for this service...
The command completed successfully
Advertisements
Categories: Oracle, SDP, Solaris Tags: , , ,
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: