Category Archives: Oracle

How-to: Quickly Configure Kerberos for Your Apache Hadoop Cluster “”

Use the scripts and screenshots below to configure a Kerberized cluster in minutes.

Kerberos is the foundation of securing your Apache Hadoop cluster. With Kerberos enabled, user authentication is required. Once users are authenticated, you can use projects like Apache Sentry (incubating) for role-based access control via GRANT/REVOKE statements.

Taming the three-headed dog that guards the gates of Hades is challenging, so Cloudera has put significant effort into making this process easier in Hadoop-based enterprise data hubs. In this post, you’ll learn how to stand-up a one-node cluster with Kerberos enforcing user authentication, using the Cloudera QuickStart VM as a demo environment.

If you want to read the product documentation, it’s available here. You should consider this reference material; I’d suggest reading it later to understand more details about what the scripts do.


You need the following downloads to follow along.

Initial Configuration

Before you start the QuickStart VM, increase the memory allocation to 8GB RAM and increase the number of CPUs to two. You can get by with a little less RAM, but we will have everything including the Kerberos server running on one node.

Start up the VM and activate Cloudera Manager as shown here:

Give this script some time to run, it has to restart the cluster.

KDC Install and Setup Script

The script does all the setup work for the Kerberos server and the appropriate configuration parameters. The comments are designed to explain what is going on inline. (Do not copy and paste this script! It contains unprintable characters that are pretending to be spaces. Rather, download it.)

Cloudera Manager Kerberos Wizard

After running the script, you now have a working Kerberos server and can secure the Hadoop cluster. The wizard will do most of the heavy lifting; you just have to fill in a few values.

To start, log into Cloudera Manager by going to http://quickstart.cloudera:7180 in your browser. The userid is cloudera and the password is cloudera. (Almost needless to say but never use “cloudera” as a password in a real-world setting.)

There are lots of productivity tools here for managing the cluster but ignore them for now and head straight for the Administration > Kerberos wizard as shown in the next screenshot.

Click on the “Enable Kerberos” button.

The four checklist items were all completed by the script you’ve already run. Check off each item and select “Continue.”

The Kerberos Wizard needs to know the details of what the script configured. Fill in the entries as follows:

  • KDC Server Host: quickstart.cloudera
  • Kerberos Security Realm: CLOUDERA
  • Kerberos Encryption Types: aes256-cts-hmac-sha1-96

Click “Continue.”

Do you want Cloudera Manager to manage the krb5.conf files in your cluster? Remember, the whole point of this blog post is to make Kerberos easier. So, please check “Yes” and then select “Continue.”

The Kerberos Wizard is going to create Kerberos principals for the different services in the cluster. To do that it needs a Kerberos Administrator ID. The ID created is: cloudera-scm/admin@CLOUDERA.

The screen shot shows how to enter this information. Recall the password is: cloudera.

The next screen provides good news. It lets you know that the wizard was able to successfully authenticate.

OK, you’re ready to let the Kerberos Wizard do its work. Since this is a VM, you can safely select “I’m ready to restart the cluster now” and then click “Continue.” You now have time to go get a coffee or other beverage of your choice.

How long does that take? Just let it work.

Congrats, you are now running a Hadoop cluster secured with Kerberos.

Kerberos is Enabled. Now What?

The old method of su - hdfs will no longer provide administrator access to the HDFS filesystem. Here is how you become the hdfs user with Kerberos:

Now validate you can do hdfs user things:

Next, invalidate the Kerberos token so as not to break anything:

The min.user parameter needs to be fixed per the message below:

This is the error message you get without fixing

Save the changes shown above and restart the YARN service. Now validate that the cloudera user can use the cluster:

If you forget to kinit before trying to use the cluster you’ll get the errors below. The simple fix is to use kinit with the principal you wish to use.

How to check which PSU is installed…if any

Oracle PSUs (Patch Set Updates) are referenced by their 5-place version number.  Unfortunately they do not change version numbers in the Oracle binaries, product banners and such though (see MOS 861152.1), so here’s how to identify which PSU your ORACLE_HOME is at…

Database Server:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 

(The first command above being for Linux)

…or using the following SQL:

select comments, version, bundle_series
from sys.registry$history
where bundle_series = 'PSU'
order by action_time;

COMMENTS                       VERSION            BUNDLE_SERIES
------------------------------ ------------------ -----------------
Patchset             PSU
PSU                  PSU

The above view is populated when catbundle.sql is executed.  If the query above ends with “ORA-00904: “BUNDLE_SERIES”: invalid identifier” then no bundle patch (PSU or CPU) has been applied.

Grid Infrastructure:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 
| grep -i 'GI PSU'

Cluster Ready Services:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 
| grep -i 'TRACKING BUG' | grep -i 'PSU'

Enterprise Manager Agent:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 
| grep -i 'ENTERPRISE MANAGER AGENT' | grep -i 'PSU'

Enterprise Manager OMS:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed 
| grep -i 'ENTERPRISE MANAGER OMS' | grep -i 'PSU'

WebLogic Server:

. $WLS_HOME/server/bin/
java weblogic.version|grep PSU

Chancing All Passwords on Exadata

All the components of an Exadata system have default passwords. We will look at each component and how to change the default passwords for each.

Database Server
An Exadata X5-2 has eight database servers. Each server has the following ID with defaults passwords:
•    Root
•    Oracle
•    Grid

As a user, you can either go in individually, change the passwords on each server or use the utility DCLI that Oracle provides on an Exadata to change all the passwords in parallel on all servers. Oracle provides files that include various server configurations. For the database component, the dbs_group file is used to change the root, grid and Oracle passwords on all database servers.

#cd /opt/oracle.SupportTools/onecommand
[root@xex1dbadm01 onecommand]# cat dbs_group
dcli -l root -g dbs_group “echo ${ROOTPASS} | passwd –stdin root”
dcli -l root -g dbs_group “echo ${ORAPASS} | passwd –stdin oracle”
dcli -l root -g dbs_group “echo ${GRIDPASS} | passwd –stdin grid”

This will allow for parallel execution of change password for all the servers in the file dbs_group and the end result being new passwords on all your database servers.
Database Server Service Processor
Each Oracle Exadata Database server comes with an ILOM (integrated lights on management) interface, which is also known as a service processor. Each service processor comes with a default password that should be changed immediately.

$ cd /opt/oracle.SupportTools/onecommand
HOSTLIST=`cat /opt/oracle.SupportTools/onecommand/dbs_group`
echo $TSOH
ipmitool -H $TSOH-ilom -U root -P <old password> set password 2 <New password>

Cell Server Password Change
A full Exadata X5-2 comes with 14 storage cells, and, as such, it is important to be able to use DCLI to change the password,
which allows for changing all the accounts on the cell server (i.e., root, celladmin and cellmonitor).

dcli -l root -g ~/cell_group “echo ${CELLADMPASS} | passwd –stdin celladmin”
dcli -l root -g ~/cell_group “echo ${CELLMONPASS} | passwd –stdin cellmonitor”
dcli -l root -g ~/cell_group “echo ${ROOTPASS} | passwd –stdin root”

Storage Cell Service Processor
Each Exadata storage cell has a service processor similar to a database server, and a similar strategy can be used to the database server for changing ILOM passwords.

$ cd /opt/oracle.SupportTools/onecommand
HOSTLIST=`cat /opt/oracle.SupportTools/onecommand/cell_group`
echo $TSOH
ipmitool -H $TSOH-ilom -U root -P <old password> set password 2 <New password>

InfiniBand Switches
A Full Rack Exadata has three InfiniBand switches, and, as with other components, it is important to change the passwords. Due to Oracle Bug 13494021,
you might have to perform some extra steps on each InfiniBand switch.

ssh root@<infiniband switch>
–only if you hit bug 13494021 you will do this
cd /conf
cp -p shadow shadow.backup
cd /etc
cp -p shadow /conf/shadow
ln -sf /etc/shadow.ilom shadow
ls -l shadow*
— End Bug Fix
#Passwd nm2user
#passwd ilom-admin
#passwd root
#passwd ilom-operator

Cisco Switch

An Exadata system also contains a Cisco brand switch. It is important to check what utility is available during install time.
It is preferable to have ssh enabled on the switch rather than telnet, which ships as default on the X5-2. Oracle My Oracle Support (MOS) Note 1415044.1 can be used to reconfigure the Cisco switch to ssh only. Once the configuration is complete, you can change the password from the default using the below commands.

ssh admin@<ciscoswitch>
Switch#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#enable password <new password>
Switch(config)#enable secret <new password>
The enable secret you have chosen is the same as your enable password.
This is not recommended. Re-enter the enable secret.
Switch#write memory

Deploy the third application – includes Elastic Java Message Service Configuration(Part -5-)

This version of the application has a Java Message Service (JMS) producer. Another tab is introduced into the application that will allow you to send JMS messages. This tab also contains a “table of processed messages”, the browser displays the messages that were processed by the message driven bean. The Enterprise Application that we will deploy later has the JMS Message Driven Bean in, this bean reads the

  1. At this point you can restart your managed servers.

Click on View changes and restart in Change Center and select Restart Checklist tab then check the box near to both managed servers and click on Restart.

Ekran Resmi 2015-09-22 13.05.51

  1. Click on ‘Lock & Edit’. Deploy the application JMSProducer-1.0.war to the wlsdevCluster, you can find this at:


Ekran Resmi 2015-09-22 13.06.01

You would have to activate the changes and also ensure that the application begins servicing requests.

  1. Make sure the application is running and working correctly by going to URL: http://localhost:7002/JMSProducer/

Ekran Resmi 2015-09-22 13.06.09

We will now use the WebLogic console to configure JMS, this is performed under Services / Messaging/JMS Servers. See screenshot below:

Ekran Resmi 2015-09-22 13.06.16

  1. Click on ‘Lock & Edit’. Create a JMS server by clicking New, using the name

Ekran Resmi 2015-09-22 13.06.22

Click Create a New Store. Use the Type: File Store, like in the screenshot below:

And Click Next.

Ekran Resmi 2015-09-22 13.06.32

  1. Use the following values in the next screen:

Name = cluster

Target to wlsdevCluster

Directory = /u01/projects/wlsdevday/jmsjta

Click OK

 Ekran Resmi 2015-09-22 13.06.44


Activate the changes. Then click on ‘Lock & Edit’ again. This time the ‘Persistent Store’ dropdown would be refreshed to include ‘cluster’ Select the Persistent store that you just created as per the screen shot below:

Ekran Resmi 2015-09-22 13.06.51

Click Next.

Target the JMS Server to wlsdevCluster.

Ekran Resmi 2015-09-22 13.06.57

Click Finish.

  1. Activate your changes, and your JMS Server should look like this:

Ekran Resmi 2015-09-22 13.07.02

  1. We are now going to create a JMS Module to hold our Connection Factory and Queue.

To do this, go back Services. Click on ‘Lock & Edit’

Select JMS Modules

Ekran Resmi 2015-09-22 13.07.09

Create a new JMS Module

Note: A JMS Module and Subdeployment can be used to group resources, including queue and topic destinations, connection factories, JMS templates, destination sort keys, destination quota, distributed destinations, foreign servers, and store-and-forward parameters. Consider how much easier it is to manage and deploy using a Module and Subdeployments rather than remembering the relationships between the JMS resources you are creating and their targeting.

Call the JMS Module jmsDevDayModule.

Ekran Resmi 2015-09-22 13.07.16

Click Next

Now target the module to the cluster.

Ekran Resmi 2015-09-22 13.07.24

Check the box next to Would you like to add resources to this JMS system module?

Ekran Resmi 2015-09-22 13.07.31

And click Finish.

On activation of changes, you should now see a screen like the one below

Ekran Resmi 2015-09-22 13.07.37

Click on ‘Lock & Edit’ again and then click New to add artifacts to the module.

Select the Connection Factory radio button and click Next.


Ekran Resmi 2015-09-22 13.07.46

Enter the following information in the next screen:

Connection Name:                              DevDayCF

Connection Factory JNDI name:          DevDayCF

 Ekran Resmi 2015-09-22 13.07.52

Click Next

Click on the Advanced Targeting (all will become clear later!)

Ekran Resmi 2015-09-22 13.07.52

Click on Create a new Subdeployment

 Ekran Resmi 2015-09-22 13.08.03

Enter the following information in the next screen:

Name: DevDaySub

 Ekran Resmi 2015-09-22 13.08.09

Click OK.

Target the subdeployment to JMS Server created.

Ekran Resmi 2015-09-22 13.08.13

Click Finish.

Activate Changes.

Now we need to create a queue in the JMS Module, this Queue will be a Distributed Destination, and this means the queue will run across the whole cluster.

Click on ‘Lock & Edit’ again. Click New.

Check the Distributed Queue radio button and click Next.

 Ekran Resmi 2015-09-22 13.08.22

On the next screen enter the following information:

Queue Name:              DevDayQueue

Queue JNDI name:      DevDayQueue

Make sure Destination Type is Uniform.

Your distributed queue configuration should look like below:

Ekran Resmi 2015-09-22 13.08.29

Click Next

Click Advanced Targeting, and choose the Subdeployment created earlier (DevDaySub)

Ekran Resmi 2015-09-22 13.08.38

Click Finish

Activate your changes.

We now need to deploy the JMS Consumer JAR. Follow the same procedure as before to deploy the application. You’ll find the application here:


The application is called EJBReceiver-1.0.jar (use the knowledge you have gained on the Developer Day so far, to deploy the application to wlsdevCluster).Dont forget to activate the changes and start the application to service all requests.

  1. Open application window in browser and type any message and click in Send Message.

 Ekran Resmi 2015-09-22 13.08.45

Again send a message.

Ekran Resmi 2015-09-22 13.08.53

Open two terminal and run the following command in both terminal. One terminal will show the log detail of wlsdevManaged-1 and another will show the log of wlsdevManaged-2.

tail -f /u01/projects/wlsdevday/domains/wlsdevdayDomain/servers/wlsdevManaged-1/logs/wlsdevManaged-1.out

tail -f /u01/projects/wlsdevday/domains/wlsdevdayDomain/servers/wlsdevManaged-2/logs/wlsdevManaged-2.out

Ekran Resmi 2015-09-22 13.09.00

As you see we have distributed queue, so our messages is processed by both managed server uniformly. Each time we send a new message it is processed uniformly by managed server.

Now it is time to clean-up. Click on Lock & Edit, stop and then delete the applications (ClustWLSessionSampleLBApp, JMSProducer-1.0 and EJBReceiver-1.0). Activate the changes finally.

Stop all the managed server instances. Ensure that only the AdminServer is up and running.

Deploy the application with Coherence*Web for HTTP session management (Part -4-)

Whilst HTTP session replication is a powerful way to enable applications to continue running even in the  event of server failure, the main issue is that the HTTP session information is held in the JVM memory of the running WebLogic server. If your application has lots or users, and or large HTTP session objects this means the application server could soon run out of memory and therefore stop your applications from scaling well. A solution to this problem is to off load the HTTP session management into a Cache. In our case we are going to off load the HTTP sessions into Coherence. This application uses Coherence*Web to achieve this.

In WebLogic Server 12C, Coherence 12C is automatically installed, and if you choose, can be tightly integrated into the WebLogic Management Framework – This means the WebLogic Console can be used to create Coherence Clusters using a concept we called Coherence Managed Servers – the process of creating these is almost identical to creating WebLogic Server Clusters. In the same way that we have just seen in the previous lab nodemanager is used to start, stop, and restart failed Coherence Servers.

The only difference in this version is the WebLogic web application deployment descriptor now has:




  1. Go to Environment -> Servers

Click Control

Select wlsdevManaged-1 and wlsdevManaged-2 and click Shutdown/Force Shutdown now.

Acknowledge the stopping of the servers by clicking Yes..

Ekran Resmi 2015-09-06 21.47.00

  1. Login to the WebLogic Console
  2. Make sure you have clicked Lock and EditEkran Resmi 2015-09-06 21.47.06

Click on Environment / Coherence Clusters

  1. Create a new Coherence Cluster

Click New in the Coherence Cluster table (as above in screenshot)

Name:  wlsdevdayCohCluster

And Click Next.

 Ekran Resmi 2015-09-06 21.47.14

Enter Unicast Listen Port as 8888 and Click Next.Ekran Resmi 2015-09-06 21.47.19

Target the Cluster at the wlsdevCluster and Click Finish.

Ekran Resmi 2015-09-06 21.47.26

Make sure you activate your changes.

  1. Click Lock & Edit. Open the Coherence cluster created, click on tab Well Known Addresses

Create a new one

Name:                          wka

Listen Address:             wins-vbox.localdomain

Listen Port:                  8088

Click OK.

Ekran Resmi 2015-09-06 21.47.32

Make sure you activate your changes.

  1. Click on ‘Loack & Edit’. Go to Environment -> Clusters, click on wlsdevCluster

Click tab Coherence. Verify that the information on the screen is the following:

Coherence Cluster: wlsdevdayCohCluster

Ensure that Local Storage Enabled and Coherence Web Local Storage are NOT CHECKED
This ensures that the WebLogic Cluster will not hold state, and will only be a client of the Coherence Cluster.

Make sure the screen looks like the screenshot below, then click Save. And activate your changes in the end

 Ekran Resmi 2015-09-06 21.47.39


  1. Click on ‘Lock & Edit’. Go to Environment, expand Clusters and click Server Templates

Click on wlsdevManaged-Template

Click on tab Coherence

Make sure the following are specified and checked:

Coherence Cluster: wlsdevdayCohCluster

Unicast Listen Address: localhost

Unicast Listen Port: 8088

Checked Unicast Port Auto Adjust

Uncheck Local Storage Enabled

 Ekran Resmi 2015-09-06 21.47.46


Click on Save and don’t forget to activate your changes

  1. Now let’s create the new Cluster for the Coherence Managed Servers
  2. As when we created the WebLogic Cluster, in the WebLogic Console go to Clusters, Click on ‘Lock & Edit’ and create a new Dynamic Cluster (This will be a dynamic Coherence cluster!!)
  3. Call the dynamic cluster CoherenceManagedCluster and Click

 Ekran Resmi 2015-09-06 21.47.52


Leave the next screen as default, and click Next.

 Ekran Resmi 2015-09-06 21.47.58


Select Use a single machine for all dynamic servers and choose Machine and click Next.

Ekran Resmi 2015-09-06 21.48.06

On the screen below, change:
Listen Port for First Server: 9100

SSL Listen Port for First Server: 10100

Click Next.

Ekran Resmi 2015-09-06 21.48.14

Review the final screen, and click Finish (Make sure you active your changes)

Finally, we need to make sure this cluster is part of the Coherence cluster, and able to store data, to do this, go to the list of clusters, click on CoherenceManagedCluster
As on the screenshot below click on the Coherence tab. Click on ‘Lock & Edit’.
Select wlsdevdayCohCluster in the selection box and click SaveEkran Resmi 2015-09-06 21.48.21

Having clicked saved, this will open up 2 check boxes. Make sure these are both ticked as per the screenshot below.

Ekran Resmi 2015-09-06 21.48.28

Click on Save again. Don’t forget to Activate changes.
Click on Server Templates, you should see a new template called
CoherenceManagedCluster-Template. Click on it

Click on tab Coherence. Click on ‘Lock & Edit’

Make sure the following are specified and checked:

Coherence Cluster:                  wlsdevdayCohCluster

Unicast Listen Adddress:          wins-vbox.localdomain

Unicast Listen Port:                  8088

Checked                                  Unicast Port Auto Adjust

Ekran Resmi 2015-09-06 21.48.35 

Click on Save and then activate the changes.

  1. Start the new Coherence servers, Go to Environment -> Servers. Click Control.
    Select CoherenceManagedCluster-1 and CoherenceManagedCluster-2 and click Start to start the servers. Before you go to the next step, wait until these servers are running
  1. Go to Environment -> Servers. Click Control. Select wlsdevManaged-1 and wlsdevManaged-2 and click Start to start the servers. Wait for the servers to get started.
  1. Now deploy the application

As before click Lock & Edit in the change center. .

Select Deployments from the Domain Structure pane.

Click Install.

Enter the Path as “/u01/content/weblogic-innovation-seminars/WInS_Demos/weblogic-basics/lab2”

And Select the ClustWLSSessionSampleCounterAppV1-1.0-SNAPSHOT.war

            Click Next.

Ekran Resmi 2015-09-06 21.48.42

Choose Install this application as an application and Click Next.

Target the Application to wlsdevCluster and Click Next.

 Ekran Resmi 2015-09-06 21.48.50


Select Copy this application onto every target for me and Click Finish.

This is the different way to install the application directly to cluster.

Click on ‘Activate Changes’. Check the box near to application and Select Start->Servicing all request->Yes.

  1. Let us do some testing……

As in lab 2, perform an uncontrolled shutdown of the managed server using the kill -9 command. In fact, if you’re brave enough kill both managed servers, Keep the Browser open, and when one of the managed servers comes back continue to use the application – the session state should be maintained as it was held in the Coherence Cache, rather than in WebLogic Servers memory.

Go to Browser and type the URL: http://localhost:7002/ClustWLSSessionSampleCounterAppV1/

Ekran Resmi 2015-09-06 21.49.02

Find out the wlsdevManaged server processes and kill them both.

Ekran Resmi 2015-09-06 21.49.08

Go to the Servers page and you would find that both the servers are being started (by the NodeManager).

Ekran Resmi 2015-09-06 21.49.14

Once one of the servers is started – the page will show the correct data. All this had been stored in Coherence.

Ekran Resmi 2015-09-06 21.49.21

  1. Before going to the next lab, let’s remove coherence configuration and also the same application (ClustWLSSessionSampleCounterAppV1-1.0-SNAPSHOT)..

Go to console, click on ‘Lock & Edit’. Go to Deployments, select ‘ClustWLSSessionSampleCounterAppV1-1.0-SNAPSHOT’ and Stop->Force Stop Now. And then Delete the application. Activate the changes.

Go to console, Lock&Edit, go to Environment -> Clusters.

Click wlsdevCluster, go to Coherence tab and select None for Coherence Cluster.

Save and then Activate your changes.
To reduce resource usage you might also want to stop the coherence servers.