NameNode, Secondary NameNode and Safe Mode – Hadoop

Hadoop Architecture is similar to the master-slave architecture. The master node is called the namenode and the slave nodes are called datanodes. The datanodes are also called as worker nodes. The namenode is the heart of the hadoop system and it manages the filesystem namespace. The namenode stores the directory, files and file to block mapping metadata on the local disk. The namenode stores this metadata in two files, the namespace image and the edit log.

Safe Mode

When you installed the hadoop, the image and edit log files will be empty. As soon as the clients starts writing the data, the namenode writes the file related metadata to the edit log and constructs the file to block and block to datanode mapping in the memory. The namenode does not write the block to data node mapping persistently on the local disk. It just maintains this mapping in the memory. When you restart the namenode, then the namenode goes into a safemode. In this safemode, the namenode merges all the edit log files and writes the metadata into the image file. The namenode then clears the edit log files and waits for the datanodes to reports the blocks. As soon as the datanodes starts reporting the blocks, the namenode constructs the block to datanode mapping in the memory. Once the datanodes completes reporting or cross a certain threshold, the namenode comes out of the safe mode. In safe mode the client cannot do any operations.

As the namenode writes the metadata into the memory, the number of files that you can store in hadoop is controlled by the namenode memory size. As a thumb rule, if the metadata of file takes 150 bytes, then to store one million files you need at least 300MB of ram on the namenode.

Without the namenode, you cannot access the hadoop system. If the namenode goes down then the complete hadoop system is inaccessible. So it is always better to take backup of the image and edit log files. Hadoop provides two ways for taking backup of the image and edit log files. One way is to use the secondary namenode and the second way is to write the files to a remote system.

Secondary NameNode

The secondary Namenode job is to periodically merge the image file with edit log file. This merge operation is to avoid the edit log file from growing into a large file. The secondary namenode runs on a separate machine and copies the image and edit log files periodically. The secondary namenode requires as much memory as the primary namenode. If the namenode crashes, then you can use the copied image and edit log files from secondary namenode and bring the primary namenode up. However, the state of secondary namenode lags from the primary namenode. So in case of namenode failure, the data loss is obvious.

Namenode and secondary namenode

Note: You cannot make the secondary namenode as the primary namenode.

Remote File System

You can configure the namenode to write the image and edit log files to multiple remote network file systems. These write operations are synchronous and atomic. The namenode first writes the metadata to the edit log on the local disk, then writes to the remote NFS system and then to the memory. Once all these operations are success, then the namenode commits the state. In case of any failure, you can copy the files from the remote NFS system and start the namenode. In this case, there is no loss of data.


Author: sercanbilgic

Current: Eproseed Position: Bigdata Senior Architect Past: Oracle Position: Engineered Systems Architect(Exadata,Exalogic,SuperCluster,Big Data Appliance) Past: Vodafone Position: Middleware Infrastructure Operations Senior Specialist Past: Accenture Position: Senior Developer/Programmer

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s