Overview of Install Hadoop
Install Hadoop involves installing and configuring the related software and utilities associated with the Hadoop framework. Hadoop is an open-source framework which Licensed from Apache software foundation for big data processing. Java needs to be installed on the system. Hadoop is installed on a Linux Operating system like CentOS for its computation. After setting up the Java in the environment, the Hadoop package software that is downloaded through The Apache website needs to be installed. The related Name Node configurations the configuration XML is known as core-site.xml, and for data nodes, the hdfs-site.xml needs to be configured alongside yarn-site.xml for resource management.
Hadoop Framework
The Apache Hadoop framework consists of the following key modules.
- Apache Hadoop Common.
- Apache Hadoop Distributed File System (HDFS).
- Apache Hadoop MapReduce
- Apache Hadoop YARN (Yet Another Resource Manager).
Apache Hadoop Common
Apache Hadoop Common module consists of shared libraries that are consumed across all other modules, including key management, generic I/O packages, libraries for metric collection and utilities for the registry, security, and streaming.
HDFS
The HDFS is based on the Google file system and is structured to run on low-cost hardware. HDFS is tolerant of faults and is designed for applications having large datasets.
MapReduce
MapReduce is an inherent parallel programming model for data processing, and Hadoop can run MapReduce programs written in various languages such as Java. MapReduce works by splitting the processing into the map phase and reduces the phase.
Apache Hadoop YARN
Apache Hadoop YARN is a core component, resource management, and job scheduling technology in the Hadoop distributed processing framework.
4.5 (6,787 ratings)
View Course
In this article, we will discuss the installation and configuration of Hadoop 2.7.4 on a single node cluster and test the configuration by running the MapReduce program called wordcount to count the number of words in the file. We will further look at few important Hadoop File System commands.
Steps to Install Hadoop
The following is a summary of the tasks involved in the configuration of Apache Hadoop.
Task 1: The first task in the Hadoop installation included setting up a virtual machine template that was configured with Cent OS7. Packages such as Java SDK 1.8 and Runtime Systems required to run Hadoop were downloaded, and the Java environment variable for Hadoop was configured by editing bash_rc.
Task 2: Hadoop Release 2.7.4 package was downloaded from the apache website and was extracted in the opt folder. Which was then renamed as Hadoop for easy access.
Task 3: Once the Hadoop packages were extracted, the next step included configuring the environment variable for the Hadoop user, followed by configuring Hadoop node XML files. In this step, NameNode was configured within core-site.xml, and DataNode was configured within hdfs-site.xml. Resource manager and node manager were configured within yarn-site.xml.
Task 4: The firewall was disabled in order to start YARN and DFS. JPS command was used to verify if relevant daemons are running in the background. The port number to access Hadoop was configured to http://localhost:50070/.
Task 5: The next few steps were used to verify and test Hadoop. For this, we have created a temporary test file in the input directory for the WordCount program. Map-reduce program Hadoop-MapReduce-examples2.7.4.jar was used to count the number of words in the file. Results were evaluated on the localhost, and logs of the submitted application were analyzed. All MapReduce applications submitted can be viewed at the online interface, the default port number being 8088.
Task 6: In the final task, we will introduce some basic Hadoop File System commands and check their usages. We will see how a directory can be created within the Hadoop file system to list the content of a directory, its size in bytes. We will further see how to delete a specific directory and file.
Results in Hadoop Installation
The following shows the results of each of the above tasks:
Result of Task 1
A new virtual machine with a cenOS7 image has been configured to run Apache Hadoop. Figure 1 shows how CenOS 7 image was configured in the Virtual machine. Figure 1.2 shows the JAVA environment variable configuration within .bash_rc.
Figure 1: Virtual Machine configuration
Figure 1.2: Java environment variable configuration
Result of Task 2
Figure 2 shows the task carried out in order to extract the Hadoop 2.7.4 package in to opt folder.
Figure 2: Extraction of Hadoop 2.7.4 package
Result of Task 3
Figure 3 shows the configuration for the environment variable for Hadoop user, Figure 3.1 to 3.4 shows the configuration for XML files required for Hadoop configuration.
Figure 3: Configuring the environment variable for Hadoop user
Figure 3.1: Configuration of core-site.xml
Figure 3.2: Configuration of hdfs-site.xml
Figure 3.3: Configuration of mapred-site.xml file
Figure 3.4: Configuration of yarn-site.xml file
Result of Task 4
Figure 4 shows the usage of the jps command to check relevant daemons are running in the background and the following figure shows Hadoop’s online user Interface.
Figure 4: jps command to verify running daemons.
Figure 4.1: Accessing Hadoop online interface at port http://hadoop1.example.com:50070/
Result of Task 5
Figure 5 shows the result for the MapReduce program called wordcount, which counts the number of words in the file. The next couple of figures displays the YARN resource manager’s online user interface for the submitted task.
Figure 5: MapReduce program results
Figure 5.1: Submitted Map-reduce application.
Figure 5.2: Logs for submitted MapReduce application.
Result of Task 6
Figure 6 shows how to create a directory within the Hadoop file system and perform a listing of the hdfs directory.
Figure 6: Creating a directory within the Hadoop file system
Figure 6.1 shows how to put a file onto the Hadoop distributed file system, and figure 6.2 shows the created file in the dirB directory.
Figure 6.1: Creating a file in HDFS.
Figure 6.2: New file created.
The next few figures show how to list the contents of particular directories:
Figure 6.3: Content of dirA
Figure 6.4: Content of dirB
The next figure shows how file and directory size can be displayed:
Figure 6.5: Display a file and directory size.
Deleting a directory or a file can be easily accomplished by the -rm command.
Figure 6.6: To delete a file.
Conclusion
Big Data has played a very important role in shaping today’s world market. Hadoop framework makes data analyst’s life easy while working on large datasets. The configuration of Apache Hadoop was quite simple, and the online user interface provided the user with multiple options to tune and manage the application. Hadoop has been used massively in organizations for data storage, machine learning analytics and backing up data. Managing a large amount of data has been quite handy because of Hadoop distributed environment and MapReduce. Hadoop development was pretty amazing when compared to relational databases as they lack tuning and performance options. Apache Hadoop is a user-friendly and low-cost solution for managing and storing big data efficiently. HDFS also goes a long way in helping in storing data.
Recommended Articles
This is a guide to Install Hadoop. Here we discuss the introduction to Install Hadoop, step-by-step installation of Hadoop, and Hadoop Installation results. You can also go through our other suggested articles to learn more –
