Difference Between Hadoop and Hive
Hadoop is a Framework or Software which was invented to manage huge data or Big Data. Hadoop is used for storing and processing large data distributed across a cluster of commodity servers. Hadoop stores the data using Hadoop distributed file system and process/query it using the Map-Reduce programming model. Hive is an application that runs over the Hadoop framework and provides SQL like interface for processing/query the data. Hive is designed and developed by Facebook before becoming part of the Apache-Hadoop project. Hive runs its query using HQL (Hive query language). Hive is having the same structure as RDBMS and almost the same commands can be used in Hive. Hive can store the data in external tables so it’s not mandatory to used HDFS also it support file formats such as ORC, Avro files, Sequence File and Text files, etc.
Hadoop’s Major Components
Figure 1, a Basic architecture of a Hadoop component.
Hadoop Base/Common: Hadoop common will provide you one platform to install all its components.
HDFS (Hadoop Distributed File System): HDFS is a major part of the Hadoop framework it takes care of all the data in the Hadoop Cluster. It works on Master/Slave Architecture and stores the data using replication.
Master/Slave Architecture & Replication
- Master Node/Name Node: Name node store the metadata of each block/file stored in HDFS, HDFS can have only one Master Node (In case of HA another Master Node will work as Secondary Master Node).
- Slave Node/Data Node: Data nodes contain actual data files in blocks. HDFS can have multiple Data Nodes.
- Replication: HDFS stores its data by dividing it into blocks. The default block size is 64 MB. Due to replication data gets stored into 3 (Default Replication factor, can be increased as per requirement) different Data Nodes hence there is the least possibility of losing the data in case of any node failure.
YARN (Yet Another Resource Negotiator): It is basically used for managing Hadoop resources also it plays an important role in the scheduling of users’ applications.
MR (Map Reduce): This is the basic programming model of Hadoop. It is used to process/query the data within Hadoop framework.
Hive’s Major Components
Figure 2, Hive’s Architecture & It’s major components.
Hive Clients: Not only SQL, Hive also supports programming languages like Java, C, Python using various drivers such as ODBC, JDBC, and Thrift. One can write any hive client application in other languages and can run in Hive using these Clients.
Hive Services: Under Hive services, execution of commands and queries take place. Hive Web Interface is having five sub-components.
4.5 (5,335 ratings)
View Course
- CLI: Default command line interface provided by Hive for the execution of Hive queries/commands.
- Hive Web Interfaces: It is a simple graphical user interface. It is an alternative to the Hive command line and used to run the queries and commands in Hive application.
- Hive Server: It is also called as Apache Thrift. It is responsible to take commands from different- different command-line interfaces and submit all the commands/queries to Hive also it retrieves the final result.
- Apache Hive Driver: It is responsible for taking the inputs from the CLI, the web UI, ODBC, JDBC or Thrift interfaces by a client and pass the information to metastore where all the file information is stored.
- Metastore: Metastore is a repository to store all Hive metadata information. Hive’s metadata stores information such as the structure of tables, partitions & column type, etc.
Hive Storage: It is the location where the actual task gets performed, All the queries that run from Hive performed the action inside Hive storage.
Head to Head Comparison Between Hadoop and Hive (Infographics)
Below is the top 8 difference between Hadoop vs Hive:
Key Differences between Hadoop and Hive
Below are the lists of points that describe the key differences between Hadoop and Hive:
1. Hadoop is a framework to process/query the Big data while Hive is an SQL Based tool that builds over Hadoop to process the data.
2. Hive process/query all the data using HQL (Hive Query Language) it’s SQL-Like Language while Hadoop can understand Map Reduce only.
3. Map Reduce is an integral part of Hadoop, Hive’s query first get converted into Map Reduce than processed by Hadoop to query the data.
4. Hive works on SQL Like query while Hadoop understands it using Java-based Map Reduce only.
5. In Hive, earlier used traditional “Relational Database’s” commands can also be used to query the big data while in Hadoop, have to write complex Map Reduce programs using Java which is not similar to traditional Java.
6. Hive can only process/query the structured data while Hadoop is meant for all types of data whether it is Structured, Unstructured or Semi-Structured.
7. Using Hive, one can process/query the data without complex programming while in the simple Hadoop ecosystem, the need to write complex Java programs for the same data.
8. One side Hadoop frameworks need 100s line for preparing Java-based MR program another side Hadoop with Hive can query the same data using 8 to 10 lines of HQL.
9. In Hive, it’s very difficult to insert the output of one query as the input of another one while the same query can be done easily using Hadoop with MR.
10. It’s not mandatory to have Metastore within the Hadoop cluster while Hadoop stores all its metadata inside HDFS (Hadoop Distributed File System).
Hadoop vs Hive Comparision Table
Let us look at the comparison table between Hadoop and Hive:
| Comparison Points | Hive | Hadoop |
|
Design and Development
|
||
| Data Storage Location |
Data can be stored in External Table, HBase or in HDFS.
|
Strictly HDFS only. |
| Language Support | HQL (Hive Query Language) |
It can use multiple programming languages such as Java, Python, Scala and many more.
|
| Data Types | It can work on Structured Data only. |
It can process Structured, Un-Structured and Semi-Structure data.
|
| Data Processing Framework |
HQL (Hive Query Language)
|
Using Java written Map Reduce program only. |
|
Computational Framework
|
SQL-Like language. | SQL and No-SQL. |
| Database |
Derby (default) also support MYSQL, Oracle…
|
HBase, Cassandra etc…. |
| Programming Framework |
SQL based programming framework.
|
Java-based programming framework. |
Conclusion
Hadoop and Hive both are used to process the Big data. Hadoop is a framework which provides a platform for other applications to query/process the Big Data while Hive is just an SQL based application which processes the data using HQL (Hive Query Language)
Hadoop can be used without Hive to process the big data while it’s not easy to use Hive without Hadoop. As a conclusion, we can’t compare Hadoop and Hive anyhow and in any aspect. Both Hadoop and Hive are completely different. Running both of the technology together can make Big Data query process much easier and comfortable for Big Data Users.
Recommended Articles:
This has been a guide to Hadoop vs Hive. Here we discuss the components of Hadoop and Hive with head to head comparison with infographics and comparison table. You may also look at the following articles to learn more –
