Pig Data Types

Introduction to Pig Data Types

Apache Hadoop is a file system it stores data but to perform data processing we need SQL like language which can manipulate data or perform complex data transformation as per our requirement this manipulation of data can be achieved by Apache PIG. It is a high-level scripting language like SQL used with Hadoop and is called as Pig Latin. Pig Data Types works with structured or unstructured data and it is translated into number of MapReduce job run on Hadoop cluster.

To understand Operators in Pig Latin we must understand Pig Data Types. Any data loaded in pig has certain structure and schema using structure of the processed data pig data types makes data model. Data model get defined when data is loaded and to understand structure data goes through a mapping. pig can handle any data due to SQL like structure it works well with Single value structure and nested hierarchical datastructure.

Pig’s Data Types

Since, pig Latin works well with single or nested data structure. Its data type can be broken into two categories:

Scalar/Primitive Types: Contain single value and simple data types.

ComplexTypes: Contains otherNested/Hierarchical data types.

Scalar Data Types

Pig’s scalar data types are also called as primitive datatypes, this is a simple data types that appears in programming languages.

Simple Type	Description and Size	Example
Int	32 -bit signed integer	26
Long	64 -bit signed integer	26L or 26l
Float	32 -bit floating point	26.5F or 26.5f or 26.5e2f or 26.5E2F
Double	64 -bit floating point	26.5 or 26.5e2
Chararray	Character array (string) in Unicode UTF-8 format	Welcome To educba
Bytearray	Blob (Byte array)
Boolean	Boolean	True/False
Datetime	Date-time	1970-01-01T00:00:00.000+00:00
Biginteger	Java BigIneger	67499292089
bigdecimal	Java BIgDecimal	150.76376256272893883

All datatypes are represented in java.lang classes except byte arrays. Default datatype is byte array in pig if type is not assigned. If schema is given in load statement, load function will apply schema and if data and datatype is different than loader will load Null values or generate error. Explicit casting is not supported like cast chararray to float. Pig does not support list or set type to store an items.

Complex Data Types

Complex datatypes are also termed as collection datatype. There are 3 complex datatypes:

• MAP
• TUPLES
• BAGS

1. MAP

Map is set of key-value pair data element mapping. Data in key-value pair can be of any type, including complex type.

Key: Index to find an element, key should be unique and must be an chararray.
Value: Any type of data can be stored in value and each key has certain dataassociated with it.Map are formed using bracket and a hash between key and values.Commas to separate more than one key-value pair.

Code:

['keyname'#'valuename']

Code:

['resource'#'EDUCBA', 'year'#2019]

Explanation: Above example creates a Map withKeys as : ‘resource’ and ‘year’ andValue as :EDUCBA and 2019

2. TUPLE

Tuple is an fixed length, ordered collection of fields formed by grouping scalar datatypes. It is similar to ROW in SQL table with field representing sql columns.
fields need not to be of same datatypes and we can refer to the field by its position as it is ordered.Tuple may or may not have schema provided with it for representing each fields type and name.

Fields: Can be of any type, field is just single/piece of data. Tuple is enclosed in parenthesis

Code:

(field[,fields....])

Code:

(EduCba,2019,kabir,1,2)

3. BAGS

A bag is unordered collection of non-unique tuples. Bag may or may not have schema associated with it and schema is flexible as each tuple can have number of fields with any type.Bag is used to store collection when grouping and bag does not need to fit into memory it can spill bags to disks if needed.

Bag is constructed using braces and tuples are separated bycommas.

Code:

{tuple,[,tuple...]}

Code:

{('Hadoop',2.7),('Hive','1.13'),('Spark',2.0)}

Null Values: A null value is a non-existent or unknown value and any type of data can null. Pig treat null value same as SQL. Pig get Null values if data is missing or error occurred during processing of data. Also null can be used as placeholder for an optional values.

Examples to Implement Pig Data Types

Below are the examples:

Example #1                             

Sample Data:

Load Data:

DATA = LOAD ‘/user/educba/data’ AS (M:map []);
DESCRIBE DATA;

Viewing The Data:

DUMP DATA;

Example #2

Sample Data:

 Load Data:

DATA= LOAD ‘/user/educba/data_tuple’ AS((F:tuple(f1:int,f2:int,f3:int),T:tuple(t1:chararray,t2:int));
DESCRIBE DATA;

Viewing The Data:

DUMP DATA;

Example #3

Sample Data:

Load Data:

DATA_BAG= LOAD ‘/user/educba/data_bag’ AS (B: bag {T: tuple(t1:int, t2:int, t3:int)});
DESCRIBE DATA_BAG;

Viewing The Data:

DUMP DATA_BAG;

Conclusion

Apache pig is a part of Hadoop ecosystem which supports SQL like structure and also It supports data types used in SQL which are represented in java.lang classes. Because of complex data types pig is used for tasks involving structured and unstructured data processing. Yahoo uses around 40% of their jobs for search as Pig extract the data, perform operations and dumps data in HDFS file system.

Recommended Articles

This is a guide to Pig Data Types. Here we discuss the introduction to Pig Data Types along with complex datatypes and examples for better understanding. You can also go through our other related articles to learn more –

1. Pig Architecture
2. Big Data Architecture
3. Pig Commands
4. Apache PIG Interview Questions