EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Hadoop Tutorial Hadoop fs Commands
Secondary Sidebar
Hadoop Tutorial
  • Commands
    • Hadoop Commands
    • Hadoop fs Commands
    • Hadoop FS Command List
    • HDFS Commands
    • HDFS ls
    • Hadoop Stack
    • HBase Commands
  • Basics
    • What is Hadoop
    • Career in Hadoop
    • Advantages of Hadoop
    • Uses of Hadoop
    • Hadoop Versions
    • HADOOP Framework
    • Hadoop Architecture
    • Hadoop Configuration
    • Hadoop Components
    • Hadoop WordCount
    • Hadoop Database
    • Hadoop Ecosystem
    • Hadoop Tools
    • Install Hadoop
    • Is Hadoop Open Source
    • What is Hadoop Cluster
    • Hadoop Namenode
    • Hadoop data lake
    • Hadoop fsck
    • HDFS File System
    • Hadoop Distributed File System
  • Advanced
    • What is Yarn in Hadoop
    • Hadoop?Administrator
    • Hadoop DistCp
    • Hadoop Administrator Jobs
    • Hadoop Schedulers
    • Hadoop Distributed File System (HDFS)
    • Hadoop Streaming
    • Apache Hadoop Ecosystem
    • Distributed Cache in Hadoop
    • Hadoop Ecosystem Components
    • Hadoop YARN Architecture
    • HDFS Architecture
    • What is HDFS
    • HDFS Federation
    • Apache HBase
    • HBase Architecture
    • What is Hbase
    • HBase Shell Commands
    • What is MapReduce in Hadoop
    • Mapreduce Combiner
    • MapReduce Architecture
    • MapReduce Word Count
    • Impala Shell
    • HBase Create Table
  • Interview Questions
    • Hadoop Admin Interview Questions
    • Hadoop Cluster Interview Questions
    • Hadoop developer interview Questions
    • HBase Interview Questions

Related Courses

Data Science Certification

Online Machine Learning Training

Hadoop Certification

MapReduce Certification Course

Hadoop fs Commands

By Priya PedamkarPriya Pedamkar

hadoop fs commands

Introduction to Hadoop fs Commands

Hadoop fs Commands are the command-line utility for working with the Hadoop system. These commands are widely used to process the data and related files. These commands are Linux based commands which control the Hadoop environment and data files. The ‘Hadoop fs’ is associated as a prefix to the command syntax. Some of the commonly used Hadoop fs commands are listing the directory structure to view the files and subdirectories, Creating directory in the HDFS file system, creating empty files, removing files and directories from HDFS, copying files from other edge nodes to HDFS and copying files from HDFS locations to edge nodes.

Commands of Hadoop fs

Now, let’s learn how to use HADOOP fs commands.

We will start with the basics. Just type these commands in PUTTY or any console you are comfortable with.

1. hadoop fs -ls

For a directory, it returns the list of files and directories whereas, for a file, it returns the statistics on the file.

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

hadoop fs -lsr: this is for recursively listing the directories and files under specific folders.

Example:

Hadoop fs -ls / or hadoop fs -lsr

All in One Data Science Bundle(360+ Courses, 50+ projects)
Python TutorialMachine LearningAWSArtificial Intelligence
TableauR ProgrammingPowerBIDeep Learning
Price
View Courses
360+ Online Courses | 50+ projects | 1500+ Hours | Verifiable Certificates | Lifetime Access
4.7 (86,584 ratings)
Note : hadoop fs -ls [-d] [-h] [-R]
  • -d: This is used to list the directories as plain files.
  • -h: This is used to format the sizes of files into a human-readable manner than just the number of bytes.
  • -R: This is used to recursively list the contents of directories.

2. hadoop fs -mkdir

This command takes the path as an argument and creates directories in hdfs.

Example:

hadoop fs -mkdir /user/datahub1/data

Note: If we try to create subdirectories, the parent directory must exist. If this condition is not met then, ‘No such file or directory’ will be appearing.

3. hadoop fs -touchz

It creates an empty file and utilizes no space

Example:

hadoop fs -touchz URI

4. hadoop fs -rm

Delete files specified as the argument. We must specify the -r option to delete the entire directory. And if -skipTrash option is specified, it will skip trash and the file will be deleted immediately.

Example:

hadoop fs -rm -r /user/test1/abc.text

5. hadoopfs -rmdir

It removes files and permissions of directories and subdirectories. Basically, it is the expanded version of the Hadoop fs -rm.

6. hadoop fs -cp

It copies the file from one location to other

Example:

hadoop fs -cp /user/data/abc.csv /user/datahub

7. hadoop fs -copyFromLocal

It copies the file from edgenode to HDFS.

8. hadoop fs -put

It copies the file from edgenode to HDFS, it is similar to the previous command but put also reads input from standard input stdin and writes to HDFS

Example:

hadoop fs -put abc.csv /user/data

Note:

  • hadoop fs -put -p: The flag preserves the access, modification time, ownership and the mode.
  • hadoop fs -put -f: This command overwrites the destination if the file already exists before the copy.

9. hadoop fs -moveFromLocal

It is similar to copy from local except that the source file is deleted from local edgenode after it is copied to HDFS

Example:

fs -moveFromLocal abc.text /user/data/acb.

10. hadoop fs -copyToLocal

It copies the file from HDFS to edgenode.

Example:

fs -copyToLocal abc.text /localpath

11. hadoop fs -chmod

This command helps us to change access of a file or directory

Example:

hadoop fs -chmod [-R] [path]

12. hadoop fs -chown

This command helps us to change the ownership of a file or directory

Example:

hadoop fs -chown [-R] [OWNER][:[GROUP]] PATH

13. hadoop fs -cat

It prints the content of an HDFS file on the terminal

Example:

hadoop fs -cat /user/data/abc.csv

14. hadoop fs -tail

It displays last KB of the HDFS file to the stdout

Example:

hadoop fs -tail /in/xyzfile

15. hadoop fs -test

This command is used for HDFS file test operations, it returns 0 if true.

  • –e: checks to see if the file exists.
  • -z: checks to see if the file is zero-length
  • -d/-f: checks to see if the path is directory/file respectively

Here, we discuss an example in detail

Example :

hadoop fs -test -[defz] /user/test/test1.text

16. hadoop fs -du

Displays sizes of files and directories contained in the given directory or the length of a file in case it is a file

17. hadoop fs -df

It displays free space

18. hadoop fs -checksum

Returns the checksum information of a file

19. hadoop fs -getfacl

It displays the access control list (ACLs) of the particular file or directory

20. hadoop fs -count

It counts the number of directories, files, and bytes under the path that matches the specified file pattern.

21. hadoop fs -setrep

Changes the replication factor of a file. And if the path is a directory then the command changes the replication factor of all the files under the directory.

Example:

hadoop fs -setrep -R /user/datahub:

It is used to accept the backward capability and has no effect.

hadoop fs – setrep -w /user/datahub: waits for the replication to be completed

22. hadoop fs -getmerge

It concatenates HDFS files in source into the destination local file

Example:

hadoop fs -getmerge /user/datahub

23. hadoop fs -appendToFile

Appends a single source or multiple sources from the local file system to the destination.

Example:

hadoop fs -appendToFile xyz.log data.csv /in/appendfile

24. hadoop fs -stat

It prints the statistics about the file or directory.

Example:

hadoop fs -stat [format]

Conclusion

So, we have gone through almost all the commands which are necessary for file handling and view the data inside the files. You can modify your files and ingest data into the Hadoop platform now.

Recommended Articles

This is a guide to Hadoop fs Commands. Here we discuss the basic concept, and various Hadoop fs commands along with its example in detail. You may also look at the following articles to learn more –

  1. Install Hadoop
  2. Hadoop Tools
  3. Hadoop Architecture
  4. Hadoop Components
  5. Learn Different Types of Hadoop Commands
Popular Course in this category
Hadoop Training Program (20 Courses, 14+ Projects, 4 Quizzes)
  20 Online Courses |  14 Hands-on Projects |  135+ Hours |  Verifiable Certificate of Completion
4.5
Price

View Course

Related Courses

Data Scientist Training (85 Courses, 67+ Projects)4.9
Machine Learning Training (20 Courses, 29+ Projects)4.8
MapReduce Training (2 Courses, 4+ Projects)4.7
0 Shares
Share
Tweet
Share
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2022 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA
Free Data Science Course

SPSS, Data visualization with Python, Matplotlib Library, Seaborn Package

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more