Hadoop hdfs常用命令

hadoop fs、 hadoop dfs 、hdfs dfs 命令区别

• Hadoop fs：使用面最广，可以操作任何文件系统。
• hadoop dfs与hdfs dfs：只能操作HDFS文件系统相关（包括与Local FS间的操作），前者已经Deprecated，一般使用后者。

model@host2 ~]$ hadoop dfs -ls
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Found 3 items
drwx------   - model supergroup          0 2020-01-20 23:00 .Trash
drwxr-xr-x   - model supergroup          0 2020-01-19 17:06 .sparkStaging
drwx------   - model supergroup          0 2020-01-21 15:34 .staging
[model@host2 ~]$ hadoop fs -ls
Found 3 items
drwx------   - model supergroup          0 2020-01-20 23:00 .Trash
drwxr-xr-x   - model supergroup          0 2020-01-19 17:06 .sparkStaging
drwx------   - model supergroup          0 2020-01-21 15:34 .staging

查看版本version

$ hadoop version
Hadoop 2.6.0-cdh5.15.0
Subversion http://github.com/cloudera/hadoop -r e3cb23a1cb2b89d074171b44e71f207c3d6ffa50
Compiled by jenkins on 2018-05-24T11:19Z
Compiled with protoc 2.5.0
From source with checksum 2efeda2e2d6b27ecf59ab56fffd9881
This command was run using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/jars/hadoop-common-2.6.0-cdh5.15.0.jar

hadoop hdfs 常用命令

hadoop fs -mkdir /创建HDFS目录

hadoop fs -ls /列出HDFS目录

hadoop fs -copyFromLocal /复制本地（local)文件到HDFS

hadoop fs -put /复制本地（local)文件到HDFS

hadoop fs -cat /列出HDFS目录下的文件内容

hadoop fs -copyToLocal /将HDFS上的文件复制到本地（local）

hadoop fs -get /将HDFS上的文件复制到本地（local）

hadoop fs -cp /复制HDFS文件

hadoop fs -rm /复制HDFS文件

实例：

hadoop  fs -ls /user/hive/warehouse/edw_ai.db/mod

hadoop  fs -rm -r  /user/hive/warehouse/edw_ai.db/mod

-put 与 -copyFromLocal不同

• -put接收stdin(标准输入)
• 如果文件已经存在，系统不会显示文件已经纯在，-put 而会直接覆盖

---

Hadoop端口

• 50070 : HDFS WEB UI端口

网址	说明
网址 http://master:50070	Hadoop HDFS WEB界面网址

---

• 创建目录

hadoop fs -mkdir

$ hadoop fs -mkdir test  //不指定路径 则默认 在 /user/账号名
$ hadoop fs -ls /user/model
Found 4 items
drwx------   - model supergroup          0 2020-01-21 16:22 /user/model/.Trash
drwxr-xr-x   - model supergroup          0 2020-01-19 17:06 /user/model/.sparkStaging
drwx------   - model supergroup          0 2020-01-21 15:34 /user/model/.staging
drwxr-xr-x   - model supergroup          0 2020-01-21 16:26 /user/model/test

• 查看根目录

$ hadoop fs -ls /
Found 2 items
drwxrwxrwt   - hdfs supergroup          0 2019-12-26 16:41 /tmp
drwxrwxrwx   - hdfs supergroup          0 2020-01-17 15:26 /user

• 创建多级HDFS目录 hadoop fs -mkdir -p /dir1/dir2/dir3
• 查看所有的子目录 hadoop fs -ls -R /

$ hadoop fs -mkdir -p /user/model/test1/test11
$ hadoop fs -ls -R /user/model
drwx------   - model supergroup          0 2020-01-21 16:22 /user/model/.Trash
drwx------   - model supergroup          0 2020-01-21 16:22 /user/model/.Trash/Current
drwx------   - model supergroup          0 2020-01-21 16:22 /user/model/.Trash/Current/user
drwx------   - model supergroup          0 2020-01-21 16:22 /user/model/.Trash/Current/user/model
drwxr-xr-x   - model supergroup          0 2020-01-21 16:19 /user/model/.Trash/Current/user/model/test2020
drwxr-xr-x   - model supergroup          0 2020-01-19 17:06 /user/model/.sparkStaging
drwx------   - model supergroup          0 2020-01-07 14:09 /user/model/.sparkStaging/application_1573720960832_5936
-rw-r--r--   3 model supergroup     121361 2020-01-07 14:09 /user/model/.sparkStaging/application_1573720960832_5936/__spark_conf__.zip
-rw-r--r--   3 model supergroup  241874791 2020-01-07 14:09 /user/model/.sparkStaging/application_1573720960832_5936/__spark_libs__8122587516618861829.zip
drwx------   - model supergroup          0 2020-01-21 15:34 /user/model/.staging
drwxr-xr-x   - model supergroup          0 2020-01-21 16:26 /user/model/test
drwxr-xr-x   - model supergroup          0 2020-01-21 16:42 /user/model/test1
drwxr-xr-x   - model supergroup          0 2020-01-21 16:42 /user/model/test1/test11

• 强制覆盖 -f

hadoop fs -copyFromLocal

hadoop fs -copyFromLocal /user/model/test1/test11 /user/mode/test

hadoop fs -copyFromLocal /user/model/test1/test11 /user/mode/test

$ hadoop fs -copyFromLocal /usr/local/doc/cmake-3.14/Copyright.txt /user/model/test
copyFromLocal: `/user/model/test/Copyright.txt': File exists
$ hadoop fs -copyFromLocal -f /usr/local/doc/cmake-3.14/Copyright.txt /user/model/test

hadoop fs -put

hadoop fs -cat

hadoop fs -cat /user/model/test/echo.txt

hadoop fs -copyToLocal

[model@host2 ~]$ mkdir test
[model@host2 ~]$ cd test/
[model@host2 test]$ ls
[model@host2 test]$ hadoop fs -copyToLocal /user/model/test/Copyright.txt
[model@host2 test]$ ll
-rw-r--r-- 1 model model 5302 Jan 21 17:33 Copyright.txt

hadoop fs -get

[model@host2 test]$ hadoop fs -get /user/model/test/Copyright.txt
[model@host2 test]$ ll
total 8
-rw-r--r-- 1 model model 5302 Jan 21 17:36 Copyright.txt

hadoop fs -cp

hadoop fs -rm

/删除目录
hadoop fs -rm -R /user/model/test