hadoop_hdfs常用shell 集群数据拷贝-优快云博客

本文链接：https://blog.youkuaiyun.com/iteye_3893/article/details/82601631

0 写法类似于linux命令

[root@hadoop0 hadoop]# hadoop
Warning: $HADOOP_HOME is deprecated.

Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin client
  fsck                 run a DFS filesystem checking utility
  fs                   run a generic filesystem user client
  balancer             run a cluster balancing utility
  fetchdt              fetch a delegation token from the NameNode
  jobtracker           run the MapReduce job Tracker node
  pipes                run a Pipes job
  tasktracker          run a MapReduce task Tracker node
  historyserver        run job history servers as a standalone daemon
  job                  manipulate MapReduce jobs
  queue                get information regarding JobQueues
  version              print the version
  jar <jar>            run a jar file
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
 or
  CLASSNAME            run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

[root@hadoop0 hadoop]# hadoop fs
Warning: $HADOOP_HOME is deprecated.

Usage: java FsShell
           [-ls <path>]
           [-lsr <path>]
           [-du <path>]
           [-dus <path>]
           [-count[-q] <path>]
           [-mv <src> <dst>]
           [-cp <src> <dst>]
           [-rm [-skipTrash] <path>]
           [-rmr [-skipTrash] <path>]
           [-expunge]
           [-put <localsrc> ... <dst>]
           [-copyFromLocal <localsrc> ... <dst>]
           [-moveFromLocal <localsrc> ... <dst>]
           [-get [-ignoreCrc] [-crc] <src> <localdst>]
           [-getmerge <src> <localdst> [addnl]]
           [-cat <src>]
           [-text <src>]
           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
           [-moveToLocal [-crc] <src> <localdst>]
           [-mkdir <path>]
           [-setrep [-R] [-w] <rep> <path/file>]
           [-touchz <path>]
           [-test -[ezd] <path>]
           [-stat [format] <path>]
           [-tail [-f] <file>]
           [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
           [-chown [-R] [OWNER][:[GROUP]] PATH...]
           [-chgrp [-R] GROUP PATH...]
           [-help [cmd]]

1. 常用命令简述

对hdfs操作的命令格式是hadoop fs
1.1 -ls <path> 表示对hdfs下一级目录的查看
1.2 -lsr <path> 表示对hdfs目录的递归查看
1.3 -mkdir <path> 创建目录
1.4 -put <src> <des> 从linux上传文件到hdfs
1.5 -get <src> <des> 从hdfs下载文件到linux
1.6 -text <path> 查看文件内容
1.7 -rm <path> 表示删除文件
1.7 -rmr <path> 表示递归删除文件

常用命令详述：

1.0 上述命令并不一定非要在 hadoop目录下执行，可以在别的linux目录下执行

1.1 -ls <path> 表示对hdfs下一级目录的查看

[root@master hadoop]# hadoop fs -ls hdfs://master:9000/
Warning: $HADOOP_HOME is deprecated.

Found 1 items
drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr
[root@master hadoop]# 

对上面信息的解释如下：
1 hdfs://master:9000是hadoop core-site.xml内配置的信息，也就是hdfs的根路径
2 hdfs://master:9000 写法可以类比于 http://192.168.1.113:8080 
  （协议类型://主机:端口）
3 请进入到hadoop目录内执行上面的命令，上述命令是展示hdfs根目录下的文件
4 drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr
  第一位为d表示目录， -表文件
  第十一位 - 表示副本个数， 副本个数只有文件才有，
  如果类型是文件夹的话(文件夹的作用仅仅是组织文件，不占用物理空间)则副本个数用-表示
  第十四位 0 表示文件大小
  第十五位 表示文件最近修改时间
  第十六位 .usr 表示文件名

综述： 查看hdfs根目录下的内容，得到 /usr这个文件

同样用上述方式，可以进入 hdfs的 /usr目录看下面的文件

[root@master hadoop]# hadoop fs -ls hdfs://master:9000/usr

1.2 -lsr <path> 表示对hdfs目录的递归查看

[root@master hadoop]# hadoop fs -lsr hdfs://master:9000/
Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr
drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr/local
drwxr-xr-x   - root supergroup          0 2014-07-19 21:08 /usr/local/hadoop
drwxr-xr-x   - root supergroup          0 2014-07-19 21:08 /usr/local/hadoop/tmp
drwxr-xr-x   - root supergroup          0 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred
drwx------   - root supergroup          0 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred/system
-rw-------   1 root supergroup          4 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred/system/jobtracker.info

执行
[root@master hadoop]# hadoop fs -lsr /
Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr
drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr/local
drwxr-xr-x   - root supergroup          0 2014-07-19 21:08 /usr/local/hadoop
drwxr-xr-x   - root supergroup          0 2014-07-19 21:08 /usr/local/hadoop/tmp
drwxr-xr-x   - root supergroup          0 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred
drwx------   - root supergroup          0 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred/system
-rw-------   1 root supergroup          4 2014-07-30 06:42 /usr/local/hadoop/tmp/mapred/system/jobtracker.info


其中在执行 # hadoop fs -lsr / 时， 因为没有发现hdfs协议的路径，因此会自动去
hadoop/conf/core-site.xml 
中找hdfs默认路径， 如下所示，此文件中配置的fs.default.name默认路径为 hdfs://master:9000 因此/就隐士的代表了配置文件中配置的hdfs根路径
<configuration>
 <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>  
</configuration>

1.3 -mkdir <path> 创建目录

[root@master conf]# hadoop fs -mkdir /d1
Warning: $HADOOP_HOME is deprecated.

[root@master conf]# hadoop fs -ls /
Warning: $HADOOP_HOME is deprecated.

Found 2 items
drwxr-xr-x   - root supergroup          0 2014-07-31 07:28 /d1
drwxr-xr-x   - root supergroup          0 2014-07-19 21:07 /usr

1.4 -put <src> <des> 从linux上传文件到hdfs

[root@master conf]# hadoop fs -put ./core-site.xml hdfs://master:9000/d1
Warning: $HADOOP_HOME is deprecated.

[root@master conf]# hadoop fs -ls /d1
Warning: $HADOOP_HOME is deprecated.

Found 1 items
-rw-r--r--   1 root supergroup        398 2014-07-31 07:35 /d1/core-site.xml
[root@master conf]# 



./core-site.xml 表示当前目录(.)下的文件 core-site.xml
上述命令是将当前目录conf下的文件 core-site.xml 上传到 hdfs根目录/d1目录内

1.5 -get <src> <des> 从hdfs下载文件到linux

[root@master sbin]# cd /root/Desktop/
[root@master Desktop]# hadoop fs -get /d1/core-site.xml .
Warning: $HADOOP_HOME is deprecated.

[root@master Desktop]# ls
core-site.xml


进入桌面，然后将 hdfs/d1/core-site.xml 下载到桌面   

hadoop fs -get /d1/core-site.xml . 中的.表示linux目录中的当前目录

1.6 -text <path> 查看文件内容

[root@master ~]# hadoop fs -text /d1/core-site.xml
Warning: $HADOOP_HOME is deprecated.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
 <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>  
</configuration>
[root@master ~]#

1.7 -rm <path> 表示删除文件

[root@master ~]# hadoop fs -rm /d1/core-site.xml
Warning: $HADOOP_HOME is deprecated.

Deleted hdfs://master:9000/d1/core-site.xml
[root@master ~]# hadoop fs -ls /d1
Warning: $HADOOP_HOME is deprecated.

Found 1 items
-rw-r--r--   1 root supergroup      71528 2014-07-31 07:48 /d1/ifconfig
[root@master ~]#

1.7 -rmr <path> 表示递归删除文件

[root@master ~]# hadoop fs -rmr hdfs://master:9000/*
Warning: $HADOOP_HOME is deprecated.

Deleted hdfs://master:9000/d1
Deleted hdfs://master:9000/usr
[root@master ~]# hadoop fs -ls /
Warning: $HADOOP_HOME is deprecated.

[root@master ~]# 



hadoop fs -rmr hdfs://master:9000/* 
上述操作是 递归删除hdfs根路径下的所有文件，  递归删除时，必须要带上全路径名称 上述命令中的*表示删除所有

1.8 查看所有命令写法：、

[root@master ~]# hadoop fs
Warning: $HADOOP_HOME is deprecated.

Usage: java FsShell
           [-ls <path>]
           [-lsr <path>]
           [-du <path>]
           [-dus <path>]
           [-count[-q] <path>]
           [-mv <src> <dst>]
           [-cp <src> <dst>]
           [-rm [-skipTrash] <path>]
           [-rmr [-skipTrash] <path>]
           [-expunge]
           [-put <localsrc> ... <dst>]
           [-copyFromLocal <localsrc> ... <dst>]
           [-moveFromLocal <localsrc> ... <dst>]
           [-get [-ignoreCrc] [-crc] <src> <localdst>]
           [-getmerge <src> <localdst> [addnl]]
           [-cat <src>]
           [-text <src>]
           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
           [-moveToLocal [-crc] <src> <localdst>]
           [-mkdir <path>]
           [-setrep [-R] [-w] <rep> <path/file>]
           [-touchz <path>]
    ..........

1.9 具体某个命令帮助查看

[root@master ~]# hadoop fs -help put
Warning: $HADOOP_HOME is deprecated.

-put <localsrc> ... <dst>:      Copy files from the local file system 
                into fs.

2.0 查看hdfs文件夹大小的两种写法：

1 统计文件夹大小    单位是bytes   需要/1024/1024/1024 = GB
[root@chinadaas109 ~]# hdfs dfs -count -q  /gsinfo
none             inf            none             inf           10           82           16634317 /gsinfo
[root@chinadaas109 ~]# hdfs dfs -du -s /gsinfo
16634317  /gsinfo

3.0 集群间复制

hadoop distcp hdfs://source hdfs://destination