Non hdfs Used

最近研究hadoop hdfs 中NonDFSUsed容量是什么，Non DFS Used为非hadoop文件系统所使用的空间，比如说本身的linux系统使用的，或者存放的其它文件。

有的时候你会发现HDFS UI界面上显示的Non DFS Used很大，但实际的服务器上都没有那么大的空间了，看看下边的解释：

Non DFS Used = Configured Capacity - Remaining Space - DFS Used
Non DFS Used = 配置的容量 - 剩余容量 - DFS使用容量

Because Configured Capacity = Total Disk Space - Reserved Space.
因为配置容量 = 总容量 - 预留空间

So Non DFS used = ( Total Disk Space - Reserved Space) - Remaining Space - DFS Used
Non DFS used=（总容量-预留空间）- 剩余容量 - DFS使用容量

Let’s take a example. Assuming I have 100 GB disk, and I set the reserved space (dfs.datanode.du.reserved) to 30 GB.
让我举个例子，架设一个100GB的磁盘，设置预留30GB

In the disk, the system and other files used up to 40 GB, DFS Used 10 GB. If you run df -h , you will see the available space is 50GB for that disk volume.
磁盘中系统和其他文件为40GB，dfs使用10GB，如果df -h 你将看到可用空间是50GB的磁盘卷

In HDFS web UI, it will show
在hdfs UI界面会显示

Non DFS used = 100GB(Total) - 30 GB( Reserved) - 10 GB (DFS used) - 50GB(Remaining) = 10 GB

So it actually means, you initially configured to reserve 30G for non dfs usage, and 70 G for HDFS. However, it turns out non dfs usage exceeds the 30G reservation and eat up 10 GB space which should belongs to HDFS!
这实际上意味着，最初配置DFS预留30GB，和HDFS 70GB，然而预留的30G和已经使用的10GB的空间，这部分属于HDFS

The term “Non DFS used” should really be renamed to something like “How much configured DFS capacity are occupied by non dfs use”
“Non DFS used” 应该解释为”配置的dfs的空间有多少空间被不是hdfs的文件占用了的”

One useful command is” lsof | grep delete”, which will help you identify those open file which has been deleted. Sometimes, Hadoop processes (like hive, yarn, and mapred and hdfs) may hold reference to those already deleted files. And these references will occupy disk space.
我们可以使用命令lsof | grep delete 这将帮助你识别那些已被删除的文件
有时,Hadoop流程(像hive, yarn, and mapred and hdfs)可能引用那些已经删除文件。这些引用将占用磁盘空间。

Also du -hsx * | sort -rh | head -10 helps list the top ten largest folders.
还可以使用 du -hsx * | sort -rh | head -10 帮助最大的十大文件夹列表

参考：

http://stackoverflow.com/questions/18477983/what-exactly-non-dfs-used-means

http://blog.youkuaiyun.com/u014297175/article/details/48679321