Why use HDFS over Lustre and / or GPFS?

最新推荐文章于 2023-07-12 20:48:13 发布

转载最新推荐文章于 2023-07-12 20:48:13 发布 · 1.2k 阅读

文章标签：

#licensing #file #hadoop #system #mapreduce #performance

Hadoop 专栏收录该内容

22 篇文章

订阅专栏

本文探讨了HDFS相较于Lustre和GPFS的优势，特别是在大规模文件系统的应用场景下。HDFS能够提供更高的扩展性，更优的价格性能比，并且在支持MapReduce任务的数据局部性和复制方面具有独特优势。

Apache Hadoop: Why use HDFS over Lustre and / or GPFS?

Curious to understand what some specific use cases would be for adopting HDFS as a file system over Lustre or GPFS. i.e. what kind of storage / computing needs are more suited to one file system over the other. thanks

Allen Wittenauer, I make yellow elephants cry.

In general (and not really geared towards the specific file systems):

Many large scale file systems do not scale as high as HDFS. (GPFS, for example, maxes out at 4PB, IIRC).

The price/perf ratio may not be as good, especially if licensing is involved.
- This can some times be attributed to POSIX support. This isn't 'cheap' and will impact either the cost or the performance or both. Ignoring the Amazon Web Services and other Cloud Services cases, this is probably the #1 reason why one would not use HDFS when using Hadoop.
Lack of MapReduce block locality support. (Of course, I'm assuming one would implement the ability to read specific blocks in the Hadoop FileSystem API. But that might be problematic for some systems as well...)
Lack of replication (i.e., only one copy of the file on disk so ability to recover data becomes more important).

Few 'real world' large scale references. No one wants to be the first on the block to try Hadoop+other fs outside of a lab. This is made worse by file system vendors having a very limited view of what Hadoop actually does and why. (The older Lustre paper at http://wiki.lustre.org/images/1/... is a funread.)