HDFS integration with Openstack Swift

The overall model in HDFS data flow with Openstack Swift integration.

The works of HDFS and Openstack Swift have been done, refer to   HADOOP-8545. It introduces an new concept in big data storage. And benefit from this marriage, Hadoop cluster would be more agile and easier to be managed.

However, there are still some debates about this new model. One of the most  controversial issues are the data locality and performance, and there is an report to address this issue, refer to AMPLab analysis . The second one is the security and how to manage user's privacy. To date, the new model doesn't have any mechanism of permission control, refer to this warning.

1. The user specifies the connection URI, such as
   

   hdfs://namenode:8020

   or
   

   swift://store.example.org

2. The HDFS client would initialize the corresponding implements for the class FileSystem.
3. The class FileSystem defines the interfaces of file operations.

Next we can drill down and take "file read" as an example to explain the data flow.

Anatomy of a File Read in case of the Swift backend

1. Before any operation of HDFS, the application should initialize the class FileSystem by following, for example in java

   String uri = "swift://storage.example.org";
   FileSystem fs = FileSystem(uri, new Configuration());

2. Because the scheme is specified as "swift" by user, the instance of FileSystem would be SwiftNativeFileSystem.