java 读取hadoop文件,用Java读取HDFS和本地文件

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the following

Configuration configuration = new Configuration();

FileSystem fileSystem = null;

if (filePath.startsWith("hdfs://")) {

fileSystem = FileSystem.get(configuration);

} else if (filePath.startsWith("file://")) {

fileSystem = FileSystem.getLocal(configuration).getRawFileSystem();

}

From here I use the API's of the FileSystem to read the file.

Can you please let me know if there is any other better way than this?

解决方案

Does this make sense,

public static void main(String[] args) throws IOException {

Configuration conf = new Configuration();

conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));

conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));

BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

System.out.println("Enter the file path...");

String filePath = br.readLine();

Path path = new Path(filePath);

FileSystem fs = path.getFileSystem(conf);

FSDataInputStream inputStream = fs.open(path);

System.out.println(inputStream.available());

fs.close();

}

You don't have to put that check if you go this way. Get the FileSystem directly from Path and then do whatever you feel like.