Java 写入hdfs程序代码

Java 写入hdfs程序代码

在这里直接上代码

 

Configurationconf = new Configuration();

conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl",org.apache.hadoop.fs.LocalFileSystem.class.getName());
conf.setBoolean("dfs.support.append",true);    

hdfs = FileSystem.get(conf);

String hdfs ="hdfs://10.20.30.91:8020/kafka/a.log";
filePath = new Path(hdfs);

FSDataOutputStream outputStream = hdfs.create(filePath);

outputStream.write(“hello”)

outputStream.flush();

outputStream.close();
hdfs.close();

 

在这里我说一下这一段代码：

 

conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl",org.apache.hadoop.fs.LocalFileSystem.class.getName());

假如不配置上面这段话，在maven 打包或者是gradle 打包运行jar的时候，运行jar会出现

错误异常：

Exception in thread "main" java.io.IOException:NoFileSystemfor scheme: file

    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375)

    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)

    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)

    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)

    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:180)

    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)

    at org.apache.mahout.classifier.naivebayes.NaiveBayesModel.materialize(NaiveBayesModel.java:100)

可能原因：

conf.getClass需要读取hadoop-common-x.jar下面的core-default.xml，打包之后，找不到系统默认的位置，需要配置上去

参考链接：

http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file