Ways to write & read HDFS files

最新推荐文章于 2025-08-10 16:33:07 发布

chaoping315

最新推荐文章于 2025-08-10 16:33:07 发布

阅读量1.1k

点赞数

分类专栏： hadoop hdfs 文章标签： path stream jobs output hadoop dos

hadoop 同时被 2 个专栏收录

5 篇文章

订阅专栏

hdfs

1 篇文章

订阅专栏

本文介绍了三种在Hadoop中操作HDFS文件的方法：使用FSDataOutputStream直接进行读写、利用BufferedReader/Writer进行文本处理及SequenceFile进行高效的数据序列化读写。这些方法适用于不同的场景需求。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Ways to write & read HDFS files

- Output Stream

FSDataOutputStream dos = fs.create(new Path("/user/tmp"), true); 
dos.writeInt(counter); 
dos.close();

- Buffered Writer/Reader

//Writer
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fs.create(new Path("/user/tmp"), true)));
bw.write(counter.toString());
bw.close();

//Reader

Configuration conf = context.getConfiguration();

FileSystem fs = FileSystem.get(conf);

DataInputStream d = new DataInputStream(fs.open(new Path(inFile)));
BufferedReader reader = new BufferedReader(new InputStreamReader(d));
while ((line = reader.readLine()) != null){
...
}
reader.close();

- SequenceFile Reader and Writer (I think most preferable way for Hadoop jobs):

//writer
SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new Path(pathForCounters, context.getTaskAttemptID().toString()), Text.class, Text.class);
   writer.append(new Text(firtUrl.toString()+"__"+ context.getTaskAttemptID().getTaskID().toString()), new Text(counter+""));
   writer.close(); 

//reader
SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path(makeUUrlFileOffsetsPathName(FileInputFormat.getInputPaths(context)[0].toString())),  conf);
   while (reader.next(key, val)){
    offsets.put(key.toString(), Integer.parseInt(val.toString()));
   }