解压
在hadoop mapreduce程序中使用如下代码解压lzo文件:
LzopDecompressor lzoDec= new LzopDecompressor(lzoBufferSize);
InputStream is = new FileInputStream("xxx");
if (is != null) {
LzopInputStream lzoIs = new LzopInputStream(is,lzoDec, lzoBufferSize);
BufferedReader reader = new BufferedReader(new InputStreamReader(lzoIs));
String line = null;
while ((line = reader.readLine()) != null) {
// to do
context.progress();
}
reader.close();
lzoIs.close();
is.close();
}
出现了如下的错误:
org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.InternalError: lzo1x_decompress_safe returned: -5
at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native Method)
at com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:309)
at com.hadoop.compression.lzo.LzopDecompressor.decompress(LzopDecompressor.java:122)
at com.hadoop.compression.lzo.LzopInputStream.decompress(LzopInputStream.java:247)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
这个主要是lzoBufferSize值设置过小导致的,将值设置大一点就可解决。