hadoop2.2.0 core-site.xml--i/o properties

本文详细介绍了Hadoop中SequenceFile的相关配置参数及其作用,包括缓存大小、校验和设置、压缩编解码器、序列化方式等,并解释了这些参数如何影响SequenceFile的操作效率。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

<!-- i/o properties -->

<property>
  <name>io.file.buffer.size</name>
  <value>4096</value>
  <description>The size of buffer for use in sequence files.
  The size of this buffer should probably be a multiple of hardware
  page size (4096 on Intel x86), and it determines how much data is
  buffered during read and write operations.</description>
</property>

注释:操作sequence文件时的缓存大小。

<property>
  <name>io.bytes.per.checksum</name>
  <value>512</value>
  <description>The number of bytes per checksum.  Must not be larger than
  io.file.buffer.size.</description>
</property>

注释:

<property>
  <name>io.skip.checksum.errors</name>
  <value>false</value>
  <description>If true, when a checksum error is encountered while
  reading a sequence file, entries are skipped, instead of throwing an
  exception.</description>
</property>

注释:

<property>
  <name>io.compression.codecs</name>
  <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
  <description>A list of the compression codec classes that can be used 
               for compression/decompression.</description>
</property>

注释:

<property>
  <name>io.serializations</name>
  <value>org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization</value>
  <description>A list of serialization classes that can be used for
  obtaining serializers and deserializers.</description>
</property>

注释:

<property>
  <name>io.seqfile.local.dir</name>
  <value>${hadoop.tmp.dir}/io/local</value>
  <description>The local directory where sequence file stores intermediate
  data files during merge.  May be a comma-separated list of
  directories on different devices in order to spread disk i/o.
  Directories that do not exist are ignored.
  </description>
</property>

注释:

<property>
  <name>io.map.index.skip</name>
  <value>0</value>
  <description>Number of index entries to skip between each entry.
  Zero by default. Setting this to values larger than zero can
  facilitate opening large MapFiles using less memory.</description>
</property>

注释:

property>
  <name>io.map.index.interval</name>
  <value>128</value>
  <description>
    MapFile consist of two files - data file (tuples) and index file
    (keys). For every io.map.index.interval records written in the
    data file, an entry (record-key, data-file-position) is written
    in the index file. This is to allow for doing binary search later
    within the index file to look up records by their keys and get their
    closest positions in the data file.
  </description>
</property>

注释:

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值