本文讨论了在Hadoop+HBase环境下，通过优化压缩算法（如LZO与Snappy）、自定义序列化机制（使用JSON除外特定数据类型）、重新设计无索引方案等方法，提升系统性能与稳定性。同时，针对部署策略，避免运行ZooKeeper实例于数据节点、不将HMaster置于NameNode之上，并建议采用额外的Master实例以防止单一故障点。
Compression
LZOor Snappy(20% better than LZO)——Block(default)
Serialization
AVROdidn’t work well——deserialization issue
Developedconfigurable serialization mechanism that uses JSON except Data type
Secondary Indexes
Wereusing ITHBase and IHBase from contrib——doesn’t work well
Redesignedschema without need for index
Westill need it though
Performance
Severaltunable parameters
Hardware(Hadoop+HBase)
DataNode -24GB RAM,8Cores,4*1TB(64GB,24Cores,8*2TB)
6mappers and 6 reducers per node (16 mappers,4reducers)
Memoryallocation by process
DataNode——1GB(2GB)
TaskTracker——1GB(2GB)
MapTasks——6*1GB(16*1.5GB)
ReduceTasks——6*1GB(4*1.5GB)
RegionServer——8GB(24GB)
TotalAllocation——24GB(64GB)
Deployment
Donot run ZK instances on DN,have a separate ZK quorum(3 minimum)
Donot run HMaster on NN
AvoidSPOF for HMaster(run additional master(s))
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
