配置
-
backup-masters
- Not present by default. A plain-text file which lists hosts on which the Master should start a backup Master process, one host per line. hadoop-metrics2-hbase.properties
- Used to connect HBase Hadoop’s Metrics2 framework. See the Hadoop Wiki entry for more information on Metrics2. Contains only commented-out examples by default. hbase-env.cmd and hbase-env.sh
- Script for Windows and Linux / Unix environments to set up the working environment for HBase, including the location of Java, Java options, and other environment variables. The file contains many commented-out examples to provide guidance. hbase-policy.xml
- The default policy configuration file used by RPC servers to make authorization decisions on client requests. Only used if HBase security is enabled. hbase-site.xml
-
HBase主要配置文件
该文件上面的配置会覆盖默认配置docs/hbase-default.xml
我们可以通过HBase浏览器界面查看全部有效配置
log4j.properties
- HBase日志配置 regionservers
-
A plain-text file containing a list of hosts which should run a RegionServer in your HBase cluster. By default, this file contains the single entry
localhost. It should contain a list of hostnames or IP addresses, one per line, and should only containlocalhostif each node in your cluster will run a RegionServer on itslocalhostinterface.
优化
内存优化
建议分配整个可用内存的70%给HBase的Java堆（Table可以缓存在内存中）
但不建议分配非常大的堆内存，因为GC过程持续太久会导致RegionServer处于长期不可用状态
允许在HDFS的文件中追加内容
配置文件：hdfs-site.xml、hbase-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
dfs.support.append | true | 开启HDFS追加同步，可以优秀的配合HBase的数据同步和持久化 |
DataNode允许的最大文件打开数
配置文件：hdfs-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
dfs.datanode.max.transfer.threads | ≥4096 | HBase一般都会同一时间操作大量的文件，根据集群的数量和规模以及数据动作 |
数据操作的等待时间
配置文件：hdfs-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
dfs.image.transfer.timeout | >60000毫秒 | 如果某次数据操作延迟非常高，socket需要等待较长时间，建议把该值设置更大，使socket不容易超时断联 |
数据的写入效率
配置文件：mapred-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
mapreduce.map.output.compress | true | 开启map输出压缩，可减少写入时间 |
mapreduce.map.output.compress.codec | GzipCodec | 也可用其它压缩方式 |
RPC监听数量
配置文件：hbase-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
hbase.regionserver.handler.count | ≥30 | Count of RPC Listener instances spun up on RegionServers. Same property is used by the Master for count of master handlers. Too many handlers can be counter-productive. Make it a multiple of CPU count. If mostly read-only, handlers count close to cpu count does well. Start with twice the CPU count and tune from there. |
HStore文件大小
配置文件：hbase-site.xml
| 属性 | 建议值 | 说明 |
|---|---|---|
hbase.hregion.max.filesize | 默认值10737418240（10GB）；跑HBase的MR时，可减小此值，因为一个Region对应一个map，单个Region过大会导致map任务执行时间过长 | 如果Region中的所有HFile的大小总和达到该数值，该Region就切为两个 |
HBase客户端缓存
配置文件：hbase-site.xml
| 属性 | 建议 | 说明 |
|---|---|---|
hbase.client.write.buffer | 增大该值可以减少RPC调用次数(但会消耗更多内存) | 指定HBase客户端缓存 |
指定scan.next扫描HBase所获取的行数
配置文件：hbase-site.xml
| 属性 | 建议 | 说明 |
|---|---|---|
hbase.client.scanner.caching | 值越大，消耗内存越大 | 指定scan.next方法获取的默认行数 |
flush、compact、split
| 属性 | 默认值 | 说明 |
|---|---|---|
hbase.hregion.memstore.flush.size | 134217728 | Memstore will be flushed to disk if size of the memstore exceeds this number of bytes. Value is checked by a thread that runs every hbase.server.thread.wakefrequency. |
hbase.regionserver.global.memstore.size.lower.limit | none | Maximum size of all memstores in a region server before flushes are forced. Defaults to 95% of hbase.regionserver.global.memstore.size (0.95). A 100% value for this value causes the minimum possible flushing to occur when updates are blocked due to memstore limiting. The default value in this configuration has been intentionally left empty in order to honor the old hbase.regionserver.global.memstore.lowerLimit property if present. |
hbase.regionserver.global.memstore.size | none | Maximum size of all memstores in a region server before new updates are blocked and flushes are forced. Defaults to 40% of heap (0.4). Updates are blocked and flushes are forced until size of all memstores in a region server hits hbase.regionserver.global.memstore.size.lower.limit. The default value in this configuration has been intentionally left empty in order to honor the old hbase.regionserver.global.memstore.upperLimit property if present. |
不知所云
建议把hbase.regionserver.global.memstore.size.lower.limit设置0.75左右
HBase启动一段时间后关不掉
关闭需要进程ID，进程ID默认存到/tmp，一段时间后自动清理，对此可修改路径
vim $HBASE_HOME/conf/hbase-env.sh
export HBASE_PID_DIR=$HBASE_HOME/logs/pids
分发
rsync.py $HBASE_HOME/conf/hbase-env.sh
Appendix
| en | 🔉 | cn |
|---|---|---|
| backup | ˈbækʌp | n. 支持；后援；adj. 支持的；候补的；vt. 做备份 |
| specify | ˈspesɪfaɪ | vt. 指定；详细说明；列举；把…列入说明书 |
| framework | ˈfreɪmwɜːrk | n. 框架，骨架；结构，构架 |
| exceed | ɪkˈsiːd | v. 超出 |
| intentionally | ɪnˈtenʃənəli | adv. 故意地 |
199
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
