<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[chenla5762的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/chenla5762</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; chenla5762]]></copyright><item><title><![CDATA[Spark-sql 读hbase]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625919</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625919</guid><author>chenla5762</author><pubDate>Fri, 15 Dec 2017 08:34:00 +0800</pubDate><description><![CDATA[SparkSQL是指整合了Hive的spark-sql cli, 本质上就是通过Hive访问HBase表，具体就是通过hive-hbase-handler 
 
 
 
 拷贝HBase的相关jar包到Spark节点上的$SPARK_HOME/lib目录下，清单如下： 
  
 
 
...]]></description><category></category></item><item><title><![CDATA[hbase 写操作产生的木桶效应]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625917</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625917</guid><author>chenla5762</author><pubDate>Thu, 07 Dec 2017 21:43:00 +0800</pubDate><description><![CDATA[生产环境中hbase的写存在木桶效应，具体体现在hbase写入数据缓慢，在参数优化极好的情况下，仍然没有解决问题。经过多次的排查，产生问题的原因有2个： 
1.网络问题 
 某个节点的网卡出现不明显故障，造成该节点网速传输速度下降。 
2.异构设备磁盘硬件I/O 
 有时候很多利旧设备...]]></description><category></category></item><item><title><![CDATA[Hadoop MapRduce 重写DBOutputFormat更新mysql数据库]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625916</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625916</guid><author>chenla5762</author><pubDate>Mon, 04 Dec 2017 22:55:00 +0800</pubDate><description><![CDATA[/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed wi...]]></description><category></category></item><item><title><![CDATA[mysql 分表]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625915</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625915</guid><author>chenla5762</author><pubDate>Sun, 03 Dec 2017 22:11:00 +0800</pubDate><description><![CDATA[创建的分表必须是 
ENGINE=MyISAM 
例如这里我们分两个表member1,member2 
DROP table IF EXISTS member1;
create table member1(
    id bigint primary key auto_increment ,...]]></description><category></category></item><item><title><![CDATA[hbase 日常运维]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625913</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625913</guid><author>chenla5762</author><pubDate>Mon, 27 Nov 2017 10:05:00 +0800</pubDate><description><![CDATA[1 Hbase日常运维 
1.1 监控Hbase运行状况 
1.1.1 操作系统 
1.1.1.1 IO  
a. 群集网络IO，磁盘IO，HDFS IO 
IO越大说明文件读写操作越多。当IO突然增加时，有可能：1.compact队列较大，...]]></description><category></category></item><item><title><![CDATA[ambari 安装过程 注意事项]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625911</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625911</guid><author>chenla5762</author><pubDate>Mon, 27 Nov 2017 09:57:00 +0800</pubDate><description><![CDATA[1.如果使用脚本部署请拔掉一根网线 2.修改进程限制 /etc/security/limits.d/90-nproc.conf  修改1024 为unlimited 3.ntpd ，httpd 服务开启，设置开机启动，防火墙关闭,设置开机关闭 4.修改文件描述符大小(65536) 打开文件数...]]></description><category></category></item><item><title><![CDATA[oozie  web ui  kerberos访问]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625909</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625909</guid><author>chenla5762</author><pubDate>Thu, 23 Nov 2017 17:42:00 +0800</pubDate><description><![CDATA[本例使用FireFox浏览器： 
1.在Firefox浏览器URL里输入：about:config 
2.设置以下两个配置项： 
network.negotiate-auth.trusted-uris=.bigdata 
network.auth.use-sspi=false 
3.重启Fir...]]></description><category></category></item><item><title><![CDATA[hive 插入中文乱码]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625907</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625907</guid><author>chenla5762</author><pubDate>Thu, 23 Nov 2017 17:40:00 +0800</pubDate><description><![CDATA[插入数据时，要将中文数据转码为 iso8859-1 
new String("测试数据".getBytes(),"iso8859-1"); 
如果是shell 将shell 工具转为iso 编码格式 

                                             ...]]></description><category></category></item><item><title><![CDATA[Hive格式各种格式下不同压缩算法的比较]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625906</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625906</guid><author>chenla5762</author><pubDate>Thu, 23 Nov 2017 17:39:00 +0800</pubDate><description><![CDATA[原始Text格式的hive分区大小为119.2G。 
 
  
   
    压缩算法  
    Text格式  
    Parquet格式  
    ORC  
    RCFile  
   
   
    不压缩  
    119.2G  
    54.1G  
    2...]]></description><category></category></item><item><title><![CDATA[Hbase集群间数据迁移方法总结]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625905</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625905</guid><author>chenla5762</author><pubDate>Thu, 23 Nov 2017 17:38:00 +0800</pubDate><description><![CDATA[今天花了一天的时间查资料做测试，略微的总结了一下hbase数据迁移的方法。 
一、需要在hbase集群停掉的情况下迁移 
步骤：(1)执行hadoop distcp -f filelist "hdfs://new cluster ip:9000/hbasetest" 
       (2)在...]]></description><category></category></item><item><title><![CDATA[ambari 干掉一个namenode 从新安装最快的方式]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625903</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625903</guid><author>chenla5762</author><pubDate>Thu, 23 Nov 2017 17:36:00 +0800</pubDate><description><![CDATA[具体实施步骤： 
1.干掉之前，将namenode迁移走，不迁移也可以（ps:安装之后需要拷贝namenode的目录） 
2.ZooKeeper Server在别的节点创建一个 
3.JournalNode ambari 没法创建，只能这样，等待安装完成之后将别的节点的JournalNode ...]]></description><category></category></item><item><title><![CDATA[hbase分页]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625902</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625902</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 19:32:00 +0800</pubDate><description><![CDATA[hbase 分页小例子，相比于PageFilter的话，效率要高（ps:这个效率高，是指设计的表rowkey很合理） 
 
 
publicList&lt;Map&lt;String,String&gt;&gt;getKkListLnHbaseNew(Stringbegin,St...]]></description><category></category></item><item><title><![CDATA[region server在写入减少睡眠等待时间]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625901</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625901</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:49:00 +0800</pubDate><description><![CDATA[region server在写入时会检查每个region对应的memstore的总大小是否超过了memstore默认大小的2倍(hbase.hregion.memstore.block.multiplier决定)，如果超过了则锁住memstore不让新写请求进来并触发flush，避免产生OOM...]]></description><category></category></item><item><title><![CDATA[jar包版本不一致问题引起的hbase region 连续挂掉]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625900</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625900</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:47:00 +0800</pubDate><description><![CDATA[hbase-annotations、hbase-client、hbase-common、hbase-protocol 版本要与集群的版本一致，否则可能会造成regionserver dead掉 

                                                ...]]></description><category></category></item><item><title><![CDATA[coprocessor]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625899</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625899</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:47:00 +0800</pubDate><description><![CDATA[添加 
*disable'emp' 
*alter'emp',METHOD=&gt;'table_att','coprocessor'=&gt;'hdfs://mycluster:8020/ldstest/coprocess.jar|com.test.coprocessor.Pla...]]></description><category></category></item><item><title><![CDATA[Hbase split的三种方式和split的过程]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625896</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625896</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:45:00 +0800</pubDate><description><![CDATA[当一个reion达到一定的大小，他会自动split称两个region。如果我们的Hbase版本是0.94 ，那么默认的有三种自动split的策略，ConstantSizeRegionSplitPolicy,IncreasingToUpperBoundRegionSplitPolicy还有Ke...]]></description><category></category></item><item><title><![CDATA[将hadoop安装部署到window]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625895</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625895</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:44:00 +0800</pubDate><description><![CDATA[1.将hadoop 安装部署包解压到window的硬盘中，比如解压到d:/hadoop 
2.设置环境变量 HADOOP_HOME 一定要大写 
3.去github上下载hadoop-common-2.2.0-bin-master 然后解压，将d:/hadoop/bin 目录全部替换为hado...]]></description><category></category></item><item><title><![CDATA[hadoop 集群txid不一致]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625894</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625894</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:43:00 +0800</pubDate><description><![CDATA[hadoop 集群txid不一致，可以执行Hadoop namenode -recovery 或者修改seen_txid 文件中的内容进行修改 

                                                    
转载于:https://my.oschi...]]></description><category></category></item><item><title><![CDATA[access control error while attempting to set up short-circuit]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625893</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625893</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:41:00 +0800</pubDate><description><![CDATA[due to InvalidToken exception. 
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-...]]></description><category></category></item><item><title><![CDATA[NameNode参数堆大小参考表]]></title><link>https://blog.youkuaiyun.com/chenla5762/article/details/100625892</link><guid>https://blog.youkuaiyun.com/chenla5762/article/details/100625892</guid><author>chenla5762</author><pubDate>Wed, 22 Nov 2017 09:24:00 +0800</pubDate><description><![CDATA[转载于:https://my.oschina.net/u/3094112/blog/1577075]]></description><category></category></item></channel></rss>