<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[菜皮的默默倾诉]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/qqpy789</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qqpy789]]></copyright><item><title><![CDATA[hadoop源码编译，导入idea中]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/86155229</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/86155229</guid><author>qqpy789</author><pubDate>Wed, 09 Jan 2019 15:32:50 +0800</pubDate><description><![CDATA[我的笔记本是deepin15.8系统，其实就是一个ubunt

1.首先安装软件

sudo apt-get install maven libssl-dev build-essential pkgconf cmake

2.安装protobuf

下载protbuf 2.5.0
https://github.com/protocolbuffers/protobuf/releases?after=...]]></description><category></category></item><item><title><![CDATA[sqoop 报错之java.lang.InternalError]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78893103</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78893103</guid><author>qqpy789</author><pubDate>Mon, 25 Dec 2017 15:14:15 +0800</pubDate><description><![CDATA[java.lang.InternalError

        at sun.security.ec.SunEC.initialize(Native Method)

        at sun.security.ec.SunEC.access$000(SunEC.java:49)

        at sun.security.ec.SunEC$1.run(SunEC.java]]></description><category></category></item><item><title><![CDATA[spark rdd转dataframe的两种方式]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78412500</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78412500</guid><author>qqpy789</author><pubDate>Wed, 01 Nov 2017 11:50:49 +0800</pubDate><description><![CDATA[方式1:
使用隐式转换
 case class Dika(skuId: Int,skudesc:String, listId: Int, listDesc:String)



 import hiveContext.implicits._
    val p = sc.parallelize(Seq(new Dika(1,"miao",1,"shu"))).toDF()


]]></description><category></category></item><item><title><![CDATA[yarn日常维护之nm健康状态为false]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78391386</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78391386</guid><author>qqpy789</author><pubDate>Mon, 30 Oct 2017 11:31:58 +0800</pubDate><description><![CDATA[最近发现yarn集群的ui上显示的nodes个数为2个,正常情况下是2个,然后就很无语了,因为以前一直都没有问题
然后差问题呗,从ui上显示丢失了206机器的nm,重新启动206上的nm 然后我查看206机器nm的日志和207上的rm的日志 
从日志上来看 没有任何问题,nm显示注册到了207机器,207机器显示收到了206机器的注册,这就无语了,我累个法克
然后磨叽了好几个小时,在查看20]]></description><category></category></item><item><title><![CDATA[hbase 开启缓冲和不开启缓冲 比较]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78343179</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78343179</guid><author>qqpy789</author><pubDate>Wed, 25 Oct 2017 16:37:22 +0800</pubDate><description><![CDATA[hbase 中的put 一个put相当于是一个rpc,如果循环的次数过多 则绝大部分时间都消耗在rpc的网路传输过程中 而不是用于传输数据的时间占比高
如果开启个缓冲 一次性提交批量数据 则会提高效率
得了 话就说那么多,直接上代码

package com.hit.test;

import org.apache.hadoop.conf.Configuration;
import org]]></description><category></category></item><item><title><![CDATA[面试杂谈]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78280699</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78280699</guid><author>qqpy789</author><pubDate>Thu, 19 Oct 2017 09:49:55 +0800</pubDate><description><![CDATA[最近面试了几家公司
第一家:可能是嫌我java功底不太厚
第二家:三面挂了 原因是 基础不够扎实 数据结构 以及jdk源码功力不是特别深厚
第三家:直接告诉我和公司业务不符合
第四家:嫌弃我没有实时流式的工作经验
总结:面试造飞机,工作补轮胎(程序员已经是红海了)]]></description><category></category></item><item><title><![CDATA[sqoop使用hcatlog生成对应的orc格式的表]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78232037</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78232037</guid><author>qqpy789</author><pubDate>Sat, 14 Oct 2017 09:25:10 +0800</pubDate><description><![CDATA[最近去面试,面试官提了一句,可以使用hcatlog来实现直接生成对应的parquet格式的表,不过我没有生成成功parquet表 但是生成成功了orc表 具体的原因还未知
不过终于可以直接生成orc格式的表了 orc基本上也够用了
sqoop import --connect $URL --username $USERNAME --password $PASSWORD --table erp_]]></description><category></category></item><item><title><![CDATA[SortShuffleManager方法头注释]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78216209</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78216209</guid><author>qqpy789</author><pubDate>Thu, 12 Oct 2017 16:16:50 +0800</pubDate><description><![CDATA[/**
 * In sort-based shuffle, incoming records are sorted according to their target partition ids, then
 * written to a single map output file. Reducers fetch contiguous regions of this file in order ]]></description><category></category></item><item><title><![CDATA[RDD五大特性]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78216195</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78216195</guid><author>qqpy789</author><pubDate>Thu, 12 Oct 2017 16:15:44 +0800</pubDate><description><![CDATA[*  - A list of partitions  rdd是分片的
*  - A function for computing each split 每个function是作用在每个split
*  - A list of dependencies on other RDDs rdd是有互相依赖的
*  - Optionally, a Partitioner for key-value RDDs]]></description><category></category></item><item><title><![CDATA[copyonwritelist源码理解]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78194730</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78194730</guid><author>qqpy789</author><pubDate>Tue, 10 Oct 2017 16:56:05 +0800</pubDate><description><![CDATA[ArrayList 的一个线程安全的变体，其中所有可变操作（add、set 等等）都是通过对底层数组进行一次新的复制来实现的。
上面这句话是官方jdk里写的 其实就是copyonwritelist是一个安全类 每次都是通过复制来实现线程安全
打开copyonwritelist的源码

   /** The lock protecting all mutators */
    final ]]></description><category></category></item><item><title><![CDATA[hdfs报错之 DisallowedDatanodeException]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78189335</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78189335</guid><author>qqpy789</author><pubDate>Tue, 10 Oct 2017 09:57:24 +0800</pubDate><description><![CDATA[先看异常堆栈,以前一直没有去看namenode的日志 今天已查看 发现一个错误 看错误的信息应该是无法解析到207机器
2017-10-10 09:45:45,124 WARN  blockmanagement.DatanodeManager (DatanodeManager.java:registerDatanode(882)) - Unresolved datanode registrati]]></description><category></category></item><item><title><![CDATA[hive日常报错之 serious problem serious problem,block 丢失]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78179538</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78179538</guid><author>qqpy789</author><pubDate>Mon, 09 Oct 2017 09:45:33 +0800</pubDate><description><![CDATA[https://community.hortonworks.com/questions/71589/after-upgrading-to-hdp-253-using-beeline-with-a-si.html

报错如下 参考链接如上 感觉是 orc的文件优化导致的这个错误 解决办法是
set hive.exec.orc.split.strategy=BI;

加上这么一个参数即可
]]></description><category></category></item><item><title><![CDATA[hadoop之二次排序]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78138901</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78138901</guid><author>qqpy789</author><pubDate>Sat, 30 Sep 2017 08:51:38 +0800</pubDate><description><![CDATA[package com.hit.ee;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.]]></description><category></category></item><item><title><![CDATA[mapreduce编程模型之WritableComparator]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78132585</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78132585</guid><author>qqpy789</author><pubDate>Fri, 29 Sep 2017 12:02:31 +0800</pubDate><description><![CDATA[WritableComparator是一个类 这个类是用于mapreduce编程模型中的比较 排序 
mapreduce中有两次排序 一次是 在环形缓冲区域之中进行分区 排序
还有一次是数据在reduce端获取文件之后进行分组
现在我讲的是后面那个
 //Define the comparator that controls which keys are grouped together ]]></description><category></category></item><item><title><![CDATA[mapreduce编程模型之partitioner]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78132507</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78132507</guid><author>qqpy789</author><pubDate>Fri, 29 Sep 2017 11:53:29 +0800</pubDate><description><![CDATA[Partitioner是一个mapreduce的顶级抽象类 用于决定map输出的kv键值对 如何按照job.setnumberruce(n)中设置的n个数进行分区 每个kv落在哪个分区之中 以及如何保证相同的key落在同一个分区中 只有相同的key落在同一个分区中 才能保证 这个相同的key可以被同一个reducer处理
上代码
 public static class MyPartition]]></description><category></category></item><item><title><![CDATA[jvm参数的配置]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78027427</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78027427</guid><author>qqpy789</author><pubDate>Tue, 19 Sep 2017 10:47:37 +0800</pubDate><description><![CDATA[1.首先获取活跃占用空间的老年代大小 和活跃占用空间的永久代大小.怎么获取呢?
(1)首先不用设置参数 然后jvm经过几次jvm之后 通过jvisualVm中的visual GC插件就能看到 或者是通过 gc的日志也能看到活跃占用空间的老年代和永久代的大小
2.然后就直接做设置
java堆 -Xms和-Xmx
3-4倍活跃老年代空间占用量
永久代  -xx:permsize
1.2~1]]></description><category></category></item><item><title><![CDATA[java如何定位锁]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/78027347</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/78027347</guid><author>qqpy789</author><pubDate>Tue, 19 Sep 2017 10:41:50 +0800</pubDate><description><![CDATA[首先使用jconsole连接上应用



点击死锁 然后会出现两个线程





也就是这两个线程出现了死锁
然后再使用jstack进行分析具体是哪一行代码出现死锁


jstack -l 18404







这个就写的比较明白了 线程1 卡在 DeadLock 43行代码 而 线程 2卡在DeadLock 56行代码
最后附上我测试使用的jav]]></description><category></category></item><item><title><![CDATA[hotspot vm调优 资料]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/77896765</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/77896765</guid><author>qqpy789</author><pubDate>Fri, 08 Sep 2017 17:06:12 +0800</pubDate><description><![CDATA[简书 中的垃圾回收器比较
http://www.jianshu.com/p/50d5c88b272d
 
oracle中关于参数的配置
http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html]]></description><category></category></item><item><title><![CDATA[kylin count distinct需要注意的东西]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/77480047</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/77480047</guid><author>qqpy789</author><pubDate>Tue, 22 Aug 2017 14:44:39 +0800</pubDate><description><![CDATA[详情请看这个链接
http://lxw1234.com/archives/2016/08/714.htm
也可以看官网给的
http://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/]]></description><category></category></item><item><title><![CDATA[kylin raw度量 需要注意的东西]]></title><link>https://blog.youkuaiyun.com/qqpy789/article/details/77480029</link><guid>https://blog.youkuaiyun.com/qqpy789/article/details/77480029</guid><author>qqpy789</author><pubDate>Tue, 22 Aug 2017 14:43:13 +0800</pubDate><description><![CDATA[大约就是 
1.避免数据倾斜
2.设置参数
3.将大的segment变成小的segment进行操作
4.raw度量 在cuboid的大小不能超过1M 不然就bufferoverflowexeption 说是说以后会进行优化,然并卵
5.raw度量不支持where条件查询


链接:http://kylin.apache.org/blog/2016/05/29/raw-measure]]></description><category></category></item></channel></rss>