<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[prometheus的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/illbehere</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; illbehere]]></copyright><item><title><![CDATA[maven 总结]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/106436901</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/106436901</guid><author>illbehere</author><pubDate>Sat, 30 May 2020 09:13:50 +0800</pubDate><description><![CDATA[测试
测试例必需以Test开头或结尾
run pom.xml
eclipse:maven build clean test  / mvn clean test
自动化测试
mvn clean
mvn clean compile 
mvn clean test
mvn clean package
mvn clean install

下面的命令包含上面的命令

maven-compiler-plugin：

&lt;plugin&gt;
    &lt;groupId&gt;org.apache.maven.]]></description><category></category></item><item><title><![CDATA[hive on spark 同时只能提交一个任务错误处理]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/105657458</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/105657458</guid><author>illbehere</author><pubDate>Tue, 21 Apr 2020 14:40:11 +0800</pubDate><description><![CDATA[同时提交几个任务时报
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 1e458098-9ef7-4709-a117-c1b0b6ea0eee_0: java....]]></description><category></category></item><item><title><![CDATA[canal kafka camus整合]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/105371837</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/105371837</guid><author>illbehere</author><pubDate>Tue, 07 Apr 2020 19:50:20 +0800</pubDate><description><![CDATA[一、camus配置
camus.job.name=Camus Job

#hdfs存放路径
etl.destination.path=/user/hive/warehouse/binlog.db
#offsets, error logs, and count files存放路径
etl.execution.base.path=/camus/exec
#完成的jobs的输出路径
etl.execut...]]></description><category></category></item><item><title><![CDATA[spark 笛卡尔积优化]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/89090355</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/89090355</guid><author>illbehere</author><pubDate>Mon, 08 Apr 2019 15:12:21 +0800</pubDate><description><![CDATA[因业务需求，需对两份数据进行关联，然后进行计算，然后想到笛卡尔积。在最开始用spark对他进行处理的时候，他总是卡死在一个地方跑不出数据。需对其进行优化。
1. 任务代码
data1 = hc.sql(" select a.*,b.*  from tableA a ,tableB b").rdd.map(list)
   

2. 优化方式1
对数据进行分区rdd.repartition(50)，...]]></description><category></category></item><item><title><![CDATA[elasticsearch多磁盘扩容]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/78202973</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/78202973</guid><author>illbehere</author><pubDate>Wed, 11 Oct 2017 14:04:02 +0800</pubDate><description><![CDATA[1、问题由于早前elasticsearch集群数据存储路径只配置了一个，所以某天磁盘突然爆满，集群差点当机。需重新配置多路径存储路径，因为在生产环境，得保证集群不死掉，只能一台一台配置重启。2、修改配置文件修改elasticsearch.yml中path.data属性，添加多路径以逗号分隔path.data : /opt/data1,/opt/data23、查看集群状态curl -XGET "htt]]></description><category></category></item><item><title><![CDATA[正规方程 Normal Equation]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/77962347</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/77962347</guid><author>illbehere</author><pubDate>Wed, 13 Sep 2017 10:03:46 +0800</pubDate><description><![CDATA[转自http://blog.youkuaiyun.com/u012328159/article/details/51036272 
前几篇博客介绍了一些梯度下降的实用技巧，特征缩放（详见http://blog.youkuaiyun.com/u012328159/article/details/51030366）和学习率（详见http://blog.youkuaiyun.com/u012328159/article/details/51]]></description><category></category></item><item><title><![CDATA[Elasticsearch之elasticsearch5.x 新特性]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/77932360</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/77932360</guid><author>illbehere</author><pubDate>Mon, 11 Sep 2017 13:51:23 +0800</pubDate><description><![CDATA[转自:http://www.cnblogs.com/zlslch/p/6619089.html1、首先看看跟性能有关的1.1 aggregation 的改进也是非常大， Instant Aggregations。 
 Elasticsearch已经在Shard层面提供了Aggregation缓存，如果你的数据没有变化，ES能够直接返回上次的缓存结果，但是有一个场景比较特殊，就是 date histo]]></description><category></category></item><item><title><![CDATA[使用spark mllib 随机森林算法对文本进行多分类]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/77162273</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/77162273</guid><author>illbehere</author><pubDate>Mon, 14 Aug 2017 16:44:45 +0800</pubDate><description><![CDATA[1、数据准备我使用的数据是公司人工标注文本数据，样本如下：1#k-v#*亮亮爱宠*波波宠物指甲钳指甲剪附送锉刀适用小型犬及猫特价
1#k-v#*顺丰包邮*宠物药品圣马利诺PowerIgG免疫力球蛋白犬猫细小病毒
1#k-v#*包邮*法国罗斯蔓草本精华宠物浴液薰衣草护色润泽香波拍套餐
1#k-v#*包邮*家朵102宠物沐浴液
1#k-v#*包邮*家朵102宠物沐浴液猫2、分词使用ansj包对文本数据去]]></description><category></category></item><item><title><![CDATA[Elasticsearch线程池配置]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/75356099</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/75356099</guid><author>illbehere</author><pubDate>Wed, 19 Jul 2017 11:42:27 +0800</pubDate><description><![CDATA[1、查看节点状态GET _nodes/stats2、查看集群配置信息GET _cluster/settings3、线程池的三种类型cached  #无限制的线程池，为每个请求创建一个线程。这种线程池是为了防止请求被阻塞或者拒绝，其中的每个线程都有一个超时时间(keep_alive)，默认5分钟，一旦超时就会回收/终止。
fixed  #有着固定大小的线程池，大小由size属性指定，默认是5*core]]></description><category></category></item><item><title><![CDATA[SVM多类划分问题 one vs rest 和 one vs one理解]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/74940506</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/74940506</guid><author>illbehere</author><pubDate>Tue, 11 Jul 2017 09:48:10 +0800</pubDate><description><![CDATA[一对多法（one-versus-rest,简称OVR SVMs）训练时依次把某个类别的样本归为一类,其他剩余的样本归为另一类，这样k个类别的样本就构造出了k个SVM。分类时将未知样本分类为具有最大分类函数值的那类。 
假如我有四类要划分（也就是4个Label），他们是A、B、C、D。于是我在抽取训练集的时候，分别抽取A所对应的向量作为正集，B,C,D所对应的向量作为负集；B所对应的向量作为正集，A,]]></description><category></category></item><item><title><![CDATA[ElasticSearch 常用的查询过滤语句]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/73739517</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/73739517</guid><author>illbehere</author><pubDate>Mon, 26 Jun 2017 14:10:17 +0800</pubDate><description><![CDATA[query 和  filter 的区别请看： http://www.cnblogs.com/ghj1976/p/5292740.html Filter DSLterm 过滤term主要用于精确匹配哪些值，比如数字，日期，布尔值或 not_analyzed 的字符串(未经分析的文本数据类型)： { “term”: { “age”:    26           }}  
{ “term”: { “d]]></description><category></category></item><item><title><![CDATA[mongodb使用]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/72910705</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/72910705</guid><author>illbehere</author><pubDate>Thu, 08 Jun 2017 10:17:44 +0800</pubDate><description><![CDATA[1、nest array querydb.inventory.find( { 'instock.0.qty': { $lte: 20 } } )2、正则查询{"$and":[{"day":{$gt:"2017-06-05"}},{"result.simpleSubject":{"$regex":"床头灯"}}]}3、查询连接超时你在用 db.collection.find() 的时候，它返回的不是所]]></description><category></category></item><item><title><![CDATA[httpclient的几种post参数格式]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/72851045</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/72851045</guid><author>illbehere</author><pubDate>Sat, 03 Jun 2017 11:26:35 +0800</pubDate><description><![CDATA[1、json格式HttpClientContext context = HttpClientContext.create();                httpPost.setURI(java.net.URI.create(url));                if(null != headers) {
                    for (String name : hea]]></description><category></category></item><item><title><![CDATA[Linux下的SVN服务器搭建]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/72279712</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/72279712</guid><author>illbehere</author><pubDate>Tue, 16 May 2017 14:20:41 +0800</pubDate><description><![CDATA[转自http://www.cnblogs.com/mymelon/p/5483215.html 
鉴于在搭建时，参考网上很多资料，网上资料在有用的同时，也坑了很多人本文的目的，也就是想让后继之人在搭建svn服务器时不再犯错，不再被网上漫天的坑爹作品所坑害，故此总结/*开始****/系统环境：Centos 6.5第一步：通过yum命令安装svnserve，命令如下：yum -y install sub]]></description><category></category></item><item><title><![CDATA[（转）Redis集群方案应该怎么做？]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/72185566</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/72185566</guid><author>illbehere</author><pubDate>Mon, 15 May 2017 17:40:12 +0800</pubDate><description><![CDATA[转自https://www.zhihu.com/question/21419897 
本来只是跟一位高人请教知识，但是看到这个问题总是忍不住想把学到的拿过来用。所以就简简单单的列上几种方案吧。那么问题来了。(这次看着不像博客了吧- -！) 为什么集群？通常，为了提高网站响应速度，总是把热点数据保存在内存中而不是直接从后端数据库中读取。Redis是一个很好的Cache工具。大型网站应用，热点数据量往往]]></description><category></category></item><item><title><![CDATA[mysql数据通过fluent同步到阿里云datahub填坑过程]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/63254486</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/63254486</guid><author>illbehere</author><pubDate>Sat, 18 Mar 2017 14:59:57 +0800</pubDate><description><![CDATA[1、ruby安装#wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.1.tar.gz
#tar zxvf ruby-2.1.1.tar.gz
#cd ruby-2.1.1
#./configure --prefix=/root/bi/ruby --with-opessl-dir=/root/bi/openssl-1.0.0l
#make &&]]></description><category></category></item><item><title><![CDATA[android apk反编译详解]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/62038488</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/62038488</guid><author>illbehere</author><pubDate>Tue, 14 Mar 2017 10:49:48 +0800</pubDate><description><![CDATA[使用工具： apktool 
 dex2jar
 jd-gui  工具介绍：apktool   作用：资源文件获取，可以提取出图片文件和布局文件进行使用查看
dex2jar 作用：将apk反编译成Java源码（classes.dex转化成jar文件）
jd-gui 作用：查看APK中classes.dex转化成出的jar文件，即源码文件
反编译流程：一、apk反编译得到程序的源代码、图片、XML配置]]></description><category></category></item><item><title><![CDATA[sparkmllib算法实例]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/54378352</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/54378352</guid><author>illbehere</author><pubDate>Thu, 12 Jan 2017 15:22:00 +0800</pubDate><description><![CDATA[逻辑回归import org.apache.spark.ml.classification.LogisticRegression// Load training data
val training = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")val lr = new LogisticRegression]]></description><category></category></item><item><title><![CDATA[sparkmllib 特征抽取、特征转换及特征选择]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/54095304</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/54095304</guid><author>illbehere</author><pubDate>Thu, 05 Jan 2017 17:08:58 +0800</pubDate><description><![CDATA[特征抽取
TF-IDF
TF-IDF一般应用于文本挖掘中，用来反映一个特征项的重要性。设特征项为 t,文档为d,文档集为D。特征频率（ term frequency）TF(t,d) 为特征项在文档d中出现在次数。 文档频率（document frequency）DF(t,D)表示含特征项t的文档数。如果只是用tf来衡量重要性，那么对于一遍文档中出现多次但含信息量极少来说是没什么用处的。因此可以用逆文]]></description><category></category></item><item><title><![CDATA[sparkmllib数据类型]]></title><link>https://blog.youkuaiyun.com/illbehere/article/details/54094208</link><guid>https://blog.youkuaiyun.com/illbehere/article/details/54094208</guid><author>illbehere</author><pubDate>Thu, 05 Jan 2017 16:06:55 +0800</pubDate><description><![CDATA[局部向量有三种方式创建局部向量import org.apache.spark.mllib.linalg.{Vector, Vectors}// Create a dense vector (1.0, 0.0, 3.0).
val dv: Vector = Vectors.dense(1.0, 0.0, 3.0)
// Create a sparse vector (1.0, 0.0, 3.0) by]]></description><category></category></item></channel></rss>