<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[iture的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/iture</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; iture]]></copyright><item><title><![CDATA[idea scala 没有错误提示]]></title><link>https://blog.youkuaiyun.com/iture/article/details/120933346</link><guid>https://blog.youkuaiyun.com/iture/article/details/120933346</guid><author>iture</author><pubDate>Sun, 24 Oct 2021 18:42:47 +0800</pubDate><description><![CDATA[本人idea版本2021.2.2
发现 Highlight node with errors 默认没有勾选上，勾上就可以了。


]]></description><category></category></item><item><title><![CDATA[HDFS fsimage文件解析]]></title><link>https://blog.youkuaiyun.com/iture/article/details/112075344</link><guid>https://blog.youkuaiyun.com/iture/article/details/112075344</guid><author>iture</author><pubDate>Fri, 01 Jan 2021 22:29:22 +0800</pubDate><description><![CDATA[指定hdfs 客户端内存
export HADOOP_CLIENT_OPTS=’-Xmx5000m -Xms5000m’
解析fsimage文件，结果以逗号分隔
nohup hdfs  oiv  -t /data/tmp/fsimage   -p Delimited -delimiter , -i fsimage_000000000xxx -o result_file &amp; 

30gb大的fsimage文件解析大概耗时2小时。
结果文件需要删除首行列名
sed -i ‘1d’ result_file]]></description><category></category></item><item><title><![CDATA[python loggin模块使用]]></title><link>https://blog.youkuaiyun.com/iture/article/details/109380796</link><guid>https://blog.youkuaiyun.com/iture/article/details/109380796</guid><author>iture</author><pubDate>Fri, 30 Oct 2020 10:46:18 +0800</pubDate><description><![CDATA[import logging
import os
if __name__ == '__main__':
    log_path = "m:/"
    log_name = "python.log"
    logger = logging.getLogger()
    handler = logging.FileHandler(os.path.join(log_path, log_name))
    formater = logging.Formatter("[%(asctime)s] [%(fun]]></description><category></category></item><item><title><![CDATA[python http请求]]></title><link>https://blog.youkuaiyun.com/iture/article/details/109318018</link><guid>https://blog.youkuaiyun.com/iture/article/details/109318018</guid><author>iture</author><pubDate>Tue, 27 Oct 2020 18:45:29 +0800</pubDate><description><![CDATA[# -*- coding: utf-8 -*-
import urllib2,urllib
import requests
import json

def get(url):
    req = urllib2.Request(url)
    res_data = urllib2.urlopen(req)
    res = res_data.read()
    return res

#data 为字典对象
def post(url, data):
    req = urllib2.Request]]></description><category></category></item><item><title><![CDATA[CDH添加外部HDFS集群nameservice]]></title><link>https://blog.youkuaiyun.com/iture/article/details/109029105</link><guid>https://blog.youkuaiyun.com/iture/article/details/109029105</guid><author>iture</author><pubDate>Mon, 12 Oct 2020 15:46:17 +0800</pubDate><description><![CDATA[如CDH自带HDFS的nameservice为nameservice-dev，现在添加对外部HDFS集群nameservice-test的支持。
在配置项”hdfs-site.xml 的 HDFS 客户端高级配置代码段（安全阀）“中添加配置
&lt;property&gt;
    &lt;name&gt;dfs.nameservices&lt;/name&gt;
    &lt;value&gt;nameservice-dev,nameservice-test&lt;/value&gt;
&lt;/pro]]></description><category></category></item><item><title><![CDATA[HBase java get scan 例子]]></title><link>https://blog.youkuaiyun.com/iture/article/details/102709317</link><guid>https://blog.youkuaiyun.com/iture/article/details/102709317</guid><author>iture</author><pubDate>Wed, 23 Oct 2019 19:34:54 +0800</pubDate><description><![CDATA[maven依赖
&lt;dependency&gt;
    &lt;groupId&gt;org.apache.hbase&lt;/groupId&gt;
    &lt;artifactId&gt;hbase-client&lt;/artifactId&gt;
    &lt;version&gt;0.98.24-hadoop2&lt;/version&gt;
&lt;/dependency&...]]></description><category></category></item><item><title><![CDATA[spark 读取orc文件]]></title><link>https://blog.youkuaiyun.com/iture/article/details/82746900</link><guid>https://blog.youkuaiyun.com/iture/article/details/82746900</guid><author>iture</author><pubDate>Mon, 17 Sep 2018 18:23:07 +0800</pubDate><description><![CDATA[&amp;lt;dependency&amp;gt;
            &amp;lt;groupId&amp;gt;org.apache.orc&amp;lt;/groupId&amp;gt;
            &amp;lt;artifactId&amp;gt;orc-mapreduce&amp;lt;/artifactId&amp;gt;
            &amp;lt;version&amp;gt;1.1.0&amp;lt;/version&amp;gt;
   ]]></description><category></category></item><item><title><![CDATA[开源调度系统hello job]]></title><link>https://blog.youkuaiyun.com/iture/article/details/80377807</link><guid>https://blog.youkuaiyun.com/iture/article/details/80377807</guid><author>iture</author><pubDate>Sat, 19 May 2018 20:55:31 +0800</pubDate><description><![CDATA[Hello Job调度系统

系统简介：


本系统使用java开发，支持时间调度、作业依赖触发、手工执行三种调度方式。

时间调度：底层基于quartz实现，支持cron命令，实现灵活的时间调度方式。
作业依赖触发：一个子作业可以依赖多个父作业，一个父作业可以有多个子作业，系统同时做死循环判断，避免作业依赖形成环。
手工执行：对任何作业都可以手动触发一次。
系统自身并不承担业务逻辑，通过ssh ...]]></description><category></category></item><item><title><![CDATA[cloudera manager 离线安装cdh hadoop]]></title><link>https://blog.youkuaiyun.com/iture/article/details/54598258</link><guid>https://blog.youkuaiyun.com/iture/article/details/54598258</guid><author>iture</author><pubDate>Wed, 18 Jan 2017 08:28:37 +0800</pubDate><description><![CDATA[软件环境:cm 5.6,cdh 5.6 
cm 离线安装包下载 
http://archive-primary.cloudera.com/cm5/repo-as-tarball/5.9.0/cdh 离线安装包下载 
http://archive-primary.cloudera.com/cdh5/parcels/kakfa 离线安装包下载 
http://archive.cloudera.com/k]]></description><category></category></item><item><title><![CDATA[spark streaming 通过zookeeper读取kafka上的数据]]></title><link>https://blog.youkuaiyun.com/iture/article/details/53583324</link><guid>https://blog.youkuaiyun.com/iture/article/details/53583324</guid><author>iture</author><pubDate>Mon, 12 Dec 2016 15:29:21 +0800</pubDate><description><![CDATA[maven 依赖如下 <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.0.2</version>
        </dependency>
        <dependen]]></description><category></category></item><item><title><![CDATA[eclipse 导入play2.5 项目]]></title><link>https://blog.youkuaiyun.com/iture/article/details/53388667</link><guid>https://blog.youkuaiyun.com/iture/article/details/53388667</guid><author>iture</author><pubDate>Tue, 29 Nov 2016 08:45:52 +0800</pubDate><description><![CDATA[安装sbt

下载安装sbt，并且把sbt添加到环境变量 
 http://www.scala-sbt.org/

创建play工程

(建议安装activator，使用activator创建新工程，而不是用sbt，点击下载activator1.3) 
启动一个终端， 在你所处的目录 创建play scala项目

sbt new playframework/play-scala-seed.g8]]></description><category></category></item><item><title><![CDATA[spark mllib 关联规则]]></title><link>https://blog.youkuaiyun.com/iture/article/details/53377162</link><guid>https://blog.youkuaiyun.com/iture/article/details/53377162</guid><author>iture</author><pubDate>Mon, 28 Nov 2016 14:27:51 +0800</pubDate><description><![CDATA[机器学习]]></description><category></category></item></channel></rss>