<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[Chenx]]></title><description><![CDATA[你来人间一趟  你要看看太阳。和你的心上人  一起走在街上。]]></description><link>https://blog.youkuaiyun.com/Anonymous_cx</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; Anonymous_cx]]></copyright><item><title><![CDATA[SparkSQL写数据到Hive的动态分区表]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/72667251</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/72667251</guid><author>Anonymous_cx</author><pubDate>Tue, 23 May 2017 21:06:01 +0800</pubDate><description><![CDATA[object HiveTableHelper extends Logging {  def hiveTableInit(sc: SparkContext): HiveContext = {
    val sqlContext = new HiveContext(sc)
    sqlContext
  }  def writePartitionTable(HCtx: HiveContext, in]]></description><category></category></item><item><title><![CDATA[Best Time to Buy and Sell Stock]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52608077</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52608077</guid><author>Anonymous_cx</author><pubDate>Wed, 21 Sep 2016 14:47:22 +0800</pubDate><description><![CDATA[Question:Say you have an array for which the ith element is the price of a given stock on day i.If you were only permitted to complete at most one transaction (ie, buy one and sell one share of the sto]]></description><category></category></item><item><title><![CDATA[Valid Anagram]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52562065</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52562065</guid><author>Anonymous_cx</author><pubDate>Sat, 17 Sep 2016 11:00:25 +0800</pubDate><description><![CDATA[Question:Given two strings s and t, write a function to determine if t is an anagram of s.For example, 
s = “anagram”, t = “nagaram”, return true. 
s = “rat”, t = “car”, return false.Note: 
You may ass]]></description><category></category></item><item><title><![CDATA[House Robber]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52558004</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52558004</guid><author>Anonymous_cx</author><pubDate>Fri, 16 Sep 2016 21:44:08 +0800</pubDate><description><![CDATA[Question:You are a professional robber planning to rob houses along a street. Each house has a certain amount of money stashed, the only constraint stopping you from robbing each of them is that adjace]]></description><category></category></item><item><title><![CDATA[Move Zeroes]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52528526</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52528526</guid><author>Anonymous_cx</author><pubDate>Tue, 13 Sep 2016 19:12:06 +0800</pubDate><description><![CDATA[Question：Given an array nums, write a function to move all 0’s to the end of it while maintaining the relative order of the non-zero elements.For example, given nums = [0, 1, 0, 3, 12], after calling y]]></description><category></category></item><item><title><![CDATA[Invert Binary Tree]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52522939</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52522939</guid><author>Anonymous_cx</author><pubDate>Tue, 13 Sep 2016 09:20:38 +0800</pubDate><description><![CDATA[Question:     4
   /   \
  2     7
 / \   / \
1   3 6   9to     4
   /   \
  7     2
 / \   / \
9   6 3   1code:class TreeNode { 
    int key; 
    String data; 
    TreeNode left; 
    TreeNode right;]]></description><category></category></item><item><title><![CDATA[Reverse Words in a String]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52503464</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52503464</guid><author>Anonymous_cx</author><pubDate>Sun, 11 Sep 2016 12:50:49 +0800</pubDate><description><![CDATA[Given an input string, reverse the string word by word.For example, 
Given s = "the sky is blue", 
return "blue is sky the".code：public class ReverseString {    public static void main(String[] args) {]]></description><category></category></item><item><title><![CDATA[Python 对文本先按词频统计，若相同按字典排序，后取TopN]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52495514</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52495514</guid><author>Anonymous_cx</author><pubDate>Sat, 10 Sep 2016 16:16:03 +0800</pubDate><description><![CDATA[Python Code:def count_words(s, n):
    dic = {}
    words = s.split(" ")
    for word in words:
        dic[word] = words.count(word)
    wordslist = sorted(dic.items(), key=lambda kv: (-kv[1], kv[0]))]]></description><category></category></item><item><title><![CDATA[Spark内存管理模型]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52215003</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/52215003</guid><author>Anonymous_cx</author><pubDate>Mon, 15 Aug 2016 21:37:55 +0800</pubDate><description><![CDATA[Spark是现在很流行的一个基于内存的分布式计算框架，既然是基于内存，那么自然而然的，内存的管理就是Spark存储管理的重中之重了。那么，Spark究竟采用什么样的内存管理模型呢？本文就为大家揭开Spark内存管理模型的神秘面纱。

        我们在《Spark源码分析之七：Task运行（一）》一文中曾经提到过，在Task被传递到Executor上去执行时，在为其分配的TaskRunne]]></description><category></category></item><item><title><![CDATA[用Flume采集多台机器上的多种日志并存储于HDFS]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51811086</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51811086</guid><author>Anonymous_cx</author><pubDate>Sat, 02 Jul 2016 14:57:29 +0800</pubDate><description><![CDATA[需求：把A、B 机器中的access.log、ugcheader.log、ugctail.log 汇总到C机器上然后统一收集到hdfs中。 
IP:   A:155 B:156 C：162 
但是在hdfs中要求的目录为： 
/source/access/20160101/** 
/source/ugcheader/20160101/** 
/source/ugctail/20160101/**结构]]></description><category></category></item><item><title><![CDATA[个性化推荐项目架构]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51771591</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51771591</guid><author>Anonymous_cx</author><pubDate>Mon, 27 Jun 2016 21:49:51 +0800</pubDate><description><![CDATA[离线与实时模块架构]]></description><category></category></item><item><title><![CDATA[Scala之隐式转换]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51589956</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51589956</guid><author>Anonymous_cx</author><pubDate>Sun, 05 Jun 2016 16:17:50 +0800</pubDate><description><![CDATA[大家都知道：1 to 10等价于1.to(10)。其实，Scala 中的Int 类型是没有to这个方法的，那为什么还能调用to方法？其实是编译器偷偷地帮我们完成Int->RichInt的转换，这个就是隐式转换。隐式转换可以丰富现有类库的功能。 
在REPL中键入：implicits -v查看编译器默认导入的隐式转换。笔者使用的Scala版本为2.11.8。 
 
第一个隐式转换的例子//从设计模式的]]></description><category></category></item><item><title><![CDATA[kafka和flume整合]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51456913</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51456913</guid><author>Anonymous_cx</author><pubDate>Thu, 19 May 2016 20:47:09 +0800</pubDate><description><![CDATA[Kafka作为source：配置文件：**#定义各个模块**
a1.sources = kafka 
a1.sinks = log
a1.channels = c1#配置kafka source
#source的类型为kafkaSource
a1.sources.kafka.type = org.apache.flume.source.kafka.KafkaSource
#消费者连接的zk集群地址]]></description><category></category></item><item><title><![CDATA[Scala之高阶函数]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51336878</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51336878</guid><author>Anonymous_cx</author><pubDate>Sat, 07 May 2016 11:09:31 +0800</pubDate><description><![CDATA[高阶函数：把函数传给函数第一个高阶函数：def formatResult(name:String,n:Int,f:Int=>Int)={
    val msg="The %s of %d is %d."
    msg.format(name,n,f(n))
}formatResult是一个高阶函数，他接受一个函数f为参数，参数的类型是Int=>Int，表示接受整型并返回一个整型结果。多态函数]]></description><category></category></item><item><title><![CDATA[Spark RDD API详解Map和Reduce]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51295634</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51295634</guid><author>Anonymous_cx</author><pubDate>Mon, 02 May 2016 11:25:06 +0800</pubDate><description><![CDATA[原始链接：https://www.zybuluo.com/jewes/note/35032  
RDD是什么？ 
RDD是Spark中的抽象数据结构类型，任何数据在Spark中都被表示为RDD。从编程的角度来看，RDD可以简单看成是一个数组。和普通数组的区别是，RDD中的数据是分区存储的，这样不同分区的数据就可以分布在不同的机器上，同时可以被并行处理。因此，Spark应用程序所做的无非是把需要处理的]]></description><category></category></item><item><title><![CDATA[Shell之网站日志的PV,UV计算]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51228945</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51228945</guid><author>Anonymous_cx</author><pubDate>Sat, 23 Apr 2016 22:59:46 +0800</pubDate><description><![CDATA[什么是PV,UV?PV:page view。即页面访问量 
UV:unique visitor。即独立访客量 
首先，more一下access.log这个文件，看一下文件的结构: 
 
我们只需要第一列的数据，所以： 
more clean.log计算PV:计算UV:首先，我们需要对IP地址进行去重，然后计算行数。 
统计访问次数在前十位的IP地址：]]></description><category></category></item><item><title><![CDATA[Strom优化]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51172278</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51172278</guid><author>Anonymous_cx</author><pubDate>Sun, 17 Apr 2016 11:09:40 +0800</pubDate><description><![CDATA[并行度worker为storm提供工作进程，程序的并行度可以设置（包括spout和bolt的并行度，如果有acker的话还包括acker的并行度），并行度即为task 数目。 
一般而言  worker和task之间的比例,即1个worker包含10~15个左右,当然根据配置和应用需要测试优化。worker(slots)CPU 16核，建议配置20个worker。CPU 24或32核，30个work]]></description><category></category></item><item><title><![CDATA[Storm集群启动与停止脚本及其注意事项]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51172225</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51172225</guid><author>Anonymous_cx</author><pubDate>Sun, 17 Apr 2016 11:02:39 +0800</pubDate><description><![CDATA[前提：集群之间首先要做免密码登陆 
集群中主节点启动：nimbus和ui进程 
从节点启动：supervisor和logviewer进程1：启动集群 
    需要先设置从节点的ip，保存到文件supervisorhost中，一行是一个节点的ip 
    192.168.1.171 
    192.168.1.172在主节点写一个脚本：start-all.sh
#!/bin/bash
    s]]></description><category></category></item><item><title><![CDATA[Storm入门程序——WordCount]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51171995</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51171995</guid><author>Anonymous_cx</author><pubDate>Sun, 17 Apr 2016 10:20:38 +0800</pubDate><description><![CDATA[spout：WordReader：package spout;import java.io.File;
import java.io.IOException;
import java.util.Collection;
import java.util.List;
import java.util.Map;import org.apache.commons.io.FileUtils;
import b]]></description><category></category></item><item><title><![CDATA[Storm入门]]></title><link>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51171874</link><guid>https://blog.youkuaiyun.com/Anonymous_cx/article/details/51171874</guid><author>Anonymous_cx</author><pubDate>Sun, 17 Apr 2016 09:58:31 +0800</pubDate><description><![CDATA[*1. 什么是StormStorm是Twitter开源的一个分布式的实时计算系统。2. Storm的设计思想 
 - Storm是对流Stream的抽象，流是一个不间断的无界的连续tuple，注意Storm在建模事件流时，把流中的事件抽象为tuple即元组。 
 - Storm将流中元素抽象为Tuple，一个tuple就是一个值列表value 
   list，list中的每个value都有一个na]]></description><category></category></item></channel></rss>