<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/imarch1</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; imarch1]]></copyright><item><title><![CDATA[Spark tungsten shuffle过程]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/51953296</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/51953296</guid><author>imarch1</author><pubDate>Tue, 19 Jul 2016 11:01:24 +0800</pubDate><description><![CDATA[1. ShuffleMapTask的runTask()方法

override def runTask(context: TaskContext): MapStatus = {
    // Deserialize the RDD using the broadcast variable.
    val deserializeStartTime = System.currentTimeMil]]></description><category></category></item><item><title><![CDATA[Spark tungsten shuffle]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/51943651</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/51943651</guid><author>imarch1</author><pubDate>Mon, 18 Jul 2016 22:03:47 +0800</pubDate><description><![CDATA[Tungsten的内存管理机制





内存管理必然涉及寻址，在介绍Tungsten的内存管理机制之前，我们先来看看内存地址的表示方式。 memory地址的表示方式在不同的内存管理模式中是不同的，这里我们考虑2种内存管理模式。第一种是off-heap模式，可以理解为是独立于JVM托管的heap之外利用c-style的malloc从os分配到的memory。这类memory不再由JVM]]></description><category></category></item><item><title><![CDATA[Spark shuffle write过程]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/51908889</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/51908889</guid><author>imarch1</author><pubDate>Thu, 14 Jul 2016 15:42:06 +0800</pubDate><description><![CDATA[1. ShuffleMapTask的runTask()方法
override def runTask(context: TaskContext): MapStatus = {
    // Deserialize the RDD using the broadcast variable.
    val deserializeStartTime = System.currentTimeMilli]]></description><category></category></item><item><title><![CDATA[Spark Shuffle Read过程]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/51890212</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/51890212</guid><author>imarch1</author><pubDate>Tue, 12 Jul 2016 17:29:37 +0800</pubDate><description><![CDATA[1. ShuffledRDD的compute()方法
  override def compute(split: Partition, context: TaskContext): Iterator[(K, C)] = {
    val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, C]]
    SparkEnv.g]]></description><category></category></item><item><title><![CDATA[字符与数字的相互转化]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/7043126</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/7043126</guid><author>imarch1</author><pubDate>Mon, 05 Dec 2011 17:21:55 +0800</pubDate><description><![CDATA[str(num); # 将数字转化位字符串


import string;
string.atoi(str); # 将字符串转化位整数
string.atof(str); # 将字符串转化位小数]]></description><category></category></item><item><title><![CDATA[操作系统的判断]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/7039669</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/7039669</guid><author>imarch1</author><pubDate>Sun, 04 Dec 2011 17:17:53 +0800</pubDate><description><![CDATA[import os;


if os.name == "nt": # windows
    blabla;
elif os.name == "posix": # linux
    blabla;]]></description><category></category></item><item><title><![CDATA[字典的遍历]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/7039663</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/7039663</guid><author>imarch1</author><pubDate>Sun, 04 Dec 2011 17:16:00 +0800</pubDate><description><![CDATA[for i in xrange(0, len(dict)):
    key = dict.keys()[i]; # key 是第i个索引]]></description><category></category></item><item><title><![CDATA[不同操作系统中的换行符]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/7039652</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/7039652</guid><author>imarch1</author><pubDate>Sun, 04 Dec 2011 17:13:41 +0800</pubDate><description><![CDATA[Windows:  \n
Linux:  \r\n
Unix:  \r]]></description><category></category></item><item><title><![CDATA[读取大文件]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/6753506</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/6753506</guid><author>imarch1</author><pubDate>Tue, 06 Sep 2011 15:08:24 +0800</pubDate><description><![CDATA[通常读取文件的方法是：
for line in open('myfile','r').readlines()   
但问题是会将整个文件读入内存，所以对于大文件此方法不适合。


下面的方法会一行一行地读取文件：

import fileinput
for lin]]></description><category></category></item><item><title><![CDATA[range 与 xrange 的区别]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/6745535</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/6745535</guid><author>imarch1</author><pubDate>Sat, 03 Sep 2011 15:04:04 +0800</pubDate><description><![CDATA[在
 Python 中，range 和 xrange 均可以用来做迭代的范围，不过 range 返回的是 list，而 xrange 则返回一个 xrange object。


for x in range(1000):
will
 generate a list]]></description><category></category></item><item><title><![CDATA[6 Lessons From Dropbox - One Million Files Saved Every 15 Minutes]]></title><link>https://blog.youkuaiyun.com/imarch1/article/details/6739000</link><guid>https://blog.youkuaiyun.com/imarch1/article/details/6739000</guid><author>imarch1</author><pubDate>Thu, 01 Sep 2011 12:51:47 +0800</pubDate><description><![CDATA[Dropbox saves one million files every 15 minutes,  more tweets than even Twitterers
 tweet. That mind blowing statistic was revealed by Ria]]></description><category></category></item></channel></rss>