<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[md_2014的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/md_2014</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; md_2014]]></copyright><item><title><![CDATA[sparkSQL将一列拆分成多列]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/142386562</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/142386562</guid><author>md_2014</author><pubDate>Fri, 20 Sep 2024 16:06:57 +0800</pubDate><description><![CDATA[sparksql 一列拆分成多列
sparksql 一列变多列]]></description><category></category></item><item><title><![CDATA[sparkSQL报Schema for type Any is not supported异常问题解决办法]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/142355368</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/142355368</guid><author>md_2014</author><pubDate>Thu, 19 Sep 2024 11:50:42 +0800</pubDate><description><![CDATA[sparkSQL报Schema for type Any is not supported异常问题解决办法]]></description><category></category></item><item><title><![CDATA[数仓Hive的使用指南]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/141400174</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/141400174</guid><author>md_2014</author><pubDate>Wed, 21 Aug 2024 18:15:34 +0800</pubDate><description><![CDATA[在日常工作中，数据工程师常常需要将处理完毕的数据存到Hadoop的dfs上或者其他分布式数据库中，本文要介绍的是如何将数据输出到hive中，借此梳理下hive的入门使用指南。]]></description><category></category></item><item><title><![CDATA[sparkSQL创建仅带schema的空目录]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/140495869</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/140495869</guid><author>md_2014</author><pubDate>Wed, 17 Jul 2024 15:48:26 +0800</pubDate><description><![CDATA[sparkSQL创建零数据的schema的空目录]]></description><category></category></item><item><title><![CDATA[sparkSQL自定义schema的方法]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/129257705</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/129257705</guid><author>md_2014</author><pubDate>Tue, 28 Feb 2023 11:11:32 +0800</pubDate><description><![CDATA[sparkSQL自定义schema的方法]]></description><category></category></item><item><title><![CDATA[数据仓库简介]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/128799712</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/128799712</guid><author>md_2014</author><pubDate>Mon, 30 Jan 2023 11:38:52 +0800</pubDate><description><![CDATA[数据仓库]]></description><category></category></item><item><title><![CDATA[编写scala版hive的自定义函数]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/128647266</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/128647266</guid><author>md_2014</author><pubDate>Wed, 11 Jan 2023 17:46:47 +0800</pubDate><description><![CDATA[Scala版Hive用户自定义函数]]></description><category></category></item><item><title><![CDATA[JTS简介]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/127775301</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/127775301</guid><author>md_2014</author><pubDate>Wed, 09 Nov 2022 18:48:32 +0800</pubDate><description><![CDATA[GIS开源架构JTS简介]]></description><category></category></item><item><title><![CDATA[spark的sortShuffleManager解析]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/120371281</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/120371281</guid><author>md_2014</author><pubDate>Sat, 18 Sep 2021 17:28:10 +0800</pubDate><description><![CDATA[标题sortShuffleManager
一、注册ShuffleHandle的策略
首先，在shuffle过程中满足以下条件，选择BypassMergeSortShuffleHandle：
1）map端没有聚合操作
2）shuffle read partitions &lt;= spark.shuffle.sort.bypassMergeThreshold(阈值默认为200)
其次，满足以下条件，选择SerializedShuffleHandle：
1）序列化方式支持对象重新定位(意思是可以对已经序列化的对]]></description><category></category></item><item><title><![CDATA[spark中RDD转DS]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/119829484</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/119829484</guid><author>md_2014</author><pubDate>Fri, 20 Aug 2021 18:25:20 +0800</pubDate><description><![CDATA[情景1. RDD[Row] -&gt; DF
方法一：先定义类型为StructType的schema，按字段先后顺序映射到schema
spark.createDataFrame(rdd, schema)
方法二：先将其转换成case class类型的RDD，然后按情景2方式转换
情景2. RDD[_] -&gt; DF/DS
调用toDF/toDS函数转换

...]]></description><category></category></item><item><title><![CDATA[sparkSQL报Unable to find encoder for type stored in a Dataset异常问题解决办法]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/119808697</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/119808697</guid><author>md_2014</author><pubDate>Thu, 19 Aug 2021 18:48:06 +0800</pubDate><description><![CDATA[背景
sparkSQL中DataFrame在聚合后按规则在每组中选取一条记录，出现异常报错: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types]]></description><category></category></item><item><title><![CDATA[spark读取嵌套数据]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/112537971</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/112537971</guid><author>md_2014</author><pubDate>Tue, 12 Jan 2021 17:26:16 +0800</pubDate><description><![CDATA[spark读取嵌套数据
背景：最近，我遇到一种场景，需要从复杂的源数据（含有嵌套字段）中抽取部分嵌套字段，经过一番摸索，发现可以通过以下方式来抽取数据。
import org.apache.spark.sql.types._
val schema = new StructType()
.add("typeId", IntegerType)
.add("offsetId", LongType)
.add("data", ArrayType(
	new StructType()
	.add("f1", Long]]></description><category></category></item><item><title><![CDATA[RoaringBitmap运行机制解析]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/111568110</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/111568110</guid><author>md_2014</author><pubDate>Tue, 22 Dec 2020 22:25:07 +0800</pubDate><description><![CDATA[背景
用于将int类型转换成bitmap类型
public static RoaringBitmap bitmapOf(final int... dat) {
    final RoaringBitmap ans = new RoaringBitmap();
    ans.add(dat);
    return ans;
  }

运行机制

初始化bitmap
final RoaringBitmap ans = new RoaringBitmap();
在初始化过程中，无参构造器会新建一个Roari]]></description><category></category></item><item><title><![CDATA[刨析Spark的shuffleManager原理]]></title><link>https://blog.youkuaiyun.com/md_2014/article/details/111051268</link><guid>https://blog.youkuaiyun.com/md_2014/article/details/111051268</guid><author>md_2014</author><pubDate>Fri, 11 Dec 2020 18:29:37 +0800</pubDate><description><![CDATA[刨析Spark的shuffleManager原理

发展史
sortShuffleManager

shuffleWriter
数据写入缓存时，如果是聚合类的shuffle算子【比如reduceByKey】，那么会选用MAP数据结构，如果是非聚合类的shuffle算子【比如join】，那么会选用Array数据结构。
private[spark] class SortShuffleWriter[K, V, C](
    shuffleBlockResolver: IndexShuffleBlockResol]]></description><category></category></item></channel></rss>