<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[qq_41848129的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/qq_41848129</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qq_41848129]]></copyright><item><title><![CDATA[spark中groupByKey、reduceByKey与sortByKey]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/101380460</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/101380460</guid><author>qq_41848129</author><pubDate>Wed, 25 Sep 2019 19:07:19 +0800</pubDate><description><![CDATA[groupByKey把相同的key的数据分组到一个集合序列当中：

[("hello",1), ("world",1), ("hello",1), ("fly",1), ("hello",1), ("world",1)] --&gt; [("hello",(1,1,1)),("word",(1,1)),("fly",(1))]

reduceByKey把相同的key的数据聚合到一起并进行相应的计算...]]></description><category></category></item><item><title><![CDATA[关系型数据库与非关系型数据库的区别]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100521378</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100521378</guid><author>qq_41848129</author><pubDate>Tue, 03 Sep 2019 16:28:38 +0800</pubDate><description><![CDATA[nosql和关系型数据库比较？
优点：
1）成本：nosql数据库简单易部署，基本都是开源软件，不需要像使用oracle那样花费大量成本购买使用，相比关系型数据库价格便宜。
2）查询速度：nosql数据库将数据存储于内存之中，关系型数据库将数据存储在硬盘中，自然查询速度远不及nosql数据库。
3）存储数据的格式：nosql的存储格式是key,value形式、文档形式、图片形式等等，所以可以存储基...]]></description><category></category></item><item><title><![CDATA[SparkSQL-通用加载和保存]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100106665</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100106665</guid><author>qq_41848129</author><pubDate>Tue, 27 Aug 2019 21:42:47 +0800</pubDate><description><![CDATA[Spark SQL的DataFrame接口支持多种数据源的操作。一个DataFrame可以进行RDDs方式的操作，也可以被注册为临时表。把DataFrame 注册为临时表之后，就可以对该DataFrame执行SQL查询

Spark SQL的默认数据源为Parquet格式。数据源为Parquet文件时，Spark SQL可以方便的执行所有的操作。修改配置项spark.sql.sources.de...]]></description><category></category></item><item><title><![CDATA[spark SQL自定义UDAF]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100086483</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100086483</guid><author>qq_41848129</author><pubDate>Mon, 26 Aug 2019 22:30:17 +0800</pubDate><description><![CDATA[json文件：


{"name":"cc", "age":18}
{"name":"qiqi", "age":19}
{"name":"xiaohei", "age":17}

继承UserDefinedAggregateFunction


import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.Row
impo...]]></description><category></category></item><item><title><![CDATA[代码块开发SparkSQL程序]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100074452</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100074452</guid><author>qq_41848129</author><pubDate>Mon, 26 Aug 2019 13:16:39 +0800</pubDate><description><![CDATA[先创建一个spark session

导包(根据自己的版本来选择,我用的是spark2.4.3)：


&lt;dependency&gt;
    &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
    &lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
    &lt;version&gt;2...]]></description><category></category></item><item><title><![CDATA[spark-DataFrame,DataSet]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100067883</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100067883</guid><author>qq_41848129</author><pubDate>Sun, 25 Aug 2019 22:12:01 +0800</pubDate><description><![CDATA[首先先看一个rdd,DF,DS的转换图



DataFrame创建：


//读取json文件  
scala&gt; val df=spark.read.json("G:/ccData/person.json")
df: org.apache.spark.sql.DataFrame = [age: bigint, id: bigint ... 1 more field]

scala&g...]]></description><category></category></item><item><title><![CDATA[spark搭建阿里云集群]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100056154</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100056154</guid><author>qq_41848129</author><pubDate>Sat, 24 Aug 2019 19:10:45 +0800</pubDate><description><![CDATA[因为现阶段使用到了spark，所以就想搭建spark集群平台在阿里云上，网上找了很多资料，发现每一个人写的都不太一样，试过几次发现搭建不成功，于是自己慢慢琢磨，总结他们的经验，最后自己成功搭建，写下此篇博客，防止以后再次需要搭建时遇到问题，以下为搭建步骤。

hadoop、scala集群安装成功

下载安装

官网下载地址：http://spark.apache.org/downloads.htm...]]></description><category></category></item><item><title><![CDATA[spark-job逻辑图]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100050721</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100050721</guid><author>qq_41848129</author><pubDate>Sat, 24 Aug 2019 11:27:35 +0800</pubDate><description><![CDATA[Job逻辑执行图


典型的Job逻辑执行图如上所示,经过下面四个步骤可以得到最终执行结果:
1.从数据源(可以是本地file,内存数据结构, HDFS,HBase等)读取数据创建最初的RDD。
2.对RDD进行一系列的transformation()操作,每一个transformation()会产生一个或多个包含不同类型RDD。
3.对最后的finalRDD进行action()操作,每个...]]></description><category></category></item><item><title><![CDATA[简单的理解zookeeper的leader选举]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100047266</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100047266</guid><author>qq_41848129</author><pubDate>Fri, 23 Aug 2019 22:55:38 +0800</pubDate><description><![CDATA[Leader选举是保证分布式数据一致性的关键所在。Leader选举分为Zookeeper集群初始化启动时选举和Zookeeper集群运行期间Leader重新选举两种情况。在讲解Leader选举前先了解一下Zookeeper节点4种可能状态和事务ID概念。

1、Zookeeper节点状态
LOOKING：寻找Leader状态，处于该状态需要进入选举流程
LEADING：领导者状态，处于该状态的节点...]]></description><category></category></item><item><title><![CDATA[spark,代码实现flatMap报错:missing parameter type]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100038509</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100038509</guid><author>qq_41848129</author><pubDate>Fri, 23 Aug 2019 15:03:32 +0800</pubDate><description><![CDATA[这里rdd1的类型是None，

所以我们需要类型转换一下，默认是JAVARDD[String]类型


]]></description><category></category></item><item><title><![CDATA[实现kafka的producer和consumer]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100025640</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100025640</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 22:37:39 +0800</pubDate><description><![CDATA[先定义一个Person对象


package test;

public class Product {
	private int id;
	private String name;
	private double price;
	public Product() {
		super();
		// TODO Auto-generated constructor stub
	}
	public ...]]></description><category></category></item><item><title><![CDATA[自定义实现kafka分区]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100024695</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100024695</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 21:36:30 +0800</pubDate><description><![CDATA[首先写一个自定义分区算法


package Test
import java.util.Map;

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
//生命周期方法
public class HashPartition implements Partitio...]]></description><category></category></item><item><title><![CDATA[kafka入门]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100022097</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100022097</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 19:01:18 +0800</pubDate><description><![CDATA[搭建什么的都不说了，先来熟悉一下基本操作

创建一个消息 replica为3, partition为3
  1)启动zk（保证你自己的zookeeper已经启动了，这里我用的是kafka自带的zookeeper）
   bin/zookeeper-server-start.sh config/zookeeper.properties 1&gt;/dev/null 2&gt...]]></description><category></category></item><item><title><![CDATA[Class<SpringJUnit4ClassRunner> cannot be resolved to a type]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100021929</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100021929</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 18:33:01 +0800</pubDate><description><![CDATA[建spring项目的时候



解决办法:在pom.xml加依赖



这里根据自己的版本去修改就好了
]]></description><category></category></item><item><title><![CDATA["javax.servlet.http.HttpServlet" was not found]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100019937</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100019937</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 17:23:18 +0800</pubDate><description><![CDATA[右键项目，选择properties

选择java build path -&gt; add library -&gt; server runtime 选中tomcat 然后 applyandclose
]]></description><category></category></item><item><title><![CDATA[SparkSQL-基础查询]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100014169</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100014169</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 15:12:22 +0800</pubDate><description><![CDATA[josn文件内容


{"id":1, "name":"cc", "age":18}
{"id":2, "name":"qiqi", "age":19}
{"id":3, "name":"xiaohei", "age":17}

{"id":1, "name":"cc2", "age":15}
{"id":2, "name":"qiqi2", "age":20}
{"id":3, "name":"...]]></description><category></category></item><item><title><![CDATA[Spark Streaming基本原理]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100012870</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100012870</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 13:11:37 +0800</pubDate><description><![CDATA[SparkStreaming

　　Spark Streaming类似于Apache Storm，用于流式数据的处理。Spark Streaming有高吞吐量和容错能力强等特点。Spark Streaming支持的数据源有很多，例如：Kafka、Flume、Twitter、ZeroMQ和简单的TCP套接字等等。数据输入后可以用Spark的高度抽象操作如：map、reduce、join、window...]]></description><category></category></item><item><title><![CDATA[SparkSQL-入门]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/100012514</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/100012514</guid><author>qq_41848129</author><pubDate>Thu, 22 Aug 2019 12:57:01 +0800</pubDate><description><![CDATA[什么是SparkSQL

Spark SQL是Spark用来处理结构化数据的一个模块，它提供了两个编程抽象分别叫做DataFrame和DataSet，它们用于作为分布式SQL查询引擎。从下图可以查看RDD、DataFrames与DataSet的关系。



SparkSQL特点

1）引入了新的RDD类型SchemaRDD，可以像传统数据库定义表一样来定义SchemaRDD。

2...]]></description><category></category></item><item><title><![CDATA[spark核心RDD-combineByKey方法解析]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/99982372</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/99982372</guid><author>qq_41848129</author><pubDate>Wed, 21 Aug 2019 18:19:46 +0800</pubDate><description><![CDATA[首先 先来看看官方文档

combineByKey[C](createCombiner: (V) ⇒C,mergeValue: (C,V) ⇒C,mergeCombiners: (C,C) ⇒C,numPartitions:Int):RDD[(K,C)]

简单的解释下3个重要的函数参数：

createCombiner: V =&gt; C ，这个函数把当前的值作为参数，...]]></description><category></category></item><item><title><![CDATA[zookeeper原理分析]]></title><link>https://blog.youkuaiyun.com/qq_41848129/article/details/99900810</link><guid>https://blog.youkuaiyun.com/qq_41848129/article/details/99900810</guid><author>qq_41848129</author><pubDate>Wed, 21 Aug 2019 00:40:46 +0800</pubDate><description><![CDATA[zookeeper是什么

ZooKeeper是一个分布式的，开放源码的分布式应用程序协调服务，是Google的Chubby一个开源的实现，它是集群的管理者，监视着集群中各个节点的状态根据节点提交的反馈进行下一步合理操作。最终，将简单易用的接口和性能高效、功能稳定的系统提供给用户

zookeeper提供了什么

简单的说，zookeeper=文件系统+通知机制。

1、 文件系统...]]></description><category></category></item></channel></rss>