<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[thyyyyyyy的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/thyyyyyyy</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; thyyyyyyy]]></copyright><item><title><![CDATA[Spark学习（四）之Spark运行流程]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84258781</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84258781</guid><author>thyyyyyyy</author><pubDate>Mon, 19 Nov 2018 19:39:48 +0800</pubDate><description><![CDATA[文章目录一、Spark中的基本概念二、Spark的运行流程
一、Spark中的基本概念
（1）Application：表示你的应用程序
（2）Driver：表示main()函数，创建SparkContext。由SparkContext负责与ClusterManager通信，进行资源的申请，任务的分配和监控等。程序执行完毕后关闭SparkContext
（3）Executor：某个Applicati...]]></description><category></category></item><item><title><![CDATA[Spark学习（三）之 RDD]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84138825</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84138825</guid><author>thyyyyyyy</author><pubDate>Fri, 16 Nov 2018 14:32:34 +0800</pubDate><description><![CDATA[文章目录1、什么是RDD
1、什么是RDD
RDD（Resilient Distributed Dataset）叫做弹性分布式数据集，是Spark中最基本的数据抽象，它代表一个不可变、可分区、里面的元素可并行计算的集合。RDD具有数据流模型的特点：自动容错、位置感知性调度和可伸缩性。RDD允许用户在执行多个查询时显式地将工作集缓存在内存中，后续的查询能够重用工作集，这极大地提升了查询速度。

...]]></description><category></category></item><item><title><![CDATA[Spark学习（二） 之集群搭建(standalone、HA-standalone、 spark on yarn)]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84109712</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84109712</guid><author>thyyyyyyy</author><pubDate>Thu, 15 Nov 2018 20:17:50 +0800</pubDate><description><![CDATA[Spark standalone
（一）Spark安装包下载
（二）安装过程
（三）启动
（四）验证
Spark standalone HA
Spark on YARN
Spark standalone
(一) Spark安装包下载
（1）官网下载
http://spark.apache.org/downloads.html

Spark standalone
(二) 安装过程
1、上传并解压缩
...]]></description><category></category></item><item><title><![CDATA[Spark学习（一）之Spark初识]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84108327</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/84108327</guid><author>thyyyyyyy</author><pubDate>Thu, 15 Nov 2018 17:48:43 +0800</pubDate><description><![CDATA[test

]]></description><category></category></item><item><title><![CDATA[Hive的三种搭建模式及远程方式详细搭建步骤]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83753275</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83753275</guid><author>thyyyyyyy</author><pubDate>Mon, 05 Nov 2018 18:53:07 +0800</pubDate><description><![CDATA[Hive的搭建hive三种方式区别和搭建按照Hive中metastore（元数据存储）不同位置分为三种方式： a)内嵌Derby方式 b)Local方式 c)Remote方式1，安装Hive	解压下载好的apache-hive-1.2.1-bin.tar.gz安装包到 /opt/apache-hive-1.2.1目录下	tar -zxvf apache-hive-1.2.1-bin.tar.gz ...]]></description><category></category></item><item><title><![CDATA[Hive基本原理]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83626445</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83626445</guid><author>thyyyyyyy</author><pubDate>Mon, 05 Nov 2018 17:27:10 +0800</pubDate><description><![CDATA[1. Hive简介
&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张数据库表，并提供完整的sql查询功能，可以将sql语句转换为MapReduce任务进行运行。其优点是学习成本低，可以通过类S]]></description><category></category></item><item><title><![CDATA[yarn集群搭建]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83624042</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83624042</guid><author>thyyyyyyy</author><pubDate>Thu, 01 Nov 2018 18:41:37 +0800</pubDate><description><![CDATA[yarn集群搭建步骤
规划
NN1NN2DNZKZKFCJNNRSNMnode01√√√√node02√√√√√√node03√√√√node04√√√
1.修改yarn-site.xml步骤
&amp;lt;property&amp;gt;
    &amp;lt;name&amp;gt;yarn.nodemanager.aux-services&amp;lt;/name&amp;gt;
    &amp;lt;value&amp;gt;mapreduce_...]]></description><category></category></item><item><title><![CDATA[Hadoop1.x版本和Hadoop2.x版本架构原理]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83151175</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83151175</guid><author>thyyyyyyy</author><pubDate>Thu, 18 Oct 2018 22:05:46 +0800</pubDate><description><![CDATA[MapReduce 1.x 架构

MapReduce 1.x 采用 Master/Slave 架构，由全局唯一的 Jobtracker 和多个 TaskTacker 组成，并且在Client中提供一系列的api供编程和管理使用。
1.client
提供api供用户编程调用，将用户编写的MapReduce程序提交到JobTracker中。
2. JobTracker

负责资源调度 主节点 发生故...]]></description><category></category></item><item><title><![CDATA[分布式MapReduce原理详解]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83097906</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83097906</guid><author>thyyyyyyy</author><pubDate>Tue, 16 Oct 2018 23:01:23 +0800</pubDate><description><![CDATA[MapReduce原理
一、什么是MapReduce？
&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;amp;nbsp;MapReduce是一个基于 java 的并行分布式计算框架，使用它来编写的数据处理应用]]></description><category></category></item><item><title><![CDATA[大数据相关知识点总结汇总]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83061852</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83061852</guid><author>thyyyyyyy</author><pubDate>Mon, 15 Oct 2018 20:05:23 +0800</pubDate><description><![CDATA[大数据课程
分布式存储HDFS
1. Hadoop的历史
作者dogcutting
Doug Cutting是Lucene、Nutch 、Hadoop等项目的发起人
1.1.Hadoop的三驾马车


GFS


MapReduce


BigTable


1.2.hadoop生态圈


hdfs


mapreduce


yarn


common


2.HDFS的存储原理
2.1 各个角...]]></description><category></category></item><item><title><![CDATA[HA-高可用HDFS集群搭建步骤]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83027237</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83027237</guid><author>thyyyyyyy</author><pubDate>Fri, 12 Oct 2018 14:57:54 +0800</pubDate><description><![CDATA[HA-高可用HDFS集群搭建步骤
首先。集群规划





NN-1
NN-2
DN
ZK
ZKFC
JNN




node01
✔



✔
✔


node02

✔
✔
✔
✔
✔


node03


✔
✔

✔


node04


✔
✔




1. 时间同步

各个节点安装ntp命令
yum install ntp
上网查找最新的时间服务器
ntp1.aliyun.com
同步...]]></description><category></category></item><item><title><![CDATA[hdfs分布式搭建出现的常见错误问题]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83004884</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/83004884</guid><author>thyyyyyyy</author><pubDate>Wed, 10 Oct 2018 23:07:39 +0800</pubDate><description><![CDATA[hdfs三种模式搭建步骤
hadoop集群启动但不能访问50070
分为两种情况
1.在虚拟机内linux系统中浏览器不能访问，

首先查看自己防火墙是否关闭

service iptables status


若未关防火墙

chkconfig iptables off



必须重启linux系统后检查防火墙状态


查看 vim  /etc/hosts文件，看是否修改成功或相对应，最后浏览...]]></description><category></category></item><item><title><![CDATA[HDFS伪分布式和完全分布式搭建步骤]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82991004</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82991004</guid><author>thyyyyyyy</author><pubDate>Wed, 10 Oct 2018 00:47:08 +0800</pubDate><description><![CDATA[HDFS的三种搭建模式
1.伪分布式搭建
搭建步骤

配置免密登录

[root@localhost ~]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
[root@localhost ~]#ssh-copy-id -i ~/.ssh/id_rsa.pub root@node01




配置jdk
linux安装jdk配置
修改hdfs-site.xml,c...]]></description><category></category></item><item><title><![CDATA[分布式存储(hdfs)思维导图]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82990914</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82990914</guid><author>thyyyyyyy</author><pubDate>Wed, 10 Oct 2018 00:20:40 +0800</pubDate><description><![CDATA[

]]></description><category></category></item><item><title><![CDATA[大数据基础知识及分布式存储原理]]></title><link>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82990891</link><guid>https://blog.youkuaiyun.com/thyyyyyyy/article/details/82990891</guid><author>thyyyyyyy</author><pubDate>Wed, 10 Oct 2018 00:14:30 +0800</pubDate><description><![CDATA[HDFS原理]]></description><category></category></item></channel></rss>