<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[lijun05的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/lijun05</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; lijun05]]></copyright><item><title><![CDATA[spark的python版本更新]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/132913886</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/132913886</guid><author>lijun05</author><pubDate>Sat, 16 Sep 2023 01:19:32 +0800</pubDate><description><![CDATA[解决python版本更新]]></description><category></category></item><item><title><![CDATA[09：RDD编程 - 如何查看应用程序信息]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/127044949</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/127044949</guid><author>lijun05</author><pubDate>Sun, 25 Sep 2022 23:26:09 +0800</pubDate><description><![CDATA[RDD编程 - 如何查看应用程序信息]]></description><category></category></item><item><title><![CDATA[08：采用Hadoop YARN管理器运行Spark应用程序]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126910117</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126910117</guid><author>lijun05</author><pubDate>Sat, 17 Sep 2022 23:31:44 +0800</pubDate><description><![CDATA[采用Hadoop YARN管理器运行Spark应用程序]]></description><category></category></item><item><title><![CDATA[07：在独立集群管理器上运行Spark应用程序]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126903099</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126903099</guid><author>lijun05</author><pubDate>Sat, 17 Sep 2022 14:39:49 +0800</pubDate><description><![CDATA[在独立集群管理器上运行Spark应用程序]]></description><category></category></item><item><title><![CDATA[06: Spark集群环境搭建]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126899030</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126899030</guid><author>lijun05</author><pubDate>Sat, 17 Sep 2022 00:45:42 +0800</pubDate><description><![CDATA[Spark集群配置]]></description><category></category></item><item><title><![CDATA[05: maven编译打包Scala程序]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126813720</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126813720</guid><author>lijun05</author><pubDate>Mon, 12 Sep 2022 13:20:24 +0800</pubDate><description><![CDATA[maven编译打包Scala程序]]></description><category></category></item><item><title><![CDATA[04：sbt编译打包Spark应用程序代码]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126811388</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126811388</guid><author>lijun05</author><pubDate>Mon, 12 Sep 2022 01:46:56 +0800</pubDate><description><![CDATA[sbt编译打包Spark程序]]></description><category></category></item><item><title><![CDATA[03：安装编译打包工具sbt和maven]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126810859</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126810859</guid><author>lijun05</author><pubDate>Sun, 11 Sep 2022 23:54:36 +0800</pubDate><description><![CDATA[安装spt和maven]]></description><category></category></item><item><title><![CDATA[02：启动spark-shell]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126807954</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126807954</guid><author>lijun05</author><pubDate>Sun, 11 Sep 2022 18:54:40 +0800</pubDate><description><![CDATA[启动spark-shell]]></description><category></category></item><item><title><![CDATA[01: Spark的安装与配置]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/126806573</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/126806573</guid><author>lijun05</author><pubDate>Sun, 11 Sep 2022 18:36:34 +0800</pubDate><description><![CDATA[Spark安装配置]]></description><category></category></item><item><title><![CDATA[13. Hive和MySQL的安装]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/121738987</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/121738987</guid><author>lijun05</author><pubDate>Mon, 06 Dec 2021 03:32:23 +0800</pubDate><description><![CDATA[转载2019级崔相元同学（数据科学与大数据技术专业）的blog：
https://blog.youkuaiyun.com/qq_46013938/article/details/121512388
其中最后一步：
schematool -dbType mysql -initSchema

如果出现以下错误：

则需要将链接MySQL的驱动jar包拷贝到hive的lib文件夹下：
cp mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar  /us]]></description><category></category></item><item><title><![CDATA[12. HBase安装配置]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120759662</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120759662</guid><author>lijun05</author><pubDate>Thu, 14 Oct 2021 11:25:27 +0800</pubDate><description><![CDATA[首先确定Hadoop已安装配置成功：



解压安装包（前提是已经将安装包传到Downloads目录下）：



修改目录名：



hbase目录权限赋予给hadoop用户：



配置环境变量：
vim ~/.bashrc
在export PATH这行追加以下内容：
:/usr/local/hbase/bin
保存退出后，执行source命令使配置生效：
source ~/.bashrc


查看HBase版本：

返回如下新信息，说明安装成功：
注：HBase 2.2.2首次安装启动出现 Erro..]]></description><category></category></item><item><title><![CDATA[11. Hadoop完全分布式配置]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120499383</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120499383</guid><author>lijun05</author><pubDate>Sun, 26 Sep 2021 23:55:24 +0800</pubDate><description><![CDATA[由于之前已经做了伪分布式配置，在做完全分布式配置之前，需要首先删除master主机hadoop安装路径下的tmp和logs文件夹中的文件：

重新配置core-site.xml和hdfs-site.xml，以及mapred-site.xml和yarn-site.xml。
（1）首先编辑core-site.xml
（2）然后编辑hdfs-site.xml

（3）编辑mapred-site.xml

（4）编辑yarn-site.xml

修改slaves文件
压缩hadoop安装文件夹，然后分发到slav.]]></description><category></category></item><item><title><![CDATA[10. Hadoop伪分布式配置]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120287611</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120287611</guid><author>lijun05</author><pubDate>Tue, 14 Sep 2021 15:39:50 +0800</pubDate><description><![CDATA[在成功安装Hadoop之后，可以开始伪分布式配置。
Hadoop的配置文件位于/usr/local/hadoop/etc/hadoop/中，伪分布式需要修改2个配置文件core-site.xml和hdfs-site.xml。以hadoop用户登录master主机。

修改配置文件core-site.xml

查看要修改的配置文件：

修改配置文件core-site.xml：


修改hdfs-site.xml


格式化NameNode（注意只格式化一次即可，重复格式化会引起问题，稍后再讲）

启动Hado]]></description><category></category></item><item><title><![CDATA[9. SSH服务配置]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120269981</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120269981</guid><author>lijun05</author><pubDate>Mon, 13 Sep 2021 16:48:22 +0800</pubDate><description><![CDATA[在CentOS系统中，执行

rpm -qa | grep ssh

可以检查机器是否安装了SSH服务，使用
ps -e | grep sshd

指令查看SSH服务是否正常开启：
下面开始配置SSH服务：
2. 分别在slave01和slave02上删除.ssh文件夹：

3. 以hadoop账户登录master，新建SSH公钥私钥对：

4. 使用ssh-copy-id命令，复制本地用户的公钥到远程主机指定用户的认证库中，实现本地用户通过SSH免密登录远程主机指定用户：(出现（yes/no），输入ye.]]></description><category></category></item><item><title><![CDATA[5. Xshell和Xftp的安装过程]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120122494</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120122494</guid><author>lijun05</author><pubDate>Sun, 05 Sep 2021 22:12:15 +0800</pubDate><description><![CDATA[Xshell和Xftp的安装过程








Xftp的安装过程如下：









]]></description><category></category></item><item><title><![CDATA[1. 安装虚拟机软件VMware]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120122388</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120122388</guid><author>lijun05</author><pubDate>Sun, 05 Sep 2021 22:00:29 +0800</pubDate><description><![CDATA[只更改了盘符：







重启电脑后，首次启动VMware时需要…
启动之后的界面如下，下面可以安装虚拟机了。


]]></description><category></category></item><item><title><![CDATA[2. 安装CentOS虚拟机]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/120045945</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/120045945</guid><author>lijun05</author><pubDate>Wed, 01 Sep 2021 18:20:21 +0800</pubDate><description><![CDATA[选择CentOS的iso虚拟光驱文件（下载的文件在哪个文件夹就到哪里找iso文件）：

点击上图的“浏览”，找到iso文件：

定义Linux主机的名称为master，用户名为hadoop：

事先在硬盘建好VM文件夹下的master、slave01、Slave02、Slave03文件夹，分别存放四台Linux主机的虚拟机文件：







磁盘大小不能太小，否则后边实验可能做不下去：



开始安装CentOS，可能需要几分钟到十几分钟时间：

正常启动后，可以登录了：

这是登录之后的界面：

...]]></description><category></category></item><item><title><![CDATA[8. 安装Hadoop]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/117935732</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/117935732</guid><author>lijun05</author><pubDate>Wed, 16 Jun 2021 00:20:10 +0800</pubDate><description><![CDATA[启动master、salve01、slave02、slave03（或者只有一个slave）。
以hadoop用户登录master，切换到root用户。
进入hadoop安装包所在路径
cd /home/hadoop/Downloads
将hadoop安装包解压到/usr/local文件夹

tar -zxvf hadoop-3.1.3.tar.gz -C /usr/local/


配置Hadoop环境变量

vi etc/profile

打开文件后，把安装JDK时的环境变量设置部分改为：


环境变量.]]></description><category></category></item><item><title><![CDATA[7. 安装JDK，为Hadoop完全分布式部署做准备]]></title><link>https://blog.youkuaiyun.com/lijun05/article/details/117935283</link><guid>https://blog.youkuaiyun.com/lijun05/article/details/117935283</guid><author>lijun05</author><pubDate>Tue, 15 Jun 2021 23:40:07 +0800</pubDate><description><![CDATA[安装JDK]]></description><category></category></item></channel></rss>