<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[大数据学习笔记]]></title><description><![CDATA[大数据学习笔记]]></description><link>https://blog.youkuaiyun.com/qq_39160721</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qq_39160721]]></copyright><item><title><![CDATA[java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMet]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/136709393</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/136709393</guid><author>qq_39160721</author><pubDate>Thu, 14 Mar 2024 14:19:57 +0800</pubDate><description><![CDATA[在hive服务器运行以下命令。]]></description><category></category></item><item><title><![CDATA[记录一下云虚拟机lvm扩展的问题]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/129614011</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/129614011</guid><author>qq_39160721</author><pubDate>Fri, 17 Mar 2023 10:46:24 +0800</pubDate><description><![CDATA[关于云上，lvm的磁盘扩展]]></description><category></category></item><item><title><![CDATA[银河麒麟v10系统维护记录]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/128969626</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/128969626</guid><author>qq_39160721</author><pubDate>Fri, 10 Feb 2023 14:38:02 +0800</pubDate><description><![CDATA[占用较大，先停用 systemctl stop audit。1.aduit进程内存占用高问题。]]></description><category></category></item><item><title><![CDATA[kingbase 创建用户报错]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/127869819</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/127869819</guid><author>qq_39160721</author><pubDate>Thu, 01 Dec 2022 11:53:34 +0800</pubDate><description><![CDATA[1）CREATE ROLE/USER: 系统管理员用户在创建普通用户时不可指定密码等选项。最近一直在根据等保要求对kingbase做安全配置，刚接触kingbase，搞得头大。2）ALTER ROLE/USER: 只有sso和普通用户本身可以更改普通用户的密码。3）ALTER ROLE/USER: 系统管理员只能修改超级用户，或超级用户权限选项。百度无解，后发现测试服务器的可以正常加，然后就对比一下两台服务器的配置。发现sso_update_user这个插件比较可疑，去除配置就正常。]]></description><category></category></item><item><title><![CDATA[在ubuntu 20.04使用微信和qq等常用工具]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/124566785</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/124566785</guid><author>qq_39160721</author><pubDate>Wed, 04 May 2022 10:27:06 +0800</pubDate><description><![CDATA[ubuntu 微信、qq、企业微信的安装使用]]></description><category></category></item><item><title><![CDATA[银河麒麟V10——安装VNC实现远程桌面]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/124273978</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/124273978</guid><author>qq_39160721</author><pubDate>Tue, 19 Apr 2022 15:11:27 +0800</pubDate><description><![CDATA[​​​​​​

安装图像界面

yum grouplist




yum groupintall -y "Server wiht UKUI GUI"
systemctl get-default
systemctl set-default graphical.target
reboot

卸载

yum groupremove "Server with UKUI GUI"


安装vnc


yum install tigervnc-server -y
rpm -qa|grep tigervnc-ser..]]></description><category></category></item><item><title><![CDATA[麒麟 Linux 挂载大于2T的硬盘]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/124150362</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/124150362</guid><author>qq_39160721</author><pubDate>Wed, 13 Apr 2022 16:12:25 +0800</pubDate><description><![CDATA[linux 大于2t磁盘挂载 lvm]]></description><category></category></item><item><title><![CDATA[基于ubuntu制作jdk1.8的镜像]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/123579825</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/123579825</guid><author>qq_39160721</author><pubDate>Fri, 18 Mar 2022 17:08:21 +0800</pubDate><description><![CDATA[ubuntu  jdk]]></description><category></category></item><item><title><![CDATA[Ubuntu 20.04 安装nginx]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/123498304</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/123498304</guid><author>qq_39160721</author><pubDate>Tue, 15 Mar 2022 11:35:23 +0800</pubDate><description><![CDATA[ubuntu 安装nginx]]></description><category></category></item><item><title><![CDATA[centos7 lvm 重新调整根分区大小]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/119954438</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/119954438</guid><author>qq_39160721</author><pubDate>Fri, 27 Aug 2021 15:49:05 +0800</pubDate><description><![CDATA[通过lvm的方式调整磁盘大小



2.查看分区情况



3.卸载删除/home



4.对根目录扩容lvextend -L +800G /dev/mapper/centos-root，然后扩展/root文件系统 ：xfs_growfs /dev/mapper/centos-root



5.重新创建数据目录lvcreate -L 1000G -n data centos,mkfs.xfs /dev/centos/data







6.挂载数据...]]></description><category></category></item><item><title><![CDATA[spark 日志带来的空间问题]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/88020139</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/88020139</guid><author>qq_39160721</author><pubDate>Thu, 28 Feb 2019 11:43:53 +0800</pubDate><description><![CDATA[环境：ambari +hdp2.6.4

最近发现hdfs的空间越来越少，少到影响数据分析处理了。

使用名称查看一下到底是谁占了空间：

hadoop fs -du -h /

发现有两个文件夹站的空间是最大的：

一个是/app-logs

一个是/spark2-history

删除文件夹下内容

设置定时清理：

## spark2-history日志

spark.history.fs...]]></description><category></category></item><item><title><![CDATA[Yarn application has already ended! It might have been killed or unable to launch application master]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/86645645</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/86645645</guid><author>qq_39160721</author><pubDate>Fri, 25 Jan 2019 14:40:02 +0800</pubDate><description><![CDATA[环境：ambari+hdp 2.7.3

出现背景：nodename服务器出现异常，发生重启。

出现问题：以前能跑的pyspark脚本，运行的时候Yarn application has already ended! It might have been killed or unable to launch application master的错误。

解决方法：

1.在ambari中重启y...]]></description><category></category></item><item><title><![CDATA[pypark处理文件异常处理。]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/82414293</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/82414293</guid><author>qq_39160721</author><pubDate>Wed, 05 Sep 2018 09:18:36 +0800</pubDate><description><![CDATA[环境：

python：3.6

spark：2.0

datanode：3台，内存8g

当使用通过spark加yarn的方式对一个4.6G的HDFS文件进行分析的时候，出现以下错误提示：

org.apache.spark.shuffle.MetadataFetchFailedException:Missing an output location for shuffle 5

 

原因：分...]]></description><category></category></item><item><title><![CDATA[spark 导入文件到hive出现多小文件的问题]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/82387328</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/82387328</guid><author>qq_39160721</author><pubDate>Tue, 04 Sep 2018 17:38:57 +0800</pubDate><description><![CDATA[环境：

ambari:2.6.1

spark 2.1

python 3.6

oracle 11.2

sqoop 1.4

将sqoop采集到HDFS中的文件导入到HIVE数据库，导入成功后，发现hive数据库中出现多个小文件的问题，严重影响后期数据分析的加载速度。



解决方法：



SJTable = spark.sql(&quot;select  *          from &quot; + t...]]></description><category></category></item><item><title><![CDATA[数据清洗分析导出遇到的一些问题]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80838947</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80838947</guid><author>qq_39160721</author><pubDate>Thu, 28 Jun 2018 10:36:56 +0800</pubDate><description><![CDATA[本文简单记录一个学习过程，无技术含量，勿喷。环境：ambari:2.6.1spark 2.1python 3.6oracle 11.2sqoop 1.4最近一直在测试数据采集、清洗、分析、导出、展示的一系列的工作，主要数据流程如下：oracle 数据库-》hdfs-》hive-》oracle总的设计：通过任务调度的方式执行各个阶段的任务，其中各个任务通过脚本的方式实现，扩展的时候只需要增加对应的脚...]]></description><category></category></item><item><title><![CDATA[Python源码安装cx_Oracle]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80838281</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80838281</guid><author>qq_39160721</author><pubDate>Thu, 28 Jun 2018 09:37:46 +0800</pubDate><description><![CDATA[最近需要将HIVE中的数据导入到oracle，采用了通过pyspark读取hive数据，然后通过python连接数据库，将数据插入到ORACLE的方案。所以涉及到了cx_orcacle的安装部署。以下是通过转载内容：https://www.cnblogs.com/sheng-247/p/7529289.html，其中比较纠结的是oracle_client的下载（屌丝没有oracle账号）。一、涉及...]]></description><category></category></item><item><title><![CDATA[通过spark sql创建HIVE的分区表]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80651256</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80651256</guid><author>qq_39160721</author><pubDate>Mon, 11 Jun 2018 14:00:50 +0800</pubDate><description><![CDATA[今天需要通过导入文本中的数据到HIVE数据库，而且因为预设该表的数据会比较大，所以采用分区表的设计方案。将表按地区和日期分区。在这个过程出现过一些BUG，记录以便后期查看。 spark.sql(&quot;use oracledb&quot;)
 spark.sql(&quot;CREATE TABLE IF NOT EXISTS &quot; + tablename + &quot; (OBUID STRING, BUS_ID STRING,R...]]></description><category></category></item><item><title><![CDATA[通过py文件将文件中的数据导入到hive遇见的问题]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80598148</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80598148</guid><author>qq_39160721</author><pubDate>Wed, 06 Jun 2018 17:41:48 +0800</pubDate><description><![CDATA[因需要通过任务的方式执行将文本的数据导入到HIVE数据库中。所以考虑每次的任务通过命令执行py文件的方式来实现定时导入文件到HIVE数据库中。所以先做了一个测试，但是出行一些问题，现记录下来，供以后查看。测试脚本如下：# -*- coding: utf-8 -*-  from __future__ import print_functionfrom pyspark.sql import Spark...]]></description><category></category></item><item><title><![CDATA[安装docker hue 并导入到内部服务器]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80271078</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80271078</guid><author>qq_39160721</author><pubDate>Thu, 10 May 2018 17:36:08 +0800</pubDate><description><![CDATA[环境：Centos 7.2Hue因管理需求，内部服务器不能上外网，离线安装Hue又是很苦逼的活，于是考虑在外网环境安装docker 版的hue，然后导出镜像，再导入到内部服务器。前提准备安装docker外部服务器和内部服务器都需要安装docker，内部服务器需要离线安装docker，安装步骤参考：https://mp.youkuaiyun.com/postedit/80269209外部服务器安装Docker ...]]></description><category></category></item><item><title><![CDATA[安装docker]]></title><link>https://blog.youkuaiyun.com/qq_39160721/article/details/80269209</link><guid>https://blog.youkuaiyun.com/qq_39160721/article/details/80269209</guid><author>qq_39160721</author><pubDate>Thu, 10 May 2018 15:57:36 +0800</pubDate><description><![CDATA[一、  安装 Docker1.    在线安装Docker 软件包和依赖包已经包含在默认的 CentOS-Extras 软件源里，安装命令如下：yum -y install docker-io或者执行Yum install docker2.    离线安装下载 docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpmhttps://download....]]></description><category></category></item></channel></rss>