<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[liuhongwei_study的专栏]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/liuhongwei_study</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; liuhongwei_study]]></copyright><item><title><![CDATA[greenplum遇到的问题]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/127684247</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/127684247</guid><author>liuhongwei_study</author><pubDate>Sat, 05 Nov 2022 15:49:05 +0800</pubDate><description><![CDATA[greenplum遇到的问题]]></description><category></category></item><item><title><![CDATA[gpbackup安装与使用]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/124747557</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/124747557</guid><author>liuhongwei_study</author><pubDate>Fri, 13 May 2022 10:54:31 +0800</pubDate><description><![CDATA[1、下载gpbackup

Download VMware Tanzu™ Greenplum® Backup and Restore — VMware Tanzu Network

2、安装gpbackup（对应自己的greenplum版本）




			
			gpscp -v -f /home/gpadmin/seg_hosts pivotal_greenplum_backup_restore-1.19.0.tar.gz =:/usr/local/greenplum-db-6.1.0/

			g..]]></description><category></category></item><item><title><![CDATA[greenplum优化]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/121084470</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/121084470</guid><author>liuhongwei_study</author><pubDate>Mon, 01 Nov 2021 17:35:28 +0800</pubDate><description><![CDATA[1、清理数据表大小

频繁进行修改删除的数据表，表占的空间大小膨胀的特别快，表查询速度也会越来越慢

使用临时表进行数据表重建



2、系统表pg_attribute,pg_class,pg_type长时间不清理，膨胀的大小很影响查询速度

使用vacuum full analyze pg_attribute



3、清理日志

greenplum对于每个查询都有保留日志，长时间磁盘容量就被日志占满

使用crontab，建立定时清理日志的跑批








...]]></description><category></category></item><item><title><![CDATA[airflow设置dag并行运行task个数]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/118568799</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/118568799</guid><author>liuhongwei_study</author><pubDate>Thu, 08 Jul 2021 10:49:31 +0800</pubDate><description><![CDATA[concurrency是设置设置dag并行运行task的个数，但是


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020,9,13),#任务启动时间点，须小于实际执行的时间的下一个间隔
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_re]]></description><category></category></item><item><title><![CDATA[kafka.errors.CommitFailedError:max_poll_interval_ms]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/115230031</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/115230031</guid><author>liuhongwei_study</author><pubDate>Tue, 27 Apr 2021 09:59:35 +0800</pubDate><description><![CDATA[kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
       .]]></description><category></category></item><item><title><![CDATA[熊猫学猿--greenplum备份]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116191016</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116191016</guid><author>liuhongwei_study</author><pubDate>Tue, 27 Apr 2021 09:45:43 +0800</pubDate><description><![CDATA[因为我的greenplum很多数据表都是从其他库同步过来，不需要进行全库备份，所以就使用定时更新需要备份的文件，然后使用gpbackup进行更新

获取需要备份的数据表名称

/home/gpadmin/backup_lxcs_data.py


import pandas as pd
import sys
from sqlalchemy import create_engine

def main():
    postgresql_config = {
        'host': '192.168.]]></description><category></category></item><item><title><![CDATA[熊猫学猿--greenplum维护清理日志]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116190527</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116190527</guid><author>liuhongwei_study</author><pubDate>Tue, 27 Apr 2021 09:35:00 +0800</pubDate><description><![CDATA[1、定期清理日志

greenplum对每个操作都有记录，但长时间下来磁盘容量就很快爆满，尤其在进行批量insert和update操作时，写一个定时清理日志脚本的命令


find /data/master/gpseg-1/pg_log  -mtime +2  -type f -name "gpdb-*.csv" -exec rm {} \;#mdw
ssh sdw1 "find /data/primary/*/pg_log  -mtime +2 -type f -name "gpdb-*.csv" -ex]]></description><category></category></item><item><title><![CDATA[熊猫学猿--sql查询]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116056305</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116056305</guid><author>liuhongwei_study</author><pubDate>Fri, 23 Apr 2021 15:01:34 +0800</pubDate><description><![CDATA[查询关键词 【select】 【from】 【where】 【order by】 【group by】【limit】

比如查询一个班级的考试成绩。语文老师只关心语文成绩 就是使用 select 语文成绩 from 表名

想查前10名select 语文成绩 from 表名 order by语文成绩 desc limit 10

查平均分 select mean(语文成绩) from表名 group by 学生学号

查不及格的学生 select 姓名from表名 where语文成绩&l...]]></description><category></category></item><item><title><![CDATA[熊猫学猿--数据库基础]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116055532</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/116055532</guid><author>liuhongwei_study</author><pubDate>Fri, 23 Apr 2021 14:42:03 +0800</pubDate><description><![CDATA[现在数据无处不在，说的一句话都能作为数据存储起来。数据库可以看作现实中的仓库，把信息按需求有规则的存储起来。这个仓库从存到取都有一套方式，仓库的每个房间就是数据库的数据表，这里每一条数据都是在一个数据表里，是不能像现实生活中没有规则的随意丢弃到仓库中，必须安排到房间里。

仓库的创建使用：CREATE DATABASE

房间(TABLE)的装修使用:CREATE TABLE,DROP ,ALTER

房间里放东西（数据）使用：INERT,DELELTE,UPDATE

找东西使用：SELECT。

...]]></description><category></category></item><item><title><![CDATA[熊猫学猿--therad和with]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/114981823</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/114981823</guid><author>liuhongwei_study</author><pubDate>Thu, 18 Mar 2021 15:58:43 +0800</pubDate><description><![CDATA[from threading import Thread
class methread(Thread):
    def __init__(self, name: str):
        super().__init__()
        self.name=name
        print(self.name)
        print('初始化'+'\n')
    def __enter__(self):
        self.start()
        print('in...]]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow与kubernetes结合]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/112647257</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/112647257</guid><author>liuhongwei_study</author><pubDate>Fri, 15 Jan 2021 09:37:53 +0800</pubDate><description><![CDATA[在接触一个新东西的时候，真的希望是一步步告诉怎么实现的，只有跑起来我才能它到底是干啥的

1、安装minikube（我是用虚拟机上Linux系统进行的测试）

下载minikube


curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 | bash -s -- 2.2.0 1.4.8 -s

sudo install kubectl /usr/local/bin/kubectl

2、安]]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow执行python脚本时能捕捉执行结果]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581633</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581633</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 16:25:09 +0800</pubDate><description><![CDATA[1、创建python脚本
def main():
	#处理脚本都用main函数下进行
	print('hello')
	
	
if __name__ == '__main__':
    try:
        main()
        sys.exit(0)
    except Exception as e:
        print(e)
        sys.exit(1)

2、执行python的sh
do
    /root/miniconda3/bin/python -u /root]]></description><category></category></item><item><title><![CDATA[熊猫学猿---airflow动态创建dag及task]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581319</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581319</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 16:15:31 +0800</pubDate><description><![CDATA["""
Code that goes along with the Airflow located at:
http://airflow.readthedocs.org/en/latest/tutorial.html
"""
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.contrib.operators.ssh_operator import SSHO]]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow创建父与子dag]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581018</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108581018</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 16:06:36 +0800</pubDate><description><![CDATA["""
Code that goes along with the Airflow located at:
http://airflow.readthedocs.org/en/latest/tutorial.html
"""
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.contrib.operators.ssh_operator import SSHOperator]]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow执行远程服务器命令]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108580396</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108580396</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 15:58:28 +0800</pubDate><description><![CDATA[添加服务器链接

使用SSHOperator
args = {
    'owner': 'xmxy',
    'depends_on_past': False,
    "catchup": False,
    'pool': 'syn_db',
    'email': ['xiongmaoxueyuan@xiongmaoxueyuan.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'start_date']]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow工作流]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108579170</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108579170</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 15:34:29 +0800</pubDate><description><![CDATA[创建脚本
在设置的dag目录下(/root/airflow/dags),新建.py文件
使用官方文档的脚本
from datetime import timedelta,datetime

# The DAG object; we'll need this to instantiate a DAG
from airflow import DAG
# Operators; we need this to operate!
from airflow.operators.bash_operator import ]]></description><category></category></item><item><title><![CDATA[熊猫学猿--airflow安装、设置时区及设置管理权限]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108577678</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108577678</guid><author>liuhongwei_study</author><pubDate>Mon, 14 Sep 2020 14:28:56 +0800</pubDate><description><![CDATA[docker下安装
1、 在Linux服务器上创建目录 mkdir airflow
2、 拉取 git clone https://github.com/puckel/docker-airflow.git /root/airflow
3、 docker run -d -p 8082:8080 puckel/docker-airflow
4、 在mysql中创建数据库airflow
5、 修改airflow支持的数据库为mysql 修改airflow.cfg
sql_alchemy_conn = mysql:]]></description><category></category></item><item><title><![CDATA[熊猫学猿--数据分析入门2%--pandas]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108551260</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108551260</guid><author>liuhongwei_study</author><pubDate>Sat, 12 Sep 2020 16:21:02 +0800</pubDate><description><![CDATA[引入pandas包
import pandas as pd

这里是引入pandas包，并起了个别名pd，方便使用时简单书写
获取数据
做开发的时候数据获取一般就两种，一种时读取文件，一种是查询数据库
从文件读取read_csv
Train_data=pd.read_csv('train.csv',index_col=['id'])

这里Test_data是DataFrame类型，DataFrame是由多个Series组成。查看

对DataFrame的操作有多种多样，可参考官方文档
在运用中比较常见的
]]></description><category></category></item><item><title><![CDATA[熊猫学猿--数据分析入门1%]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108549840</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/108549840</guid><author>liuhongwei_study</author><pubDate>Sat, 12 Sep 2020 15:27:25 +0800</pubDate><description><![CDATA[大数据，智能AI就像程序界的明星一样，各行都想搞一搞。就连小学生都在开启python编程，我一个搞电路的朋友在问python是什么，怎么这么多广告。一个会计的朋友都说，他也在学python。一个老的php程序猿启动探索python之迷。

接触新的语言有三步

1、搭环境

搭环境真的不是一件简单的事情，新手越不懂，越遇到各种兼容性问题，感觉环境搭成功了，语言之门就跨进了一半了。

2、选开发软件

在什么软件上开发，选对了真的会事半功倍。python安装后，会让你打开命令窗口，输入脚本看运行结果。即使写.]]></description><category></category></item><item><title><![CDATA[nvm安装node,npm总是失败]]></title><link>https://blog.youkuaiyun.com/liuhongwei_study/article/details/105999364</link><guid>https://blog.youkuaiyun.com/liuhongwei_study/article/details/105999364</guid><author>liuhongwei_study</author><pubDate>Fri, 08 May 2020 16:03:31 +0800</pubDate><description><![CDATA[1、下载安装nvm

2、使用nvm安装node指定版本，这里需要设置npm下载镜像


nvm  npm_mirror  https://npm.taobao.org/mirrors/npm/
nvm node_mirror http://npm.taobao.org/mirrors/node/
nvm  install 10.15.2



这里npm总是不成功

3、解决办法，下载npm...]]></description><category></category></item></channel></rss>