<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[donger__chen的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/donger__chen</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; donger__chen]]></copyright><item><title><![CDATA[Python argparse使用方法介绍]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/126331440</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/126331440</guid><author>donger__chen</author><pubDate>Sun, 14 Aug 2022 15:34:02 +0800</pubDate><description><![CDATA[python argparse使用方法介绍]]></description><category></category></item><item><title><![CDATA[pyspark窗口windows]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/123011984</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/123011984</guid><author>donger__chen</author><pubDate>Fri, 18 Feb 2022 22:32:31 +0800</pubDate><description><![CDATA[pyspark窗口windows
1. 根据滑窗计算统计量
from pyspark.sql import Windows, SparkSession
import pyspark.sql.functions as F
data = spark.createDataFrame(...)
# data.columns = ['A', 'B', 'C', 'D]
# 对data按A作groupby分区，然后在每个分区内根据B做排序，并将当前行和前两行作为一个组合，计算每个组合内列C的总和和列D的平均值
win=]]></description><category></category></item><item><title><![CDATA[python连接hbase]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/118071641</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/118071641</guid><author>donger__chen</author><pubDate>Sun, 20 Jun 2021 16:51:46 +0800</pubDate><description><![CDATA[1.happybase

访问使用的是thrift，因此必须开启thrift(hbase thrift start)
如果hbase-site.xml的hbase.regionserver.thrift.http设置为true，则该方式无法访问。会有No protocol version 的错误

# pip install happybase
# 官方文档 https://happybase.readthedocs.io/en/latest/api.html#connection
connection =]]></description><category></category></item><item><title><![CDATA[kafka基本操作]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/117911613</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/117911613</guid><author>donger__chen</author><pubDate>Mon, 14 Jun 2021 21:23:01 +0800</pubDate><description><![CDATA[0.消息队列(Message queue,MQ)

0.0 什么是消息队列？
0.1 消息队列的种类

0.点对点模式
一对一，一个生产者对应一个消费者；
消费者主动拉取数据，消息被消费后会被清除；
一条消息只会被一个消费者所读取。
1.发布-订阅模式
一对多，一个生产者对应多个消费者；
消息被消费后不会立马被清除，而是间隔一定时长后才会作清除动作。


0.2 各类别的优缺点

1.kafka中的基本概念

生产者
消费者
消费组
broker
partition
replication
leader
I]]></description><category></category></item><item><title><![CDATA[PYCHARM连接SPARK的配置]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/117231469</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/117231469</guid><author>donger__chen</author><pubDate>Mon, 24 May 2021 20:24:07 +0800</pubDate><description><![CDATA[PYCHARM连接SPARK
新增环境变量

SPARK_HOME
PYSPARK_HOME
PYTHONPATH

配置方式
pycharm --&gt; run --&gt; edit configurations --&gt; environment -&gt; environment variables --&gt; user environment variables 新增SPARK_HOME、PYSPARK_HOME、PYTHONPATH($SPARK_HOME/python/lib/py4j-]]></description><category></category></item><item><title><![CDATA[PYTHON调用GPU之HelloWorld]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/116904368</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/116904368</guid><author>donger__chen</author><pubDate>Sun, 16 May 2021 22:21:22 +0800</pubDate><description><![CDATA[PYTHON调用GPU之HelloWorld

import cv2
import numpy as np
from numba import cuda
import time
import math
import os
# os.environ['CUDA_VISIBLE_DEVICES'] = "0"


@cuda.jit
def process_gpu(img,channels):
    tx = cuda.blockIdx.x*cuda.blockDim.x + cuda.threadIdx.x]]></description><category></category></item><item><title><![CDATA[Scala基础]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/112095774</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/112095774</guid><author>donger__chen</author><pubDate>Sat, 02 Jan 2021 18:15:30 +0800</pubDate><description><![CDATA[Scala基础
1.变量的声明

 1.1 变量声明语法
val | var 变量名 [:变量类型] = 变量值，如：

val age:Int = 123
val height = 123.3
var weight:Double = 123.3
var price:Float = 123.4f
var isTrue:Boolean = true


1.var表明变量是可变的，val表明变量是不可变的(与java的final一样)；
2.scala语法是变量名在前，数据类型在后，与java相反；
3.sc]]></description><category></category></item><item><title><![CDATA[MapReduce之join]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/111822758</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/111822758</guid><author>donger__chen</author><pubDate>Sun, 27 Dec 2020 17:13:00 +0800</pubDate><description><![CDATA[1.ReduceJoin

Bean

import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

public class OrderBean implements Writable {
    private String orderId;
    private String prodId;
    private int]]></description><category></category></item><item><title><![CDATA[用WordCount读懂MapReduce框架]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/111768722</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/111768722</guid><author>donger__chen</author><pubDate>Sat, 26 Dec 2020 20:18:20 +0800</pubDate><description><![CDATA[1.MapReduce框架介绍

2.MapReduce之WordCount代码
一个MapReduce代码可以包括以下几个部分

Mapper
Partitioner
Combiner
Reducer


自定义类

import org.apache.hadoop.io.WritableComparable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
public class WCBe]]></description><category></category></item><item><title><![CDATA[Hadoop之MapReduce编程]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/111060313</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/111060313</guid><author>donger__chen</author><pubDate>Sun, 13 Dec 2020 15:33:23 +0800</pubDate><description><![CDATA[MapReduce编程基础
0. MR与Java的数据类型对比




MR
Java




boolean
BooleanWritable


byte
ByteWritable


int
IntWritable


float
FloatWritable


long
LongWritable


double
DoubleWritable


String
Text


map
MapWritable


array
ArrayWritable


//数据类型所在java包
import org]]></description><category></category></item><item><title><![CDATA[大数据入门交流]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110940713</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110940713</guid><author>donger__chen</author><pubDate>Wed, 09 Dec 2020 21:07:43 +0800</pubDate><description><![CDATA[非计算机专业，接触hadoop生态近两年，基本从零开始，期间踩过无数坑，深知这条路的不易。虽说现在仍是大数据小白，但是这两年也积累了一点经验，如能利用这有限的经验帮助到一些刚接触hadoop生态的人，那将甚是开心。

    当你在hadoop、spark、hbase、hive、zookeeper、sqoop等组件的安装与使用过程中遇到了问题，可私信我，让我们共同探讨、共同进步。

   
...]]></description><category></category></item><item><title><![CDATA[Maven基础]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110826331</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110826331</guid><author>donger__chen</author><pubDate>Mon, 07 Dec 2020 21:12:18 +0800</pubDate><description><![CDATA[Maven基础

Maven与Intellij IDEA的关系
Maven是Maven，Intellij IDEA是Intellij IDEA，两者是独立的，但是一般Intellij IDEA内嵌Maven。

1. 安装

安装地址
linux系统选择tar.gz安装包；windows选择zip包
下载完后，解压
修改配置文件
解压路径下有一个conf文件夹，打开settings.xml

找到
&lt;localRepository&gt;/path/to/local/repo&lt;/localRep]]></description><category></category></item><item><title><![CDATA[Hive基础]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110688478</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110688478</guid><author>donger__chen</author><pubDate>Sun, 06 Dec 2020 16:44:32 +0800</pubDate><description><![CDATA[Hive的语法基本与mysql相同 ，且同样不区分大小写
1.安装
apache hive安装教程，安装过程有问题可以留言。

2.数据类型




Hive数据类型
对应java类型





TINYINT
byte
1byte


SMALLINT
short
2byte


INT
int
4byte


BIGINT
long
8byte


BOOLEAN
boolean



FLOAT
float



DOUBLE
double



STRING
string
单引号或双引号


TIM]]></description><category></category></item><item><title><![CDATA[sqoop基础]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110268983</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110268983</guid><author>donger__chen</author><pubDate>Sun, 29 Nov 2020 22:37:55 +0800</pubDate><description><![CDATA[sqoop基础

sqoop的含义是sql to hadoop，目的是实现关系型数据库与hdaoop之间的数据转换

Linux版安装教程
1.到清华镜像下载sqoop安装包
2.解压以下载的安装包
tar -zxvf 安装包路径 -C 解压路径

3.修改配置文件

...]]></description><category></category></item><item><title><![CDATA[Hive安装]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110286932</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110286932</guid><author>donger__chen</author><pubDate>Sat, 28 Nov 2020 17:52:55 +0800</pubDate><description><![CDATA[Hive安装

下载hive安装包
清华镜像安装地址
解压

tar -zxvf 安装包路径 -C 解压路径


修改配置文件

cd 2的解压路径
cp hive-env.sh-template hive-env.sh
vim hive-env.sh
#添加
export HADOOP_HOME=/home/dong/hadoop/hadoop
export HIVE_CONF_DIR=/home/dong/hadoop/hive/conf


添加环境变量

export HIVE_HOME=/home]]></description><category></category></item><item><title><![CDATA[linux下mysql密码忘记解决办法]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110275033</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110275033</guid><author>donger__chen</author><pubDate>Sat, 28 Nov 2020 15:17:42 +0800</pubDate><description><![CDATA[linux下mysql密码忘记解决办法

将停掉mysql服务

sudo service mysql stop


将mysql切换为免密码登录状态

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

#添加
skip-grant-tables


重启mysql服务

sudo service mysql start


进入mysql

mysql -uroot


修改密码

#直接修改密码如果报
#ERROR 1290 (HY000): The MySQL ]]></description><category></category></item><item><title><![CDATA[html入门之简历制作]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/110124654</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/110124654</guid><author>donger__chen</author><pubDate>Sat, 28 Nov 2020 13:44:31 +0800</pubDate><description><![CDATA[&lt;!DOCTYPE html&gt;

&lt;html xmlns="http://www.w3.org/1999/xhtml"&gt;
&lt;head runat="server"&gt;
    &lt;title&gt;&lt;/title&gt;
    &lt;style type="text/css"&gt;
        #form1 {
            width:800px;
            height:1000px;
            backgro.]]></description><category></category></item><item><title><![CDATA[HTML基础]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/109896179</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/109896179</guid><author>donger__chen</author><pubDate>Sat, 21 Nov 2020 16:59:11 +0800</pubDate><description><![CDATA[HTML

HTML全称HyperText Markup Language，中文译为超文本标记语言，所有的Web无论以什么语言、什么架构开发，最终其结构都是以html形式呈现。
HTML使用标签的形式来标识网页中的不同组成部分，相较于文本，超文本不仅可以保存文字，还能保存图片、音频、视频，其中“超”是指代超链接，使用链接实现不同页面的跳转。
超文件标记语言中的标记指的是标签，其作用是用于指定网页呈现的内容的格式，比如呈现的是标题或是内容

&lt;!--  注释
标签是成对出现的，用&lt;&gt;&lt;]]></description><category></category></item><item><title><![CDATA[JAVA面向对象]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/109013890</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/109013890</guid><author>donger__chen</author><pubDate>Sun, 08 Nov 2020 16:49:42 +0800</pubDate><description><![CDATA[JAVA面向对象


java类及类成员
1.属性
2.方法
3.构造器
4.代码块
5.内部类


面向对象三大特性
1.封装
2.继承
3.多态


关键字
1.this
2.supper
3.static
4.final
5.abstract
6.interface
7.package
8.import


类与对象(实例)
实例化的类即为对象


//类
class Person{
	//属性
	String name;
	int age;
	float weight;
	float height;]]></description><category></category></item><item><title><![CDATA[Java通过API操作HBase]]></title><link>https://blog.youkuaiyun.com/donger__chen/article/details/109557773</link><guid>https://blog.youkuaiyun.com/donger__chen/article/details/109557773</guid><author>donger__chen</author><pubDate>Sun, 08 Nov 2020 16:34:48 +0800</pubDate><description><![CDATA[java通过API操作HBase

依赖包安装

&lt;-- 在pom.xml文件里添加以下两个依赖，version版本为对应的hbase版本--&gt;
&lt;dependencies&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hbase&lt;/groupId&gt;
        &lt;artifactId&gt;hbase-server&lt;/artifactId&gt;
        &lt;version]]></description><category></category></item></channel></rss>