<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[xiaomengzhang的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/xiaomengzhang</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; xiaomengzhang]]></copyright><item><title><![CDATA[（14）pandas基础1：Series和DataFrame模块]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121360457</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121360457</guid><author>xiaomengzhang</author><pubDate>Tue, 16 Nov 2021 18:17:42 +0800</pubDate><description><![CDATA[pandas的常用工具数据结构：Series和DataFrame]]></description><category></category></item><item><title><![CDATA[（13）numpy基础5：random模块 伪随机数生成及使用]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121355719</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121355719</guid><author>xiaomengzhang</author><pubDate>Tue, 16 Nov 2021 15:34:33 +0800</pubDate><description><![CDATA[1.numpy.random 模块

1.1 函数详解

seed 向随机数生成器传递随机状态种子

permutation 返回一个序列的随机排序，或者返回一个乱序的整数范围序列

shuffle 随机排序一个序列

rand 从均匀分布中抽取样本

randint 根据给定的由低到高的范围抽取随机整数

randn 从均值0方差1的正态分布中抽取样本（MATLAB型接口）

binomial 从二项分布中抽取样本

normal 从正态（高斯）分布中抽取样本

beta 从bata分布中抽取样本

c.]]></description><category></category></item><item><title><![CDATA[（12）numpy基础4：数组进行文件输入或输出及 线性代数等操作]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121290153</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121290153</guid><author>xiaomengzhang</author><pubDate>Fri, 12 Nov 2021 17:19:33 +0800</pubDate><description><![CDATA[1 numpy数组进行文件输入和输出

numpy在硬盘中将数据以文本或二进制文件的形式存入硬盘或由硬盘载入

np.save和np.load是高效存取硬盘数据的函数


arr = np.arange(10)
np.save('some_array', arr) #将数组数据存储，文件会加后缀 .npy
np.load('some_array.npy') #将数组的输出
np.savez('array_archive.npz', a=arr, b=arr) #将要存储的数组，作为参数传入，保存
arch ]]></description><category></category></item><item><title><![CDATA[（11）numpy基础3：面向数组编程]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121142436</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121142436</guid><author>xiaomengzhang</author><pubDate>Thu, 04 Nov 2021 15:30:26 +0800</pubDate><description><![CDATA[1 牛刀小试

对网格数据可视化样例


points = np.arange(-5, 5, 0.01) # 生成起始为-5，步长为0.01，结束为5的数组
points
xs, ys = np.meshgrid(points, points) #据两个数组生成网格采样点的函数，结果是二维矩阵
ys
注意：[X,Y] =meshmeshgrid(x,y); 返回的两个meshX、Y必定是行数、列数相等的且X、Y的行数都等于输入参数y中元素的总个数，X、Y的列数都等于输入参数x中元素总个数





...]]></description><category></category></item><item><title><![CDATA[（10）numpy基础2：通用函数]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121127970</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121127970</guid><author>xiaomengzhang</author><pubDate>Wed, 03 Nov 2021 18:53:19 +0800</pubDate><description><![CDATA[通用函数，也可称为ufunc，是一种在ndarray数据中进行逐元元素操作的函数

后续更新···
]]></description><category></category></item><item><title><![CDATA[sql 拉链表详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121123831</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121123831</guid><author>xiaomengzhang</author><pubDate>Wed, 03 Nov 2021 16:16:31 +0800</pubDate><description><![CDATA[参考链接：

拉链表详细讲解 - 知乎 (zhihu.com)

代码部分：

CREATE TABLE source_table(
 userid string,
 loginname string,
 regiondate string,
 phonenum string,
 birthday string,
 status string,
 lastlogindate string)
PARTITIONED BY (datatime string)
ROW FORMAT D...]]></description><category></category></item><item><title><![CDATA[（9）numpy基础1：数组与向量化计算]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121080645</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121080645</guid><author>xiaomengzhang</author><pubDate>Mon, 01 Nov 2021 15:54:36 +0800</pubDate><description><![CDATA[1 初识numpy

1.1定义

是 Numerical Python 简称，它是python数值计算最为重要的基础包，基于numpy的科学函数功能，将numpy的数值对象作为数据交换的通用语

NumPy 是一个运行速度非常快的数学库，主要用于数组计算，包含：

（1）ndarray，高效多维数组，基于数组计算及广播功能函数

（2）对硬盘中数组数据进行读写的工具，并对内存映射文件进行操作

（3）用numpy连接C/C++/Fortran语言类库的c语言API

将数据传给底层语言编写的外部类库，再.]]></description><category></category></item><item><title><![CDATA[（7）python 之 文件操作（open函数） 与Unicode编码]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121063218</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/121063218</guid><author>xiaomengzhang</author><pubDate>Sun, 31 Oct 2021 14:25:40 +0800</pubDate><description><![CDATA[1 操作文件 open()函数详解

open()函数用于创建或打开指定文件，语法格式：

file = open(file_name , mode='r' , buffering=-1 , encoding = 'utf-8')

file：表示要创建的文件对象。

file_name：要创建或打开文件的文件名称（最好绝对路径）

mode：可选，用于指定文件的打开模式。如果不写，则默认以只读（r）模式打开文件。

可选模式：

r：只读(文件必须存在); w:只写 (若文件存在，会覆盖文件；反之，则..]]></description><category></category></item><item><title><![CDATA[（8）python 实战详情 常用模块和函数]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120990150</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120990150</guid><author>xiaomengzhang</author><pubDate>Wed, 27 Oct 2021 12:02:29 +0800</pubDate><description><![CDATA[1 运行python脚本传参

1.1 指定参数名传参

python xx/xx.py -t $a -s $b -k $c

实际使用：

usage = "usage: %prog -t token -s secret -k key"
  parser = OptionParser(usage)
  parser.add_option("-t", "--token", dest="token", help="Token")
  parser.add_option("-s", "--sec...]]></description><category></category></item><item><title><![CDATA[shell 之变量详解及实战]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120988170</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120988170</guid><author>xiaomengzhang</author><pubDate>Wed, 27 Oct 2021 11:11:11 +0800</pubDate><description><![CDATA[1 定义

定义变量时 your_name="xx

除了显式地直接赋值，还可以用语句给变量赋值，如：


for file in `ls /etc`
或
for file in $(ls /etc)


2 变量类型

运行shell时，会同时存在三种变量：

（1）局部变量局部变量在脚本或命令中定义，仅在当前shell实例中有效，其他shell启动的程序不能访问局部变量。

（2）环境变量所有的程序，包括shell启动的程序，都能访问环境变量，有些程序需要环境变量来保证其正常运行。必要的时候she...]]></description><category></category></item><item><title><![CDATA[（6）python 之 函数 详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120954930</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120954930</guid><author>xiaomengzhang</author><pubDate>Mon, 25 Oct 2021 17:49:37 +0800</pubDate><description><![CDATA[1.定义

函数python中最重要、最基础的代码组织和代码复用方式；函数声明时使用def，返回时用return

如达到函数尾部时仍没有return，就会自动返回none

方法定义 例子：

def my_function(x, y, z=1.5):
  if z &gt; 1:
    return z * (x + y)
  else:
    return z / (x + y)

调用例子：

my_function(5, 6, z=0.7)
my_functio...]]></description><category></category></item><item><title><![CDATA[（5）python 列表、集合和字典的推导式]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120888950</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120888950</guid><author>xiaomengzhang</author><pubDate>Thu, 21 Oct 2021 17:05:09 +0800</pubDate><description><![CDATA[1 定义

可过滤一个容器的元素，用一种简明的表达式：转换传递过滤的元素，生成新的列表

基本形式：[expr for val in collection if condition]
]]></description><category></category></item><item><title><![CDATA[shell 常用的语句整合]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120563057</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120563057</guid><author>xiaomengzhang</author><pubDate>Thu, 21 Oct 2021 16:01:39 +0800</pubDate><description><![CDATA[1.执行脚本

（1）调用hive(hive -e/-f )

hive --hiveconf tez.application.tags="xx/xx" --hiveconf hive.metastore.uris="xx/xx" -hivevar tableName=$tableName -hivevar partition=$partition -hivevar date=$date -f ${xx}/xx.sql

（2）调用spark

spark-sql --hiveconf ...]]></description><category></category></item><item><title><![CDATA[（4）python 之 集合 set 详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120886876</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120886876</guid><author>xiaomengzhang</author><pubDate>Thu, 21 Oct 2021 15:43:04 +0800</pubDate><description><![CDATA[1 定义

集合set 是一种无序且元素唯一的容器，集合像字典，但是只有键没有值；

创建方式有两种：通过set函数或用字面值集与大括号的语法


set([2, 2, 2, 1, 3, 3]) 或 {2, 2, 2, 1, 3, 3}




2 集合set 常见方法

支持数学上的集合操作，联合、交集、差集、对称差集


合并：a.union(b)          a | b
交集：a.intersection(b)   a &amp; b
c = a.copy()
c |= b 将c的内容 设置为.]]></description><category></category></item><item><title><![CDATA[spark-submit 样例]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120780736</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120780736</guid><author>xiaomengzhang</author><pubDate>Thu, 21 Oct 2021 14:49:12 +0800</pubDate><description><![CDATA[spark-submit --conf spark.driver.host=xx \

--conf hive.metastore.uris=xx \

--class com.mainclass\

--num-executors 1 \

--executor-cores 2 \

--master local[2] \

--driver-cores 1 \

--executor-memory 4g \

--jars xx/fastjson-1.2.62.jar,xx.jar \

xx.jar.]]></description><category></category></item><item><title><![CDATA[java 打成jar包，并指定main方法的插件]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120886341</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120886341</guid><author>xiaomengzhang</author><pubDate>Thu, 21 Oct 2021 14:48:18 +0800</pubDate><description><![CDATA[1 插件 maven方法

&lt;plugin&gt;
    &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
    &lt;configuration&gt;
        &lt;descriptorRefs&gt;
            &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
        &lt;/descriptorRefs&g.]]></description><category></category></item><item><title><![CDATA[（3）python之 字典dict 详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120866221</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120866221</guid><author>xiaomengzhang</author><pubDate>Wed, 20 Oct 2021 17:15:42 +0800</pubDate><description><![CDATA[dict
]]></description><category></category></item><item><title><![CDATA[（2）python 之 列表 list 详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120851428</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120851428</guid><author>xiaomengzhang</author><pubDate>Tue, 19 Oct 2021 19:02:05 +0800</pubDate><description><![CDATA[（1）定义

列表的长度可变，包括的内容也可变，使用[]或list来定义


a_list = [2, 3, 7, None]  创建list
tup = ('foo', 'bar', 'baz')
b_list = list(tup)  创建list
b_list
b_list[1] = 'peekaboo' 修改对应索引值
b_list  返回 ['foo', 'peekaboo', 'baz']
gen = range(10)
gen
list(gen)
(2)增减元素
增加元素
b_list.app]]></description><category></category></item><item><title><![CDATA[（1） python 之 元组 详解]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120849900</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120849900</guid><author>xiaomengzhang</author><pubDate>Tue, 19 Oct 2021 17:42:05 +0800</pubDate><description><![CDATA[1 元组 tuple

（1）定义，修改


tup = 4, 5, 6   返回 (4, 5, 6)
nested_tup = (4, 5, 6), (7, 8) 返回 ((4, 5, 6), (7, 8))
tuple([4, 0, 2]) 返回 (4, 0, 2)  tuple('string') 返回('s', 't', 'r', 'i', 'n', 'g')
tup[0] 返回's'  
tuple(['foo', [1, 2], True])
tup = tuple(['foo', [1, 2]]]></description><category></category></item><item><title><![CDATA[sql 常用统计函数]]></title><link>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120545719</link><guid>https://blog.youkuaiyun.com/xiaomengzhang/article/details/120545719</guid><author>xiaomengzhang</author><pubDate>Tue, 19 Oct 2021 16:42:07 +0800</pubDate><description><![CDATA[（1）一列中不同值的数据量统计

1.使用count 

count(channel = 0 OR NULL) AS '注册',

count(channel = 1 OR NULL) AS '新增',

2.使用sum+if

sum(IF(channel = 0, 1, 0)) AS '注册',

sum(IF(channel = 1, 1, 0)) AS '新增',

3.使用sum

sum(channel = 0) AS '注册',

sum(channel = 1) AS '新增',
...]]></description><category></category></item></channel></rss>