python数据分析期末复习归纳（更新中）
文章目录
前言
可以通过《利用Python进行数据分析》的GitHub仓库获得本书的数据文件和相关材料。
链接：GitHub仓库地址
提示：以下是本篇文章正文内容
一、python语言基础
分值：1~2分
- python使用缩进来组织代码，而不是其他语言比如R、C++、java和Perl那样用大括号。
- 你见到的python语句都不是以分号结尾，而分号也是可以用于在一行内将多条语句进行分隔：
a = 5;b = 6;c = 7
- python语言的一个重要特征就是对象模型的一致性。每一个数值、字符串、数据结构、函数、类、模块以及所有存在于python解释器中的事物都是python的对象。每一个对象都会关联到一种数据类型和内部数据。
- 检查两个引用是否指向同一个对象可以用 is 关键字。is not 在你检查两个关键字是不是相同对象时也是有效的。
In []: a = [1, 2, 3]
In []: b = a
In []: c = list(a)
In []: a is b
Out[]: True
In []: a is not c
Out[]: True
'''因为list函数总是创建一个新的Python列表（即一份拷贝），我们可以确定c与a是不同的。
is和==是不同的，因为在这种情况下我们可以得到:'''
In []: a == c
Out[]: True
#is 和 is not的常用之处是检查一个变量是否为None，因为None只有一个实例：
In []: a = None
In []: a is None
Out[]: True
- 可变对象与不可变对象。Python中的大部分对象，例如列表、字典、Numpy数组都是可变对象，大多数用户定义的类型（类）也是可变的。可变对象中包含的对象和值是可以被修改的。还有其他的对象是不可变的，比如字符串、元组。
- 数值类型。基础的python数字类型就是int 和 float。int 可以存储任意大小的数字。float表示浮点数，每一个浮点数都是双精度64位数值。
- 字符串。字符串是Unicode字符的序列，因此可以被看作是除了列表和元组外的一种序列。
In []: s = 'python'
In []: list(s)
Out[]: ['p','y','t','h','o','n']
In []: s[:3]
Out[]: 'pyt'
二、内建数据结构、函数、文件（重点）
元组
定义元组
In []: tup = 4,5,6
In []: tup
Out[]: (4, 5, 6)
使用tuple函数将任意序列或迭代器转换为元组：
In []: tuple([4, 0, 2])
Out[]: (4, 0, 2)
In []: tup = tuple('string')
In []: tup
Out[]: ('s', 't', 'r', 'i', 'n', 'g')
In []: (4,None,'foo')+(6,0)+('bar',)
Out[]: (4, None, 'foo', 6, 0, 'bar')
In []: ('foo','bar')*4
Out[]: ('foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'bar')
列表
创建列表
In []: alist = []
In []: list(range(10))
Out[]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
In []: tup
Out[]: ('foo', 'bar', 'baz')
In []: b_list(tup)
In []: b_list
Out[]: ['foo', 'bar', 'baz']
In []: b_list.append('dasf')
In []: b_list
Out[]: ['foo', 'bar', 'baz', 'dasf']
insert()、pop()、remove()方法
In []: b_list.insert(2,'hug')
In []: b_list
Out[]: ['foo', 'bar', 'hug', 'baz', 'dasf']
In []: b_list.pop(3)
Out[]: 'baz'
In []: b_list
Out[]: ['foo', 'bar', 'hug', 'dasf']
In []: b_list.remove('dasf')
In []: b_list
Out[]: ['foo', 'bar', 'hug']
连接两个列表
In []: ['hj',6,(2,3)]+['re',7]
Out[]: ['hj', 6, (2, 3), 're', 7]
#extend()方法消耗更小
In []: x=['hj',6,(2,3)]
In []: x.extend(['re',7])
In []: x
Out[]: ['hj', 6, (2, 3), 're', 7]
排序
In []: a=[4,6,1,9,2,8]
In []: a.sort()
In []: a
Out[]: [1, 2, 4, 6, 8, 9]
In []: a = [4, 6, 1, 9, 2, 8]
In []: sorted(a)
Out[]: [1, 2, 4, 6, 8, 9]
In []: a
Out[]: [4, 6, 1, 9, 2, 8]
In []: b=['df','rewg','fsdvsfdvsdv','d','gfg']
In []: b.sort(key=len)
In []: b
Out[]: ['d', 'df', 'gfg', 'rewg', 'fsdvsfdvsdv']
切片
In []: seq
Out[]: [7, 2, 3, 6, 3, 5, 6, 0, 1]
In []: seq[4:3]
Out[]: []
In []: seq[3:4]
Out[]: [6]
In [35]: seq[3:4]=[6,3]
In []: seq
Out[]: [7, 2, 3, 6, 3, 3, 5, 6, 0, 1]
In []: seq[-6:-2]
Out[]: [3, 3, 5, 6]
内建序列函数
zip可以将列表、元组或者其他序列的元素配对，新建一个元组构成的列表
In []: seq1 = ['foo', 'bar', 'baz']
In []: seq2 = ['one', 'two', 'three']
In []: zipped = zip(seq1, seq2)
In []: list(zipped)
Out[]: [('foo', 'one'), ('bar', 'two'), ('baz', 'three')]
reversed()将序列的元素倒序排列
In []: list(reversed(range(10)))
Out[]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
字典
创建字典
In []: d1={'a':'some value','b':[1,2,3,4]}
In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4]}
#向字典中添加元素
In []: d1[7]=0
In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4], 7: 0}
In []: d1[4] = 'banace'
In []: d1
Out[]: {'a': 'some value', 'b': [1, 2, 3, 4], 7: 0, 4: 'banace'}
In []: 'b' in d1
Out[]: True
In []: list(d1.keys())
Out[]: ['a', 'b', 7, 4]
In []: list(d1.values())
Out[]: ['some value', [1, 2, 3, 4], 0, 'banace']
使用update()方法将两个字典合并
In [55]: d1.update({'r':'goo','h':'integer'})
In [56]: d1
Out[56]:
{'a': 'some value',
'b': [1, 2, 3, 4],
7: 0,
4: 'banace',
'r': 'goo',
'h': 'integer'}
从序列生成字典
In []: mapping={}
In []: mapping = dict(zip(range(5),reversed(range(5))))
In []: mapping
Out[]: {0: 4, 1: 3, 2: 2, 3: 1, 4: 0}
有效的字典类型
尽管字典的值可以是任何Python对象，但键必须是不可变的对象，比如标量类型（整数、浮点数、字符串）或元组（且元组内对象也必须是不可变对象）
In []: x={[3,4]:54} #key值为一个可变对象列表，报错 unhashable
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-61-abe1ff6f266b> in <module>
----> 1 x={[3,4]:54}
TypeError: unhashable type: 'list'
函数
三、Numpy基础（重点）
生成ndarray
In []: data2=[[4,5,2],[3,4,5]]
In []: arr1=np.array(data2)
In []: arr1
Out[]:
array([[4, 5, 2],
[3, 4, 5]])
In []: arr1.ndim
Out[]: 2
In []: arr1.shape
Out[]: (2, 3)
In []: arr1.dtype
Out[]: dtype('int32')
In []: np.zeros(10)
Out[]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
In []: np.zeros((3,7))
Out[]:
array([[0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0.]])
In []: arr2=np.array([1,2,3],dtype=np.int64)
In []: arr2
Out[]: array([1, 2, 3], dtype=int64)
In []: arr2.dtype
Out[]: dtype('int64')
Numpy数组算数
四、pandas入门（重点）
pandas采用了很多NumPy的代码风格，但最大的不同在于pandas是用来处理表格型或异质型数据的。而NumPy则相反，它更适合处理同质型的数值类数组数据。
Series
In []: obj = pd.Series([4, 7, -5, 3])
In []: obj
Out[]:
0 4
1 7
2 -5
3 3
dtype: int64
In []: obj.values
Out[]: array([ 4, 7, -5, 3], dtype=int64)
In []: obj.index #与range(4)类似
Out[]: RangeIndex(start=0, stop=4, step=1)
In []: obj2 = pd.Series([4, 7, -5, 3],index=['d','b','a','c'])
In []: obj2
Out[]:
d 4
b 7
a -5
c 3
dtype: int64
In []: obj2.index
Out[]: Index(['d', 'b', 'a', 'c'], dtype='object')
如果已经有数据包含在Python字典中，你可以使用字典生成一个Series：
In []: sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
In []: obj3=Series(sdata)
In []: obj3
Out[]:
Ohio 35000
Texas 71000
Oregon 16000
Utah 5000
dtype: int64
DataFrame
#嵌套字典
In []: pop
Out[]: {'Nevada': {2001: 2.4, 2002: 2.9}, 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}
In []: frame3=DataFrame(pop)
In []: frame3
Out[]:
Nevada Ohio
2001 2.4 1.7
2002 2.9 3.6
2000 NaN 1.5
转置操作
In []: frame3.T
Out[]:
2001 2002 2000
Nevada 2.4 2.9 NaN
Ohio 1.7 3.6 1.5
生成时指定索引
In []: pd.DataFrame(pop,index=[2000,2001,2002])
Out[]:
Nevada Ohio
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6
DataFrame的values属性
In []: frame3.values
Out[]:
array([[2.4, 1.7],
[2.9, 3.6],
[nan, 1.5]])
fill方法会把值前向填充
In []: obj3 = Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])
In []: obj3.reindex(range(6), method='ffill')
Out[]:
0 blue
1 blue
2 purple
3 purple
4 yellow
5 yellow
dtype: object
reindex()
In []: frame = DataFrame(np.arange(9).reshape((3, 3)), index=['a', 'c', 'd'],
...: columns=['Ohio', 'Texas', 'California'])
In []: frame
Out[]:
Ohio Texas California
a 0 1 2
c 3 4 5
d 6 7 8
In []: frame2 = frame.reindex(['a', 'b', 'c', 'd'])
In []: frame2
Out[]:
Ohio Texas California
a 0.0 1.0 2.0
b NaN NaN NaN
c 3.0 4.0 5.0
d 6.0 7.0 8.0
In []: states = ['Texas', 'Utah', 'California']
In []: frame.reindex(columns=states)#使用column关键字重建索引
Out[]:
Texas Utah California
a 1 NaN 2
c 4 NaN 5
d 7 NaN 8
drop()
#Series中使用
In []: obj=pd.Series(np.arange(5.),index=['a','b','c','d','e'])
In []: obj
Out[]:
a 0.0
b 1.0
c 2.0
d 3.0
e 4.0
dtype: float64
In []: new_obj=obj.drop(['a','c'])
In []: new_obj
Out[]:
b 1.0
d 3.0
e 4.0
dtype: float64
#DataFrame中使用
In []: data = DataFrame(np.arange(16).reshape((4, 4)),^M
...: index=['Ohio', 'Colorado', 'Utah', 'New York'],^M
...: columns=['one', 'two', 'three', 'four'])
In []: data
Out[]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
Utah 8 9 10 11
New York 12 13 14 15
In []: data.drop(['Colorado', 'Ohio'])
Out[]:
one two three four
Utah 8 9 10 11
New York 12 13 14 15
In []: data.drop(['two','four'],axis=1)
Out[]:
one three
Ohio 0 2
Colorado 4 6
Utah 8 10
New York 12 14
普通的python切片是不包含尾部的，Series的切片与之不同
In []: obj[2:4] #普通切片
Out[]:
c 2.0
d 3.0
dtype: float64
In []: obj['b':'d'] #按索引切片
Out[]:
b 1.0
c 2.0
d 3.0
dtype: float64
In []: data
Out[]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
Utah 8 9 10 11
New York 12 13 14 15
In []: data['two']
Out[]:
Ohio 1
Colorado 5
Utah 9
New York 13
Name: two, dtype: int32
In []: data[['three','one']]
Out[]:
three one
Ohio 2 0
Colorado 6 4
Utah 10 8
New York 14 12
In []: data[:2]
Out[]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
使用轴标签（loc）和整数标签（iloc）选择数据
In []: data.loc['Colorado', ['two', 'three']]
Out[]:
two 5
three 6
Name: Colorado, dtype: int32
In []: data.iloc[1,[1,2]]
Out[]:
two 5
three 6
Name: Colorado, dtype: int32
In []: data.iloc[[1,2], [3, 0, 1]]
Out[]:
four one two
Colorado 7 4 5
Utah 11 8 9
In []: ser[:2]
Out[]:
0 0.0
1 1.0
dtype: float64
In []: ser.iloc[:2]
Out[]:
0 0.0
1 1.0
dtype: float64
In []: ser.loc[:2]
Out[]:
0 0.0
1 1.0
2 2.0
dtype: float64
了解广播机制
#NumPy
In []: arr=np.arange(12.).reshape((3,4))
In []: arr
Out[]:
array([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
In []: arr[0]
Out[]: array([0., 1., 2., 3.])
In []: arr-arr[0]
Out[]:
array([[0., 0., 0., 0.],
[4., 4., 4., 4.],
[8., 8., 8., 8.]])
#DataFrame与Series之间也是类似的
In []: frame = DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),
...: index=['Utah', 'Ohio', 'Texas', 'Oregon'])
In []: series=frame.iloc[0]
In []: frame
Out[]:
b d e
Utah 0.0 1.0 2.0
Ohio 3.0 4.0 5.0
Texas 6.0 7.0 8.0
Oregon 9.0 10.0 11.0
In []: series
Out[]:
b 0.0
d 1.0
e 2.0
Name: Utah, dtype: float64
In []: frame - series
Out[]:
b d e
Utah 0.0 0.0 0.0
Ohio 3.0 3.0 3.0
Texas 6.0 6.0 6.0
Oregon 9.0 9.0 9.0
五、数据载入
考察点：数据读入与写入
#数据读入
In []: !cat examples/ex1.csv
In []: df = pd.read_csv('examples/ex1.csv')
In []: pd.read_table('examples/ex1.csv', sep=',') #指定分隔符
In []: !cat examples/csv_mindex.csv
In []: parsed = pd.read_csv('examples/csv_mindex.csv',
index_col=['key1', 'key2'])
#数据写入
In []: data = pd.read_csv('examples/ex5.csv')
In []: data
In []: data.to_csv('examples/out.csv')
In []: !cat examples/out.csv
六、数据清洗与准备
分值：6分
处理缺失值
In [56]: string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])
In []: string_data
Out[]:
0 aardvark
1 artichoke
2 NaN
3 avocado
dtype: object
In []: string_data.isnull()
Out[]:
0 False
1 False
2 True
3 False
dtype: bool
In []: from numpy import nan as NA
In []: data = pd.Series([1, NA, 3.5, NA, 7])
In []: data.dropna()
Out[]:
0 1.0
2 3.5
4 7.0
dtype: float64
In []: data = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA],
...: [NA, NA, NA], [NA, 6.5, 3.]])
In []: cleaned = data.dropna()
In []: data
Out[]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
2 NaN NaN NaN
3 NaN 6.5 3.0
In []: cleaned
Out[]:
0 1 2
0 1.0 6.5 3.0
#传入how='all'时，将删除所有值均为NA的行
In []: data.dropna(how='all')
Out[]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
3 NaN 6.5 3.0
In []: data[4]=NA
In []: data
Out[]:
0 1 2 4
0 1.0 6.5 3.0 NaN
1 1.0 NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN 6.5 3.0 NaN
In []: data.dropna(axis=1,how='all')
Out[]:
0 1 2
0 1.0 6.5 3.0
1 1.0 NaN NaN
2 NaN NaN NaN
3 NaN 6.5 3.0
使用fillna()方法补全缺失值
In []: df
Out[]:
0 1 2
0 0.291028 NaN NaN
1 -0.247507 NaN NaN
2 -0.846066 NaN 1.815868
3 -2.644441 NaN -1.593109
4 0.687419 -0.576368 -1.207267
5 1.028088 -0.199093 1.090297
6 -1.356587 2.840271 0.588919
In []: df.fillna(0)
Out[]:
0 1 2
0 0.291028 0.000000 0.000000
1 -0.247507 0.000000 0.000000
2 -0.846066 0.000000 1.815868
3 -2.644441 0.000000 -1.593109
4 0.687419 -0.576368 -1.207267
5 1.028088 -0.199093 1.090297
6 -1.356587 2.840271 0.588919
#可以使用字典，为不同列设定不同的填充值
In []: df.fillna({1: 0.5, 2: 0})
Out[]:
0 1 2
0 0.291028 0.500000 0.000000
1 -0.247507 0.500000 0.000000
2 -0.846066 0.500000 1.815868
3 -2.644441 0.500000 -1.593109
4 0.687419 -0.576368 -1.207267
5 1.028088 -0.199093 1.090297
6 -1.356587 2.840271 0.588919
In []: df
Out[]:
0 1 2
0 -0.749045 0.120431 -0.524772
1 -1.170878 0.449045 -0.009419
2 -1.522980 NaN -0.932252
3 0.245718 NaN -0.584712
4 -0.611673 NaN NaN
5 0.205112 NaN NaN
In []: df.fillna(method='ffill')
Out[]:
0 1 2
0 -0.749045 0.120431 -0.524772
1 -1.170878 0.449045 -0.009419
2 -1.522980 0.449045 -0.932252
3 0.245718 0.449045 -0.584712
4 -0.611673 0.449045 -0.584712
5 0.205112 0.449045 -0.584712
In []: df.fillna(method='ffill', limit=1) #limit参数表示处理范围
Out[]:
0 1 2
0 -0.749045 0.120431 -0.524772
1 -1.170878 0.449045 -0.009419
2 -1.522980 0.449045 -0.932252
3 0.245718 NaN -0.584712
4 -0.611673 NaN -0.584712
5 0.205112 NaN NaN
#替代值
In []: data
Out[]:
0 1.0
1 -999.0
2 2.0
3 -999.0
4 -1000.0
5 3.0
dtype: float64
In []: data.replace(-999,NA)
Out[]:
0 1.0
1 NaN
2 2.0
3 NaN
4 -1000.0
5 3.0
dtype: float64
In []: data.replace([-999,-1000],[NA,0])
Out[]:
0 1.0
1 NaN
2 2.0
3 NaN
4 0.0
5 3.0
dtype: float64
#参数也可通过字典传递
In []: data.replace({-999:NA,-1000:0})
Out[]:
0 1.0
1 NaN
2 2.0
3 NaN
4 0.0
5 3.0
dtype: float64
七、数据规整：连接、联合与重塑
分值：6分
分层索引
In []: data = pd.Series(np.random.randn(9),^M
...: index=[['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'd'],^M
...: [1, 2, 3, 1, 3, 1, 2, 2, 3]])
In []: data
Out[]:
a 1 -1.218316
2 -0.331598
3 1.511461
b 1 0.443087
3 0.080628
c 1 0.088635
2 -2.549623
d 2 -0.793741
3 -0.266901
dtype: float64
#使用DataFrame的列进行索引
In [109]: frame = pd.DataFrame(np.arange(12).reshape((4, 3)),^M
...: index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
...: columns=[['Ohio', 'Ohio', 'Colorado'],
...: ['Green', 'Red', 'Green']])
In []: frame
Out[]:
Ohio Colorado
Green Red Green
a 1 0 1 2
2 3 4 5
b 1 6 7 8
2 9 10 11
In []: frame.columns.names=['state','color']
In []: frame
Out[]:
state Ohio Colorado
color Green Red Green
key1 key2
a 1 0 1 2
2 3 4 5
b 1 6 7 8
2 9 10 11
In []: frame['Ohio']
Out[]:
color Green Red
key1 key2
a 1 0 1
2 3 4
b 1 6 7
2 9 10
重排序与层级排序
In []: frame.swaplevel('key1', 'key2')
Out[]:
state Ohio Colorado
color Green Red Green
key2 key1
1 a 0 1 2
2 a 3 4 5
1 b 6 7 8
2 b 9 10 11
In []: frame.sort_index(level=1)
Out[]:
state Ohio Colorado
color Green Red Green
key1 key2
a 1 0 1 2
b 1 6 7 8
a 2 3 4 5
b 2 9 10 11
In []: frame.swaplevel(0, 1).sort_index(level=0)
Out[]:
state Ohio Colorado
color Green Red Green
key2 key1
1 a 0 1 2
b 6 7 8
2 a 3 4 5
b 9 10 11
使用DataFrame的列进行索引
In []: frame = pd.DataFrame({'a': range(7), 'b': range(7, 0, -1),
...: 'c': ['one', 'one', 'one', 'two', 'two',
...: 'two', 'two'],
...: 'd': [0, 1, 2, 0, 1, 2, 3]})
In []: frame
Out[]:
a b c d
0 0 7 one 0
1 1 6 one 1
2 2 5 one 2
3 3 4 two 0
4 4 3 two 1
5 5 2 two 2
6 6 1 two 3
In []: frame2=frame.set_index(['c','d'])
In []: frame2
Out[]:
a b
c d
one 0 0 7
1 1 6
2 2 5
two 0 3 4
1 4 3
2 5 2
3 6 1
#默认情况下，设置为索引的列会从DateFrame中移除，但是也可以留在DataFrame中
In []: frame2=frame.set_index(['c','d'],drop=False)
In []: frame2
Out[]:
a b c d
c d
one 0 0 7 one 0
1 1 6 one 1
2 2 5 one 2
two 0 3 4 two 0
1 4 3 two 1
2 5 2 two 2
3 6 1 two 3
八、绘图与可视化
分值：6~10分
In []: import matplotlib.pyplot as plt
In []: import numpy as np
In []: data=np.arange(10)
In []: data
Out[]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In []: plt.plot(data)
Out[]: [<matplotlib.lines.Line2D at 0x283375ca160>]
GOOD LUCK !
author : Haoyu
school : CSUFT
扫一扫
7692
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
