pandas在web开发中的使用场景：删除文章的重复评论、统计某个用户的评论数量

id,news_id,user_name,content,create_at
1,101,zhangsan,垃圾,2017-12-07 15:44:44
2,102,lisi,很好,2017-12-07 15:44:44
3,102,zhangsan,好文章,2017-12-07 15:44:44
4,103,wangwu,真不错,2017-12-07 15:44:44
5,104,zhangsan,垃圾,2017-12-07 15:44:44
6,105,lisi,垃圾,2017-12-07 15:44:44
7,101,zhangsan,垃圾,2017-12-07 15:44:44
8,109,lily,不错哦,2017-12-07 15:44:44
9,101,zhangsan,垃圾,2017-12-07 15:44:44
10,102,zhaoliu,什么玩意儿,2017-12-07 15:44:44

我们有一个csv文件，内容如上，可以理解为网站用户评论表。
肉眼就可以看出，有某个用户针对某个新闻评论了相同的内容。

找出重复数据

# coding: utf-8
import pandas as pd

reviews = pd.read_csv("./csv/reviews.csv", dtype={"id": pd.np.str_}, index_col="id")
print(type(reviews)) #<class 'pandas.core.frame.DataFrame'>

# 找重复
# "news_id","user_name","content" 这3个字段数据重复
ret = reviews.duplicated(subset=["news_id","user_name","content"])
print(ret)

打印结果如下：

id
1     False
2     False
3     False
4     False
5     False
6     False
7      True
8     False
9      True
10    False
dtype: bool

id为7和9的为True，意思是他俩是重复的，实际我们知道id为1的数据和他俩也是相同的。

# keep=False 只要是重复的都拿出来
ret = reviews.duplicated(subset=["news_id","user_name","content"], keep=False)

print(reviews[ret]) # 打印 这些内容
"""
    news_id user_name content            create_at
id                                                
1       101  zhangsan      垃圾  2017-12-07 15:44:44
7       101  zhangsan      垃圾  2017-12-07 15:44:44
9       101  zhangsan      垃圾  2017-12-07 15:44:44
"""

找出重复内容的id：

# coding: utf-8
import pandas as pd

reviews = pd.read_csv("./csv/reviews.csv")

# 取出重复的内容
duplicate_reviews = reviews[reviews.duplicated(subset=["news_id","user_name","content"], keep=False)]
# 取出重复内容的id
ids = duplicate_reviews['id']

# 用逗号拼接id
print(",".join(str(id) for id in ids)) # 1,7,9

1,7,9就是我们需要的重复记录的id，有了这个id字符串就可以组装sql去删除数据表中的重复记录了。

统计某个用户的评论数量

# coding: utf-8
import pandas as pd

reviews = pd.read_csv("./csv/reviews.csv")

# 按用户分组 统计数量
result  = reviews.groupby("user_name").size()
print(result)
"""
user_name
lily        1
lisi        2
wangwu      1
zhangsan    5
zhaoliu     1
dtype: int64
"""