爬取孔夫子旧书网的评论

这次带来的是爬取孔夫子旧书网的书店评论，并写入txt文档和数据库

这个用到了json模块
json模块是格式转换模块，json是为了将爬取下来的评论网页解析成Python方便处理的字典格式。json有.load()和.loads()，dump()和dumps()这几种方法。

• loads()是将json格式转换成Python方便处理的字典格式。
• dumps()方法是将Python的字典格式转换成json格式。

还有就是评论信息在浏览器的检查元素的Network中，是 js或者 xhr格式

完整代码展示：

#-*- coding:utf-8 -*-
from bs4 import BeautifulSoup
import requests
import json
import MySQLdb
#解决出现的写入错误
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

print('连接到mysql服务器...')
conn = MySQLdb.connect(host='127.0.0.1', user='root', passwd='123mysql', db='onefive',charset='utf8')
print('连接上了!')
cur = conn.cursor()
sql = """CREATE TABLE comment(
                         store  CHAR(10),
                         book  CHAR(60),
                         comment  CHAR(100),
                         time  CHAR(20),
                         reviewer  CHAR(20))"""
cur.execute(sql)
conn.commit()
url1='http://book.kongfz.com/256332/935885648/'
user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134'
headers = {'User-Agent': user_agent}
html1 = requests.get(url1,headers=headers).content
soup = BeautifulSoup(html1,'html.parser')
article = soup.find('div',attrs={'class':'main-box'})
#书店名
store = article.find('div',attrs={'class':'shop_top_title'}).get_text()
#print title
f=open('comment2.txt','a')
f.write(store)
f.close()
cur.execute("INSERT INTO comment(store) VALUES ('%s');" % (store))
conn.commit()
#评论的相关信息
for page in range(1,3):
    url2 = 'http://book.kongfz.com/Pc/Ajax/getShopReviewList?userId=1710684&itemId=&page=' + str(page) + '&needEmpty=0&rating=all'
    html2 = requests.get(url2, headers=headers).content
    b = json.loads(html2)
    dic = b['result']
    for k in dic['reviewList']:
        book = k['itemName'].encode('UTF-8')
        print book
        comment = k['content'].encode('UTF-8')
        print comment
        time = k['reviewTime'].encode('UTF-8')
        print time
        reviewer = k['appraiserNickname'].encode('UTF-8')
        print reviewer
        end = book + '\n' + comment + '\n' + time + '\n' + reviewer + '\n'
        f=open('comment2.txt','a')
        f.write(end)
        f.close()
        cur.execute("INSERT INTO comment(book,comment,time,reviewer) VALUES ('%s', '%s', '%s','%s');" % (book,comment,time,reviewer))
        conn.commit()
conn.close()