雪球（初步）_雪球组合擂台 json-优快云博客

本文链接：https://blog.youkuaiyun.com/q810935819/article/details/81734981

本文介绍了一种使用Python从雪球网站抓取金融数据的方法，并将其存储到MySQL数据库中。通过构建代理、设置请求头和利用requests库发起GET请求，实现了对特定URL的访问。获取的数据经过JSON解析后，提取了所需字段如uid、text和target，并通过格式化的SQL语句将这些数据插入到名为xueqiu的数据库表中。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import random,json,pymysql
from mysql_test import mysql_conn
import requests

a = -1
count_=10
for i in range(4):
    url='https://xueqiu.com/v4/statuses/public_timeline_by_category.json?since_id=-1&max_id={}&count={}&category=6'.format(a,count_)
    # url='https://xueqiu.com/v4/statuses/public_timeline_by_category.json?since_id=-1&max_id=539084&count=15&category=6'
    # url = 'https://xueqiu.com/v4/statuses/public_timeline_by_category.json?since_id=-1&max_id=539012&count=15&category=6'
    headers={
        "Accept":"*/*",
        # "Accept-Encoding":"gzip, deflate, sdch, br",
        "Accept-Language":"zh-CN,zh;q=0.8",
        "Connection":"keep-alive",
        "Cookie":"aliyungf_tc=AQAAAJ9Du32BtQkAQLLA3TukJ1QYLXH3; xq_a_token=584d0cf8d5a5a9809761f2244d8d272bac729ed4; xq_a_token.sig=x0gT9jm6qnwd-ddLu66T3A8KiVA; xq_r_token=98f278457fc4e1e5eb0846e36a7296e642b8138a; xq_r_token.sig=2Uxv_DgYTcCjz7qx4j570JpNHIs; Hm_lvt_1db88642e346389874251b5a1eded6e3=1534337781,1534337793; Hm_lpvt_1db88642e346389874251b5a1eded6e3=1534337793; _ga=GA1.2.1362703266.1534337782; _gid=GA1.2.1866667702.1534337782; u=591534337824036; device_id=cf2794d80e7c04fc7608f8d911fbf125",
        "Host":"xueqiu.com",
        "Referer":"https://xueqiu.com/",
        "User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4620.400 QQBrowser/9.7.13014.400",
        "X-Requested-With":"XMLHttpRequest",
    }
    # 构建代理
    proxy = {
        'http' : 'http://alice:123456@120.78.166.84:6666',
        'https' : 'http://alice:123456@120.78.166.84:6666'
    }
    # 发起get请求
    response=requests.get(url,headers=headers,proxies=proxy)
    # response.text接收到bytes类型； json.loads转换成字典
    res_dict = json.loads(response.text)
    # 提取下一次循环的max_id 重新赋值
    a=res_dict['next_max_id']
    # 重新赋值获取的数据条数
    count_=15
    for i in res_dict['list']:

        data_str = i['data']
        # 将字符串转为字典
        data_dict=json.loads(data_str)
        # 根据键提取值
        uid=data_dict['id']
        text=data_dict['text']
        target=data_dict['target']
        # print(data_dict['id'],data_dict['text'],data_dict['target'])
        # 存入数据库语句
        sql = "insert into xueqiu(uid,text,target) values ({uid},'{text}','{target}')".format(uid=uid,text=text,target=target)

        mc = mysql_conn()
        mc.execute_modify_mysql(sql)