京东爬虫，数据转为elasticsearch以及csv文件

qq_20546587

于 2024-04-23 19:57:37 发布

阅读量438

点赞数 2

文章标签：爬虫 elasticsearch 大数据

本文链接：https://blog.youkuaiyun.com/qq_20546587/article/details/138136892

版权

本文介绍了如何使用Python的requests、parsel库以及Elasticsearch进行网络请求，从京东搜索结果中抓取商品名称、价格、店铺和链接，并将数据存储到Elasticsearch中以备后续分析。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import requests
import parsel
import csv
from elasticsearch import Elasticsearch

es = Elasticsearch(["https://localhost:9200"], basic_auth=('elastic', 'hw-FxQTckOuswZ1jRikA'), verify_certs=False,
                   request_timeout=3600)
print(es)
INDEX_NAME = "jd-produict-list"
mappings = {
    "mappings": {

        "properties": {
            "name": {
                "type": "text"
            },
            "price": {
                "type": "text"
            },
            "link": {
                "type": "text",
                "index": "false"
            },
            "shop": {
                "type": "text",
            }
        }

    }
}
res1 = es.options(ignore_status=404).indices.delete(index=INDEX_NAME)
res = es.indices.create(index=INDEX_NAME, body=mappings)
print(res)
url = ("https://api.m.jd.com/?appid=search-pc-java&functionId=pc_search_s_new&client=pc&clientVersion=1.0.0&t"
       "=1713168142494&body=%7B%22keyword%22%3A%22java%22%2C%22pvid%22%3A%22555ce3e0d8264f2d91d2a39ce764254d%22%2C"
       "%22page%22%

最低0.47元/天解锁文章