19092501es scroll-scan的python滚动翻页

最新推荐文章于 2024-01-11 11:20:31 发布

原创最新推荐文章于 2024-01-11 11:20:31 发布 · 309 阅读

1 ·

CC 4.0 BY-SA版权

Python 专栏收录该内容

5 篇文章

订阅专栏

本文介绍了一种利用Elasticsearch进行大规模数据检索的方法，通过设置特定的查询条件，实现对某一店铺所有商品信息的批量获取。示例代码展示了如何配置搜索参数、执行滚动搜索以及处理返回结果。

from elasticsearch import Elasticsearch
es = Elasticsearch([
        'http://conf:9200/'
    ])
page = es.search(
    index ='ware_index',
    doc_type ='ware_type',
    scroll ='2m',
    search_type ='scan',
    size =1000,
    body ={
        "query":{
            "match_all":{}
        },
        "filter":{
            "term":{
                "shop":"230"
            }
        }
    }
)

sid = page['_scroll_id']
scroll_size = page['hits']['total']
 
# Start scrolling
while(scroll_size >0):
    print "Scrolling..."
    page = es.scroll(scroll_id = sid, scroll ='2m')
    # Update the scroll ID
    sid = page['_scroll_id']
    # Get the number of results that we returned in the last scroll
    scroll_size = len(page['hits']['hits'])
    print "scroll size: "+ str(scroll_size)
    # Do something with the obtained page