Java实现es的scroll滚动查询

原创已于 2022-03-23 17:52:09 修改 · 8.9k 阅读

13 ·

CC 4.0 BY-SA版权

文章标签：

#java #elasticsearch #jar #大数据

于 2021-11-30 15:11:47 首次发布

es 专栏收录该内容

9 篇文章

订阅专栏

这篇博客介绍了如何在Java中使用Elasticsearch的滚动查询（Scroll API）来遍历大量数据，同时提供了清除滚动ID的方法，以防止内存溢出。示例代码展示了如何设置滚动查询、获取搜索结果并逐条处理，以及在达到一定条件后断开滚动并清除滚动ID。

    public void selectData() throws IOException {
        int i=1,size=2;
        SearchRequest searchRequest = new SearchRequest("index");
        Scroll scroll = new Scroll(TimeValue.timeValueMinutes(5L));
        searchRequest.scroll(scroll);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        MatchAllQueryBuilder matchAllQueryBuilder = QueryBuilders.matchAllQuery();
        searchSourceBuilder.query(matchAllQueryBuilder);
        searchSourceBuilder.size(size);

        searchRequest.source(searchSourceBuilder);
        SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
        String scrollId = response.getScrollId();
        SearchHit[] searchHits = response.getHits().getHits();

        System.out.println(response.getHits().getTotalHits());
        for (SearchHit searchHit : searchHits) {
            System.out.println(searchHit.getSourceAsString());
        }
        while (searchHits != null && searchHits.length > 0) {
            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(scroll);
            response = client.scroll(scrollRequest, RequestOptions.DEFAULT);
            scrollId = response.getScrollId();
            searchHits = response.getHits().getHits();

            for (SearchHit searchHit : searchHits) {
                i++;
                System.out.println(searchHit.getSourceAsString());
            }
            if (i > 10) {
                break;
            }
        }
    }

scroll滚动查询，es默认是存储500条scroll_id，如果超过500条继续使用滚动查询时，会报错，下面代码是对screll_id进行删除操作。

public static boolean clearScrollIds(RestHighLevelClient client,String... scrollIds){
    List<String> sIds = new ArrayList<>();
    for (String scrollId : scrollIds) {
        sIds.add(scrollId);
    }
    ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
    //添加单个id
    clearScrollRequest.addScrollId("滚动id");
    //添加多个id
    clearScrollRequest.setScrollIds(sIds);
    try {
        client.clearScroll(clearScrollRequest,RequestOptions.DEFAULT);
        return true;
    } catch (IOException e) {
        return false;
    }

}