Java自动创建ES索引，自动滚动ES索引（Easy-Es版）

原创 已于 2024-10-25 09:55:32 修改 · 898 阅读 · 5 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#1024程序员节 #java #elasticsearch

于 2024-10-24 18:34:10 首次发布

一、前言

        本文旨在解决项目中ElasticSearch的索引自动化问题，集中管理索引模板，自动滚动索引。搭配Easy-Es开源框架，提供很大的便利性，当然，不用Easy-Es也有实现的方式。

二、引入依赖

<!-- elasticsearch -->
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.14.0</version>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>7.14.0</version>
</dependency>

<!-- easy-es -->
<dependency>
    <groupId>org.dromara.easy-es</groupId>
    <artifactId>easy-es-boot-starter</artifactId>
    <version>2.0.0-beta3</version>
</dependency>

三、EasyES配置

easy-es:
  enable: true
  address: 127.0.0.1:9200
  username: admin
  password: admin

四、代码示例

4.1 初始化配置类

核心配置类，用于初始化系统索引，检查索引版本，滚动索引，与EasyES开关联动，可设置 easy-es.enable 为 false 关闭改功能。

package com.mory.framework.es.config;

import cn.hutool.core.date.DateUtil;
import cn.hutool.core.thread.ThreadUtil;
import cn.hutool.core.util.ObjectUtil;
import com.alibaba.fastjson2.JSONObject;
import com.mory.framework.es.contants.ESConstants;
import com.mory.framework.es.contants.ESIndexConstants;
import com.mory.framework.es.domain.MoryMetaIndex;
import com.mory.framework.es.domain.MoryVersionIndex;
import com.mory.framework.es.mapper.MoryMetaIndexMapper;
import com.mory.framework.es.mapper.MoryVersionIndexMapper;
import com.mory.framework.es.monitor.ESMonitor;
import com.mory.framework.es.service.EsManageService;
import com.mory.framework.es.service.impl.EsManageServiceImpl;
import com.mory.framework.es.task.EsIndexListTask;
import com.mory.framework.es.utils.IndexUtil;
import lombok.extern.slf4j.Slf4j;
import org.dromara.easyes.core.conditions.select.LambdaEsQueryWrapper;
import org.elasticsearch.client.RestHighLevelClient;
import org.redisson.api.RedissonClient;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;

import javax.annotation.Resource;
import java.io.IOException;
import java.util.Date;
import java.util.List;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.Lock;

/**
 * @author mory
 * @date 2024-07-22 15:06
 */
@Configuration
@Slf4j
@ConditionalOnClass({RestHighLevelClient.class})
@ConditionalOnExpression("'${easy-es.address:x}'!='x'")
@ConditionalOnProperty(
        prefix = "easy-es",
        name = {"enable"},
        havingValue = "true",
        matchIfMissing = true
)
public class EsInitializeConfig {

    @Resource
    private MoryMetaIndexMapper metaIndexMapper;

    @Resource
    private MoryVersionIndexMapper versionIndexMapper;

    @Resource
    private RedissonClient redissonClient;

    private final AtomicBoolean completedLatch = new AtomicBoolean(false);

    @Bean
    @DependsOn("restHighLevelClient")
    public EsManageService esManagerApi(RestHighLevelClient restHighLevelClient) {
        return new EsManageServiceImpl(restHighLevelClient);
    }

    @Bean
    public ESMonitor esMonitor(EsManageService esManageService) {
        ScheduledThreadPoolExecutor executor = ThreadUtil.createScheduledExecutor(1);
        ESMonitor esMonitor = new ESMonitor(esManageService);
        EsIndexListTask esIndexListTask = new EsIndexListTask(esManageService);
		
		// 每10秒检查一遍ES的状态
        executor.scheduleAtFixedRate(esMonitor, 0, 10, TimeUnit.SECONDS);
		// 每30秒拉取一遍当前所有的索引
        executor.scheduleAtFixedRate(esIndexListTask, 0, 30, TimeUnit.SECONDS);
		// 避免因ES状态异常，导致索引创建或滚动失败，定时做检查
        executor.scheduleAtFixedRate(() -> {
            if (!completedLatch.get() && !ESMonitor.currentColor().equals(ESConstants.ES_HEALTH_RED)) {
                Lock lock = redissonClient.getLock("initEsIndexLock");
                try {
                    lock.lock();
                    try {
						// 遍历所有索引模板，创建索引模板和生命周期
                        for (int i = 0; i < IndexUtil.getTemplates().size(); i++) {
                            JSONObject template = IndexUtil.getTemplates().getJSONObject(i);
                            esManageService.lowLevelPutIndexTemplate(template);
                            esManageService.lowLevelPutILMPolicy(template);
                        }
                        esManageService.initPollInterval();
						// 创建索引信息index
                        if (!metaIndexMapper.existsIndex(ESIndexConstants.MORY_META_INDEX)) {
                            metaIndexMapper.createIndex(ESIndexConstants.MORY_META_INDEX);
                        }
						// 创建索引版本index
                        if (!versionIndexMapper.existsIndex(ESIndexConstants.MORY_VERSION_INDEX)) {
                            versionIndexMapper.createIndex(ESIndexConstants.MORY_VERSION_INDEX);
                        }
						// 再次遍历所有索引模板，创建/滚动索引
                        for (int i = 0; i < IndexUtil.getTemplates().size(); i++) {
                            JSONObject template = IndexUtil.getTemplates().getJSONObject(i);
                            if (!createIndex(esManageService, template)) {
                                rolloverEsTemplate(esManageService, template);
                            }
                        }
                        completedLatch.set(true);
                    } catch (Exception e) {
                        log.error(e.getMessage(), e);
                    }
                } finally {
                    lock.unlock();
                }
            }
        }, 0, 10, TimeUnit.SECONDS);

        return esMonitor;
    }

    private boolean createIndex(EsManageService esManageService, JSONObject template) throws IOException {
        JSONObject set = template.getJSONObject(ESConstants.KEY_SET);
        String indexName = set.getString(ESConstants.INDEX_NAME);
        List<MoryMetaIndex> metaIndices = metaIndexMapper.selectList(new LambdaEsQueryWrapper<MoryMetaIndex>()
                .eq(MoryMetaIndex::getIndexName, indexName));
        if (ObjectUtil.isEmpty(metaIndices)) {
            // 初次创建索引记录索引记录
            esManageService.createIndex(indexName, set.getString(ESConstants.KEY_ALIAS));
            MoryMetaIndex metaIndex = new MoryMetaIndex();
            metaIndex.setIndexName(indexName);
            metaIndex.setCreateTime(DateUtil.formatDateTime(new Date()));
            metaIndex.setStatus("create");
            metaIndexMapper.insert(metaIndex);

            // 初次创建索引记录索引版本
            MoryVersionIndex versionIndex = new MoryVersionIndex();
            versionIndex.setIndexName(indexName);
            versionIndex.setVersion(String.valueOf(template.getJSONObject(ESConstants.KEY_TEMPLATE).toJSONString().hashCode()));
            versionIndexMapper.insert(versionIndex);

            return true;
        }
        return false;
    }

    private void rolloverEsTemplate(EsManageService esManageService, JSONObject template) {
        JSONObject set = template.getJSONObject(ESConstants.KEY_SET);
        String indexName = set.getString(ESConstants.INDEX_NAME);
        String version = String.valueOf(template.getJSONObject(ESConstants.KEY_TEMPLATE).toJSONString().hashCode());
        List<MoryVersionIndex> versionIndices = versionIndexMapper.selectList(new LambdaEsQueryWrapper<MoryVersionIndex>()
                .eq(MoryVersionIndex::getIndexName, indexName)
                .eq(MoryVersionIndex::getVersion, version));
		// 判断索引模板是否变动，是则滚动索引
        if (ObjectUtil.isEmpty(versionIndices)) {
            MoryVersionIndex versionIndex = new MoryVersionIndex();
            versionIndex.setIndexName(indexName);
            versionIndex.setVersion(version);
            versionIndexMapper.insert(versionIndex);
            esManageService.rolloverIndex(set.getString(ESConstants.KEY_ALIAS));
        }
    }

}

4.2 常量池

package com.mory.framework.es.contants;

/**
 * @author mory
 * @date 2024-07-19 10:48
 */
public interface ESConstants {

    /**
     * ES健康值 RED
     */
    String ES_HEALTH_RED = "red";
    /**
     * ES健康值 YELLOW
     */
    String ES_HEALTH_YELLOW = "yellow";
    /**
     * ES健康值 GREEN
     */
    String ES_HEALTH_GREEN = "green";

    String KEY_ILM_POLICY = "ilmPolicy";

    String KEY_SET = "set";

    String KEY_SETTING = "settings";

    String KEY_INDEX = "index";

    String KEY_ALIASES = "aliases";

    String KEY_ALIAS = "alias";

    String KEY_MAPPING = "mappings";

    String METHOD_PUT = "PUT";

    String INDEX_NAME = "indexName";

    String KEY_TEMPLATE = "template";

    String KEY_INDEX_PATTERNS = "index_patterns";

    String TEMPLATE_PREFIX = "_template";

    String POLICY_PREFIX = "_ilm/policy";

    String URL_SEPARATE = "/";

    String TEMPLATE_SUFFIX = "-template";

    String POLICY_SUFFIX = "-policy";

    String ALIAS_SUFFIX = "-alias";

    int PUT_LIST_MAX_SIZE = 5000;

    String ILM_INDEX_EXPRESSION = "<${indexName}-{now/m{yyyy.MM.dd.HH.mm|Asia/Shanghai}}-000001>";

    String ILM_INDEX_PATTERN = "^.*-(\\d{4}(\\.\\d{2}){4})-(\\d+)$";

    String LIFECYCLE_NAME = "lifecycle.name";

    String LIFECYCLE_ROLLOVER_ALIAS = "lifecycle.rollover_alias";

}

package com.mory.framework.es.contants;

/**
 * @author mory
 * @date 2024-07-22 17:21
 */
public interface ESIndexConstants {

    String MORY_META_INDEX = "mory-meta-index";

    String MORY_VERSION_INDEX = "mory-version-index";

    String MORY_DEMO_ALIAS = "mory-demo-alias";

}

4.3 ES健康监听器

package com.mory.framework.es.monitor;

import com.mory.framework.es.contants.ESConstants;
import com.mory.framework.es.service.EsManageService;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;

/**
 * @author mory
 * @date 2024-07-22 15:14
 */
@Data
@Slf4j
public class ESMonitor implements Runnable {

    private static String esColor = ESConstants.ES_HEALTH_GREEN;

    private EsManageService esManageService;

    public ESMonitor(EsManageService esManageService) {
        this.esManageService = esManageService;
    }

    @Override
    public void run() {
        try {
            ClusterHealthResponse clusterHealth = esManageService.getClusterHealth();
            esColor = clusterHealth.getStatus().name();
        } catch (Exception e) {
            // ignore
        }
    }

    public static String currentColor(){
        return esColor;
    }

}

4.4 获取ES所有索引任务

package com.mory.framework.es.task;

import com.mory.framework.es.service.EsManageService;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * 定时获取ES全部索引
 *
 * @author mory
 * @date 2024-07-22 15:23
 */
@Data
@Slf4j
public class EsIndexListTask implements Runnable {

    private static Map<String, List<String>> aliasIndicesMap = new HashMap<>();

    private EsManageService esManageService;

    public EsIndexListTask(EsManageService esManageService) {
        this.esManageService = esManageService;
    }

    @Override
    public void run() {
        aliasIndicesMap = esManageService.groupIndexByAlias();
    }

    public static List<String> getIndicesByAlias(String alias) {
        return aliasIndicesMap.getOrDefault(alias, new ArrayList<>());
    }

}

4.5 工具类

比较核心的工具类，通过解析实体类，生成索引模板

package com.mory.framework.es.utils;

import cn.hutool.core.date.DatePattern;
import cn.hutool.core.date.DateUtil;
import cn.hutool.core.util.ClassUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil;
import com.alibaba.fastjson2.JSONArray;
import com.alibaba.fastjson2.JSONObject;
import com.alibaba.fastjson2.JSON;
import com.mory.common.exception.ServiceException;
import com.mory.framework.es.contants.ESConstants;
import com.mory.framework.es.domain.IndexConfig;
import com.mory.framework.es.task.EsIndexListTask;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.dromara.easyes.annotation.IndexName;
import org.dromara.easyes.core.biz.CreateIndexParam;
import org.dromara.easyes.core.biz.EntityInfo;
import org.dromara.easyes.core.toolkit.EntityInfoHelper;
import org.dromara.easyes.core.toolkit.IndexUtils;

import java.lang.reflect.InvocationTargetException;
import java.util.*;
import java.util.regex.Pattern;
import java.util.stream.Collectors;

/**
 * @author mory
 * @date 2024-07-19 10:57
 */
@Slf4j
public class IndexUtil {

    private static final List<String> TEMPLATE_PACKAGES = Arrays.asList(
            "com.mory.framework.es.domain"
    );

    private static final Pattern INDEX_PATTERN = Pattern.compile(ESConstants.ILM_INDEX_PATTERN);

    @Getter
    private static final JSONArray templates = new JSONArray();

    static {
        try {
            loadTemplates();
        } catch (InstantiationException | IllegalAccessException | InvocationTargetException e) {
            log.error("加载ES索引模板失败: {}", e.getMessage());
        }
    }

    /**
     * 根据别名获取索引数组
     *
     * @param alias 索引别名
     * @param startTime 搜索开始时间
     * @param endTime 搜索结束时间
     * @return 索引数组
     */
    public static String[] getIndexByAlias(String alias, String startTime, String endTime) {
        return getIndexByAlias(alias, DateUtil.parse(startTime, DatePattern.NORM_DATETIME_FORMAT), DateUtil.parse(endTime, DatePattern.NORM_DATETIME_FORMAT));
    }

    /**
     * 根据别名获取索引数组
     *
     * @param alias 索引别名
     * @param startTime 搜索开始时间
     * @param endTime 搜索结束时间
     * @return 索引数组
     */
    public static String[] getIndexByAlias(String alias, Date startTime, Date endTime) {
        List<String> hitIndex = new ArrayList<>();
        if (StringUtils.isNotEmpty(alias) && startTime != null && endTime != null) {
            long searchStartMs = startTime.getTime();
            long searchEndMs = endTime.getTime();

            if (searchStartMs > searchEndMs) {
                throw new ServiceException("开始时间不能大于结束时间");
            }

            String[] aliases = alias.split(",");
            for (String _alias: aliases) {
                List<String> indices = EsIndexListTask.getIndicesByAlias(_alias);
                if (ObjectUtil.isEmpty(indices)) {
                    continue;
                }
                List<String> indicesOfAlias = indices.stream().filter(indexName -> INDEX_PATTERN.matcher(indexName).matches()).sorted(IndexUtil::sortByAsc).collect(Collectors.toList());
                if (indicesOfAlias.isEmpty()) {
                    continue;
                } else if (indicesOfAlias.size() == 1) {
                    hitIndex.add(indicesOfAlias.get(0));
                    continue;
                }

                int start = 0;
                int end;
                for (int i = 0; i < indicesOfAlias.size(); i++) {
                    long timeMs = indexDatetimeMs(indicesOfAlias.get(i));
                    if (searchStartMs < timeMs) {
                        if (i == 0) {
                            start = i - 1;
                        }
                        break;
                    } else {
                        start++;
                    }
                }
                if (start == indicesOfAlias.size()) {
                    start--;
                }

                end = findEndIndex(0, indicesOfAlias, searchEndMs);
                if (start == -1 && end == -1) {
                    hitIndex.add(indicesOfAlias.get(0));
                    continue;
                }
                if (start < 0) {
                    start = 0;
                }
                if (end < 0) {
                    end = 0;
                }
                if (start == end) {
                    hitIndex.add(indicesOfAlias.get(start));
                } else {
                    for (int i = start; i <= end; i++) {
                        hitIndex.add(indicesOfAlias.get(i));
                    }
                }
            }
        }

        return hitIndex.toArray(new String[]{});
    }

    /**
     * 索引顺序排序
     *
     * @param a1 索引1
     * @param a2 索引2
     * @return 比较结果
     */
    public static int sortByAsc(String a1, String a2) {
        String[] a1Arr = a1.split("-");
        String[] a2Arr = a2.split("-");
        int a1SortNumber = Integer.parseInt(a1Arr[a1Arr.length - 1]);
        int a2SortNumber = Integer.parseInt(a2Arr[a2Arr.length - 1]);
        return Long.valueOf(a1SortNumber - a2SortNumber).intValue();
    }

    /**
     * 获取索引时间戳
     *
     * @param indexName 索引名，需符合EsConstants.ILM_INDEX_PATTERN
     * @return 时间戳
     */
    private static long indexDatetimeMs(String indexName) {
        String[] indexArr = indexName.split("-");
        String indexDatetimeStr = indexArr[indexArr.length - 2];
        return DateUtil.parse(indexDatetimeStr, "yyyy.MM.dd.HH.mm").getTime();
    }

    /**
     * 查找符合时间范围的索引下标
     *
     * @param index 下标
     * @param indices 索引集合
     * @param endMs 搜索结束时间戳
     * @return 找到的下标
     */
    private static int findEndIndex(int index, List<String> indices, long endMs) {
        long timeMs = indexDatetimeMs(indices.get(index));
        if (endMs < timeMs) {
            return index - 1;
        } else {
            if (index == indices.size() - 1) {
                return index;
            }
            return findEndIndex(index + 1, indices, endMs);
        }
    }

    /**
     * 加载索引模板
     *
     * @throws InstantiationException
     * @throws IllegalAccessException
     * @throws InvocationTargetException
     */
    private static void loadTemplates() throws InstantiationException, IllegalAccessException, InvocationTargetException {
        for (String templatePackage: TEMPLATE_PACKAGES) {
            Set<Class<?>> classes = ClassUtil.scanPackageBySuper(templatePackage, IndexConfig.class);
            for (Class<?> clazz: classes) {
                IndexName annotation = clazz.getAnnotation(IndexName.class);
                if (!ClassUtil.isAbstract(clazz) && annotation != null) {
                    templates.add(genTemplate(clazz));
                }
            }
        }
    }

    /**
     * 生成索引模板
     *
     * @param clazz 索引实体类Class
     * @return 索引模板
     * @throws InstantiationException
     * @throws IllegalAccessException
     * @throws InvocationTargetException
     */
    private static JSONObject genTemplate(Class<?> clazz) throws InstantiationException, IllegalAccessException, InvocationTargetException {
        JSONObject result = new JSONObject();
        JSONObject set = new JSONObject();
        JSONObject template = new JSONObject();

        IndexName annotation = clazz.getAnnotation(IndexName.class);
        String indexName = ObjectUtil.isEmpty(annotation.value()) ? StrUtil.toUnderlineCase(clazz.getSimpleName()).toLowerCase().replace("_", "-") : annotation.value();
        String indexAlias = annotation.aliasName();
        if (indexName.endsWith(ESConstants.ALIAS_SUFFIX)) { // 如果实体索引名为别名
            indexAlias = ObjectUtil.isEmpty(indexAlias) || "ee_default_alias".equals(indexAlias) ? indexName : indexAlias;
            indexName = indexName.substring(0, indexName.length() - ESConstants.ALIAS_SUFFIX.length());
        } else {
            indexAlias = ObjectUtil.isEmpty(indexAlias) || "ee_default_alias".equals(indexAlias) ? indexName.concat(ESConstants.ALIAS_SUFFIX) : indexAlias;
        }
        String indexPolicy = indexName.concat(ESConstants.POLICY_SUFFIX);

        set.put(ESConstants.INDEX_NAME, indexName);
        set.put(ESConstants.KEY_ALIAS, indexAlias);

        Object instance = clazz.newInstance();
        JSONObject impPolicy = (JSONObject) ClassUtil.getPublicMethod(clazz, "ilmPolicy").invoke(clazz.newInstance());
        String indexPatterns = (String) ClassUtil.getPublicMethod(clazz, "indexPatterns").invoke(instance);
        indexPatterns = "-*".equals(indexPatterns) ? indexName.concat(indexPatterns) : indexPatterns;
        JSONObject setting = (JSONObject) ClassUtil.getPublicMethod(clazz, "settings").invoke(instance);
        JSONObject mappingsMerge = (JSONObject) ClassUtil.getPublicMethod(clazz, "mappingsMerge").invoke(instance);

        if (StrUtil.isEmpty(setting.getJSONObject(ESConstants.KEY_INDEX).getString(ESConstants.LIFECYCLE_NAME))) {
            setting.getJSONObject(ESConstants.KEY_INDEX).put(ESConstants.LIFECYCLE_NAME, indexPolicy);
        }
        if (StrUtil.isEmpty(setting.getJSONObject(ESConstants.KEY_INDEX).getString(ESConstants.LIFECYCLE_ROLLOVER_ALIAS))) {
            setting.getJSONObject(ESConstants.KEY_INDEX).put(ESConstants.LIFECYCLE_ROLLOVER_ALIAS, indexAlias);
        }

        EntityInfo entityInfo = EntityInfoHelper.getEntityInfo(clazz);
        CreateIndexParam createIndexParam = IndexUtils.getCreateIndexParam(entityInfo);
        Map<String, Object> indexMappings = IndexUtils.initMapping(entityInfo, createIndexParam.getEsIndexParamList());
        JSONObject mappings = JSONObject.parseObject(JSON.toJSONString(indexMappings));
        mappings.putAll(mappingsMerge);

        template.put(ESConstants.KEY_INDEX_PATTERNS, indexPatterns);
        template.put(ESConstants.KEY_SETTING, setting);
        template.put(ESConstants.KEY_MAPPING, mappings);

        result.put(ESConstants.KEY_SET, set);
        result.put(ESConstants.KEY_ILM_POLICY, impPolicy);
        result.put(ESConstants.KEY_TEMPLATE, template);

        return result;
    }

}

4.6 核心服务

本片用到的ES的操作接口和实现

package com.mory.framework.es.service;

import com.alibaba.fastjson2.JSONObject;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;

import java.io.IOException;
import java.util.List;
import java.util.Map;

/**
 * @author mory
 * @date 2024-07-19 10:26
 */
public interface EsManageService {

    ClusterHealthResponse getClusterHealth() throws IOException;

    void lowLevelPutIndexTemplate(JSONObject template) throws IOException;

    void lowLevelPutILMPolicy(JSONObject template) throws IOException;

    void initPollInterval() throws IOException;

    boolean isIndexExist(String index) throws IOException;

    void createIndex(String indexName, String alias) throws IOException;

    void rolloverIndex(String alias);

    Map<String, List<String>> groupIndexByAlias();

    <T> String addOne(T entity, String indexName) throws IOException;

}


package com.mory.framework.es.service.impl;

import com.mory.framework.es.contants.ESConstants;
import com.mory.framework.es.service.EsManageService;
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest;
import org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsResponse;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;

import cn.hutool.core.io.IoUtil;
import cn.hutool.core.util.ObjectUtil;
import com.alibaba.fastjson2.JSONObject;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
import org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.client.*;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.cluster.metadata.AliasMetadata;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.core.TimeValue;

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.stream.Collectors;

/**
 * @author mory
 * @date 2024-07-19 10:26
 */
@Slf4j
public class EsManageServiceImpl implements EsManageService {

    private final RestHighLevelClient highLevelClient;

    public EsManageServiceImpl(RestHighLevelClient highLevelClient) {
        this.highLevelClient = highLevelClient;
    }

    /**
     * 获取ES状况
     *
     * @return ClusterHealthResponse
     * @throws IOException
     */
    @Override
    public ClusterHealthResponse getClusterHealth() throws IOException {
        ClusterHealthRequest request = new ClusterHealthRequest();
        request.timeout(TimeValue.timeValueSeconds(50)).local(true);
        return highLevelClient.cluster().health(request, RequestOptions.DEFAULT);
    }

    /**
     * 更新索引模板
     * @param template 索引模板
     * @throws IOException
     */
    @Override
    public void lowLevelPutIndexTemplate(JSONObject template) throws IOException {
        JSONObject templateBody = template.getJSONObject(ESConstants.KEY_TEMPLATE);
        String indexName = template.getJSONObject(ESConstants.KEY_SET).getString(ESConstants.INDEX_NAME);
        Request templateRequest = new Request(ESConstants.METHOD_PUT, ESConstants.TEMPLATE_PREFIX + ESConstants.URL_SEPARATE + indexName + ESConstants.TEMPLATE_SUFFIX);
        templateRequest.setJsonEntity(templateBody.toJSONString());
        RestClient lowLevelClient = highLevelClient.getLowLevelClient();
        lowLevelClient.performRequest(templateRequest);
    }

    /**
     * 更新ILM策略
     *
     * @param template 索引模板
     * @throws IOException
     */
    @Override
    public void lowLevelPutILMPolicy(JSONObject template) throws IOException {
        String policy = template.getString(ESConstants.KEY_ILM_POLICY);
        String indexName = template.getJSONObject(ESConstants.KEY_SET).getString(ESConstants.INDEX_NAME);
        Request policyRequest = new Request(ESConstants.METHOD_PUT, ESConstants.POLICY_PREFIX + ESConstants.URL_SEPARATE + indexName + ESConstants.POLICY_SUFFIX);
        policyRequest.setJsonEntity(policy);
        RestClient lowLevelClient = highLevelClient.getLowLevelClient();
        lowLevelClient.performRequest(policyRequest);
    }

    /**
     * 设置ILM定期运行时间
     *
     * @throws IOException
     */
    @Override
    public void initPollInterval() throws IOException {
        ClusterUpdateSettingsRequest clusterUpdateSettingsRequest = new ClusterUpdateSettingsRequest();
        Map<String, Object> setting = new HashMap<>();
        setting.put("indices.lifecycle.poll_interval", "5m");
        clusterUpdateSettingsRequest.persistentSettings(setting);
        ClusterUpdateSettingsResponse response = highLevelClient.cluster().putSettings(clusterUpdateSettingsRequest, RequestOptions.DEFAULT);
        boolean acknowledged = response.isAcknowledged();
        if (!acknowledged) {
            log.error("ILM 索引滚动刷新时间设置错误，请检查ES cluster 设置信息");
        }
    }

    /**
     * 判断索引是否存在
     * @param index 索引
     * @return true-存在；false-不存在
     * @throws IOException
     */
    @Override
    public boolean isIndexExist(String index) throws IOException {
        GetIndexRequest request = new GetIndexRequest(index);
        return highLevelClient.indices().exists(request, RequestOptions.DEFAULT);
    }

    /**
     * 创建索引
     * @param indexName 索引名
     * @param alias 索引别名
     * @throws IOException
     */
    @Override
    public void createIndex(String indexName, String alias) throws IOException {
        String expression = ESConstants.ILM_INDEX_EXPRESSION.replace("${indexName}", indexName);
        CreateIndexRequest request = new CreateIndexRequest(expression);
        Map<String, Object> aliasMap = new HashMap<>();
        aliasMap.put("aliases", new HashMap<String, Map<String, Object>>() {{
            put(alias, new HashMap<String, Object>() {{
                put("is_write_index", true);
            }});
        }});
        request.source(JSONObject.toJSONString(aliasMap), XContentType.JSON);
        highLevelClient.indices().create(request, RequestOptions.DEFAULT);
    }

    /**
     * 滚动索引
     *
     * @param alias 索引别名
     */
    @Override
    public void rolloverIndex(String alias) {
        RestClient lowLevelClient = highLevelClient.getLowLevelClient();
        JSONObject requestBody = new JSONObject();
        JSONObject conditions = new JSONObject();
        conditions.put("max_age","1s");
        requestBody.put("conditions",conditions);
        Request post = new Request("POST", alias + "/_rollover");
        post.setJsonEntity(requestBody.toJSONString());
        try {
            Response response = lowLevelClient.performRequest(post);
            String resStr = IoUtil.read(response.getEntity().getContent(), StandardCharsets.UTF_8);
            JSONObject jsonResponse = JSONObject.parseObject(resStr);
            Boolean acknowledged = jsonResponse.getBoolean("acknowledged");
            if (acknowledged){
                log.info("roll {} result:{}", alias, jsonResponse.getString("new_index"));
            }else {
                log.error("index rollover exception, cause: [{}]", resStr);
                throw new RuntimeException("index rollover exception, cause: " + resStr);
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    /**
     * 根据索引别名对索引分组
     *
     * @return 按别名分组的集合
     */
    @Override
    public Map<String, List<String>> groupIndexByAlias() {
        try {
            GetAliasesRequest request = new GetAliasesRequest();
            GetAliasesResponse res = highLevelClient.indices().getAlias(request, RequestOptions.DEFAULT);
            Map<String, Set<AliasMetadata>> aliases = res.getAliases();
            aliases = aliases.entrySet().stream()
                    .filter(entry -> entry.getKey().startsWith("mory-") || entry.getKey().startsWith("third-"))
                    .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
            HashMap<String, List<String>> aliasGroup = new HashMap<>();
            aliases.forEach((k, v) -> {
                for (AliasMetadata aliasMetadata : v) {
                    String alias = aliasMetadata.getAlias();
                    if (!aliasGroup.containsKey(alias)) {
                        aliasGroup.put(alias, new ArrayList<String>() {{
                            add(k);
                        }});
                    } else {
                        aliasGroup.get(alias).add(k);
                    }
                }
            });
            return aliasGroup;
        } catch (IOException e) {
            log.error("Group Index By Alias Fail: {}", e.getMessage());
        }
        return Collections.emptyMap();
    }

    /**
     * 添加一条数据
     *
     * @param entity 实体
     * @param indexName 索引名
     * @return id
     * @throws IOException
     */
    @Override
    public <T> String addOne(T entity, String indexName) throws IOException {
        String id = null;
        IndexRequest indexRequest = genPutRequest(entity, indexName);
        indexRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
        IndexResponse response = highLevelClient.index(indexRequest, RequestOptions.DEFAULT);
        if (ObjectUtil.isNotNull(response)) {
            id = response.getId();
        }
        return id;
    }

    /**
     * 生成IndexRequest
     *
     * @param entity 实体
     * @param index 索引
     * @return IndexRequest
     */
    private <T> IndexRequest genPutRequest(T entity, String index) {
        return new IndexRequest(index)
                .timeout(TimeValue.timeValueSeconds(10))
                .source(JSONObject.toJSONString(entity), XContentType.JSON);
    }

}

4.7 基础索引实体

记录索引信息和索引版本

package com.mory.framework.es.domain;

import com.mory.framework.es.contants.ESIndexConstants;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.dromara.easyes.annotation.IndexField;
import org.dromara.easyes.annotation.IndexName;
import org.dromara.easyes.annotation.rely.FieldType;

/**
 * @author mory
 * @date 2024-07-22 14:55
 */
@NoArgsConstructor
@AllArgsConstructor
@Data
@IndexName(value = ESIndexConstants.MORY_META_INDEX)
public class MoryMetaIndex {

    private String id;

    private String indexName;

    @IndexField(fieldType = FieldType.DATE, dateFormat = "yyyy-MM-dd HH:mm:ss")
    private String createTime;

    private String status;

}


package com.mory.framework.es.domain;

import com.mory.framework.es.contants.ESIndexConstants;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.dromara.easyes.annotation.IndexName;

/**
 * @author mory
 * @date 2024-07-22 16:16
 */
@NoArgsConstructor
@AllArgsConstructor
@Data
@IndexName(value = ESIndexConstants.MORY_VERSION_INDEX)
public class MoryVersionIndex {

    private String id;

    private String indexName;

    private String version;

}


package com.mory.framework.es.domain;

import com.alibaba.fastjson2.JSONObject;

/**
 * @author mory
 * @date 2024-07-22 17:19
 */
public interface IndexConfig {

    default JSONObject ilmPolicy() {
        return JSONObject.parseObject("{\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_size\":\"10GB\"}}}}}}");
    }

    default String indexPatterns() {
        return "-*";
    }

    default JSONObject settings() {
        return JSONObject.parseObject("{\"index\":{\"number_of_shards\":\"1\",\"number_of_replicas\":\"0\",\"refresh_interval\":\"60s\",\"translog.durability\":\"async\",\"translog.flush_threshold_size\":\"1024mb\",\"translog.sync_interval\":\"60s\"}}");
    }

    default JSONObject mappingsMerge() {
        return JSONObject.parseObject("{\"dynamic\":\"false\"}");
//        return JSONObject.parseObject("{\"dynamic\":\"strict\",\"_field_names\":{\"enabled\":\"false\"}}");
    }

}

4.8 示例索引实体

当实体类的字段有变化，会自动滚动索引

package com.mory.framework.es.domain;

import com.alibaba.fastjson.annotation.JSONField;
import com.mory.framework.es.contants.ESIndexConstants;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.NoArgsConstructor;
import org.dromara.easyes.annotation.IndexField;
import org.dromara.easyes.annotation.IndexName;
import org.dromara.easyes.annotation.rely.FieldType;

import java.util.Date;
import java.util.List;

/**
 * @author mory
 * @date 2024-08-26 10:03
 */
@EqualsAndHashCode(callSuper = true)
@Data
@IndexName(value = ESIndexConstants.MORY_DEMO_ALIAS)
public class MoryDemo implements IndexConfig {

    private String id;

    private Long userId;

    @IndexField(fieldType = FieldType.DATE, dateFormat = "yyyy-MM-dd")
    @JSONField(format = "yyyy-MM-dd")
    private Date startTime;

    @IndexField(fieldType = FieldType.DATE, dateFormat = "yyyy-MM-dd")
    @JSONField(format = "yyyy-MM-dd")
    private Date endTime;

    private Long operatorId;

    @IndexField(fieldType = FieldType.NESTED, nestedClass = DemoType.class)
    private List<DemoType> demoTypes;

    private Integer status;

    @AllArgsConstructor
    @NoArgsConstructor
    @Data
    public static class DemoType {
        private Long typeId;
        private String typeName;
    }

}

4.9 应用

基础使用参见：Easy-Es

查询时，参考以下方式

String[] indices = IndexUtil.getIndexByAlias(ESIndexConstants.MORY_DEMO_ALIAS, request.getStartTime(), request.getEndTime());
LambdaEsQueryWrapper<MoryDemo> queryWrapper = new LambdaEsQueryWrapper<>();
if (indices.length > 0) {
	queryWrapper.indexName(indices);
}

五、后记

        整体实现思路

        1. 基于实体类的字段生成索引模板

        2. 根据索引模板创建索引和索引生命周期

        3. 对实体类做版本控制，监听实体类变化，更新索引模板，滚动索引

        4. 定时获取全量索引，便于检索使用

六、其他

        核心代码大致就上述那些，一些扩展性的能力可自行发掘，比如，自动填充实体创建和修改时间，再比如，封装服务层，抽象公共方法等。

java连接ElasticSearch 8.2.3、新建索引、删除索引、填充数据
weixin_44307338的博客
07-04 2319
java连接ElasticSearch 8.2.3、新建索引、删除索引、填充数据，同步ES库与数据库数据
Spring Boot 集成 Elasticsearch 指南：使用 Easy-ES-Starter
一起coding，一起嗨。
08-11 604
Spring Boot 集成 Elasticsearch 指南：使用 Easy-ES-Starter
JAVA——ES简介(史上最全！！)
最新发布
bigdealandgood的博客
07-12 1275
Elasticsearch 概述Elasticsearch是一个基于lucene、分布式、通过Restful方式进行交互的近实时搜索平台框架。ELK 技术栈是Elasticsearch、Logstash、Kibana三大开元框架首字母大写简称。而Elasticsearch 是一个开源的高扩展的分布式全文搜索引擎， 是整个 ELK技术栈的核心。Elasticsearch是一个基于lucene、分布式、通过Restful方式进行交互的近实时搜索平台框架。Logstash是中央数据流引擎，用于从不同目标(文件/数
elasticsearch5.5.1的javaapi，含创建索引、mapping、查询、增加删除文档
08-19
elasticsearch5.5.1的javaapi，含创建索引、mapping、查询、增加删除文档
java es自动创建索引
weixin_33156085的博客
07-15 442
如何在Java中使用ES自动创建索引 1. 理解整个流程 在Java中使用Elasticsearch（ES）自动创建索引可以分为以下步骤： 步骤 描述 1 连接Elasticsearch集群 2 创建Index对象 3 设置Index的Mapping 4 将Index对象添加到Elasticsea...
Java之ES创建索引
清风自来
10-08 4618
下面来继续分享并记录ES相关： 通过Java创建ES索引如下所示： /** * create by wxy_jdhk */ public class ElasticSearchUtils { public void createIndexOne(TransportClient client) { // 创建索引 try { IndexResponse response = clie...
java 建立索引_Java创建ES索引实现
weixin_32150221的博客
02-12 881
1、pom.xml文件org.springframework.bootspring-boot-starter-testorg.springframework.cloudspring-cloud-starter-eurekaorg.elasticsearchelasticsearch6.2.3org.elasticsearch.clienttransport6.2.3org.elasticsearc...
java用es客户端创建索引
weixin_64638001的博客
03-10 2088
【代码】java用es客户端创建索引。
使用java开发ElasticSearch Demo，整合了easy-es，并在mybatisplus的拦截器中，完成数据同步
11-05
最后，easy-es将变更数据同步到ElasticSearch中，确保ElasticSearch索引与数据库的数据保持一致。 为了达到上述目标，本Demo展示了一种开发模式，其中包含以下几个关键点： 1. 初始化配置：在Java项目中集成easy-...
Easy-Es是一款简化ElasticSearch搜索引擎操作的开源框架,全自动智能索引托管
03-18
Elasticsearch 国内顶尖elasticsearch搜索引擎框架es ORM框架,索引全自动智能托管,如丝般顺滑,与Mybatis-plus一致的API,屏蔽语言差异,开发者只需要会MySQL语法即可完成对Es的相关操作,零额外学习成本.底层采用...
easy-ES框架使用，轻松操作查询Elasticsearch
06-01
easy-ES框架是为了解决在Java开发中与Elasticsearch交互时的复杂性而诞生的。它提供了一套简洁、直观的API，使得开发者能够更高效地创建索引、插入文档、执行查询以及进行其他常见的ES操作。通过使用easy-ES，你可以...
es自动化脚本:创建索引和结构映射
02-07
以脚本方式创建es 索引和结构映射. 莎士比亚在不经意间这样说过，本来无望的事，大胆尝试，往往能成功。我希望诸位也能好好地体会这句话。 一般来讲，我们都必须务必慎重的考虑考虑。 就我个人来说，我秃头了对我的意义，不能不说非常重大。 我认为， 带着这些问题，我们来审视一下我秃头了。 就我个人来说，我秃头了对我的意义，不能不说非常重大。 在这种困难的抉择下，本人思来想去，寝食难安。 邓拓曾经提到过，越是没有本领的就越加自命不凡。这不禁令我深思。 西班牙曾经提到过，自己的鞋子，自己知道紧在哪里。这不禁令我深思。 了解清楚我秃头了到底是一种怎么样的存在，是解决一切问题的关键。 不用谢,我热愛滑铲.
Elasticsearch——使用Java API实现ES中的索引、映射、文档操作
热门推荐
贝尔摩德苦艾酒
08-26 2万+
Elasticsearch——使用Java API实现ES中的索引、映射、文档操作
使用java来创建es索引（基于es7.8）
晓缔的专栏
01-25 2332
使用java来创建es索引
【ES】JavaAPI学习-01 环境配置 索引创建
锥栗的博客
04-23 2370
前言 本文档及关联项目用于ElasticSearch的JavaAPI的学习。 环境 JDK8 IntelliJ IDEA 2021.2.3 (Community Edition) 实现 项目创建 创建一个新项目 取名为：ElasticSearch-JavaAPI-learning 依赖配置 pom.xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0"
ElasticSearch 滚动索引
weixin_45277567的博客
04-02 1112
elasticsearch 滚动索引
ElasticSearch 利用Java接口创建索引以及操作文档
qq_18600061的专栏
10-25 1904
ElasticSearch 利用Java接口创建索引以及操作文档 pom.xml： &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM...
Java创建ES索引实现
weixin_30725467的博客
06-12 1479
1、pom.xml文件 <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> </dependency> <dependency> &...
Elasticsearch：运用 Java 创建索引并写入数据
Elastic 中国社区官方博客
10-21 8642
在我之前的文章 “Elasticsearch：Java 运用示例”，我讲述了如何在 Java 应用中创建一个索引，并写入数据。在今天的例子中，我来着重讲述如何有目的地创建按照我们需求的索引，并介绍几种常见的方法写入数据。 安装 我们首先参考如下的文章来安装我们需要的 Elasticsearch 及 Kibana： 如何在 Linux，MacOS 及 Windows 上进行安装 Elasticsearch Kibana：如何在 Linux，MacOS 及 Windows上安装 Elastic 栈中
简化操作的Easy-Es开源框架：智能托管ElasticSearch索引
资源摘要信息:"Easy-Es是一款简化Elasticsearch搜索引擎操作的开源框架，旨在通过自动化和智能化的方式降低开发者在使用Elasticsearch时的技术门槛。它提供了一套与Mybatis-plus类似的API，这意味着开发者可以利用...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值