elasticsearch语法和整合springboot

版本控制

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-M74Okngl-1680862475871)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230403110845071.png)]

安装elasticsearch

简介

elasticsearch

下载

安装包下载地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch

解压

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ctOsKh6r-1680862475872)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172230290.png)]

启动

方式一：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IrO2qWwp-1680862475872)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406171852255.png)]

方式二：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IezmkhfD-1680862475873)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172357949.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qiGttJ86-1680862475873)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172545328.png)]

检查是否安装成功

访问：http://localhost:9200/

出现下面信息表示成功

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-c9WSawg3-1680862475874)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406172759807.png)]

安装kibana

备注

是一个es可视化工具，功能很多，跟es一样，下载解压，双击bin下面的kibana.bat文件或者打开控制台输入kibana启动

访问kibana页面：http://localhost:5601/app/kibana

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-j97KxUjO-1680862475874)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180538887.png)]

语言设置转中文

网页纯英文，可自选设置成中文

编辑config下面的配置文件，文件最下端设置语言

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cbXHj6lM-1680862475875)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180331511.png)]

#英文

#i18n.locale: "en"

#中文

i18n.locale: "zh-CN"

es语法

参考资料：https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

rest风格

PUT //_doc/<_id>
POST //_doc/
PUT //_create/<_id>
POST //_create/<_id>

索引

创建索引

PUT /my-index-000001
{
  "settings": {
    "index": {
      "number_of_shards": 3,  
      "number_of_replicas": 2 
    }
  }
}

PUT /my-index-000001
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 2
  }
}

PUT /test
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "field1": { "type": "text" }
    }
  }
}

GET /my-index-000001/_mapping

修改索引

// 查看是否存在
HEAD my-data-stream

post /my-index-000001

获取索引

GET /my-index-000001

删除索引

DELETE /my-index-000001

文档

PUT /<target>/_doc/<_id>
POST /<target>/_doc/
PUT /<target>/_create/<_id>
POST /<target>/_create/<_id>

创建文档

POST my-index-000001/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}

修改文档

POST //_update/<_id>

PUT test/_doc/1
{
  "counter" : 1,
  "tags" : ["red"]
}

获取文档

GET /_doc/<_id>

GET my-index-000001/_doc/0

GET my-index-000001/_doc/0?_source=false

删除文档

DELETE //_doc/<_id>

单个删除

DELETE /my-index-000001/_doc/1?timeout=5m

查询删除

// Deletes documents that match the specified query.
POST /my-index-000001/_delete_by_query
{
  "query": {
    "match": {
      "user.id": "elkbee"
    }
  }
}

批量删除

POST /decode/resolve/_bulk
{"index":{"_id":"1"}}
{"vin": "D1445266366" }
{"index":{"_id":"2"}}
{"vin": "C4526365588" }

搜索

参考资料：

https://nasuyun.com/docs/api-search-text/api-search-text-multi-match

https://blog.youkuaiyun.com/qq_41911898/article/details/110089644

GET /<target>/_search
GET /_search
POST /<target>/_search
POST /_search

字段匹配 最简单的搜索，使用match_all来表示，例如搜索全部

GET /decode/resolve/_search
{
  "query": { "match_all": {} }
}

分页搜索，from表示偏移量，从0开始，size表示每页显示的数量

GET /decode/resolve/_search
{
  "query": { "match_all": {} },
  "from": 0,
  "size": 10
}

搜索并返回指定字段内容，使用_source表示，例如只返回id和createTime两个字段内容

GET /decode/resolve/_search
{
  "query": { "match_all": {} },
  "sort": { "createTime": { "order": "desc" } }
}

条件搜索，使用match表示匹配条件，例如搜索出id为20的文档：

GET /decode/resolve/_search
{
  "query": {
    "match": {
      "id": 20
    }
  }
}

短语匹配搜索，使用match_phrase表示，例如搜索address字段中同时包含mill和lane的文档：

GET /decode/resolve/_search
{
  "query": {
    "match_phrase": {
      "address": "mill lane"
    }
  }
}

组合搜索，使用bool来进行组合，must表示同时满足，例如搜索address字段中同时包含mill和lane的文档

GET /decode/resolve/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

9、组合搜索，should表示满足其中任意一个，搜索address字段中包含mill或者lane的文档

GET /decode/resolve/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

10、组合搜索，must_not表示同时不满足，例如搜索address字段中不包含mill且不包含lane的文档

GET /decode/resolve/_search
{
  "query": {
    "bool": {
      "must_not": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}

11、组合搜索，组合must和must_not，例如搜索age字段等于40且state字段不包含ID的文档

GET /decode/resolve/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "age": "40" } }
      ],
      "must_not": [
        { "match": { "state": "ID" } }
      ]
    }
  }
}

12、搜索过滤，使用filter来表示，例如过滤出balance字段在20000~30000的文档

GET /decode/resolve/_search
{
  "query": {
    "bool": {
      "must": { "match_all": {} },
      "filter": {
        "range": {
          "balance": {
            "gte": 20000,
            "lte": 30000
          }
        }
      }
    }
  }
}

13、对搜索结果进行聚合，使用aggs来表示，类似于MySql中的group by，例如对vin字段进行聚合，统计出相同state的文档数量

GET /decode/resolve/_search
{
  "size": 0,
  "aggs": {
    "group_by_vin": {
      "terms": {
        "field": "vin.keyword"
      }
    }
  }
}

14、嵌套聚合，例如对state字段进行聚合，统计出相同vin的文档数量，再统计出balance的平均值

GET /decode/resolve/_search
{
  "size": 0,
  "aggs": {
    "group_by_vin": {
      "terms": {
        "field": "vin.keyword"
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}

15、对聚合搜索的结果进行排序，例如按balance的平均值降序排列

GET /decode/resolve/_search
{
  "size": 0,
  "aggs": {
    "group_by_vin": {
      "terms": {
        "field": "vin.keyword",
        "order": {
          "average_balance": "desc"
        }
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}

16、按字段值的范围进行分段聚合，例如分段范围为age字段的[20,30] [30,40] [40,50]，之后按gender统计文档个数和balance的平均值

GET /decode/resolve/_search
{
  "size": 0,
  "aggs": {
    "group_by_age": {
      "range": {
        "field": "age",
        "ranges": [
          {
            "from": 20,
            "to": 30
          },
          {
            "from": 30,
            "to": 40
          },
          {
            "from": 40,
            "to": 50
          }
        ]
      },
      "aggs": {
        "group_by_gender": {
          "terms": {
            "field": "gender.keyword"
          },
          "aggs": {
            "average_balance": {
              "avg": {
                "field": "balance"
              }
            }
          }
        }
      }
    }
  }
}

分词

elasticsearch-head-master

简介

elasticsearch-head是一个前端项目，需要node环境，功能是作为一个es可视化界面。个人感觉比kibana更实用，可以看到索引，文档和进行增删改查，效果图如下

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-g99xoE3E-1680862475876)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406180844170.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JTc31GxK-1680862475876)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406181107042.png)]

安装启动

下载依赖

npm i

启动

npm run start

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OZi36gKf-1680862475877)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406181516568.png)]

跨域

因为elasticsearch-head-master 是前端项目，访问elasticsearch 存在跨域问题，通过修改elasticsearch 配置解决

打开elasticsearch -> config -> elasticsearch.yml

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NAYUvg6L-1680862475878)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406182226585.png)]

拉到配置文件的最低端

#跨域问题
#设置可跨域
http.cors.enabled: true
#设置所有可访问
http.cors.allow-origin: "*"

springboot整合es

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SqJyEarX-1680862475879)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407140422589.png)]

环境

springboot == 2.4.4

elasticsearch==7.9.3

canal==1.1.5

导入依赖

<!--  elasticsearch  -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<!--   spring    -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

目录结构

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-pMxDkG9r-1680862475879)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407161338552.png)]

添加配置

打开application.yml

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-6m8Fh51h-1680862475880)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406184751183.png)]

elasticsearch:
  cluster:
    name: http
  host: 127.0.0.1
  port: 9200

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1MI5fuY8-1680862475880)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406184815677.png)]

import org.apache.http.HttpHost;
import org.elasticsearch.client.ElasticsearchClient;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;

/**
 * @BelongsProject: es01
 * @BelongsPackage: com.dhl.es01.client
 * @Author: HuiLong.Ding
 * @Description: TODO
 * @Date: 2023/03/29  18:46
 */
@Configuration
public class ElasticSearchConfig {

    private static final Logger logger = LoggerFactory.getLogger(ElasticsearchClient.class);

    @Value("${elasticsearch.cluster.name}")
    private String clusterName;
    @Value("${elasticsearch.host}")
    private String host;
    @Value("${elasticsearch.port}")
    private int port;

    @Bean
    public RestHighLevelClient restHighLevelClient(){
        RestHighLevelClient client = new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost(host, port, clusterName)
                )
        );
        logger.info("elasticsearch 连接成功");
        return client;
    }
}

java的es操作api

找官方文档 （7.6.x）

https://www.elastic.co/guide/index.html

索引

增删改查

import org.elasticsearch.client.RestHighLevelClient;

private RestHighLevelClient restHighLevelClient;

//-------------es rhlc 客户端
    /**
     * 创建索引
     *
     * @author HuiLong.Ding
     * @date 2023/4/7 14:24
     */
    @GetMapping("/index/put")
    CreateIndexResponse putIndex() throws IOException {
        // 1.创建索引请求
        CreateIndexRequest request = new CreateIndexRequest("yd_pop_order");
        // 2.执行创建请求，请求后获得响应createIndexResponse
        CreateIndexResponse createIndexResponse = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
        System.out.println("createIndexResponse = " + createIndexResponse);
        return createIndexResponse;
    }

    // 测试获取索引，只能判断其是否存在
    @GetMapping("/index/exist")
    boolean existIndex() throws IOException {
        // 1.创建索引请求
        GetIndexRequest request = new GetIndexRequest("yd_pop_order");
        // 2.获取索引，是否存在
        return restHighLevelClient.indices().exists(request, RequestOptions.DEFAULT);
    }

    // 测试删除索引
    @GetMapping("/index/del")
    AcknowledgedResponse testDeleteIndex() throws IOException {
        // 1.创建删除索引请求
        DeleteIndexRequest request = new DeleteIndexRequest("yd_pop_order");
        // 2.获取删除状态
        AcknowledgedResponse delete = restHighLevelClient.indices().delete(request, RequestOptions.DEFAULT);
        System.out.println(delete.isAcknowledged());
        return delete;
    }

    // 测试添加文档
    @GetMapping("/index/add")
    IndexResponse testAddDocument() throws IOException {
        // 1.创建对象
        YdPopOrder ydPopOrder = new YdPopOrder();
        ydPopOrder.setId(2609731592282L);
        ydPopOrder.setState("6346453454");
        ydPopOrder.setCreated(new Date());
        ydPopOrder.setResponseJson("我是数据");
        // 2.创建请求
        IndexRequest request = new IndexRequest("yd_pop_order");
        // 3.规则 put /yd_pop_order/_doc/1
        request.id("1");
        request.timeout(TimeValue.timeValueSeconds(1));// 或者 request.timeout("1s");
        // 4.把数据放入请求 json
        IndexRequest source = request.source(JSON.toJSONString(ydPopOrder), XContentType.JSON);
        // 5.客户端发送请求，获取响应结果
        IndexResponse indexResponse = restHighLevelClient.index(request, RequestOptions.DEFAULT);
        System.out.println(indexResponse.toString());// 结果的json
        System.out.println(indexResponse.status());// 对应我们命令返回的状态 created
        return indexResponse;
    }

    // 获取文档的信息
    @GetMapping("/index/get")
    void testGetDocument() throws IOException {
        GetRequest getRequest = new GetRequest("yd_pop_order", "1");
        GetResponse getResponse = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
        System.out.println(getResponse.getSourceAsString());// 打印文档的内容
        System.out.println(getResponse);// 返回的全部内容和密令一样
    }

文档

原生api参考资料：https://www.kuangstudy.com/bbs/1552226602677882882#header15

添加文档

@GetMapping("/put")
public boolean put() throws Exception {
    YdPopOrder ydPopOrder = new YdPopOrder();
    ydPopOrder.setId(2609731522282L);
    ydPopOrder.setState("6346453454");
    ydPopOrder.setCreated(new Date());
    ydPopOrder.setResponseJson("数据");
    ydPopOrderRepository.save(ydPopOrder);
    return true;
}

批量添加

@GetMapping("/putall")
public boolean putall() throws Exception {
    List<YdPopOrder> list = new ArrayList<>();
    Date date = new Date();
    for (int i = 0; i < 1000000; i++) {
        YdPopOrder ydPopOrder = new YdPopOrder();
        ydPopOrder.setId(2609731692282L + i);
        ydPopOrder.setState("6346453454" + i);
        ydPopOrder.setCreated(date);
        ydPopOrder.setResponseJson("数据");
        list.add(ydPopOrder);
    }
    ydPopOrderRepository.saveAll(list);
    return true;
}

删除文档

@GetMapping("/del")
public boolean del(Long value) throws Exception {
    ydPopOrderRepository.deleteById(value);
    return true;
}

@GetMapping("/delall")
public boolean delall() throws Exception {
    ydPopOrderRepository.deleteAll();
    return true;
}

获取文档

@GetMapping("/get")
public Optional<YdPopOrder> get(Long value) throws Exception {
    return ydPopOrderRepository.findById(value);
}

自定义

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-C9YwhFZy-1680862475881)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407141227855.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gKrSDz68-1680862475881)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230407141300823.png)]

高亮

安装canal

下载无需安装，解压即可用，打开canal -> bin 替换startup.bat,然后双击startup.bat启动

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-furB6jHk-1680862475882)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20230406183554658.png)]

springboot整合canal

pom.xml导入依赖

<!--   canal     -->
<dependency>
    <groupId>com.alibaba.otter</groupId>
    <artifactId>canal.client</artifactId>
    <version>1.1.5</version>
</dependency>
<dependency>
    <groupId>com.alibaba.otter</groupId>
    <artifactId>canal.protocol</artifactId>
    <version>1.1.5</version>
</dependency>

application.yml

# canal
canal:
  host: 127.0.0.1
  port: 11111
  destination: example
  username: canal
  password: canal

CanalClient

@Component
public class CanalClient implements DisposableBean {
    private static final Logger logger = LoggerFactory.getLogger(CanalClient.class);
    private CanalConnector canalConnector;

    @Value("${canal.host}")
    private String canalHost;
    @Value("${canal.port}")
    private String canalPort;
    @Value("${canal.destination}")
    private String canalDestination;
    @Value("${canal.username}")
    private String canalUsername;
    @Value("${canal.password}")
    private String canalPassword;

    @Bean
    public CanalConnector getCanalConnector() {
        canalConnector = CanalConnectors.newClusterConnector(Lists.newArrayList(new InetSocketAddress(canalHost, Integer.valueOf(canalPort))), canalDestination, canalUsername, canalPassword);
        canalConnector.connect();
        // 指定filter，格式 {database}.{table}，这里不做过滤，过滤操作留给用户
        // .*\..*   可访问所有数据库的所有表    test\\..* 代表可访问名字为test的数据库下面的所有表
        canalConnector.subscribe(".*\\..*");
        // 回滚寻找上次中断的位置
        canalConnector.rollback();
        logger.info("canal客户端启动成功");
        return canalConnector;
    }

    @Override
    public void destroy() throws Exception {
        if (canalConnector != null) {
            canalConnector.disconnect();
        }
    }
}

注意 canalConnector.subscribe(“.\…”); 是对需要同步的数据库的过滤，可以通过正则匹配

参考资料：https://blog.youkuaiyun.com/Alice_qixin/article/details/108121895

全库全表	connector.subscribe(“.\…”)
指定库全表	connector.subscribe(“test\…*”)
单表	connector.subscribe(“test.user”)
多规则组合使用	connector.subscribe(“test\…*,test2.user1,test3.user2”)

抛出问题

es

searchSimilar函数

时间格式问题：