【Spring Boot 2.0学习项目】SpringBoot+ElasticSearch博客检索系统

SpringBoot+ElasticSearch博客检索系统

一、初识ElasticSearch

1.ElasticSearch概念和适用场景

• 分布式；
• 全文检索；
• 实时快速；
• Restful API；

2.ElasticSearch VS MySQL

例子：

MySql：select * from user.user_info where name = '张三';

ES：GET /user/user_info/_search?q=name:张三

3.ElasticSearch、kibana安装

ElasticSearch下载

启动：/bin/elasticsearch.bat

展示：

Kibana下载地址

启动：/bin/kibana.bat

展示：

4.演示postman、kibana对es的交互

postman对es的交互API：

• 查看所有索引： GET localhost:9200/_all
• 创建索引-test：PUT localhost:9200/test
• 删除索引-test：DELETE localhost:9200/test
• 创建索引-person： PUT localhost:9200/person
• 新增数据-person-1：PUT localhost:9200/person/_doc/1

{
    "first_name":"John",
    "last_name":"Smith",
    "age":25,
    "about":"i love  to go rock climbing",
    "interests":["sports","music"]
}

• 新增数据-person-2：PUT localhost:9200/person/_doc/2

{
    "first_name":"Eric",
    "last_name":"Smith",
    "age":23,
    "about":"i love  basketball",
    "interests":["sports","reading"]
}

• 搜索数据-person-id： GET localhost:9200/person/_doc/1
• 搜索数据-person-name：GET localhost:9200/person/_doc/_search?q=first_name:john

kibana对es的交互

/dev tools/console下

查看所有索引: GET _all

查看id为1的数据：GET /person/_doc/1

搜索查询数据：

POST /person/_doc/_search
{
  "query":{
    "bool": {
      "should": [
        {"match": {
          "first_name": "Eric"
        }}
      ]
    }
  }
}

二、博客网站全文检索

1.MySql与ElasticSearch语句测试检索速度

1.1 Mysql建表测试检索速度

/*
Navicat MySQL Data Transfer

Source Server         : lcz
Source Server Version : 50729
Source Host           : localhost:3306
Source Database       : blog

Target Server Type    : MYSQL
Target Server Version : 50729
File Encoding         : 65001

Date: 2022-01-16 13:14:03
*/

SET FOREIGN_KEY_CHECKS=0;

-- ----------------------------
-- Table structure for `t_blog`
-- ----------------------------
DROP TABLE IF EXISTS `t_blog`;
CREATE TABLE `t_blog` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增id',
  `title` varchar(60) DEFAULT NULL COMMENT '博客标题',
  `author` varchar(60) DEFAULT NULL COMMENT '博客作者',
  `content` mediumtext COMMENT '博客内容',
  `create_time` datetime DEFAULT NULL COMMENT '创建时间',
  `update_time` datetime DEFAULT NULL COMMENT '更新时间',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8mb4;

-- ----------------------------
-- Records of t_blog
-- ----------------------------
INSERT INTO `t_blog` VALUES ('1', 'Springboot 为什么这', 'bywind', '没错 Springboot ', '2019-12-08 01:44:29', '2019-12-08 01:44:34');
INSERT INTO `t_blog` VALUES ('3', 'Springboot 中 Redis', 'bywind', 'Spring Boot', '2019-12-08 01:44:29', '2019-12-08 01:44:29');
INSERT INTO `t_blog` VALUES ('4', 'Springboot 中如何优化', 'bywind', null, '2019-12-08 01:44:29', '2019-12-08 01:44:29');
INSERT INTO `t_blog` VALUES ('5', 'Springboot 消息队列', 'bywind', null, '2019-12-08 01:44:29', '2019-12-08 01:44:29');
INSERT INTO `t_blog` VALUES ('6', 'Docker Compose + Springboot', 'bywind', null, '2019-12-08 01:44:29', '2019-12-08 01:44:29');

查询语句：

select * from t_blog where title like "%spring%" or content like "%spring%";

1.2 ElastciSearch为什么搜索快呢？

底层基于倒排索引

分布式

2.Mysql与ElasticSearch同步中间件

2.1 开源的中间件介绍

• binlog订阅

  

  • alibaba/canal：阿里巴巴开源组件。MySQL binlog增量订阅&消费组件
  • go-mysql-elasticsearch：go语言的组件。
  • logstash:官方提供的组件

  

2.2 logstash来增量、全量同步数据解决方案

Logstash下载地址

mysql-connector-java下载地址

在下载的logstash中放入mysql-connector-java

在config中新建一个mysql.conf文件，内容如下：

input {
	jdbc {
		# jdbc驱动包位置
		jdbc_driver_library => "D:\software\elasticsearch\logstash-6.3.2\logstash-6.3.2\\mysql-connector-java-5.1.31.jar"
		# 要使用的驱动包类
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        # mysql数据库的连接信息
        jdbc_connection_string => "jdbc:mysql://localhost:3306/blog?serverTimezone=UTC&characterEncoding=utf8"
        # mysql用户
        jdbc_user => "root"
        # mysql密码
        jdbc_password => "123"

        # 定时任务，默认一分钟，"* * * * *"代表设置为无延迟
        schedule => "* * * * *"

        # 清空上一次sql_last_value记录
        clean_run => true
        # 要执行的sql语句
        statement => "SELECT * FROM t_blog WHERE update_time > date_add(:sql_last_value, interval 8 hour) AND update_time<date_add(NOW(), interval 8 hour) ORDER BY update_time desc"
	}
}


output {
    elasticsearch {
        # es host：port
        hosts => ["127.0.0.1:9200"]
        #索引
        index => "blog"
        # ——id
        document_id => "%{id}"

    }

}

启动方式：

D:\software\elasticsearch\logstash-6.3.2\logstash-6.3.2\bin>logstash -f ../config/mysql.conf

在kibana中验证：

GET /blog/_stats, 查看其_all底下的count字段。

3.ElasticSearch内置分词器

3.1 内置分词器

• standard;

standard : ES默认分词器，将单词转换为小写，去除停用词与符号，支持中文——单字切分

• simple;

simple ：通过非字母字符进行切分，统一化为小写，去除数字类型字符

• whitespace;

whitespace ：不支持中文，不转换为小写，只去除空格，

• language;

language ：特定语言的分词器，不支持中文

测试分词效果：

3.2 引入elasticsearch-analysis-ik分词器

从github下载对应版本的分词器。解压之后，在elasticsearch中的plugins中新建一个ik文件夹，放入解压之后的文件即可。重启之后

三、springboot+elasticsearch实现博客检索功能

1.环境配置以及项目结构

（1）pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.1.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.lcz</groupId>
    <artifactId>elasticsearch_blog</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>elasticsearch_blog</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>1.8</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

（2）application.properties配置选项

通用数据源配置
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/blog?useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai
spring.datasource.username=root
spring.datasource.password=123
#Hikari数据源专用配置
spring.datasource.hikari.maximum-pool-size=20
spring.datasource.hikari.minimum-idle=5
#JPA相关配置
spring.jpa.database-platform=org.hibernate.dialect.MySQLDialect
#es
spring.data.elasticsearch.cluster-nodes=127.0.0.1:9300
spring.data.elasticsearch.cluster-name=elasticsearch
#mVc
spring.mvc.static-path-pattern=/**
spring.devtools.livereload.enabled=true
spring.devtools.restart.additional-paths=static/**
#日期格式化4
spring.jackson.date-format=yyyy-MM-dd HH:mm:ss

#spring.resources.static-locations=classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/,\
#  classpath:/public/,classpath:/templates/,

（3）项目结构

2.具体实现

MySQL出发

package com.lcz.elasticsearch_blog.entity.mysql;

/**
 * @author : codingchao
 * @date : 2022-01-17 10:28
 * @Description:
 **/


import lombok.Data;

import javax.persistence.*;
import java.util.Date;

/**
 * CREATE TABLE `t_blog` (
 *   `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增ID',
 *   `title` varchar(60) DEFAULT NULL COMMENT '博客标题',
 *   `author` varchar(60) DEFAULT NULL COMMENT '博客作者',
 *   `content` mediumtext COMMENT '博客内容',
 *   `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
 *   `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间',
 *   PRIMARY KEY (`id`)
 * ) ENGINE=InnoDB AUTO_INCREMENT=112 DEFAULT CHARSET=utf8mb4
 */

@Data
@Table(name = "t_blog")
@Entity
public class MySqlBlog {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String title;
    private String author;
    @Column(columnDefinition = "mediumtext")
    private String content;
    private Date createTime;
    private Date updateTime;

}

package com.lcz.elasticsearch_blog.respository.mysql;

import com.lcz.elasticsearch_blog.entity.mysql.MySqlBlog;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;

import java.util.List;

/**
 * @author : codingchao
 * @date : 2022-01-17 10:39
 * @Description:
 **/
public interface MySqlBlogRespository extends JpaRepository<MySqlBlog,Integer> {
    @Query("select e from MySqlBlog e order by e.createTime desc")
    List<MySqlBlog> queryAll();
    @Query("select e from MySqlBlog e where e.title like concat('%',:keyword,'%') " +
            "or e.content like concat('%',:keyword,'%') order by e.createTime desc")
    List<MySqlBlog> queryBlogs(@Param("keyword") String keyword);
}

ES出发

package com.lcz.elasticsearch_blog.entity.es;

import com.fasterxml.jackson.annotation.JsonAlias;
import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.DateFormat;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;

import javax.persistence.*;
import java.util.Date;


/**
 * @author : codingchao
 * @date : 2022-01-17 11:37
 * @Description:
 **/
import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.DateFormat;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;

import java.util.Date;

/**
 * @program: estest
 * @description: ES实体类
 * @author: Mr.Huang
 * @create: 2020-03-30 21:21
 **/
@Data
@Document(indexName = "blog", type = "doc",
        useServerConfiguration = true, createIndex = false)
public class EsBlog {
    @Id
    private Integer id;
    @Field(type = FieldType.Text, analyzer = "ik_max_work")
    private String title;
    @Field(type = FieldType.Text, analyzer = "ik_max_work")
    private String author;
    @Field(type = FieldType.Text, analyzer = "ik_max_work")
    private String content;
    @Field(type = FieldType.Date, format = DateFormat.custom,
            pattern = "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis")
    @JsonAlias(value = "create_time")
    private Date createTime;
    @Field(type = FieldType.Date, format = DateFormat.custom,
            pattern = "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis")
    @JsonAlias(value = "update_time")
    private Date updateTime;

    public String getTitle() {
        return title;
    }
}

package com.lcz.elasticsearch_blog.respository.es;

import com.lcz.elasticsearch_blog.entity.es.EsBlog;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

public interface EsBlogRepository extends ElasticsearchRepository<EsBlog,Integer> {

}

controller

package com.lcz.elasticsearch_blog.controller;

import com.lcz.elasticsearch_blog.entity.mysql.MySqlBlog;
import com.lcz.elasticsearch_blog.respository.mysql.MySqlBlogRespository;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

import java.util.List;


/**
 * @author : codingchao
 * @date : 2022-01-17 11:27
 * @Description:
 **/
@Controller
@Slf4j
public class IndexController {
    @Autowired
    private MySqlBlogRespository mySqlBlogRespository;

    @RequestMapping("/")
    public String index(){
        List<MySqlBlog> all = mySqlBlogRespository.findAll();
        System.out.println(all.size());
        return "index.html";
    }
}

package com.lcz.elasticsearch_blog.controller;

import com.lcz.elasticsearch_blog.entity.es.EsBlog;
import com.lcz.elasticsearch_blog.entity.mysql.MySqlBlog;
import com.lcz.elasticsearch_blog.respository.es.EsBlogRepository;
import com.lcz.elasticsearch_blog.respository.mysql.MySqlBlogRespository;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.util.StopWatch;
import org.springframework.web.bind.annotation.*;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;


/**
 * @author : codingchao
 * @date : 2022-01-17 11:54
 * @Description:
 **/
@RestController
@Slf4j
public class DataController {

    @Autowired
    private MySqlBlogRespository mysqlBlogRepository;
    @Autowired
    private EsBlogRepository esBlogRepository;

    @GetMapping("/blogs")
    public Object blog(){
        List<MySqlBlog> mysqlBlogs = mysqlBlogRepository.queryAll();
        return mysqlBlogs;
    }

    @PostMapping("/search")
    public Object search(@RequestBody Param param){
        Map<String, Object> map = new HashMap<>();
        String type = param.getType();
        StopWatch watch = new StopWatch();
        watch.start();
        if(type.equalsIgnoreCase("mysql")){
            List<MySqlBlog> mysqlBlogs = mysqlBlogRepository.queryBlogs(param.getKeyword());
            map.put("list",mysqlBlogs);
        }else if(type.equalsIgnoreCase("es")){
            BoolQueryBuilder builder = QueryBuilders.boolQuery();
            builder.should(QueryBuilders.matchPhraseQuery("title",param.getKeyword()));
            builder.should(QueryBuilders.matchPhraseQuery("content",param.getKeyword()));
            String s = builder.toString();
            System.out.println("======");
            System.out.println(s);
            System.out.println("======");
            Page<EsBlog> esBlogs = (Page<EsBlog>) esBlogRepository.search(builder);
            List<EsBlog> content = esBlogs.getContent();
            map.put("list",content);
        }else {
            return ">>> 不知道 <<<";
        }
        watch.stop();
        long totalTimeMillis = watch.getTotalTimeMillis();
        map.put("duration",totalTimeMillis);
        return map;
    }

    @GetMapping("/blog/{id}")
    public Object blog(@PathVariable Integer id){
        Optional<MySqlBlog> byId = mysqlBlogRepository.findById(id);
        return byId.get();
    }


    @Data
    public static class Param{
        // String,es
        private String type;
        private String keyword;

        public String getType() {
            return type;
        }

        public void setType(String type) {
            this.type = type;
        }

        public String getKeyword() {
            return keyword;
        }

        public void setKeyword(String keyword) {
            this.keyword = keyword;
        }
    }
}

3.源码

github下载地址