1.ES简介
2.es倒排索引
3.es的初步使用
4.索引库
5.java代码操作es
6.添加文档-数据
7.查询语法
8.java代码实现
9.开发中的使用
10.分组 / 统计（平均值/最大值/…）
11.java代码实现
12.数据聚合 -> 点击选项查询，其他选项存在数据久显示该选项
13.输入内容自动补全
14.sql与es 数据同步
15.集群
es简介
倒排索引
根据搜索条件进行分词，得到分词结果去词条列表进行查询，获取文档ID，根据文档集对应的ID，查询出对应的文档.
- 介绍
与mysql的对比
总结
安装es - kibana
## 1.1.创建网络
因为我们还需要部署kibana容器，因此需要让es和kibana容器互联。这里先创建一个网络：
```sh
docker network create es-net
1.2.加载镜像
这里我们采用elasticsearch的7.12.1版本的镜像，这个镜像体积非常大，接近1G。不建议大家自己pull。
课前资料提供了镜像的tar包：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QqN8f2Ae-1671012481661)(assets/image-20210510165308064.png)]
大家将其上传到虚拟机中，然后运行命令加载即可：
# 导入数据
docker load -i es.tar
同理还有kibana的tar包也需要这样做。
1.3.运行
运行docker命令，部署单点es：
docker run -d \
--name es \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
-e "discovery.type=single-node" \
-v es-data:/usr/share/elasticsearch/data \
-v es-plugins:/usr/share/elasticsearch/plugins \
--privileged \
--network es-net \
-p 9200:9200 \
-p 9300:9300 \
elasticsearch:7.12.1
命令解释：
-e "cluster.name=es-docker-cluster"：设置集群名称-e "http.host=0.0.0.0"：监听的地址，可以外网访问-e "ES_JAVA_OPTS=-Xms512m -Xmx512m"：内存大小-e "discovery.type=single-node"：非集群模式-v es-data:/usr/share/elasticsearch/data：挂载逻辑卷，绑定es的数据目录-v es-logs:/usr/share/elasticsearch/logs：挂载逻辑卷，绑定es的日志目录-v es-plugins:/usr/share/elasticsearch/plugins：挂载逻辑卷，绑定es的插件目录--privileged：授予逻辑卷访问权--network es-net：加入一个名为es-net的网络中-p 9200:9200：端口映射配置
在浏览器中输入：http://192.168.150.101:9200 即可看到elasticsearch的响应结果：
2.部署kibana
kibana可以给我们提供一个elasticsearch的可视化界面，便于我们学习。
2.1.部署
运行docker命令，部署kibana
docker run -d \
--name kibana \
-e ELASTICSEARCH_HOSTS=http://es:9200 \
--network=es-net \
-p 5601:5601 \
kibana:7.12.1
es初步使用
使用LK分词器
新兴的词语要用扩展词条
停用词词典
索引库
文档的增删改查
java操作es
- 初始化操作
- 关闭操作
新增/删除索引库
@Test
public void insert() throws IOException {
// 1. 本次向es发送请求为的是创建heima索引库
CreateIndexRequest request = new CreateIndexRequest("heima");
// 2. 发送es请求，接收响应结果
// client.indices() ：通过客户端创建索引库
// create(request, RequestOptions.DEFAULT) 参数1：请求创建的参数 ， 请求方式-固定
CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT);
// 3.处理请求结果
boolean flag = response.isAcknowledged();
System.out.println("当前请求的结果为："+flag);
}
@Test
public void Delete() throws IOException {
// 1. 本次向es发送请求为的是创建heima索引库
DeleteIndexRequest request = new DeleteIndexRequest("heima");
// 2. 删除索引库
AcknowledgedResponse delete = client.indices().delete(request, RequestOptions.DEFAULT);
// 3.处理请求结果
boolean flag = delete.isAcknowledged();
System.out.println("当前删除的结果为 ："+flag);
}
查看索引库是否存在/结构
// 判断API是否存在
@Test
public void exists() throws IOException {
// 1. 本次向es发送请求为的是创建heima索引库
GetIndexRequest request = new GetIndexRequest("heima");
// 2. 查看索引库是否存在
boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);
// 3.处理请求结果
if (exists){
System.out.println("当前所有存在");
}else {
System.out.println("当前索引不存在");
}
}
// 查看索引库结果
@Test
public void get() throws IOException {
// 1. 本次向es发送请求为的是创建heima索引库
GetIndexRequest request = new GetIndexRequest("heima");
// 2. 查看索引库结果
GetIndexResponse response = client.indices().get(request, RequestOptions.DEFAULT);
// 3.处理请求结果
Map<String, MappingMetadata> map = response.getMappings();
System.out.println(map);
MappingMetadata heima = map.get("heima");
Map<String, Object> sourceAsMap = heima.getSourceAsMap();
System.out.println(sourceAsMap);
}
添加文档 / 数据
import org.apache.http.HttpHost;
import org.elasticsearch.action.DocWriteRequest;
import org.elasticsearch.action.DocWriteResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.get.GetRequest;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
@SpringBootTest
public class HelloTest {
private RestHighLevelClient client;
@BeforeEach
void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.136.131:9200") ));
}
// 添加文档数据
@Test
public void indexs() throws IOException {
IndexRequest request = new IndexRequest("heima");
request.id("1");
String docJson = "{\n" +
" \"info\":\"黑马\",\n" +
" \"email\":\"zy@qq.com\",\n" +
" \"name\":{\n" +
" \"firstName\":\"云\",\n" +
" \"lastName\":\"找\"\n" +
" }\n" +
"}";
request.source(docJson , XContentType.JSON);
// 2.发送请求给es数据，接收响应数据
IndexResponse response = client.index(request, RequestOptions.DEFAULT);
// 3.处理响应结果
DocWriteResponse.Result result = response.getResult();
System.out.println("添加数据"+result);
}
// 查看文档数据
@Test
public void Query() throws IOException {
GetRequest request = new GetRequest("heima","1");
// 2.发送请求给es数据，接收响应数据
GetResponse documentFields = client.get(request, RequestOptions.DEFAULT);
// 3.处理响应结果
String sourceAsString = documentFields.getSourceAsString();
System.out.println(sourceAsString);
}
// 修改
@Test
public void Update1() throws IOException {
UpdateRequest request = new UpdateRequest("heima","1");
// 2.发送请求给es数据，接收响应数据
request.doc(
"email","66666666666@qq.com"
);
UpdateResponse update = client.update(request, RequestOptions.DEFAULT);
// 3.获取请求结果
DocWriteResponse.Result result = update.getResult();
System.out.println(result);
}
// 删除
@Test
public void delete() throws IOException {
DeleteRequest request = new DeleteRequest("heima" , "1");
DeleteResponse delete = client.delete(request, RequestOptions.DEFAULT);
DocWriteResponse.Result result = delete.getResult();
System.out.println("删除的结果为"+result);
}
@AfterEach
void tearDown() throws IOException {
this.client.close();
}
}
SQL添加es
import cn.itcast.hotel.constants.HotelConstants;
import cn.itcast.hotel.pojo.Hotel;
import cn.itcast.hotel.pojo.HotelDoc;
import cn.itcast.hotel.service.IHotelService;
import com.alibaba.fastjson.JSON;
import net.minidev.json.JSONUtil;
import org.apache.http.HttpHost;
import org.elasticsearch.action.DocWriteResponse;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.CreateIndexResponse;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.rest.RestStatus;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.io.IOException;
import java.util.List;
@SpringBootTest
public class HotelTest {
@BeforeEach
void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.136.131:9200") ));
}
@Autowired
private IHotelService iHotelService ;
private RestHighLevelClient client = null;
// 创建索引库
@Test
public void test03() throws IOException {
// 1. 本次向es发送请求为的是创建heima索引库
CreateIndexRequest request = new CreateIndexRequest("hotel");
request.mapping(HotelConstants.MAPPING_TEMPLATE, XContentType.JSON);
// 2. 发送es请求，接收响应结
CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT);
boolean fragment = response.isAcknowledged();
if (fragment){
System.out.println("操作成功");
}
}
// SQL数据添加到es中
@Test
public void test01() throws IOException {
Hotel hotel = iHotelService.findByid(36934l);
// 将数据修改为es的数据格式
HotelDoc hotelDoc = new HotelDoc(hotel);
// 将java数据修改为json
String hotelJson = JSON.toJSONString(hotelDoc);
System.out.println(hotelJson);
// 1. 创建添加语义对象
IndexRequest request = new IndexRequest("hotel");
request.id("36934");
// 设置添加的文档数据
request.source(hotelJson , XContentType.JSON);
// 发送给es
IndexResponse response = client.index(request, RequestOptions.DEFAULT);
DocWriteResponse.Result result = response.getResult();
System.out.println(result);
}
/**
* 向es中，批量添加数据 / 将数据库中的数据全部添加到es中
* @throws IOException
*/
@Test
public void bulk() throws IOException {
// 从mysql中查询所有的数据
List<Hotel> all = iHotelService.findAll();
// 批量创建请求的语义对象
BulkRequest bulkRequest = new BulkRequest("hotel");
// 处理对象并转化
for (Hotel hotel : all ){
HotelDoc hotelDoc = new HotelDoc(hotel);
String hotelJson = JSON.toJSONString(hotelDoc);
// 1. 创建添加语义对象
IndexRequest request = new IndexRequest("hotel");
request.id(hotel.getId()+" ");
// 设置添加的文档数据
request.source(hotelJson , XContentType.JSON);
bulkRequest.add(request);
}
BulkResponse responses = client.bulk(bulkRequest, RequestOptions.DEFAULT);
RestStatus status = responses.status();
System.out.println(responses +" " + status);
}
@AfterEach
void tearDown() throws IOException {
this.client.close();
}
}
查询语法
ES查询语句
全文检索查询 where or
#match 分词查询，匹配一个字段
GET /hotel/_search
{
"query": {
"match": {
"name": "上海"
}
},
"size": 200
}
#math 查询所有字段中包含xx的数据
GET /hotel/_search
{
"query": {
"match": {
"all": "上海"
}
},
"size": 200
}
#multi_match 分词查询,匹配多个字段
GET /hotel/_search
{
"query": {
"multi_match": {
"query": "上海",
"fields": ["name" ,"brand"]
}
},
"size": 200
}
精确查询 (查询条件不分词 between and )
#范围查询 100 - 200
# gt > gte >= lt < lt <=
GET /hotel/_search
{
"query": {
"range": {
"price": {
"gte": 100,
"lte": 200
}
}
}
}
GET /hotel/_search
{
"query": {
"ids": {
"price": [123 , 124]
}
}
}
地理坐标查询 （根据经纬度查询）
#地理坐标查询 矩形范围查询
GET /hotel/_search
{
"query": {
"geo_bounding_box": {
"location": {
"top_left": {
"lat": 31.1,
"lon": 121.5
},
"bottom_right": {
"lat": 30.9,
"lon": 121.7
}
}
}
}
}
#附近查询 geo_distance 查询 圓心坐標+方圓多少公里
GET /hotel/_search
{
"query": {
"geo_distance": {
"distance": "15km",
"location": "31.21,121.5"
}
}
}
复合查询
#在搜索时，es将结果自动排序，根据分值进行排序
GET /hotel/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": "如家酒店"
}
},
"functions": [
{
"filter": {
"term": {
"brand": "如家"
}
},
"weight": 2
}
],
"boost_mode": "sum"
}
}
}
#在搜索时，es将结果自动排序，根据分值进行排序
GET /hotel/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": "如家酒店"
}
},
"functions": [
{
"filter": {
"term": {
"id": "2359697"
}
},
"weight": 200
}
],
"boost_mode": "multiply"
}
}
}
布尔查询 多条件组合
#布尔查询
GET /hotel/_search
{
"query": {
"bool": {
"must": [
{"term": {"city": "上海" }}
],
"should": [
{"term": {"brand": "皇冠假日" }},
{"term": {"brand": "华美达" }}
],
"must_not": [
{ "range": { "price": { "lte": 500 } }}
],
"filter": [
{ "range": {"score": { "gte": 45 } }}
]
}
}
}
排序
# 排序 默认为分值排序 , 如果设置了排序字段，则不再计较分值
# 根据第一个字段进行排序，如果结果一样，根据第二个字段进行排序
GET /hotel/_search
{
"query": {
"term": {
"brand": "如家"
}
},
"sort": [
{
"price": "asc"
},
{
"score":"asc"
}
]
}
GET /hotel/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"_geo_distance" : {
"location": "31.034661 , 121.612282",
"order" : "asc",
"unit" : "km"
}
}
]
}
分页
随机翻页页码
#分页查询 from : 当前页码-1 size：每页显示条数
GET /hotel/_search
{
"query": {
"term": {
"brand": "如家"
}
},
"from": 0 ,
"size": 20
}
手机翻页
高亮搜索
java代码实现
import cn.itcast.hotel.pojo.HotelDoc;
import com.alibaba.fastjson.JSON;
import org.apache.http.HttpHost;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.elasticsearch.search.sort.SortOrder;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.io.IOException;
public class esTset {
private RestHighLevelClient client;
@BeforeEach
void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.136.131:9200")
));
}
@Test // 查询所有
public void EsTest() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.matchAllQuery());
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
// 解析结果
private void handleResponse(SearchResponse response) {
// 4.解析响应
SearchHits searchHits = response.getHits();
// 4.1.获取总条数
long total = searchHits.getTotalHits().value;
System.out.println("共搜索到" + total + "条数据");
// 4.2.文档数组
SearchHit[] hits = searchHits.getHits();
// 4.3.遍历
for (SearchHit hit : hits) {
// 获取文档source
String json = hit.getSourceAsString();
// 反序列化 将json转化为对象
HotelDoc hotelDoc = JSON.parseObject(json, HotelDoc.class);
System.out.println("hotelDoc = " + hotelDoc);
}
}
@Test // 分词查询
public void EsTest1() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.matchQuery("name" , "上海外滩"));
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
@Test // 分词查询多个字段
public void EsTest2() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.multiMatchQuery("上海外滩" , "name" , "brand"));
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
@Test // 范围查询
public void EsTest3() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.rangeQuery("price").gte(100).lte(200));
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
@Test // 词条查询 // 查询内容作为一个整体进行查询
public void EsTest4() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.termQuery("city" , "北京"));
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
// 布尔查询
@Test
public void EsTest5() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(QueryBuilders.termQuery("city" , "上海"));
boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(2000));
// boolQueryBuilder.filter(QueryBuilders.termQuery("city","上海"));
request.source().query(boolQueryBuilder);
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
//排序喝分页
@Test
public void EsTest6() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().query(QueryBuilders.termQuery("city" , "北京"));
// todo：排序
request.source().sort("price" , SortOrder.ASC);
// todo：分页
request.source().from(0);
request.source().size(5);
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
// 高亮排序
@Test
public void EsTest7() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.构建条件
request.source().query(QueryBuilders.matchQuery("all","如家酒店"));
// todo: 高亮查询
request.source().highlighter(
new HighlightBuilder()
.field("name")
.requireFieldMatch(false)
);
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponses(response);
}
// 解析高亮结果数据
private void handleResponses(SearchResponse response) {
// 4.解析响应
SearchHits searchHits = response.getHits();
// 4.1.获取总条数
long total = searchHits.getTotalHits().value;
System.out.println("共搜索到" + total + "条数据");
// 4.2.文档数组
SearchHit[] hits = searchHits.getHits();
// 4.3.遍历
for (SearchHit dohit : hits) {
// todo: 解析高亮
HighlightField highlightField = dohit.getHighlightFields().get("name");
String name = " " ;
if (highlightField!=null){
name = highlightField.getFragments()[0].string();
System.out.println(name);
}
}
}
@AfterEach
void tearDown() throws IOException {
this.client.close();
}
}
查询数据的全部方法
@Test // 查询所有
public void EsTest() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.查询所有
request.source().query(QueryBuilders.matchAllQuery());
// 2.分词查询
request.source().query(QueryBuilders.matchQuery("name" , "上海外滩"));
// 3.范围查询
request.source().query(QueryBuilders.rangeQuery("price").gte(100).lte(200));
// 4. 词条查询查询内容作为一个整体进行查询
request.source().query(QueryBuilders.termQuery("city" , "北京"));
// 5. 布尔查询
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.must(QueryBuilders.termQuery("city" , "上海"));
boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(2000));
request.source().query(boolQueryBuilder);
// todo：排序
request.source().sort("price" , SortOrder.ASC);
// todo：分页
request.source().from(0);
request.source().size(5);
// 6. 高亮查询
// 2.构建条件
request.source().query(QueryBuilders.matchQuery("all","如家酒店"));
// todo: 高亮查询
request.source().highlighter(
new HighlightBuilder()
.field("name")
.requireFieldMatch(false)
);
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
handleResponse(response);
}
开发中的使用
查询的是这个：
// 创建es客户端对象，存入ioc容器中
@Bean
public RestHighLevelClient RestHighLevelClient(){
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.136.131:9200")
));
return client ;
}
--------------------------------------------------------------------------------------------
@Autowired
private RestHighLevelClient client ;
@Override
public PageResult search(RequestParams params) throws IOException {
PageResult pageResult = new PageResult();
Integer page = params.getPage();
Integer size = params.getSize();
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
if (params.getKey()==null || "".equals(params.getKey())){
// 用户没有输入 ， 构建查询所有的DSL
boolQueryBuilder.must(QueryBuilders.matchAllQuery());
}else {
// 用户输入了，使用分词查询
boolQueryBuilder.must(QueryBuilders.matchQuery("all" , params.getKey()));
}
// 构造查询方法
if (params.getCity()!= null && !params.getCity().equals("")){
boolQueryBuilder.filter(QueryBuilders.termQuery("city" , params.getCity() ));
}
if (params.getBrand()!= null && !params.getBrand().equals("")){
boolQueryBuilder.filter(QueryBuilders.termQuery("brand" , params.getBrand() ));
}
if (params.getStarName()!= null && !params.getStarName().equals("")){
boolQueryBuilder.filter(QueryBuilders.termQuery("starName" , params.getStarName() ));
}
if (params.getMinPrice() !=null && params.getMaxPrice()!=null){
boolQueryBuilder.filter(QueryBuilders
.rangeQuery("price")
.gte(params.getMinPrice())
.lte(params.getMaxPrice()));
}
// 2.算分控制
FunctionScoreQueryBuilder functionScoreQuery =
QueryBuilders.functionScoreQuery(
// 原始查询，相关性算分的查询
boolQueryBuilder ,
// function score的数组
new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{
// 其中的一个function score 元素
new FunctionScoreQueryBuilder.FilterFunctionBuilder(
// 过滤条件
QueryBuilders.termQuery("isAD", true),
// 算分函数
ScoreFunctionBuilders.weightFactorFunction(100000)
)
});
// 放布尔查询条件放入request中
request.source().query(functionScoreQuery);
// 地理坐标 排序
String location = params.getLocation();
if (location != null && !location.equals("")) {
request.source().sort(SortBuilders
.geoDistanceSort("location", new GeoPoint(location))
.order(SortOrder.ASC)
.unit(DistanceUnit.KILOMETERS)
);
}
// todo：分页
request.source().from( (page - 1) * size );
request.source().size(10);
// 3.发送请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析响应
SearchHits searchHits = response.getHits();
// 4.1.获取总条数
long total = searchHits.getTotalHits().value;
pageResult.setTotal(total);
System.out.println("共搜索到" + total + "条数据");
// 4.2.文档数组
SearchHit[] hits = searchHits.getHits();
// 4.3.遍历
List<HotelDoc> list = new ArrayList<>();
for (SearchHit hit : hits) {
// 获取文档source
String json = hit.getSourceAsString();
// 反序列化 将json转化为对象
HotelDoc hotelDoc = JSON.parseObject(json, HotelDoc.class);
// 取出距离值
Object[] sortValues = hit.getSortValues();
if (sortValues.length > 0){
hotelDoc.setDistance(sortValues[0]);
}
list.add(hotelDoc);
}
pageResult.setHotels(list);
return pageResult ;
}
自动补全
聚合
分组
多个分组
#聚合函数 根据酒店品牌进行分组
GET /hotel/_search
{
"size": 0, #不需要展示命中的文档数据
"aggs": { # 固定属性 ， 聚合内容
"brandName": { #自定义聚合名称，根据名称获取集合的结果
"terms": { #根据词条进行分桶
"field": "brand", # 分桶字段
"size": 20 # 展示多少个桶
}
}
}
}
# 聚合结果
GET /hotel/_search
{
"size": 0,
"aggs": {
"brandName": {
"terms": {
"field": "brand",
"order":{
"_count": "asc"
},
"size": 20
}
},
"dizi": {
"terms": {
"field": "city",
"size": 10
}
}
}
}
多次分组
GET /hotel/_search
{
"size": 0,
"aggs": {
"brandName": {
"terms": {
"field": "brand",
"size": 20
},
"aggs": {
"cityName": {
"terms": {
"field": "city",
"size": 10
}
}
}
}
}
}
限定查询范围分组
# 限定查询范围
GET /hotel/_search
{
"size": 0,
"query": {
"term": {
"city": "上海"
}
},
"aggs": {
"brandName": {
"terms": {
"field": "brand",
"order":{
"_count": "asc"
},
"size": 20
}
}
}
}
对当前数据进行分组在分组后，对分组后的数据进行求和
GET /hotel/_search
{
"size": 0,
"aggs": {
"brandName": {
"terms": {
"field": "brand",
"size": 20
},
"aggs": {
"cityName": {
"terms": {
"field": "city",
"size": 10
},
"aggs": {
"price_status_name": {
"avg": {
"field": "price"
}
}
}
}
}
}
}
}
最大值 / 最小值 / 平均值 / 求和 -> 必须要带查询
srats：统计所有
# 先对数据进行分组，对组内的数据进行度量
GET /hotel/_search
{
"size": 0,
"aggs": {
"brandName": {
"terms": {
"field": "brand",
"order":{
"price_score_name.max": "desc"
},
"size": 20
},
"aggs": {
"price_status_name": {
"stats": {
"field": "price"
}
},
"price_score_name": {
"stats": {
"field": "score"
}
}
}
}
}
}
java代码实现分组 / 平均值…
分组的代码实现
import java.io.IOException;
import java.util.List;
public class Dotets {
private RestHighLevelClient client;
@BeforeEach
void setUp() {
this.client = new RestHighLevelClient(RestClient.builder(
HttpHost.create("http://192.168.136.131:9200")
));
}
@Test
public void test02() throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
request.source().size(0);
// 2. 设置分组数据
request.source().aggregation(
AggregationBuilders.terms("brandName")
.field("brand")
.size(10)
);
// 城市分组
request.source().aggregation(
AggregationBuilders.terms("citydName")
.field("city")
.size(10)
);
// 3. 发送请求给es，并接收结果
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
handleResponse(response , "brandName");
System.out.println("---------------------------------------------");
handleResponse(response , "citydName");
}
private void handleResponse(SearchResponse response , String name) {
// 4. 解析聚合结果
Aggregations aggregations = response.getAggregations();
Terms terms = aggregations.get(name);
List<? extends Terms.Bucket> buckets = terms.getBuckets();
for (Terms.Bucket bucket : buckets){
Object key = bucket.getKey();
long docCount = bucket.getDocCount();
System.out.println(key +" "+ docCount);
}
}
@AfterEach
void tearDown() throws IOException {
this.client.close();
}
}
查询
@Autowired
private RestHighLevelClient client ;
/**
* 聚合查询酒店的星级
* @param params
* @return
*/
@Override
public Map<String, List<String>> getFilters(RequestParams params) throws IOException {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
buildBasicQuery(params , request);
request.source().size(0);
// 2. 设置分组数据
request.source().aggregation(
AggregationBuilders.terms("brandAgg")
.field("brand")
.size(100)
);
// 城市分组
request.source().aggregation(
AggregationBuilders.terms("cityAgg")
.field("city")
.size(100)
);
//
request.source().aggregation(AggregationBuilders
.terms("starAgg")
.field("starName")
.size(100));
// 3. 发送请求给es，并接收结果
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析结果
Aggregations aggregations = response.getAggregations();
List<String> list1 = handleResponse(aggregations, "brandAgg");
List<String> list2 = handleResponse(aggregations, "cityAgg");
List<String> list3 = handleResponse(aggregations, "starAgg");
Map<String, List<String>> map = new HashMap<>(3);
map.put("brand" , list1);
map.put("city" , list2);
map.put("starName" , list3);
return map ;
}
private List<String> handleResponse(Aggregations aggregations , String name) {
// 4. 解析聚合结果
Terms terms = aggregations.get(name);
List<String> list = new ArrayList<>();
List<? extends Terms.Bucket> buckets = terms.getBuckets();
if (buckets != null){
for (Terms.Bucket bucket : buckets){
String key = bucket.getKeyAsString();
list.add(key);
}
}
return list ;
}
private void buildBasicQuery(RequestParams params, SearchRequest request) {
// 1.准备Boolean查询
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
// 1.1.关键字搜索，match查询，放到must中
String key = params.getKey();
if (StringUtils.isNotBlank(key)) {
// 不为空，根据关键字查询
boolQuery.must(QueryBuilders.matchQuery("all", key));
} else {
// 为空，查询所有
boolQuery.must(QueryBuilders.matchAllQuery());
}
// 1.2.品牌
String brand = params.getBrand();
if (StringUtils.isNotBlank(brand)) {
boolQuery.filter(QueryBuilders.termQuery("brand", brand));
}
// 1.3.城市
String city = params.getCity();
if (StringUtils.isNotBlank(city)) {
boolQuery.filter(QueryBuilders.termQuery("city", city));
}
// 1.4.星级
String starName = params.getStarName();
if (StringUtils.isNotBlank(starName)) {
boolQuery.filter(QueryBuilders.termQuery("starName", starName));
}
// 1.5.价格范围
Integer minPrice = params.getMinPrice();
Integer maxPrice = params.getMaxPrice();
if (minPrice != null && maxPrice != null) {
maxPrice = maxPrice == 0 ? Integer.MAX_VALUE : maxPrice;
boolQuery.filter(QueryBuilders.rangeQuery("price").gte(minPrice).lte(maxPrice));
}
// 2.算分函数查询
FunctionScoreQueryBuilder functionScoreQuery = QueryBuilders.functionScoreQuery(
boolQuery, // 原始查询，boolQuery
new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{ // function数组
new FunctionScoreQueryBuilder.FilterFunctionBuilder(
QueryBuilders.termQuery("isAD", true), // 过滤条件
ScoreFunctionBuilders.weightFactorFunction(10) // 算分函数
)
}
);
// 3.设置查询条件
request.source().query(functionScoreQuery);
}
自动补全
拼音分词器
自动补全
// 自动补全
@Override
public List<String> getsuggetions(String prefix) {
try {
// 1.准备Request
SearchRequest request = new SearchRequest("hotel");
// 2.准备DSL
request.source().suggest(new SuggestBuilder().addSuggestion(
"suggestions", // 自定义补全字段名称
SuggestBuilders.completionSuggestion("suggestion") // 补全字段
.prefix(prefix) // 补全字段的关键字
.skipDuplicates(true) // 跳过重复的
.size(10) // 获取前10条结果
));
// 3.发起请求
SearchResponse response = client.search(request, RequestOptions.DEFAULT);
// 4.解析结果
Suggest suggest = response.getSuggest();
// 4.1.根据补全查询名称，获取补全结果
CompletionSuggestion suggestions = suggest.getSuggestion("suggestions");
// 4.2.获取options
List<CompletionSuggestion.Entry.Option> options = suggestions.getOptions();
// 4.3.遍历
List<String> list = new ArrayList<>(options.size());
for (CompletionSuggestion.Entry.Option option : options) {
String text = option.getText().toString();
list.add(text);
}
return list;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
数据同步
spring:
rabbitmq:
host: 192.168.136.131
port: 5672
virtual-host: /
username: xiaowang
password: 123321
- 解决方式
- 1.同步调用，在操作sql完成新增/修改/删除操作时，对es完成同样的操作
-
- 利用rabbq发送消息，当新增成功后，生产者向rabbq发送消息，
生产者：
消费者：
- 利用rabbq发送消息，当新增成功后，生产者向rabbq发送消息，
es集群
集群下的每一台服务器叫做节点，多个节点组成集群，先将海量数据进行分片，在对数据进行分配
- 解决海量数据存储
- 答：将索引拆分为每一片存储到不同的服务器上
- 如果服务器宕机，单点故障
- 答：分片构建副本
节点划分：
节点的作用
脑裂问题
在es7.0之后修复了这个问题，添加了hash运算，当主节点发生为问题后，副本节点进行投票（当前集群+1）%2 的方法进行推选主节点，当宕机的主节点修复后，变为从节点
故障转移 / 动态伸缩
- 当主节点发生问题后没推选新的主节点，会将宕机的主节点的分片数据迁移到其他节点上，保证数据的安全，当宕机的主节点恢复后，变为从节点，其他节点将分片信息返回给宕机的主节点
新增流程过程
- 新增文档后，进行hash运算，由主节点进行分发，路由到对应的节点分片中，分片进行保存文档的操作，同步给分片副本，将结果返回给主节点
- 查询
- 分散阶段，先会找到协同节点（主节点），将请求分散到每一个分片上
- 聚集阶段，进行数据的操作，将数据返回至协调节点，操作数据，将数据返回给用户
扫一扫
2103
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
