elasticsearch学习笔记

最新推荐文章于 2022-05-20 10:31:04 发布

原创最新推荐文章于 2022-05-20 10:31:04 发布 · 1k 阅读

0 ·

CC 4.0 BY-SA版权

elasticsearch 专栏收录该内容

3 篇文章

订阅专栏

本文介绍了解决Elasticsearch在写入数据时遇到的异常情况，包括配置synonym.txt文件来避免错误，以及如何通过调整Spark配置来实现更灵活的数据写入操作。

1.Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot determine write shards for

在config/analysis下创建 synonym.txt文件即可

The index operation automatically creates an index if it has not been created before (check out thecreate index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the put mapping API for manually creating a type mapping)

es 会自动创建index和type如果不存在的话

//利用隐式转换，native的支持

implicit class ESConfig(conf: SparkConf) {

def build(index: String, typeTable: String, mapId: Option[String] = None): SparkConf = {
val sc = conf
// .set("spark.buffer.pageSize", "8m")
.set("es.nodes", config.nodes)
.set("es.port", config.port)
.set("es.scroll.size", "2000")
.set("es.resource", s"$index/$typeTable")
.set("es.index.auto.create", "true")
.set("es.write.operation", "upsert")
.set("es.batch.size.entries", config.batchSize)
mapId match {
case Some(id) => sc.set("es.mapping.id", id)
case None => sc
}
}

}