Hadoop集群版本必须和Druid.io中版本同一，可以通过pull-deps下载相同hadoop-dependencies版本,e.g. :

Druid：Druid.io 部署&使用文档

时间  2016-12-23

标签 海量数据 栏目 Java开源

原文   http://blog.youkuaiyun.com/silentwolfyh/article/details/53843335

 

参考：

---

http://lxw1234.com/archives/2015/11/554.htm 海量数据实时OLAP分析系统-Druid.io安装配置和体验 
http://druid.io/docs/0.9.2/design/design.html Druid官网搭建

Druid.io 部署&使用文档

---

1.集群模式下部署

Prerequisites : Java 7 or higher & Zookeeper & mysql

下载Druid.io :

curl -O http://static.druid.io/artifacts/releases/druid-0.9.1.1-bin.tar.gz
tar -xzf druid-0.9.1.1-bin.tar.gz
cd druid-0.9.1.1

文件夹目录结构 ：

• LICENSE - the license files.
• bin/ - scripts related to the single-machine quickstart.
• conf/* - template configurations for a clustered setup.
• conf-quickstart/* - configurations for the single-machine quickstart.
• extensions/* - all Druid extensions.
• hadoop-dependencies/* - Druid Hadoop dependencies.
• lib/* - all included software packages for core Druid.
• quickstart/* - files related to the single-machine quickstart.

所有配置文件均在 conf/* 目录下.

配饰HDFS为Druid.io的deep storage & 配置zk & 配置mysql

修改 conf/druid/_common/common.runtime.properties 文件.

#
# Extensions
#

# This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
# based on your particular setup.
#使用 "mysql-metadata-storage" 作为metadata的存储
#使用 "druid-hdfs-storage" 作为 deep storage
#使用 "druid-parquet-extensions" 向druid中插入parquet数据
druid.extensions.loadList=["druid-kafka-eight", "druid-histogram", "druid-datasketches",  "mysql-metadata-storage", "druid-hdfs-storage", "druid-avro-extensions", "druid-parquet-extensions"]

# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
# and uncomment the line below to point to your directory.
#druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies

#
# Logging
#

# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true

#
# Zookeeper
#

druid.zk.service.host=tagtic-slave01:2181,tagtic-slave02:2181,tagtic-slave03:2181
druid.zk.paths.base=/druid

#
# Metadata storage
#

# For MySQL:
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://tagtic-master:3306/druid
druid.metadata.storage.connector.user=root
druid.metadata.storage.connector.password=123456

#
# Deep storage
#

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
druid.storage.type=hdfs
druid.storage.storageDirectory=/druid/segments

#
# Indexing service logs
#

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=/druid/indexing-logs

#
# Service discovery
#

druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator

#
# Monitoring
#

druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]
druid.emitter=logging
druid.emitter.logging.logLevel=info

将 Hadoop 的配置文件(core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml) cp 到 conf/druid/_common 目录下

修改 conf/druid/middleManager/runtime.properties 文件.

druid.service=druid/middleManager
druid.port=18091

# Number of tasks per middleManager
druid.worker.capacity=3

# Task launch parameters
# **CDH版本添加 -Dhadoop.mapreduce.job.classloader=true 来解决hadoop indexer导入时jar包冲突问题**
druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dhadoop.mapreduce.job.classloader=true

druid.indexer.task.baseTaskDir=var/druid/task

# HTTP server threads
druid.server.http.numThreads=25

# Processing threads and buffers
druid.processing.buffer.sizeBytes=536870912
druid.processing.numThreads=2

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=/tmp/druid-indexing
druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]

Hadoop集群版本必须和Druid.io中版本同一，可以通过pull-deps下载相同hadoop-dependencies版本,e.g. : 
java -classpath "lib/*" io.druid.cli.Main tools pull-deps --defaultVersion 0.9.1.1 -c io.druid.extensions:mysql-metadata-storage:0.9.1.1 -c druid-hdfs-storage -h org.apache.hadoop:hadoop-client:2.6.0

项目中Druid.io配置端口号：

• druid.service=druid/coordinator druid.port=18081
• druid.service=druid/broker druid.port=18082
• druid.service=druid/historical druid.port=18083
• druid.service=druid/overlord druid.port=18090
• druid.service=druid/middleManager druid.port=18091

2.启动Druid.io

java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &>> logs/coordinator.log &

java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &>> logs/overlord.log &

java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &>> logs/historical.log &

java `cat conf/druid/middleManager/jvm.config | xargs` -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &>> logs/middleManager.log &

java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &>> logs/broker.log &

3.从HDFS导入数据到Druid.io

批量导入Batch Data Ingestion 
导入Parquet文件

Druid作业查看 Coordinator : http://tagtic-master:18090/console.html 
Druid集群查看Cluster : http://tagtic-master:18081/#/

解决传入数据时区问题 Hadoop Configuration，在conf/druid/_common/mapred-site.xml中添加

<property>
    <name>mapreduce.map.java.opts</name>
    <value>-server -Xmx1536m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-server -Xmx2560m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value>
</property>