实时数仓、基于Flink1.11的SQL构建实时数仓 之kafka集群搭建
目录
一:环境准备
jdk1.8
kafka_2.11-2.4.1
集群规划详见:实时数仓、基于Flink1.11的SQL构建实时数仓 环境说明
zookeeper集群 详见:实时数仓、基于Flink1.11的SQL构建实时数仓 之 zookeeper集群搭建
集群节点:
192.168.137.121 m1
192.168.137.122 m2
192.168.137.123 s1
二:安装配置
1.下载安装包:
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.11-2.4.1.tgz
2.上传安装包到 m1节点 并解压到:/opt/
tar -xzf kafka_2.11-2.4.1.tgz
3.配置 server.properties
进入目录:/opt/kafka_2.11-2.4.1/config
[root@m1 config]# cat server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
#注意修改每台机器不要重复
broker.id=1
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# &nb

本文档详细介绍了如何在集群环境中搭建基于Flink1.11的实时数仓,重点是Kafka集群的配置和启动。首先,明确了环境需求,包括JDK1.8和Kafka2.4.1。接着,详细阐述了从下载安装包、上传到节点、配置server.properties和zookeeper.properties,再到分发到其他节点的步骤。最后,提供了启动Kafka集群的命令,并给出批量执行脚本,帮助快速启动集群。请注意修改配置文件中的broker.id和zookeeper.connect以适应集群设置。
最低0.47元/天 解锁文章
1253

被折叠的 条评论
为什么被折叠?



