实时数仓、基于Flink1.11的SQL构建实时数仓之kafka集群搭建

最新推荐文章于 2023-12-20 10:01:30 发布

原创

最新推荐文章于 2023-12-20 10:01:30 发布 · 645 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #zookeeper

本文档详细介绍了如何在集群环境中搭建基于Flink1.11的实时数仓，重点是Kafka集群的配置和启动。首先，明确了环境需求，包括JDK1.8和Kafka2.4.1。接着，详细阐述了从下载安装包、上传到节点、配置server.properties和zookeeper.properties，再到分发到其他节点的步骤。最后，提供了启动Kafka集群的命令，并给出批量执行脚本，帮助快速启动集群。请注意修改配置文件中的broker.id和zookeeper.connect以适应集群设置。

实时数仓、基于Flink1.11的SQL构建实时数仓之kafka集群搭建

一：环境准备

二：安装配置

1.下载安装包：

2.上传安装包到 m1节点并解压到：/opt/

3.配置 server.properties

4. 配置 zookeeper.properties

5.分发到 m2 s1

三.启动集群：

一：环境准备

jdk1.8

kafka_2.11-2.4.1

集群规划详见：实时数仓、基于Flink1.11的SQL构建实时数仓环境说明

zookeeper集群详见：实时数仓、基于Flink1.11的SQL构建实时数仓之 zookeeper集群搭建

集群节点：

192.168.137.121 m1
192.168.137.122 m2
192.168.137.123 s1

二：安装配置

1.下载安装包：

https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.11-2.4.1.tgz

2.上传安装包到 m1节点并解压到：/opt/

tar -xzf kafka_2.11-2.4.1.tgz

3.配置 server.properties

进入目录：/opt/kafka_2.11-2.4.1/config

[root@m1 config]# cat server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.

#注意修改每台机器不要重复
broker.id=1

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# &nb