Kafka单机部署

最新推荐文章于 2025-05-09 09:45:00 发布

YongYu_IT

最新推荐文章于 2025-05-09 09:45:00 发布

阅读量626

点赞数

CC 4.0 BY-SA版权

分类专栏：分布式系统

本文链接：https://blog.youkuaiyun.com/YongYu_IT/article/details/81453880

分布式系统专栏收录该内容

23 篇文章

订阅专栏

一般而言，kafka集群和zookeeper集群是分开（分机）部署的，我们尝试一下。

所有配置文件下载地址：https://download.youkuaiyun.com/download/yongyu_it/10586668

1、参照《Zookeeper集群环境搭建》搭建zookeeper集群

环境配置如

$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── cmd
│   ├── start1.sh
│   ├── start2.sh
│   └── start3.sh
└── docker-compose-zookeeper.yml

1 directory, 4 files

docker-compose-zookeeper.yml

version: '2'
 
networks:
  thinking:
  
services:
  test_1.thinking.com:
    image: ubuntu
    container_name: test_1.thinking.com
    command: sh /cmd/start1.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin
 
 
  test_2.thinking.com:
    image: ubuntu
    container_name: test_2.thinking.com
    command: sh /cmd/start2.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin
 
  test_3.thinking.com:
    image: ubuntu
    container_name: test_3.thinking.com
    command: sh /cmd/start3.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12
      - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin

start1.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 1 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

start2.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 2 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

start3.sh

mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 3 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bash

启动集群

$ docker-compose -f docker-compose-zookeeper.yml up -d
测试集群

$ docker exec -it test_1.thinking.com /bin/bash
root@ed717de94636:/# cd /home/zookeeper-3.4.12/bin
root@ed717de94636:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:52:37,981 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-08-06 06:52:37,984 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=ed717de94636
2018-08-06 06:52:37,984 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_171
2018-08-06 06:52:37,986 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-08-06 06:52:37,986 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper-3.4.12/bin/../build/classes:/home/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/zookeeper-3.4.12/bin/../conf:
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.13.0-45-generic
2018-08-06 06:52:37,987 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2018-08-06 06:52:37,988 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2018-08-06 06:52:37,988 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper-3.4.12/bin
2018-08-06 06:52:37,989 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@255316f2
Welcome to ZooKeeper!
2018-08-06 06:52:38,005 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-08-06 06:52:38,068 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2018-08-06 06:52:38,104 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100003216670000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
create /zk-book 123
Created /zk-book
[zk: localhost:2181(CONNECTED) 1] get /zk-book
123
cZxid = 0x100000002
ctime = Mon Aug 06 06:52:44 GMT 2018
mZxid = 0x100000002
mtime = Mon Aug 06 06:52:44 GMT 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 2] close
2018-08-06 06:52:55,164 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x100003216670000 closed
[zk: localhost:2181(CLOSED) 3] 2018-08-06 06:52:55,165 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x100003216670000
root@ed717de94636:/home/zookeeper-3.4.12/bin#

$ docker exec -it test_2.thinking.com /bin/bash
root@53c259066ce4:/# cd /home/zookeeper-3.4.12/bin
root@53c259066ce4:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:55:25,814 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-08-06 06:55:25,817 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=53c259066ce4
2018-08-06 06:55:25,817 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_171
2018-08-06 06:55:25,819 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper-3.4.12/bin/../build/classes:/home/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/zookeeper-3.4.12/bin/../conf:
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-08-06 06:55:25,820 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=4.13.0-45-generic
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2018-08-06 06:55:25,821 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper-3.4.12/bin
2018-08-06 06:55:25,822 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@255316f2
Welcome to ZooKeeper!
2018-08-06 06:55:25,843 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-08-06 06:55:25,913 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-08-06 06:55:25,939 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x200003212ac0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] get /zk-book
123
cZxid = 0x100000002
ctime = Mon Aug 06 06:52:44 GMT 2018
mZxid = 0x100000002
mtime = Mon Aug 06 06:52:44 GMT 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] close
2018-08-06 06:55:36,662 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x200003212ac0000 closed
[zk: localhost:2181(CLOSED) 2] 2018-08-06 06:55:36,663 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x200003212ac0000
root@53c259066ce4:/home/zookeeper-3.4.12/bin#

2、启动kafka服务器（单机模式）

参考《kafka入门与实践》（牟大恩）

先编写docker compose启动脚本

$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── docker-compose-kafka-sig.yml
└── kafka-sig-cmd
    └── kafka-sig.sh

1 directory, 2 files

docker-compose-kafka-sig.yml

version: '2'
 
networks:
  thinking:

services:
  kafka.thinking.com:
    image: ubuntu
    container_name: kafka.thinking.com
    command: sh /cmd/kafka-sig.sh
    tty: true
    networks:
      - thinking
    volumes:
      - /etc/java-8-openjdk:/etc/java-8-openjdk
      - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
      - /home/yong/Desktop/kafka-test-20180806/kafka-sig-cmd:/cmd
      - /home/yong/Desktop/kafka_2.11-1.1.1:/usr/local/kafka
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
      - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin

改写kafka配置文件/usr/local/kafka/config/server. properties文件，主要是改写

broker.id

host.name

log.dirs

zookeeper.connect

broker.id=1
host.name=kafka.thinking.com
log.dirs=/tmp/kafka-logs
zookeeper.connect=test_1.thinking.com:2181,test_2.thinking.com:2181,test_3.thinking.com:2181

编写kafka-sig.sh文件

mkdir -p /tmp/kafka-logs
cd /usr/local/kafka/bin
ln -sf bash /bin/sh
./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
/bin/bash

启动kafka

$ docker-compose -f docker-compose-kafka-sig.yml up -d

检验kafka

$ docker exec -it kafka.thinking.com /bin/bash
# cd /usr/local/kafka/bin
# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
# ./kafka-console-producer.sh --broker-list kafka.thinking.com:9092 --topic test
>Fuck you 1001
>Fuck you 1002
# ./kafka-console-consumer.sh --zookeepertest_1.thinking.com:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Fuck you 1001
Fuck you 1002
^CProcessed a total of 2 messages

成功！