Kafka

官网地址:http://kafka.apache.org

版本记录:http://kafka.apache.org/documentation/#upgrade

下载链接:http://archive.apache.org/dist/kafka/

一、基本环境说明

系统版本

Centos6.8

运行用户root

Kafka版本

2.11-1.1.0

二、Kafka文件说明

安装目录:/usr/local/kafka

配置文件:/usr/local/kafka/config

启动脚本:/usr/local/kafka/bin

三、安装部署

1.准备 Java 运行环境;

2.搭建ZK;

2.下载源码包,上传到/usr/local/src目录;

3.解压:tar zxvf kafka_2.11-0.11.0.3.tgz -C /usr/local/

4. 修改配置文件;

四、配置文件说明

官方链接:http://kafka.apache.org/documentation.html#brokerconfigs

server.properties

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

 

# see kafka.server.KafkaConfig for additional details and defaults

 

############################# Server Basics #############################

 

# The id of the broker. This must be set to a unique integer for each broker.

# 节点的ID,必须与其它节点不同

broker.id=0

 

# Switch to enable topic deletion or not, default value is false

# 选择启用删除主题功能,默认false

#delete.topic.enable=true

 

############################# Socket Server Settings #############################

 

# The address the socket server listens on. It will get the value returned from

# java.net.InetAddress.getCanonicalHostName() if not configured.

# 套接字服务器坚挺的地址。如果没有配置,就使用java.net.InetAddress.getCanonicalHostName()的返回值

#   FORMAT:

#     listeners = listener_name://host_name:port

#   EXAMPLE:

#     listeners = PLAINTEXT://your.host.name:9092

#listeners=PLAINTEXT://:9092

 

# Hostname and port the broker will advertise to producers and consumers. If not set,

# it uses the value for "listeners" if configured.  Otherwise, it will use the value

# returned from java.net.InetAddress.getCanonicalHostName().

# 节点的主机名会通知给生产者和消费者。如果没有设置,如果配置了"listeners"就使用"listeners"的值。

# 否则就使用java.net.InetAddress.getCanonicalHostName()的返回值

#advertised.listeners=PLAINTEXT://your.host.name:9092

 

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details

# 将侦听器的名称映射到安全协议,默认情况下它们是相同的。有关详细信息,请参阅配置文档

#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

 

# The number of threads that the server uses for receiving requests from the network and sending responses to the network

# 服务器用来接受请求或者发送响应的线程数

num.network.threads=3

 

# The number of threads that the server uses for processing requests, which may include disk I/O

# 服务器用来处理请求的线程数,可能包括磁盘IO

num.io.threads=8

 

# The send buffer (SO_SNDBUF) used by the socket server

# 套接字服务器使用的发送缓冲区大小

socket.send.buffer.bytes=102400

 

# The receive buffer (SO_RCVBUF) used by the socket server

# 套接字服务器使用的接收缓冲区大小

socket.receive.buffer.bytes=102400

 

# The maximum size of a request that the socket server will accept (protection against OOM)

# 单个请求最大能接收的数据量

socket.request.max.bytes=104857600

 

 

############################# Log Basics #############################

 

# A comma seperated list of directories under which to store log files

# 一个逗号分隔的目录列表,用来存储日志文件

log.dirs=/tmp/kafka-logs

 

# The default number of log partitions per topic. More partitions allow greater

# parallelism for consumption, but this will also result in more files across

# the brokers.

# 每个主题的日志分区的默认数量。更多的分区允许更大的并行操作,但是它会导致节点产生更多的文件

num.partitions=1

 

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

# This value is recommended to be increased for installations with data dirs located in RAID array.

# 每个数据目录中的线程数,用于在启动时日志恢复,并在关闭时刷新。

num.recovery.threads.per.data.dir=1

 

############################# Internal Topic Settings  #############################

# 内部主题设置

# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"

# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.

# 对于除了开发测试之外的其他任何东西,group元数据内部主题的复制因子“__consumer_offsets”和“__transaction_state”,建议值大于1,以确保可用性(如3)。

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

 

############################# Log Flush Policy #############################

 

# Messages are immediately written to the filesystem but by default we only fsync() to sync

# the OS cache lazily. The following configurations control the flush of data to disk.

# 消息直接被写入文件系统,但是默认情况下我们仅仅调用fsync()以延迟的同步系统缓存

# There are a few important trade-offs here:

# 这些有一些重要的权衡

#    1. Durability: Unflushed data may be lost if you are not using replication.

#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.

#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.

#    1. 持久性:如果不使用复制,未刷新的数据可能会丢失。

#    2. 延迟:非常大的刷新间隔可能会在刷新时导致延迟,因为将会有大量数据刷新。

#    3. 吞吐量:刷新通常是最昂贵的操作,而一个小的刷新间隔可能会导致过多的搜索。

# The settings below allow one to configure the flush policy to flush data after a period of time or

# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# 下面的设置允许你去配置刷新策略,每隔一段时间刷新或者一次N个消息(或者两个都配置)。这可以在全局范围内完成,并在每个主题的基础上重写。

 

# The number of messages to accept before forcing a flush of data to disk

# 在强制刷新数据到磁盘之前允许接收消息的数量

#log.flush.interval.messages=10000

 

# The maximum amount of time a message can sit in a log before we force a flush

# 在强制刷新之前,消息可以在日志中停留的最长时间

#log.flush.interval.ms=1000

 

############################# Log Retention Policy #############################

 

# The following configurations control the disposal of log segments. The policy can

# be set to delete segments after a period of time, or after a given size has accumulated.

# 以下的配置控制了日志段的处理。策略可以配置为每隔一段时间删除片段或者到达一定大小之后。

# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens

# from the end of the log.

# 当满足这些条件时,将会删除一个片段。删除总是发生在日志的末尾。

 

# The minimum age of a log file to be eligible for deletion

# 一个日志的最小存活时间,可以被删除

log.retention.hours=168

 

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining

# segments don't drop below log.retention.bytes.

# 一个基于大小的日志保留策略。段将被从日志中删除只要剩下的部分段不低于log.retention.bytes。

#log.retention.bytes=1073741824

 

# The maximum size of a log segment file. When this size is reached a new log segment will be created.

# 每一个日志段大小的最大值。当到达这个大小时,会生成一个新的片段。

log.segment.bytes=1073741824

 

# The interval at which log segments are checked to see if they can be deleted according

# to the retention policies

# 检查日志段的时间间隔,看是否可以根据保留策略删除它们

log.retention.check.interval.ms=300000

 

############################# Zookeeper #############################

 

# Zookeeper connection string (see zookeeper docs for details).

# Zookeeper连接字符串(具体见Zookeeper文档)

# This is a comma separated host:port pairs, each corresponding to a zk

# 这是一个以逗号为分割的部分,每一个都匹配一个Zookeeper

# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

# You can also append an optional chroot string to the urls to specify the

# root directory for all kafka znodes.

# 您还可以将一个可选的chroot字符串附加到url,以指定所有kafka znode的根目录。

zookeeper.connect=localhost:2181

 

# Timeout in ms for connecting to zookeeper

# 连接到Zookeeper的超时时间

zookeeper.connection.timeout.ms=6000

 

 

############################# Group Coordinator Settings #############################

 

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.

# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.

# The default value for this is 3 seconds.

# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.

# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.

group.initial.rebalance.delay.ms=0

 

 

五、其他

1、启动服务

cd /usr/local/kafka ; nohup ./bin/kafka-server-start.sh config/server.properties &

2、停止服务

cd /usr/local/kafka ; sh  ./bin/kafka-server-stop.sh

3、创建TOPIC

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic test

4、查看TOPIC

./bin/kafka-topics.sh --list --zookeeper localhost:2181 
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

5、删除TOPIC

./bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic test --zookeeper localhost:2181 

6、生产者
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
7、消费者 
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092--topic test --from-beginning

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值