kcat - Apache Kafka producer and consumer tool

原创已于 2025-06-16 18:33:42 修改 · 624 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #分布式

于 2024-12-10 13:58:17 首次发布

DBA 专栏收录该内容

1002 篇文章

订阅专栏

目录标题

- ✅ `kcat` 使用详解（原 `kafkacat`）

✅ `kcat` 使用详解（原 `kafkacat`）

kcat 是一个轻量级、命令行下的 Kafka 消费/生产/元数据查看工具，基于 librdkafka 构建，适合测试、调试 Kafka 消息流。支持 SASL、TLS、Schema Registry、Avro、JSON 等高级功能。

🔧 典型使用命令说明

docker run --rm --name=kcat --network=host \
  k8smaster.qfusion.irds/irds/kcat:1.7.1 \
  -b 10.10.x.x:2187 \
  -X security.protocol=SASL_PLAINTEXT \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username='kafka-6c7c3c69-admin' \
  -X sasl.password='xxx' \
  -t bpx \
  -L

🔍 参数解释

参数	说明
`--rm`	容器退出后自动删除
`--network=host`	共享宿主机网络，用于连接 Kafka 主机端口
`-b <ip:port>`	Kafka Broker 地址（多个逗号分隔）
`-X security.protocol=...`	设置 Kafka 安全协议（如 `SASL_PLAINTEXT`, `SASL_SSL`）
`-X sasl.mechanism=...`	使用的认证机制，如 `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`
`-X sasl.username=...`	Kafka 登录用户名
`-X sasl.password=...`	Kafka 登录密码
`-t <topic>`	指定 Topic 名称（如 bpx）
`-L`	查看元数据（列出所有 Topic、分区等）

🧭 使用模式总览

模式	命令	描述
消费模式	`-C`	从 Kafka 消费消息
生产模式	`-P`	向 Kafka 发送消息
元数据模式	`-L`	列出 Broker 和 Topic 元数据信息
查询模式	`-Q`	查询某时间戳对应的 offset

📥 消费 Kafka 消息

kcat -C -b 10.10.x.x:2187 -t test-topic -o beginning -e -c 10

-C：消费模式
-t：Topic 名
-o beginning：从起始位置读取（支持：beginning、end、具体 offset、相对 offset 如 -10）
-c 10：最多读取 10 条消息
-e：读取完最后一条消息后退出

📤 生产 Kafka 消息

echo "hello kafka" | kcat -P -b 10.10.x.x:2187 -t test-topic

-P：生产模式，从 stdin（或文件）读取消息并发送
-t：指定发送的 Topic

🧾 查询指定时间戳的 offset

kcat -Q -b 10.10.x.x:2187 -t mytopic:0:1684200000000

-Q：查询模式
-t topic:partition:timestamp：查询某分区在给定时间戳（ms）对应的 offset

🗂 输出格式控制

-f：自定义输出格式
常用格式变量如下：

变量	含义
`%s`	消息内容
`%k`	消息键
`%t`	Topic 名
`%p`	分区编号
`%o`	消息 offset
`%T`	时间戳（ms）

示例：

kcat -C -b broker -t topic -f 'Topic %t [%p] offset %o: %s\n'

🔐 安全连接选项示例（SASL）

-X security.protocol=SASL_PLAINTEXT
-X sasl.mechanism=SCRAM-SHA-512
-X sasl.username=...
-X sasl.password=...

支持的协议：PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
支持的机制：PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, OAUTHBEARER

🧪 实用命令汇总

1. 查看 Kafka 所有 Topic 和分区信息

kcat -L -b broker_ip:port

2. 消费某 Topic 的最近一条消息

kcat -C -b broker -t topic -o -1 -c 1 -e

3. 消费消息并以 JSON 输出

kcat -C -b broker -t topic -J

4. 生产多条消息（批量发送）

printf "msg1\nmsg2\nmsg3\n" | kcat -P -b broker -t topic

📌 Tips

如果你频繁使用 kcat，可以将配置保存到文件中，使用 -F 引用：

cat ~/.config/kcat.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.username=kafka-user
sasl.password=secret

执行时只需：

kcat -F ~/.config/kcat.conf -b broker -t topic -C

如果你希望我生成一份 企业标准格式的 kcat 使用手册文档（带截图、示例脚本等），也可以告诉我。

Usage: kcat <options> [file1 file2 .. | topic1 topic2 ..]]
kcat - Apache Kafka producer and consumer tool
https://github.com/edenhill/kcat
Copyright (c) 2014-2021, Magnus Edenhill
Version 1.7.1 (JSON, Avro, Transactions, IncrementalAssign, JSONVerbatim, librdkafka 1.8.2 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)


General options:
  -C | -P | -L | -Q  Mode: Consume, Produce, Metadata List, Query mode
  -G <group-id>      Mode: High-level KafkaConsumer (Kafka >=0.9 balanced consumer groups)
                     Expects a list of topics to subscribe to
  -t <topic>         Topic to consume from, produce to, or list
  -p <partition>     Partition
  -b <brokers,..>    Bootstrap broker(s) (host[:port])
  -D <delim>         Message delimiter string:
                     a-z | \r | \n | \t | \xNN ..
                     Default: \n
  -K <delim>         Key delimiter (same format as -D)
  -c <cnt>           Limit message count
  -m <seconds>       Metadata (et.al.) request timeout.
                     This limits how long kcat will block
                     while waiting for initial metadata to be
                     retrieved from the Kafka cluster.
                     It also sets the timeout for the producer's
                     transaction commits, init, aborts, etc.
                     Default: 5 seconds.
  -F <config-file>   Read configuration properties from file,
                     file format is "property=value".
                     The KCAT_CONFIG=path environment can also be used, but -F takes precedence.
                     The default configuration file is $HOME/.config/kcat.conf
  -X list            List available librdkafka configuration properties
  -X prop=val        Set librdkafka configuration property.
                     Properties prefixed with "topic." are
                     applied as topic properties.
  -X schema.registry.prop=val Set libserdes configuration property for the Avro/Schema-Registry client.
  -X dump            Dump configuration and exit.
  -d <dbg1,...>      Enable librdkafka debugging:
                     all,generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf
  -q                 Be quiet (verbosity set to 0)
  -v                 Increase verbosity
  -E                 Do not exit on non-fatal error
  -V                 Print version
  -h                 Print usage help

Producer options:
  -z snappy|gzip|lz4 Message compression. Default: none
  -p -1              Use random partitioner
  -D <delim>         Delimiter to split input into messages
  -K <delim>         Delimiter to split input key and message
  -k <str>           Use a fixed key for all messages.
                     If combined with -K, per-message keys
                     takes precendence.
  -H <header=value>  Add Message Headers (may be specified multiple times)
  -l                 Send messages from a file separated by
                     delimiter, as with stdin.
                     (only one file allowed)
  -T                 Output sent messages to stdout, acting like tee.
  -c <cnt>           Exit after producing this number of messages
  -Z                 Send empty messages as NULL messages
  file1 file2..      Read messages from files.
                     With -l, only one file permitted.
                     Otherwise, the entire file contents will
                     be sent as one single message.
  -X transactional.id=.. Enable transactions and send all
                     messages in a single transaction which
                     is committed when stdin is closed or the
                     input file(s) are fully read.
                     If kcat is terminated through Ctrl-C
                     (et.al) the transaction will be aborted.

Consumer options:
  -o <offset>        Offset to start consuming from:
                     beginning | end | stored |
                     <value>  (absolute offset) |
                     -<value> (relative offset from end)
                     s@<value> (timestamp in ms to start at)
                     e@<value> (timestamp in ms to stop at (not included))
  -e                 Exit successfully when last message received
  -f <fmt..>         Output formatting string, see below.
                     Takes precedence over -D and -K.
  -J                 Output with JSON envelope
  -s key=<serdes>    Deserialize non-NULL keys using <serdes>.
  -s value=<serdes>  Deserialize non-NULL values using <serdes>.
  -s <serdes>        Deserialize non-NULL keys and values using <serdes>.
                     Available deserializers (<serdes>):
                       <pack-str> - A combination of:
                                    <: little-endian,
                                    >: big-endian (recommended),
                                    b: signed 8-bit integer
                                    B: unsigned 8-bit integer
                                    h: signed 16-bit integer
                                    H: unsigned 16-bit integer
                                    i: signed 32-bit integer
                                    I: unsigned 32-bit integer
                                    q: signed 64-bit integer
                                    Q: unsigned 64-bit integer
                                    c: ASCII character
                                    s: remaining data is string
                                    $: match end-of-input (no more bytes remaining or a parse error is raised).
                                       Not including this token skips any
                                       remaining data after the pack-str is
                                       exhausted.
                       avro       - Avro-formatted with schema in Schema-Registry (requires -r)
                     E.g.: -s key=i -s value=avro - key is 32-bit integer, value is Avro.
                       or: -s avro - both key and value are Avro-serialized
  -r <url>           Schema registry URL (when avro deserializer is used with -s)
  -D <delim>         Delimiter to separate messages on output
  -K <delim>         Print message keys prefixing the message
                     with specified delimiter.
  -O                 Print message offset using -K delimiter
  -c <cnt>           Exit after consuming this number of messages
  -Z                 Print NULL values and keys as "NULL" instead of empty.
                     For JSON (-J) the nullstr is always null.
  -u                 Unbuffered output

Metadata options (-L):
  -t <topic>         Topic to query (optional)

Query options (-Q):
  -t <t>:<p>:<ts>    Get offset for topic <t>,
                     partition <p>, timestamp <ts>.
                     Timestamp is the number of milliseconds
                     since epoch UTC.
                     Requires broker >= 0.10.0.0 and librdkafka >= 0.9.3.
                     Multiple -t .. are allowed but a partition
                     must only occur once.

Format string tokens:
  %s                 Message payload
  %S                 Message payload length (or -1 for NULL)
  %R                 Message payload length (or -1 for NULL) serialized
                     as a binary big endian 32-bit signed integer
  %k                 Message key
  %K                 Message key length (or -1 for NULL)
  %T                 Message timestamp (milliseconds since epoch UTC)
  %h                 Message headers (n=v CSV)
  %t                 Topic
  %p                 Partition
  %o                 Message offset
  \n \r \t           Newlines, tab
  \xXX \xNNN         Any ASCII character
 Example:
  -f 'Topic %t [%p] at offset %o: key %k: %s\n'

JSON message envelope (on one line) when consuming with -J:
 { "topic": str, "partition": int, "offset": int,
   "tstype": "create|logappend|unknown", "ts": int, // timestamp in milliseconds since epoch
   "broker": int,
   "headers": { "<name>": str, .. }, // optional
   "key": str|json, "payload": str|json,
   "key_error": str, "payload_error": str, //optional
   "key_schema_id": int, "value_schema_id": int //optional
 }
 notes:
   - key_error and payload_error are only included if deserialization fails.
   - key_schema_id and value_schema_id are included for successfully deserialized Avro messages.

Consumer mode (writes messages to stdout):
  kcat -b <broker> -t <topic> -p <partition>
 or:
  kcat -C -b ...

High-level KafkaConsumer mode:
  kcat -b <broker> -G <group-id> topic1 top2 ^aregex\d+

Producer mode (reads messages from stdin):
  ... | kcat -b <broker> -t <topic> -p <partition>
 or:
  kcat -P -b ...

Metadata listing:
  kcat -L -b <broker> [-t <topic>]

Query offset by timestamp:
  kcat -Q -b broker -t <topic>:<partition>:<timestamp>