【C#】Confluent.Kafka快速开始

原创已于 2025-01-10 08:38:53 修改 · 521 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#c# #kafka

于 2025-01-10 01:03:59 首次发布

【C#】Confluent.Kafka快速开始

一、安装Kafka

http://kafka.apache.org/downloads

下载 Binary 版本：kafka_2.13-3.9.0.tgz (116 MB)
修改配置 zookeeper.properties

dataDir=C:/App/kafka_2.13-3.9.0/zookeeper

修改配置 server.properties

log.dirs=C:/App/kafka_2.13-3.9.0/kafka-logs

二、启动

启动zookeeper

cd bin\windows
zookeeper-server-start.bat ..\..\config\zookeeper.properties

启动kafka

cd bin\windows
kafka-server-start.bat ..\..\config\server.properties

三、创建/查看 Topic

创建Topic

kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

replication-factor：主题副本数。副本位于集群中不同的broker上，副本数不能超过broker的数量。
partitions ：主题分区数。分区数越多 => 提升消息处理的吞吐量 => 增加性能开销 => 日志分段越多

查看Topic

kafka-topics.bat --list --bootstrap-server localhost:9092

四、生产者/消费者

启动生产者 producer

kafka-console-producer.bat --broker-list localhost:9092 --topic test

启动消费者 customer

kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning

五、C# 实例

引用类库

Confluent.Kafka

生产者

void Main()
{
	const string topic = "test";
	const string host = "localhost:9092";

	var cfg = new ProducerConfig
	{
		BootstrapServers = host
	};
	
	using var producer = new ProducerBuilder<Null,string>(cfg).Build();
	for(int i=0;i<100;i++) // for test
	{
		try
		{
			producer.Produce(topic, new Message<Null, string>
			{
				Value = $"[{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff}] { new Faker("en").Address.FullAddress()}"
			});
		}
		catch (ProduceException<Null, string> ex)
		{
			Console.WriteLine($"ex: {ex.Error.Reason}");
		}
	}
}

消费者

void Main()
{
	const string topic = "test";
	const string host = "localhost:9092";

	var cfg = new ConsumerConfig() {
		GroupId="test-consumer-group",
		BootstrapServers = host,
		AutoOffsetReset =  AutoOffsetReset.Earliset,
		EnableAutoCommit = true  // 自动提交
	};

	using var builder = new ConsumerBuilder<Ignore,string>(cfg).Build();
	builder.Subscribe(topic);

	var cts = new CancellationTokenSource();
	try
	{
		while(true)
		{
            try
            {
				var consumer = builder.Consume(cts.Token);
				Console.WriteLine($"Message: {consumer.Message.Value} received from {consumer.TopicPartitionOffset}");
				
				// builder.Commit(consumer); // 手动提交
            }
            catch (ConsumeException e)
            {
                Console.WriteLine($"Error occured: {e.Error.Reason}");
            }
		}
	}
	catch(OperationCanceledException ex)
	{
		builder.Close();
		Console.WriteLine($"ex: {ex}");
	}
}