Kafka(1)Install on Redhat and Try Java Client

最新推荐文章于 2025-12-17 20:09:02 发布

原创最新推荐文章于 2025-12-17 20:09:02 发布 · 97 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #java

Distributed 专栏收录该内容

160 篇文章

订阅专栏

本文详细介绍了如何在红帽系统上安装Kafka并使用Java客户端发送和接收消息，包括下载源码、编译构建相关jar包、启动Zookeeper和Kafka服务，以及使用命令行工具进行消息交互。同时，还提供了如何在本地Maven仓库中更新相关jar包的方法，并展示了Java客户端示例。

Kafka(1)Install on Redhat and Try Java Client

1. Install and Try Kafka
>wget http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
>tar zxvf kafka-0.7.0-incubating-src.tar.gz
>mv kafka-0.7.0-incubating-src /opt/tools/kafka-0.7.0
>cd /opt/tools/kafka-0.7.0
>./sbt update
>./sbt package

2. Start the Server and try messages
Start the zookeeper first
>bin/zookeeper-server-start.sh config/zookeeper.properties

>bin/kafka-server-start.sh config/server.properties

We can send the message from producer and consumer
Producer, we type in this window
>bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test

Consumer, we got message which we typed.
>bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

3. Try to Run JAVA Client
First problem, how to find the jar package for kafka. This is a problem. There is not easy way like maven or pom to do that.

Kafka is not build via maven, there is a build tool named sbt. Simple Build Tool. I have no time to learn this right now. So I just use
this command to download and build all the related jar packages.
>./sbt release-zip

There is example using java, it is under this directory /opt/tools/kafka-0.7.0/examples/bin. How do I know where the jar packages live in classpath? Check this in the file java-producer-consumer-demo.sh and java-simple-consumer-demo.sh.

I download the jar package from there directories:
$base_dir/project/boot/scala-2.8.0/lib/*.jar
$base_dir/core/lib_managed/scala_2.8.0/compile/*.jar
$base_dir/core/lib/*.jar
$base_dir/core/target/scala_2.8.0/*.jar
$base_dir/examples/target/scala_2.8.0/*.jar

Update these related jar package to my local maven repository.

The 2 demos can run well on redhat server, localhost.

The POM.xml will be as follow:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>kafka</groupId>
<artifactId>kafka</artifactId>
<version>0.7.0</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.8.0.RC7</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>org.I0Itec</groupId>
<artifactId>zkclient</artifactId>
<version>20110412</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.3.3</version>
</dependency>

references:
http://www.54chen.com/java-ee/linkedin-kafka-usage.html
http://incubator.apache.org/kafka/
http://incubator.apache.org/kafka/quickstart.html