Kafka环境搭建及与Spring的整合测试

【前言】 Kafka作为一种分布式消息队列的实现，采用发布订阅的消息模型。通过生产者把消息Topic存储在broke,消息者可以订阅多个感兴趣的topic进行消费。消费者需要自己保留一个offset，从kafka 获取消息时，只拉去当前offset 以后的消息。

Kafka相比于其他MQ（activemq、rabbitmq）优点：

1.高性能高吞吐量 ；  Kafka 集群可以透明的扩展，增加新的服务器进集群。
高性能。Kafka 的性能大大超过传统的ActiveMQ、RabbitMQ等MQ 实现，尤其是Kafka 还支持batch 操作。

2.容错能力。kafka把每个partition的数据复制到几台服务器上，一台broke 出故障时，zookeeper服务将通知生产者和消费者，从而使用其他的broke节点。

【环境搭建】Linux centos下安装zookeeper-3.3.6，bin目录下启动服务

 ./zkServer.sh start 

，安装kafka_2.10-0.10.0.1 ，bin目录下启动服务【 注： kafka客户端jar包版本需要和服务端版本保持一致，如客户端为kafka-client-0.10需要和本次安装的服务端版本保持一致】

./kafka-server-start.sh  ../config/server.properties 

【程序测试】项目目录：

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.kafka</groupId>
    <artifactId>kafkaDemo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <spring.version>4.3.3.RELEASE</spring.version>
        <slf4j.version>1.7.5</slf4j.version>
    </properties>
    <dependencies>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-tx</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>1.1.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-kafka</artifactId>
            <version>2.1.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.15</version>
            <exclusions>
                <exclusion>
                    <artifactId>jmxtools</artifactId>
                    <groupId>com.sun.jdmk</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jmxri</artifactId>
                    <groupId>com.sun.jmx</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jms</artifactId>
                    <groupId>javax.jms</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>mail</artifactId>
                    <groupId>javax.mail</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.101tec</groupId>
            <artifactId>zkclient</artifactId>
            <version>0.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>jmxri</artifactId>
                    <groupId>com.sun.jmx</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jmxtools</artifactId>
                    <groupId>com.sun.jdmk</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context-support</artifactId>
            <version>3.2.6.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.12</version>
        </dependency>
    </dependencies>
    <build>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <excludes>
                    <exclude>*.xml</exclude>
                    <exclude>spring/*.xml</exclude>
                    <exclude>conf*/*</exclude>
                </excludes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration><!--描述执行的文件路径-->
                    <descriptor>src/main/assembly/assembly.xml</descriptor>
                    <appendAssemblyId>false</appendAssemblyId>
                    <finalName>kafkaDemo</finalName>
                </configuration>
                <executions><!--执行器，执行器的名称，绑定到package的生命周期上，single表示只执行一次-->
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

producerSerivce接口：

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
public class ProducerService {
    @Autowired
    private KafkaTemplate<String,String> kafkaTemplate;

    protected Logger logger  = LoggerFactory.getLogger(this.getClass());

    public void sendMessage(String msg){
        logger.info("----------------进入sendMessage------------");
        kafkaTemplate.sendDefault(msg);
        logger.info("----------------sendMessage完成------------");
    }

    public KafkaTemplate<String, String> getKafkaTemplat