kafka部署二

分享一下我老师大神的人工智能教程！零基础，通俗易懂！http://blog.youkuaiyun.com/jiangjunshow

也欢迎大家转载本篇文章。分享知识，造福人民，实现我们中华民族伟大复兴！

               

现在来测试一下，使用kafka内置的命令行工具发送数据，结果出现问题

bin/kafka-console-producer.sh --broker-list 10.149.11.151:9092 --topic vehicle_test[2015-09-05 14:18:50,877] WARN Property topic is not valid (kafka.utils.VerifiableProperties)

无法将消息添加到kafka中，报错为leader is not valid,

之后用consumer读取也出错

bin/kafka-console-consumer.sh --zookeeper 10.149.11.146:2181/vehicle_kafka --from-beginning --topic vehicle_test[2015-09-05 14:41:07,630] WARN [console-consumer-72151_10-149-11-147-1441435267500-9d20d013], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector)

这个原因可以参考kafka FAQ

https://cwiki.apache.org/confluence/display/KAFKA/FAQ

Why can't my consumers/producers connect to the brokers?

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the "hostname" property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

打开server.properties，可以看到host.name和advertised.host.name, 默认都被禁用，因为kafka从InetAddress.getLocalHost.getHostAddress API中获取IP地址。但是这个API有时候不能得到正确的IP地址，比如这里就得到localhost，然后填入zookeeper

[zk: localhost:2181(CONNECTED) 24] get /vehicle_kafka/brokers/ids/148{"jmx_port":-1,"timestamp":"1441377326345","host":"localhost","version":1,"port":9092}cZxid = 0x2cctime = Fri Sep 04 22:35:26 CST 2015mZxid = 0x2cmtime = Fri Sep 04 22:35:26 CST 2015pZxid = 0x2ccversion = 0dataVersion = 0aclVersion = 0ephemeralOwner = 0x14f976bd628000adataLength = 86numChildren = 0

在my-server.properties文件中，启用host.name，如下：

# Hostname the broker will bind to. If not set, the server will bind to all interfaces                                                                 host.name=10.149.11.147

重新启动kafka server，现在zookeeper中的配置被恢复正常了

[zk: localhost:2181(CONNECTED) 27] get /vehicle_kafka/brokers/ids/147{"jmx_port":-1,"timestamp":"1441438553787","host":"10.149.11.147","version":1,"port":9092}cZxid = 0x8dctime = Sat Sep 05 15:35:53 CST 2015mZxid = 0x8dmtime = Sat Sep 05 15:35:53 CST 2015pZxid = 0x8dcversion = 0dataVersion = 0aclVersion = 0ephemeralOwner = 0x14f976bd6280021dataLength = 90numChildren = 0

再测试，现在warning信息仍然有，但是数据能够读取到了。

# bin/kafka-console-consumer.sh --zookeeper 10.149.11.146:2181/vehicle_kafka --from-beginning --topic vehicle_testhelloworld

Kafka还有很多话题，可以参考：

https://cwiki.apache.org/confluence/display/KAFKA/FAQ

本系列会在后面对更多的实践中遇到的问题进行讨论。

           

给我老师的人工智能教程打call！http://blog.youkuaiyun.com/jiangjunshow