kafka部署二

现在来测试一下，使用kafka内置的命令行工具发送数据，结果出现问题

bin/kafka-console-producer.sh --broker-list 10.149.11.151:9092 --topic vehicle_test
[2015-09-05 14:18:50,877] WARN Property topic is not valid (kafka.utils.VerifiableProperties)

无法将消息添加到kafka中，报错为leader is not valid,

之后用consumer读取也出错

bin/kafka-console-consumer.sh --zookeeper 10.149.11.146:2181/vehicle_kafka --from-beginning --topic vehicle_test
[2015-09-05 14:41:07,630] WARN [console-consumer-72151_10-149-11-147-1441435267500-9d20d013], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector)

这个原因可以参考kafka FAQ

https://cwiki.apache.org/confluence/display/KAFKA/FAQ

Why can't my consumers/producers connect to the brokers?

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the "hostname" property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

打开server.properties，可以看到host.name和advertised.host.name, 默认都被禁用，因为kafka从InetAddress.getLocalHost.getHostAddress API中获取IP地址。但是这个API有时候不能得到正确的IP地址，比如这里就得到localhost，然后填入zookeeper

[zk: localhost:2181(CONNECTED) 24] get /vehicle_kafka/brokers/ids/148
{"jmx_port":-1,"timestamp":"1441377326345","host":"localhost","version":1,"port":9092}
cZxid = 0x2c
ctime = Fri Sep 04 22:35:26 CST 2015
mZxid = 0x2c
mtime = Fri Sep 04 22:35:26 CST 2015
pZxid = 0x2c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14f976bd628000a
dataLength = 86
numChildren = 0

在my-server.properties文件中，启用host.name，如下：

# Hostname the broker will bind to. If not set, the server will bind to all interfaces                                                                 
host.name=10.149.11.147

重新启动kafka server，现在zookeeper中的配置被恢复正常了

[zk: localhost:2181(CONNECTED) 27] get /vehicle_kafka/brokers/ids/147
{"jmx_port":-1,"timestamp":"1441438553787","host":"10.149.11.147","version":1,"port":9092}
cZxid = 0x8d
ctime = Sat Sep 05 15:35:53 CST 2015
mZxid = 0x8d
mtime = Sat Sep 05 15:35:53 CST 2015
pZxid = 0x8d
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14f976bd6280021
dataLength = 90
numChildren = 0

再测试，现在warning信息仍然有，但是数据能够读取到了。

# bin/kafka-console-consumer.sh --zookeeper 10.149.11.146:2181/vehicle_kafka --from-beginning --topic vehicle_test
hello
world

Kafka还有很多话题，可以参考：

https://cwiki.apache.org/confluence/display/KAFKA/FAQ

本系列会在后面对更多的实践中遇到的问题进行讨论。