背景,公司使用了AWS 的MSK,在命令行连接配置的时候,报错:
报错1:
WARN Session 0x0 for server z-2.sconceptmqprd.y5xuv6.c1.kafka.me-central-1.amazonaws.com/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)
报错2:
zookeeper is not a recognized option
joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at org.apache.kafka.tools.TopicCommand$TopicCommandOptions.<init>(TopicCommand.java:830)
at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:100)
at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:90)
at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:85)
报错3:
zookeeper is not a recognized option
joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at org.apache.kafka.tools.TopicCommand$TopicCommandOptions.<init>(TopicCommand.java:830)
at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:100)
at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:90)
at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:85)
以上的报错都是因为MSK的连接真的很麻烦。不能通过Apache ZooKeeper 的方式连接。其次要保证跳板机和MSK在一个VPC中。
解决方式:
1.登录跳板机,下载kafka命令
#wget https://archive.apache.org/dist/kafka/{YOUR MSK VERSION}/kafka_2.13-{YOUR MSK VERSION}.tgz
#我的3.5.1的版本
wget https://archive.apache.org/dist/kafka/3.5.1/kafka_2.13-3.5.1.tgz
#下载慢的,可以在本地电脑下载之后,再上传到服务器。
tar -xzf kafka_2.13-3.5.1.tgz
2.必做,下载 Amazon MSK IAM JAR 文件
java要用,这里要保证java-11以上的版本
cd kafka_2.13-3.5.1/libs
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
3.编辑验证方式的文件,必须,不然必定出现报错:
vim kafka_2.13-3.5.1/bin/client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
4.尝试连接,创建topic:
.kafka-topics.sh --create --bootstrap-server BootstrapServerString --command-config client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic
#BootstrapServerString为端点地址