Kafka(4)Multiple Kafka and Scala Client

Kafka(4)Multiple Kafka and Scala Client

1. Create Multiple Nodes
>cp server.properties server1.properties
>cp server.properties server2.properties
The content are as follow:
config/sever1.properties
broker.id = 1
port =9093
log.dir=/tmp/kafka-logs-1

config/server2.properties
broker.id = 2
port = 9094
log.dir=/tmp/kafka-logs-2

Start the 2 nodes
>JMX_PORT=9997 bin/kafka-server-start.sh config/server1.properties &
>JMX_PORT=9998 bin/kafka-server-start.sh config/server2.properties &

Create the topic
>bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 2 --partition 1 --topic my-replicated-topic

List the topics
>bin/kafka-list-topic.sh --zookeeper localhost:2181

topic: my-replicated-topicpartition: 0leader: 1replicas: 1,2isr: 1,2 topic: testpartition: 0leader: 0replicas: 0isr: 0

Here is the command to kill one node
>pkill -9 -f config/server1.properties

2. The Scala Client
The Scala Class will be as follow:
        "com.rabbitmq"        %   "amqp-client"               % "3.1.4",
        "org.apache.kafka"          %   "kafka_2.10"            % "0.8.0" intransitive(),
        "com.yammer.metrics"        %   "metrics-core"          % "2.2.0",
        "com.twitter"         %%  "util-collection"           % "6.3.6",
        "com.101tec"          %   "zkclient"                  % "0.3"

The producer class
package com.sillycat.superduty.jobs.producer

import java.util.Properties

import kafka.javaapi.producer.Producer
import kafka.producer.{ KeyedMessage, ProducerConfig }

object NewTaskKafka extends App {
  val props2: Properties = new Properties()
  props2.put("zk.connect", "localhost:2181")
  props2.put("metadata.broker.list", "localhost:9092");
  props2.put("serializer.class", "kafka.serializer.StringEncoder")
  props2.put("zk.connectiontimeout.ms", "15000")

  val config: ProducerConfig = new ProducerConfig(props2)

  val producer: Producer[String, String] = new Producer[String, String](config)

  val data = new KeyedMessage[String, String]("test", "test-message, it is ok")
  producer.send(data)
  producer.close
}

The Consumer class
package com.sillycat.superduty.jobs.consumer

import kafka.api.{ FetchRequestBuilder, FetchRequest }
import kafka.javaapi.consumer.SimpleConsumer
import kafka.javaapi.FetchResponse
import kafka.javaapi.message.ByteBufferMessageSet
import scala.collection.JavaConversions._
import java.nio.ByteBuffer
import com.typesafe.scalalogging.slf4j.Logging
import java.util.Properties
import kafka.consumer.{Consumer, ConsumerConfig}
import scala.collection.JavaConverters._

object WorkerKafka extends App with Logging {

  val props = new Properties()

  props.put("group.id", "console-consumer-2222222")
  props.put("socket.receive.buffer.bytes", (2 * 1024 * 1024).toString)
  props.put("socket.timeout.ms", (ConsumerConfig.SocketTimeout).toString)
  props.put("fetch.message.max.bytes", (1024 * 1024).toString)
  props.put("fetch.min.bytes", (1).toString)
  props.put("fetch.wait.max.ms", (100).toString)
  props.put("auto.commit.enable", "true")
  props.put("auto.commit.interval.ms", (ConsumerConfig.AutoCommitInterval).toString)
  props.put("auto.offset.reset", "smallest")
  props.put("zookeeper.connect", "localhost:2181")
  props.put("consumer.timeout.ms", (-1).toString)
  props.put("refresh.leader.backoff.ms", (ConsumerConfig.RefreshMetadataBackoffMs).toString)


  val config   = new ConsumerConfig(props)
  val consumer = Consumer.createJavaConsumerConnector(config)

  val topicMap =  Map[String, Integer]("test" -> 1)

  println("about to get the comsumerMsgStreams")
  val consumerMap = consumer.createMessageStreams(topicMap.asJava)

  val streamz = consumerMap.get("test")

  val stream = streamz.iterator().next()

  println("listening... (?) ")

  val consumerIter = stream.iterator()
  while(consumerIter.hasNext()){
    System.out.println("MSG -> " + new String(consumerIter.next().message))
  }

}


We can use zkCli.sh to view some of the configuration.
>zkCli.sh -server localhost:2181
zkCli>ls /
zkCli>get /brokers/topics/my-replicated-topic
zkCli>get /brokers/topics/test

And here is the document for the zookeeper data structure
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

We can also configure the multiple nodes for zookeeper in server.properties, there is a configuration like this.
zookeeper.connect=localhost:2182,localhost:2183


References:
http://kafka.apache.org/08/quickstart.html

https://cwiki.apache.org/confluence/display/KAFKA/Index