How-to: write own Kafka Partitioner based on requirement

Kafka's default partitioner is based on hash first element which is generated by spliting log via tab. In our usage, this is not a normal case. Our logs are normal log which the first element should be split by blank, and we hope the partitioner is based on the first word.

Then we could generate a new console producer class, and in the main class set the properties like following:

object OurConsoleProducer {

 ...

 def main(args: Array[String]) {
    val config = new ProducerConfig(args)
    val reader = Class.forName(config.readerClass).newInstance().asInstanceOf[MessageReader]
    val props = new Properties
    ...
    props.put("key.separator", " ")
    ...

    props.put("partitioner.class", "OurPartitioner")

    ...

In the partitioner class, we are going to put log started with some word to some spacified partitioner.

  /*
   * p-01:1(log started with p-01(splited by blank) will be stored in partitioner 1)

   * p-02:2

   * p-03:3

   * p-04:4

   * p-05:5

   * p-06:6

   * p-07:7

   * Otheres:0
   */ 
package kafka.producer


import kafka.utils._

class OurPartitioner(props: VerifiableProperties = null) extends Partitioner {
  private val random = new java.util.Random

  def partition(key: Any, numPartitions: Int): Int = {
//    Utils.abs(key.hashCode) % numPartitions
    if(numPartitions != 8){
      System.err.print("Warning: the partition number should be 7 while is "+Int)
    }
    val a = key.asInstanceOf[Array[Byte]].mkString(" ")
    if(a.equals("102 105 114 115 116 112 50 112 45 48 49")){//p-01 ascii code  
      System.out.println("1")
      1
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 50")){//p-02 ascii code 
      2
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 51")){//p-03 ascii code  
      3
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 52")){//p-04 ascii code  
      4
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 53")){//p-05 ascii code  
      5
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 54")){//p-06 ascii code  
      6
    }else if(a.equals("102 105 114 115 116 112 50 112 45 48 55")){//p-07 ascii code  
      7
    }else{
      0
    }
  }
}

The console shell should be like:

>> cat our-console-producer.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx512M"
fi
exec $(dirname $0)/kafka-run-class.sh kafka.tools.OurConsoleProducer $@

Then create our-topic with 8 partitioner, and transfer our logs to our-console-producer.sh via pipleline.