Flume: avro source and sink

In order to flow the data across multiple agents or hops, the sink of the previous agent and source of the current hop need to be avro type with the sink pointing to the hostname (or IP address) and port of the source.

 

 

Hop 1:

a1.channels.ch1.type = memory

a1.sources.avro-source1.channels = ch1
a1.sources.avro-source1.type = avro
a1.sources.avro-source1.bind = 0.0.0.0
a1.sources.avro-source1.port = 41414

a1.sinks.avro-sink1.channel = ch1
a1.sinks.avro-sink1.type = avro
a1.sinks.avro-sink1.hostname = 192.168.10.204
a1.sinks.avro-sink1.port = 5545
a1.sinks.avro-sink1.batch-size = 1

a1.channels = ch1
a1.sources = avro-source1
a1.sinks = avro-sink1

 

Hop2

a1.channels.ch1.type = memory

a1.sources.avro-source1.channels = ch1
a1.sources.avro-source1.type = avro
a1.sources.avro-source1.bind = 0.0.0.0
a1.sources.avro-source1.port = 5545

a1.sinks.log-sink1.channel = ch1
a1.sinks.log-sink1.type = logger

a1.channels = ch1
a1.sources = avro-source1
a1.sinks = log-sink1

 

 

 

test using avro-client

flume-ng avro-client -H dmining05 --port 41414 -F /etc/passwd

 

 

 

 

 

 

 

 

 

 

 

 

 

 

References

http://flume.apache.org/FlumeUserGuide.html