Spark 1.3.1+Flume 1.4.0环境搭建

一个简单的过程,走了很多弯路,记录下来,总结经验,愿助路人.

首先你要确定的一件事,你的spark版本,以及跟他适应的Flume版本,我就是因为这个,浪费了很多功夫,所以,我们要完成一个任务,首先要十分确定它所有的前提条件.(这里我的spark版本是1.3.1,与他兼容的是flume1.4.0: Spark1.3.1+Flume的官方教程.

---

我们采用教程中更好的方法二: Pull-based Approach using a Custom Sink
第一步Flume将数据推送到sink中,data先缓存在此sink中,然后,Spark Streaming用一个receiver从这个sink中拉过来数据.只有当数据接收到,这个交易才算成功.

1. Flume 的配置

Flume如果使用其他第三方插件,需要(根据版本)把下面的两个JAR包加入到Flume的classpath中(这里,有两种办法:1.直接将它们拷贝到FLUME_HOME/lib下;2.在 FLUME_HOME/路径下建plugins.d路径的方法,我建议用两种方法都加入进去).
(1)Flume中配置sink的依赖, 注意这个是带-sink后缀的(link)

 groupId = org.apache.spark
 artifactId = spark-streaming-flume-sink_2.10
 version = 1.3.1

(2)主机的scala版本(link)

 groupId = org.scala-lang
 artifactId = scala-library
 version = 2.10.4

将上面两个jar加入到Flume的classpath后,根据Flume官网的教程写配置文件,如下:

a1.channels = c1
a1.sources = r1
a1.sinks = k1

a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = [your file path ]
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1

a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = [your hostname/ip]
a1.sinks.k1.port = [port]

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

OK, Flume段配置好了.

---

2.构建Spark Streaming工程

我采用maven来构建scala项目,首先加载构建spark streaming工程的flume的依赖,注意这个是不带-sink后缀的!

 groupId = org.apache.spark
 artifactId = spark-streaming-flume_2.10
 version = 1.3.1

我的代码如下:(在控制台输出接收到的数据)

package ada.himon

import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.flume.FlumeUtils
import org.apache.spark.streaming.{StreamingContext, Milliseconds}

/**
 * Created by himon on 16-12-19.
 */
object app1 {
  def main(args: Array[String]): Unit = {
    if (args.length < 2){
      println("Usage: Spark_Flume <hostname> <port>")
      System.exit(1)
    }
    val hostname = args(0)
    val port = Integer.parseInt(args(1))
    println("hostName:"+hostname+" port:"+port)
    val batchInterval = Milliseconds(2000)
    val conf = new SparkConf().setAppName("approach2")
    val ssc = new StreamingContext(conf,batchInterval)
    val stream = FlumeUtils.createPollingStream(ssc,hostname,port)
    stream.map(e => "Event:header:" + e.event.get(0).toString + "body: " + new   String(e.event.getBody.array)).print()

    ssc.start()
    ssc.awaitTermination()
  }
}

---

然后Idea来build出项目的JAR包,这里提醒,最好把spark-streaming-flume_2.10依赖也打包进这个项目的JAR包中.
我提交spark项目的启动脚本如下:

spark-submit \
--master local[5] \
--num-executors 5 \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 1 \
--class ada.himon.app1 \
--name approach2 \
--jars /home/liyang/humeng/Jars/flume-ng-sdk-1.7.0.jar,/home/liyang/humeng/Jars/spark-streaming-flume_2.10-1.3.1.jar,/home/liyang/humeng/Jars/spark-streaming-flume-sink_2.10-1.3.1.jar \
approach2.jar [master_ip] [port]

---

下面列举我遇到一些问题:
错误1:

java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/sink/SparkFlumeProtocol$Callbackat

解决: spark提交任务时, –jar后面一定要把带-sink的jar包依赖加入进来.

错误2:

ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - java.io.IOException: Error connecting to /[master_ip]:[my_port]

解决: 如果用local模式来提交spark任务,需要先运行Flume来创建sink,再提交执行spark任务.