spark-structstreaming-结果数据存入hbase

最新推荐文章于 2022-05-15 00:54:38 发布

freshghost1234

最新推荐文章于 2022-05-15 00:54:38 发布

阅读量2.8k

点赞数

分类专栏：大数据-计算框架-spark 文章标签： spark

本文链接：https://blog.youkuaiyun.com/qq_34969081/article/details/78983857

版权

前言

本节描述通过spark-structstreaming将结果结果数据存入hbase

正文

object testWriteResultToHbase{
def main(args:Array[String]){

  val kafkaservers=args(0)
  val topic=args(1)     
  val zookeeperservers=args(2)
   val tablename=args(3)
   val spark=SparkSession
  .builder
  .appName("testWriteResultToHbase")
  .master("local")
  .getOrCreate()

  import spark.implicits._
   val lines = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers",kafkaservers)
  .option("subscribe", topic)
  .load()
  .selectExpr("cast(topic as String) ","cast(key as String)","CAST(value AS STRING)")
  .as[(String,String,String)]
   lines.createTempView("Originalkafka")
  import spark.sql
  val count=sql("select value from Originalkafka ")
  val query =count.writeStream
  .outputMode("append")
  .foreach(new ForeachWriter[Row]{

    va