spark-shell 统计hbase数据条数

YiqiangXu

于 2016-09-29 13:51:36 发布

阅读量1.3k

点赞数

分类专栏： spark

本文链接：https://blog.youkuaiyun.com/u012871493/article/details/52701260

版权

spark 专栏收录该内容

6 篇文章

订阅专栏

sc.stop
sc.stop

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor, TableName}
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
import org.apache.spark._

val sparkConf = new SparkConf().setAppName("HbaseCount")
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
val sc = new SparkContext(sparkConf)
val hbaseConf: Configuration = HBaseConfiguration.create()
hbaseConf.set(TableInputFormat.INPUT_TABLE, "wenda")
val hBaseRDD = sc.newAPIHadoopRDD(hbaseConf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
hBaseRDD.count