spark源码阅读一-spark读写hbase代码分析

1.读取hbase代码

val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat],

   classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],  

   classOf[org.apache.hadoop.hbase.client.Result])  

看看 newAPIHadoopRDD函数实现

def newAPIHadoopRDD[K, V, F <: NewInputFormat[K, V]](

    conf: Configuration = hadoopConfiguration,

    fClass: Class[F],

    kClass: Class[K],

    vClass: Class[V]): RDD[(K, V)] = withScope {

  assertNotStopped()

  // This is a hack to enforce loading hdfs-site.xml.   // See SPARK-11227 for details.   FileSystem. getLocal(conf)

  // Add necessary security credentials to the JobConf. Required to access secure HDFS.   val jconf = new JobConf(conf)

  SparkHadoopUtil. get.addCredentials(jconf)

  new NewHadoopRDD( this, fClass, kClass, vClass, jconf)

}

是新的hadoop rdd接口和类。我们可以看到input format是 TableInputFormat，那么读出来数据的分区也是根据<