MapReduce 操作 hbase

最新推荐文章于 2023-03-03 14:20:52 发布

墨荷琼林

最新推荐文章于 2023-03-03 14:20:52 发布

阅读量682

点赞数

分类专栏：大数据存储 & 分布式预算框架文章标签： hbase mapreduce TableMapper TableReducer TableMapReduceUtil

本文链接：https://blog.youkuaiyun.com/moheqionglin/article/details/42739039

版权

大数据存储 & 分布式预算框架专栏收录该内容

4 篇文章

订阅专栏

hbase MapReduce程序三种类型

类型1 输入hbase 输出hbase

类型2 输入非hbase 输出 hbase

类型3 输入hbase 输出非hbase

======================类型1===============================

1、先看一个标准的hbase作为数据读取源和输出源的样例：

View Code JAVA

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, "job name ");
job.setJarByClass(test.class);
Scan scan = new Scan();

//---------------------------------map---------------------------------------------- 
TableMapReduceUtil.initTableMapperJob(inputTable, scan, mapper.class,Writable.class, Writable.class, job);

  //---------------------------------reduce----------------------------------------------                  
TableMapReduceUtil.initTableReducerJob(outputTable, reducer.class, job);

job.waitForCompletion(true);

接下来就是mapper类：

public class mapper extends	TableMapper<KEYOUT, VALUEOUT> {

	public void map(Writable key, Writable value, Context context)	throws IOException, InterruptedException {
		        //mapper逻辑
			context.write(key, value);
		}
 	}
}

然后reducer类：

public class countUniteRedcuer extends	TableReducer<KEYIN, VALUEIN, KEYOUT> {
	public void reduce(Text key, Iterable<VALUEIN> values, Context context) throws IOException, InterruptedException {
                //reducer逻辑
		context.write(null, put or delete);
	}
}

======================类型2===============================

2、有时候我们需要数据源是hdfs的文本，输出对象是hbase。这时候变化也很简单：

View Code JAVA

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, "job name ");
job.setJarByClass(test.class);
 
job.setMapperClass(mapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(job, path);
 
TableMapReduceUtil.initTableReducerJob(tableName,reducer.class, job);

public class mapper extends Mapper<LongWritable,Writable,Writable,Writable> {
	public void map(LongWritable key, Text line, Context context) {
           	 //mapper逻辑
		 context.write(k, one);
	}
}
public class redcuer extends
		TableReducer<KEYIN, VALUEIN, KEYOUT> {
	public void reduce(Writable key, Iterable<Writable> values, Context context)
			throws IOException, InterruptedException {
                //reducer逻辑
		context.write(null, put or delete);
	}
}

======================类型3===============================

3、最后就是从hbase中的表作为数据源读取，hdfs作为数据输出，简单的如下：

View Code JAVA

Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, "job name ");
job.setJarByClass(test.class);
Scan scan = new Scan();
TableMapReduceUtil.initTableMapperJob(inputTable, scan, mapper.class, Writable.class, Writable.class, job);
job.setOutputKeyClass(Writable.class);
job.setOutputValueClass(Writable.class);
FileOutputFormat.setOutputPath(job, Path);
job.waitForCompletion(true);

mapper和reducer简单如下：

public class mapper extends
		TableMapper<KEYOUT, VALUEOUT>{
 
	public void map(Writable key, Writable value, Context context)
			throws IOException, InterruptedException {
		        //mapper逻辑
			context.write(key, value);
		}
 
	}
}
 
public class reducer extends
		Reducer<Writable,Writable,Writable,Writable>  {
 
	public void reducer(Writable key, Writable value, Context context)
			throws IOException, InterruptedException {
		        //reducer逻辑
			context.write(key, value);
		}
	}
}

最后说一下TableMapper和TableReducer的本质，其实这俩类就是为了简化一下书写代码，因为传入的4个泛型参数里都会有固定的参数类型，所以是Mapper和Reducer的简化版本，本质他们没有任何区别。源码如下：

View Code JAVA

public abstract class TableMapper<KEYOUT, VALUEOUT>
extends Mapper<ImmutableBytesWritable, Result, KEYOUT, VALUEOUT> {
}
 
public abstract class TableReducer<KEYIN, VALUEIN, KEYOUT>
extends Reducer<KEYIN, VALUEIN, KEYOUT, Writable> {
}