Hadoop学习笔记 6 - eclipse远程连接Hadoop

eclipse连接Hadoop

1.下载Hadoop-eclipse-plugin-2.6.0.jar包放在eclipse目录的plugin文件夹下

2.下载hadoop-2.6.0解压，设置环境变量

新建一个HADOOP_HOME，路径为解压的路径

3.在Path变量后加上：%HADOOP_HOME%\bin

4.重启eclipse，点击Window>>Preference>>Hadoop Map/Reduce

地址为解压的路径

5.点击Window>>Show Viwe>>Other>>MapReduce Tools>>Map/Reduce Locations

右键New Hadoop Location

这两个要和core-site.xml中的一样，点击finish
右上角出现DFS Locations

运行一个WordCount项目

在eclipse中Ctrl+N，选择Map/Reduce Project


新建Class

编写WordCount代码：

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {
        public static class WordCountMap extends
                        Mapper<LongWritable, Text, Text, IntWritable> {
                private final IntWritable one = new IntWritable(1);
                private Text word = new Text();

                public void map(LongWritable key, Text value, Context context)
                                throws IOException, InterruptedException {
                        String line = value.toString();
                        StringTokenizer token = new StringTokenizer(line);
                        while (token.hasMoreTokens()) {
                                word.set(token.nextToken());
                                context.write(word, one);
                        }
                }
        }

        public static class WordCountReduce extends
                        Reducer<Text, IntWritable, Text, IntWritable> {
                public void reduce(Text key, Iterable<IntWritable> values,
                                Context context) throws IOException, InterruptedException {
                        int sum = 0;
                        for (IntWritable val : values) {
                                sum += val.get();
                        }
                        context.write(key, new IntWritable(sum));
                }
        }

        public static void main(String[] args) throws Exception {
                Configuration conf = new Configuration();
                Job job = new Job(conf);
                job.setJarByClass(WordCount.class);
                job.setJobName("wordcount");
                job.setOutputKeyClass(Text.class);
                job.setOutputValueClass(IntWritable.class);
                job.setMapperClass(WordCountMap.class);
                job.setReducerClass(WordCountReduce.class);
                job.setInputFormatClass(TextInputFormat.class);
                job.setOutputFormatClass(TextOutputFormat.class);
                FileInputFormat.addInputPath(job, new Path("hdfs://192.168.2.172:9000/WordCountInput"));//和namenode的IP一致，此目录需手动创建并在其中放入要处理的文件
                FileOutputFormat.setOutputPath(job, new Path("hdfs://192.168.2.172:9000/WordCountOutput"));//此目录不需创建，会自动生成
                job.waitForCompletion(true);
        }
}

新建两个文件file1，file2

在hdfs上新建一个目录WordCountInput

将file1，file2上传至此目录

在eclipse上运行，右键WordCount>>Run As>>Run Configuration

运行成功！

在HDFS本地运行

这里直接使用Hadoop自带的wordcount程序：

可以看到运行成功后生成的文件