hadoop学习笔记

一.安装jdk和hadoop并配置环境变量,环境变量配置如下

jdk下载:wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie"
http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz

hadoop 下载: wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz 
解压缩 :tar -zxvf  hadoop-1.2.1.tar.gz 

export JAVA_HOME=/usr/java/jdk1.8.0_144
export JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/opt/hadoop-1.2.1
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CALSSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH

二.修改hadoop下的conf文件下的四个文件

2.1 hadoop-env.sh文件

修改jdk地址 export JAVA_HOME=/usr/java/jdk1.8.0_144

2.2 core-site.xml 文件

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop</value>
</property>

<property>
<name>dfs.name.dir</name>
<value>/hadoop/name</value>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://ray123:9000</value>
</property>
</configuration>

2.3  hdfs-site.xml 文件

<configuration>
<property>
<name>dfs.data.dir</name>
<value>/hadoop/data</value>
</property>
</configuration>

2.4 mapred-site.xml文件

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>ray123:9001</value>
</property>
</configuration>

三.启动hadoop

3.1 运行 hadoop namenode -format

3.2 切换到bin目录 运行 start-all.sh

3.3 运行jps出现如下表示安装成功

2480 JobTracker
2404 SecondaryNameNode
3958 Jps
2279 DataNode
2154 NameNode
2878 TaskTracker

四.编写java程序编译并打包

示例WordCount.java如下

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {
	public static class WordCountMap extends
			Mapper<LongWritable, Text, Text, IntWritable> {
		private final IntWritable one = new IntWritable(1);
		private Text word = new Text();

		public void map(LongWritable key, Text value, Context context)
				throws IOException, InterruptedException {
			String line = value.toString();
			StringTokenizer token = new StringTokenizer(line);
			while (token.hasMoreTokens()) {
				word.set(token.nextToken());
				context.write(word, one);
			}
		}
	}

	public static class WordCountReduce extends
			Reducer<Text, IntWritable, Text, IntWritable> {
		public void reduce(Text key, Iterable<IntWritable> values,
				Context context) throws IOException, InterruptedException {
			int sum = 0;
			for (IntWritable val : values) {
				sum += val.get();
			}
			context.write(key, new IntWritable(sum));
		}
	}

	public static void main(String[] args) throws Exception {
		Configuration conf = new Configuration();
		Job job = new Job(conf);
		job.setJarByClass(WordCount.class);
		job.setJobName("wordcount");
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);
		job.setMapperClass(WordCountMap.class);
		job.setReducerClass(WordCountReduce.class);
		job.setInputFormatClass(TextInputFormat.class);
		job.setOutputFormatClass(TextOutputFormat.class);
		FileInputFormat.addInputPath(job, new Path(args[0]));
		FileOutputFormat.setOutputPath(job, new Path(args[1]));
		job.waitForCompletion(true);
	}
}

编译命令: javac  -classpath  /opt/hadoop-1.2.1/hadoop-core-1.2.1.jar: /opt/hadoop-1.2.1/commons-cli-1.2.1.jar  -d  word_count_class/  WordCount.java

打包命令:jar -cvf wordcount.jar *.class

新建input文件夹放如两个输入文件file1和file2

将文件放到hadoop的input_wordcount 下  hadoop fs -put input/* input_wordcount/

查看hadoop 下的文件 hadoop fs -ls

运行jar文件 hadoop jar word_count_class/wordcount.jar  WordCount(主类)  input_wordcount(输入文件地址)  output_wordcount(输出文件地址)

运行结果在output_wordcount下的part-r-00000中