PIG安装使用

上一篇博客中记录了用hive执行mapreduce任务实现统计最大值以及总量，这里用另外一个强大的工具PIG实现同样的功能。

首先下载pig-0.10.1.tar.gz版本解压到hadoop/pig下面，配置好HADOOP_HOME以及PATH环境变量

#读取HDFS中的数据到变量中
grunt> Line = load 'inputTest/test.log' using PigStorage(' ') as (day, bytes, tag, user);
grunt> describe Line;
Line: {day: bytearray,bytes: bytearray,tag: bytearray,user: bytearray}
grunt> dump Line;
(20121221,04567,user,s00001)
(20121221,75531,user,s00003)
(20121222,52369,user,s00002)
(20121222,01297,user,s00001)
(20121223,61223,user,s00002)
(20121223,33121,user,s00003)

#对变量进行group操作
grunt> Groupd_Line = group Line by day;
grunt> describe Groupd_Line;
Groupd_Line: {group: bytearray,Line: {(day: bytearray,bytes: bytearray,tag: bytearray,user: bytearray)}}
grunt> dump Groupd_Line;
(20121221,{(20121221,04567,user,s00001),(20121221,75531,user,s00003)})
(20121222,{(20121222,52369,user,s00002),(20121222,01297,user,s00001)})
(20121223,{(20121223,61223,user,s00002),(20121223,33121,user,s00003)})

#计算总量
grunt> Sum_Groupd_Line = foreach Groupd_Line generate group, SUM(Line.bytes);
grunt> describe Sum_Groupd_Line;
Sum_Groupd_Line: {group: bytearray,double}
grunt> dump Sum_Groupd_Line;
(20121221,80098.0)
(20121222,53666.0)
(20121223,94344.0)

#保存到HDFS中
grunt> store Sum_Groupd_Line into 'sumOutput';

#通过HDFS查看结果
root:~/hadoop # hadoop fs -cat /user/root/sumOutput/part-r-00000
20121221        80098.0
20121222        53666.0
20121223        94344.0


#计算最大值也是类似操作
grunt> Max_Groupd_Line = foreach Groupd_Line generate group, MAX(Line.bytes);
grunt> dump Max_Groupd_Line;
(20121221,75531.0)
(20121222,52369.0)
(20121223,61223.0)
grunt> store Max_Groupd_Line into 'maxOutput';

对lzo压缩格式的文件一样的操作

grunt> Line = load 'inputTest/test.log.lzo' using PigStorage(' ') as (day, bytes, tag, user);
grunt> Filt = FILTER Line by day == 20121221;
grunt> dump Filt;
(20121221,04567,user,s00001)
(20121221,75531,user,s00003)

参考资料：pig的安装和使用