hive加载大文件(3G)

wind520 wind520 博客等级 墨衍会员 · AI 创作全网分发 墨衍智能分发 本文由作者通过墨衍一键同步至各平台 1分发平台 2.3k累计阅读 2.4k平均单平台 已同步平台 优快云 微信公众号 微博 知乎 掘金 百家号 博客园 抖音 小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年，写一次发全网 我也要 码龄25年
关注
原创 最新推荐文章于 2026-05-02 10:41:02 发布 2.3k 阅读 AI 写同款· GEO 优化 ›
Hive常见调优总结 基础设置 队列选择 yarn队列分配，选择合适的队列进行运行，避免都在同一个繁忙的队列中运行。 SET mapreduce.job.queuename=xx; 控制mapper&reducer数目 合理控制文件切分的各种参数，使Mapper和Reducer数目达到一个合适的值，可以加快处理速度。 // HiveInputFormat文件格式 # 文件分割最大大小 set mapreduce.input.fileinputformat.split.maxsize=536870912; # 文件分割最 阅读详情

1：上传文件到hadoop

[jifeng@jifeng01 ~]$ ls -l
总计 3835464
-rw-r--r--  1 jifeng jifeng    7118627 2012-02-01 access_log.txt
drwxrwxr-x  6 jifeng jifeng       4096 04-30 11:14 apache-ant-1.9.4
-rw-r--r--  1 jifeng jifeng    5564593 08-05 16:59 apache-ant-1.9.4-bin.tar.gz
drwxr-xr-x  2 jifeng jifeng       4096 2014-02-26 Desktop
-rw-r--r--  1 jifeng jifeng 3415284428 09-11 16:51 firewallb-week2-origin.sql

[jifeng@jifeng01 ~]$ hadoop fs -mkdir /user/jifeng/fire
Warning: $HADOOP_HOME is deprecated.

[jifeng@jifeng01 ~]$ hadoop fs -put /home/jifeng/firewallb-week2-origin.sql /user/jifeng/fire/
Warning: $HADOOP_HOME is deprecated.

2：hive总创建外部表 

hive> CREATE EXTERNAL TABLE firewall (
    >   idauto INT,
    >   dt TIMESTAMP,
    >   dtstr STRING,
    >   timeSeconds INT,
    >   priority STRING,
    >   operation STRING,
    >   messageCode STRING,
    >   protocol STRING,
    >   srcIp STRING,
    >   destIp STRING,
    >   srcIPNumber TINYINT,
    >   destIPNumber TINYINT,
    >   srcPort INT,
    >   destPort INT,
    >   destService STRING,
    >   direction STRING,
    >   flags STRING,
    >   command STRING,
    >   slot1min INT,
    >   slot5min INT,
    >   slot30min INT,
    >   slot60min INT 
    > ) row format delimited fields terminated by '\t';
OK
Time taken: 0.405 seconds


CREATE EXTERNAL TABLE firewall (  
       idauto INT,  
       dt TIMESTAMP,  
       dtstr STRING,  
       timeSeconds INT,  
       priority STRING,  
       operation STRING,  
       messageCode STRING,  
       protocol STRING,  
       srcIp STRING,  
       destIp STRING,  
       srcIPNumber bigINT,  
       destIPNumber bigINT,  
       srcPort INT,  
       destPort INT,  
       destService STRING,  
       direction STRING,  
       flags STRING,  
       command STRING,  
       slot1min INT,  
       slot5min INT,  
       slot30min INT,  
       slot60min INT   
     ) row format delimited fields terminated by '\t';


3：加载hdfs文件 

hive> load data inpath '/user/jifeng/fire/firewallb-week2-origin.sql' into table firewall; 
Loading data to table default.firewall
Table default.firewall stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 3415284428, raw_data_size: 0]
OK
Time taken: 0.574 seconds

4：统计IP出现次数 

hive> select srcIp,count(*) as num  from firewall  group by srcIp order by  num desc ; 
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 4
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201409072150_0030, Tracking URL = http://jifeng01:50030/jobdetails.jsp?jobid=job_201409072150_0030
Kill Command = /home/jifeng/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job  -kill job_201409072150_0030
Hadoop job information for Stage-1: number of mappers: 13; number of reducers: 4
2014-09-11 18:00:17,094 Stage-1 map = 0%,  reduce = 0%
2014-09-11 18:00:28,139 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:29,145 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:30,148 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:31,158 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:32,162 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:33,165 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:34,168 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:35,171 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 3.22 sec
2014-09-11 18:00:36,176 Stage-1 map = 8%,  reduce = 1%, Cumulative CPU 3.22 sec
2014-09-11 18:00:37,179 Stage-1 map = 10%,  reduce = 1%, Cumulative CPU 3.22 sec
2014-09-11 18:00:38,182 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:39,185 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:40,190 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:41,193 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:42,197 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:43,201 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:44,203 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:45,209 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:46,212 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:47,215 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:48,218 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:49,221 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:50,225 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:51,229 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:52,233 Stage-1 map = 10%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:53,236 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:54,240 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:55,243 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:56,247 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:57,251 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 3.22 sec
2014-09-11 18:00:58,256 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:00:59,260 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:00,265 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:01,271 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:02,275 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:03,279 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:04,286 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:05,289 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:06,292 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:07,295 Stage-1 map = 13%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:08,298 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:09,302 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:10,306 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:11,308 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:12,312 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:13,317 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:14,321 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:15,324 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:16,327 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:17,331 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:18,335 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:19,340 Stage-1 map = 15%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:20,343 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:21,348 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:22,351 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:23,354 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:24,359 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 12.05 sec
2014-09-11 18:01:25,363 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:26,368 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:27,371 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:28,374 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:29,378 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:30,381 Stage-1 map = 17%,  reduce = 2%, Cumulative CPU 17.21 sec
2014-09-11 18:01:31,383 Stage-1 map = 17%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:32,388 Stage-1 map = 17%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:33,391 Stage-1 map = 19%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:34,394 Stage-1 map = 19%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:35,398 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:36,401 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:37,405 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:38,408 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:39,410 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:40,413 Stage-1 map = 21%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:41,416 Stage-1 map = 23%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:42,419 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:43,422 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:44,424 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:45,429 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:46,432 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:47,436 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:48,438 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:49,442 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:50,446 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:51,448 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:52,453 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:53,456 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:54,460 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:55,462 Stage-1 map = 25%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:56,465 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:57,468 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 17.21 sec
2014-09-11 18:01:58,472 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:01:59,476 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:00,479 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:01,481 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:02,485 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:03,487 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:04,490 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:05,492 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:06,495 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:07,498 Stage-1 map = 27%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:08,502 Stage-1 map = 29%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:09,506 Stage-1 map = 29%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:10,510 Stage-1 map = 29%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:11,513 Stage-1 map = 29%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:12,517 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:13,519 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:14,522 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:15,525 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:16,529 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:17,531 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:18,534 Stage-1 map = 35%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:19,537 Stage-1 map = 35%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:20,540 Stage-1 map = 35%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:21,543 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:22,548 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:23,553 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 25.15 sec
2014-09-11 18:02:24,556 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:25,559 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:26,562 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:27,565 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:28,568 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:29,571 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:30,573 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:31,576 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:32,579 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:33,583 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:34,585 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:35,588 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:36,591 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:37,594 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:38,596 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:39,600 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:40,602 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:41,605 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:42,607 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:43,610 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:44,613 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:45,615 Stage-1 map = 40%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:46,619 Stage-1 map = 40%,  reduce = 3%, Cumulative CPU 28.38 sec
2014-09-11 18:02:47,622 Stage-1 map = 42%,  reduce = 3%, Cumulative CPU 29.53 sec
2014-09-11 18:02:48,626 Stage-1 map = 42%,  reduce = 3%, Cumulative CPU 29.53 sec
2014-09-11 18:02:49,629 Stage-1 map = 42%,  reduce = 3%, Cumulative CPU 29.53 sec
2014-09-11 18:02:50,631 Stage-1 map = 42%,  reduce = 3%, Cumulative CPU 29.53 sec
2014-09-11 18:02:51,633 Stage-1 map = 44%,  reduce = 4%, Cumulative CPU 29.53 sec
2014-09-11 18:02:52,637 Stage-1 map = 44%,  reduce = 4%, Cumulative CPU 29.53 sec
2014-09-11 18:02:53,641 Stage-1 map = 46%,  reduce = 4%, Cumulative CPU 30.81 sec
2014-09-11 18:02:54,643 Stage-1 map = 48%,  reduce = 4%, Cumulative CPU 33.52 sec
2014-09-11 18:02:55,646 Stage-1 map = 48%,  reduce = 4%, Cumulative CPU 33.52 sec
2014-09-11 18:02:56,650 Stage-1 map = 48%,  reduce = 4%, Cumulative CPU 33.52 sec
2014-09-11 18:02:57,653 Stage-1 map = 48%,  reduce = 5%, Cumulative CPU 33.52 sec
2014-09-11 18:02:58,656 Stage-1 map = 48%,  reduce = 5%, Cumulative CPU 36.95 sec
2014-09-11 18:02:59,659 Stage-1 map = 48%,  reduce = 6%, Cumulative CPU 36.95 sec
2014-09-11 18:03:00,663 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 37.23 sec
2014-09-11 18:03:01,667 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 37.23 sec
2014-09-11 18:03:02,671 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 37.23 sec
2014-09-11 18:03:03,674 Stage-1 map = 50%,  reduce = 11%, Cumulative CPU 37.23 sec
2014-09-11 18:03:04,678 Stage-1 map = 50%,  reduce = 11%, Cumulative CPU 37.23 sec
2014-09-11 18:03:05,681 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 37.23 sec
2014-09-11 18:03:06,683 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 37.23 sec
2014-09-11 18:03:07,688 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:08,692 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:09,695 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:10,700 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:11,703 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:12,709 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:13,712 Stage-1 map = 54%,  reduce = 13%, Cumulative CPU 38.38 sec
2014-09-11 18:03:14,714 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:15,717 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:16,719 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:17,722 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:18,725 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:19,728 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:20,730 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:21,735 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:22,739 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:23,741 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 38.38 sec
2014-09-11 18:03:24,744 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:25,746 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:26,748 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:27,752 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:28,754 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:29,758 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:30,761 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:31,764 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:32,767 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:33,774 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:34,777 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:35,780 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:36,782 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:37,785 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:38,788 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:39,790 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:40,793 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:41,796 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 41.66 sec
2014-09-11 18:03:42,800 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:43,804 Stage-1 map = 63%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:44,808 Stage-1 map = 63%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:45,811 Stage-1 map = 63%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:46,815 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:47,818 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 42.04 sec
2014-09-11 18:03:48,821 Stage-1 map = 65%,  reduce = 16%, Cumulative CPU 42.04 sec
2014-09-11 18:03:49,823 Stage-1 map = 65%,  reduce = 16%, Cumulative CPU 42.04 sec
2014-09-11 18:03:50,826 Stage-1 map = 65%,  reduce = 17%, Cumulative CPU 42.04 sec
2014-09-11 18:03:51,829 Stage-1 map = 65%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:52,831 Stage-1 map = 65%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:53,834 Stage-1 map = 65%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:54,837 Stage-1 map = 65%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:55,840 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:56,843 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:57,846 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 42.04 sec
2014-09-11 18:03:58,850 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:03:59,854 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:00,859 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:01,863 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:02,867 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:03,870 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:04,872 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:05,877 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:06,881 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:07,884 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:08,888 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:09,891 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:10,894 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:11,897 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:12,899 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:13,903 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:14,908 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:15,911 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:16,915 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:17,917 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:18,920 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:19,923 Stage-1 map = 73%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:20,925 Stage-1 map = 73%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:21,928 Stage-1 map = 75%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:22,931 Stage-1 map = 75%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:23,934 Stage-1 map = 75%,  reduce = 18%, Cumulative CPU 52.31 sec
2014-09-11 18:04:24,937 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:25,940 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:26,943 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:27,948 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:28,952 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:29,955 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:30,957 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:31,960 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:32,964 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:33,967 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:34,970 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:35,972 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:36,976 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:37,978 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:38,982 Stage-1 map = 77%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:39,985 Stage-1 map = 79%,  reduce = 18%, Cumulative CPU 56.59 sec
2014-09-11 18:04:40,988 Stage-1 map = 81%,  reduce = 18%, Cumulative CPU 57.94 sec
2014-09-11 18:04:41,995 Stage-1 map = 81%,  reduce = 18%, Cumulative CPU 57.94 sec
2014-09-11 18:04:42,998 Stage-1 map = 81%,  reduce = 18%, Cumulative CPU 57.94 sec
2014-09-11 18:04:44,000 Stage-1 map = 81%,  reduce = 19%, Cumulative CPU 57.94 sec
2014-09-11 18:04:45,003 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:46,005 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:47,010 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:48,014 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:49,017 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:50,019 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:51,022 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:52,025 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:53,028 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:54,030 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:55,033 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:56,036 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:57,039 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:58,043 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 57.94 sec
2014-09-11 18:04:59,047 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:00,050 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:01,052 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:02,055 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:03,058 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:04,062 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:05,065 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:06,069 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:07,073 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:08,075 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:09,079 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:10,082 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:11,085 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:12,088 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:13,090 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:14,092 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:15,095 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:16,099 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:17,102 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:18,105 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:19,108 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:20,110 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:21,114 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:22,117 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:23,119 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:24,122 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 64.35 sec
2014-09-11 18:05:25,124 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:26,127 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:27,130 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:28,133 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:29,137 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:30,140 Stage-1 map = 91%,  reduce = 21%, Cumulative CPU 67.35 sec
2014-09-11 18:05:31,142 Stage-1 map = 92%,  reduce = 21%, Cumulative CPU 68.52 sec
2014-09-11 18:05:32,146 Stage-1 map = 92%,  reduce = 21%, Cumulative CPU 68.52 sec
2014-09-11 18:05:33,149 Stage-1 map = 92%,  reduce = 21%, Cumulative CPU 68.52 sec
2014-09-11 18:05:34,152 Stage-1 map = 92%,  reduce = 21%, Cumulative CPU 68.52 sec
2014-09-11 18:05:35,157 Stage-1 map = 92%,  reduce = 21%, Cumulative CPU 68.52 sec
2014-09-11 18:05:36,160 Stage-1 map = 94%,  reduce = 21%, Cumulative CPU 68.94 sec
2014-09-11 18:05:37,164 Stage-1 map = 94%,  reduce = 21%, Cumulative CPU 68.94 sec
2014-09-11 18:05:38,167 Stage-1 map = 94%,  reduce = 22%, Cumulative CPU 68.94 sec
2014-09-11 18:05:39,170 Stage-1 map = 96%,  reduce = 23%, Cumulative CPU 70.0 sec
2014-09-11 18:05:40,172 Stage-1 map = 96%,  reduce = 23%, Cumulative CPU 70.0 sec
2014-09-11 18:05:41,175 Stage-1 map = 96%,  reduce = 23%, Cumulative CPU 70.0 sec
2014-09-11 18:05:42,178 Stage-1 map = 96%,  reduce = 23%, Cumulative CPU 70.0 sec
2014-09-11 18:05:43,181 Stage-1 map = 96%,  reduce = 23%, Cumulative CPU 70.0 sec
2014-09-11 18:05:44,185 Stage-1 map = 98%,  reduce = 26%, Cumulative CPU 70.73 sec
2014-09-11 18:05:45,188 Stage-1 map = 98%,  reduce = 29%, Cumulative CPU 70.73 sec
2014-09-11 18:05:46,190 Stage-1 map = 100%,  reduce = 29%, Cumulative CPU 71.43 sec
2014-09-11 18:05:47,193 Stage-1 map = 100%,  reduce = 29%, Cumulative CPU 71.43 sec
2014-09-11 18:05:48,196 Stage-1 map = 100%,  reduce = 29%, Cumulative CPU 71.43 sec
2014-09-11 18:05:49,199 Stage-1 map = 100%,  reduce = 29%, Cumulative CPU 71.43 sec
2014-09-11 18:05:50,203 Stage-1 map = 100%,  reduce = 65%, Cumulative CPU 72.67 sec
2014-09-11 18:05:51,209 Stage-1 map = 100%,  reduce = 65%, Cumulative CPU 73.35 sec
2014-09-11 18:05:52,212 Stage-1 map = 100%,  reduce = 83%, Cumulative CPU 73.35 sec
2014-09-11 18:05:53,214 Stage-1 map = 100%,  reduce = 83%, Cumulative CPU 73.35 sec
2014-09-11 18:05:54,217 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 74.94 sec
2014-09-11 18:05:55,221 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 74.94 sec
2014-09-11 18:05:56,225 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 74.94 sec
2014-09-11 18:05:57,229 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 74.94 sec
MapReduce Total cumulative CPU time: 1 minutes 14 seconds 940 msec
Ended Job = job_201409072150_0030
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201409072150_0031, Tracking URL = http://jifeng01:50030/jobdetails.jsp?jobid=job_201409072150_0031
Kill Command = /home/jifeng/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job  -kill job_201409072150_0031
Hadoop job information for Stage-2: number of mappers: 3; number of reducers: 1
2014-09-11 18:05:59,481 Stage-2 map = 0%,  reduce = 0%
2014-09-11 18:06:01,490 Stage-2 map = 33%,  reduce = 0%, Cumulative CPU 0.36 sec
2014-09-11 18:06:02,494 Stage-2 map = 33%,  reduce = 0%, Cumulative CPU 0.36 sec
2014-09-11 18:06:03,498 Stage-2 map = 33%,  reduce = 0%, Cumulative CPU 0.36 sec
2014-09-11 18:06:04,500 Stage-2 map = 67%,  reduce = 0%, Cumulative CPU 0.71 sec
2014-09-11 18:06:05,503 Stage-2 map = 67%,  reduce = 0%, Cumulative CPU 0.71 sec
2014-09-11 18:06:06,506 Stage-2 map = 67%,  reduce = 0%, Cumulative CPU 0.71 sec
2014-09-11 18:06:07,509 Stage-2 map = 67%,  reduce = 0%, Cumulative CPU 0.71 sec
2014-09-11 18:06:08,512 Stage-2 map = 67%,  reduce = 22%, Cumulative CPU 0.71 sec
2014-09-11 18:06:09,515 Stage-2 map = 100%,  reduce = 22%, Cumulative CPU 1.19 sec
2014-09-11 18:06:10,518 Stage-2 map = 100%,  reduce = 22%, Cumulative CPU 1.19 sec
2014-09-11 18:06:11,520 Stage-2 map = 100%,  reduce = 22%, Cumulative CPU 1.19 sec
2014-09-11 18:06:12,524 Stage-2 map = 100%,  reduce = 22%, Cumulative CPU 1.19 sec
2014-09-11 18:06:13,527 Stage-2 map = 100%,  reduce = 22%, Cumulative CPU 1.19 sec
2014-09-11 18:06:14,531 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 1.81 sec
2014-09-11 18:06:15,534 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 1.81 sec
2014-09-11 18:06:16,538 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 1.81 sec
MapReduce Total cumulative CPU time: 1 seconds 810 msec
Ended Job = job_201409072150_0031
MapReduce Jobs Launched: 
Job 0: Map: 13  Reduce: 4   Cumulative CPU: 74.94 sec   HDFS Read: 3415495752 HDFS Write: 45773 SUCCESS
Job 1: Map: 3  Reduce: 1   Cumulative CPU: 1.81 sec   HDFS Read: 47452 HDFS Write: 23372 SUCCESS
Total MapReduce CPU Time Spent: 1 minutes 16 seconds 750 msec
OK
10.12.15.152    3875019
10.13.77.49     1954742
10.6.6.7        1769857
10.138.235.111  1630920
10.12.14.15     756475
10.17.15.10     720936
172.10.0.4      482568
10.78.100.150   300530
10.247.58.182   292384
10.38.217.48    282588
172.30.0.4      232850
10.10.11.102    205579
10.170.32.110   202735
10.170.32.181   200539
10.138.214.18   197864
10.247.106.27   193912
10.156.165.120  183929
10.15.7.85      172422
10.0.0.42       165063
10.200.20.2     164519
10.70.68.127    161735
172.20.0.15     145924
10.250.178.101  140053
172.20.0.4      122768
172.30.1.223    51329
172.20.1.81     51328
172.10.2.66     51304
172.20.1.47     51298
172.30.1.218    51236

Time taken: 366.904 seconds, Fetched: 1389 row(s)


hive> desc firewall;
OK
idauto                  int                     None                
dt                      timestamp               None                
dtstr                   string                  None                
timeseconds             int                     None                
priority                string                  None                
operation               string                  None                
messagecode             string                  None                
protocol                string                  None                
srcip                   string                  None                
destip                  string                  None                
srcipnumber             tinyint                 None                
destipnumber            tinyint                 None                
srcport                 int                     None                
destport                int                     None                
destservice             string                  None                
direction               string                  None                
flags                   string                  None                
command                 string                  None                
slot1min                int                     None                
slot5min                int                     None                
slot30min               int                     None                
slot60min               int                     None                
Time taken: 0.106 seconds, Fetched: 22 row(s)
hive> select * from firewall order by dt desc limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201409072150_0032, Tracking URL = http://jifeng01:50030/jobdetails.jsp?jobid=job_201409072150_0032
Kill Command = /home/jifeng/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job  -kill job_201409072150_0032
Hadoop job information for Stage-1: number of mappers: 13; number of reducers: 1
2014-09-11 18:53:36,120 Stage-1 map = 0%,  reduce = 0%
2014-09-11 18:54:03,210 Stage-1 map = 8%,  reduce = 0%
2014-09-11 18:54:05,217 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 19.33 sec
2014-09-11 18:54:06,221 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 19.33 sec
2014-09-11 18:54:07,264 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:08,267 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:09,270 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:10,272 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:11,277 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:12,282 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:13,286 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:14,289 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:15,294 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:16,299 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:17,301 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:18,304 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:19,307 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:20,312 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:21,315 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:22,320 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:23,325 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:24,329 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:25,331 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:26,335 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:27,337 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:28,341 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:29,344 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:30,444 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 40.04 sec
2014-09-11 18:54:31,448 Stage-1 map = 20%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:32,451 Stage-1 map = 24%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:33,455 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:34,458 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:35,461 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:36,464 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:37,469 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:38,473 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:39,476 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:40,480 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:41,482 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:42,486 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 69.96 sec
2014-09-11 18:54:43,490 Stage-1 map = 30%,  reduce = 0%, Cumulative CPU 75.79 sec
2014-09-11 18:54:44,493 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:45,496 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:46,501 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:47,506 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:48,509 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:49,513 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:50,517 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:51,521 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:52,525 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:53,530 Stage-1 map = 37%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:54,533 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:55,536 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:56,539 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:57,543 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:58,546 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 81.82 sec
2014-09-11 18:54:59,551 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:00,559 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:01,563 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:02,568 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:03,572 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:04,578 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 104.83 sec
2014-09-11 18:55:05,582 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 148.55 sec
2014-09-11 18:55:06,586 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 148.55 sec
2014-09-11 18:55:07,591 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:08,594 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:09,598 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:10,601 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:11,605 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:12,608 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 171.88 sec
2014-09-11 18:55:13,610 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:14,617 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:15,620 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:16,625 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:17,627 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:18,632 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 172.61 sec
2014-09-11 18:55:19,636 Stage-1 map = 46%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:20,639 Stage-1 map = 46%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:21,643 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:22,646 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:23,652 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:24,656 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:25,660 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:26,664 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:27,667 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:28,671 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 173.86 sec
2014-09-11 18:55:29,675 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 173.86 sec
2014-09-11 18:55:30,678 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 173.86 sec
2014-09-11 18:55:31,684 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 194.76 sec
2014-09-11 18:55:32,689 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 194.76 sec
2014-09-11 18:55:33,693 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 194.76 sec
2014-09-11 18:55:34,696 Stage-1 map = 50%,  reduce = 13%, Cumulative CPU 194.76 sec
2014-09-11 18:55:35,700 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:36,703 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:37,707 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:38,851 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:39,854 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:40,859 Stage-1 map = 52%,  reduce = 13%, Cumulative CPU 201.47 sec
2014-09-11 18:55:41,862 Stage-1 map = 52%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:42,866 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:43,870 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:44,873 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:45,878 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:46,884 Stage-1 map = 54%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:47,888 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:48,894 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:49,898 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:50,902 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:51,905 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:52,908 Stage-1 map = 56%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:53,912 Stage-1 map = 58%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:54,915 Stage-1 map = 58%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:55,919 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:56,923 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:57,927 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:58,929 Stage-1 map = 60%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:55:59,933 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:00,937 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:01,939 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:02,943 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:03,947 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:04,952 Stage-1 map = 62%,  reduce = 15%, Cumulative CPU 201.47 sec
2014-09-11 18:56:05,955 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 233.97 sec
2014-09-11 18:56:06,958 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 255.02 sec
2014-09-11 18:56:07,960 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 255.02 sec
2014-09-11 18:56:08,965 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 255.02 sec
2014-09-11 18:56:09,969 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 255.02 sec
2014-09-11 18:56:10,973 Stage-1 map = 65%,  reduce = 15%, Cumulative CPU 255.02 sec
2014-09-11 18:56:11,976 Stage-1 map = 65%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:12,979 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:13,984 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:14,987 Stage-1 map = 67%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:15,991 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:16,995 Stage-1 map = 69%,  reduce = 18%, Cumulative CPU 255.02 sec
2014-09-11 18:56:18,000 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 267.57 sec
2014-09-11 18:56:19,003 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 267.57 sec
2014-09-11 18:56:20,007 Stage-1 map = 71%,  reduce = 18%, Cumulative CPU 267.57 sec
2014-09-11 18:56:21,011 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:22,014 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:23,022 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:24,025 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:25,028 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:26,033 Stage-1 map = 71%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:27,037 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:28,041 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:29,047 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:30,050 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:31,053 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 267.57 sec
2014-09-11 18:56:32,056 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:33,060 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:34,064 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:35,067 Stage-1 map = 73%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:36,071 Stage-1 map = 75%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:37,074 Stage-1 map = 75%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:38,077 Stage-1 map = 77%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:39,080 Stage-1 map = 77%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:40,084 Stage-1 map = 79%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:41,087 Stage-1 map = 79%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:42,092 Stage-1 map = 79%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:43,095 Stage-1 map = 79%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:44,099 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:45,102 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:46,105 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:47,109 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:48,113 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:49,117 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:50,120 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:51,124 Stage-1 map = 81%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:52,129 Stage-1 map = 83%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:53,136 Stage-1 map = 85%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:54,139 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:55,144 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:56,149 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:57,152 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:58,155 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:56:59,164 Stage-1 map = 87%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:57:00,169 Stage-1 map = 89%,  reduce = 21%, Cumulative CPU 272.61 sec
2014-09-11 18:57:01,171 Stage-1 map = 90%,  reduce = 21%, Cumulative CPU 288.97 sec
2014-09-11 18:57:02,175 Stage-1 map = 90%,  reduce = 21%, Cumulative CPU 288.97 sec
2014-09-11 18:57:03,178 Stage-1 map = 90%,  reduce = 21%, Cumulative CPU 288.97 sec
2014-09-11 18:57:04,181 Stage-1 map = 90%,  reduce = 21%, Cumulative CPU 288.97 sec
2014-09-11 18:57:05,184 Stage-1 map = 90%,  reduce = 21%, Cumulative CPU 288.97 sec
2014-09-11 18:57:06,188 Stage-1 map = 90%,  reduce = 23%, Cumulative CPU 323.8 sec
2014-09-11 18:57:07,192 Stage-1 map = 90%,  reduce = 23%, Cumulative CPU 345.22 sec
2014-09-11 18:57:08,195 Stage-1 map = 92%,  reduce = 23%, Cumulative CPU 345.22 sec
2014-09-11 18:57:09,200 Stage-1 map = 92%,  reduce = 23%, Cumulative CPU 345.22 sec
2014-09-11 18:57:10,204 Stage-1 map = 92%,  reduce = 23%, Cumulative CPU 345.22 sec
2014-09-11 18:57:11,209 Stage-1 map = 92%,  reduce = 23%, Cumulative CPU 345.22 sec
2014-09-11 18:57:12,213 Stage-1 map = 94%,  reduce = 23%, Cumulative CPU 349.49 sec
2014-09-11 18:57:13,217 Stage-1 map = 94%,  reduce = 23%, Cumulative CPU 349.49 sec
2014-09-11 18:57:14,220 Stage-1 map = 94%,  reduce = 23%, Cumulative CPU 349.49 sec
2014-09-11 18:57:15,224 Stage-1 map = 94%,  reduce = 23%, Cumulative CPU 349.49 sec
2014-09-11 18:57:16,227 Stage-1 map = 98%,  reduce = 23%, Cumulative CPU 372.21 sec
2014-09-11 18:57:17,230 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 378.56 sec
2014-09-11 18:57:18,233 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 378.56 sec
2014-09-11 18:57:19,236 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 378.56 sec
2014-09-11 18:57:20,240 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 378.56 sec
2014-09-11 18:57:21,245 Stage-1 map = 100%,  reduce = 33%, Cumulative CPU 378.56 sec
2014-09-11 18:57:22,248 Stage-1 map = 100%,  reduce = 33%, Cumulative CPU 378.56 sec
2014-09-11 18:57:23,251 Stage-1 map = 100%,  reduce = 33%, Cumulative CPU 378.56 sec
2014-09-11 18:57:24,255 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 380.17 sec
2014-09-11 18:57:25,258 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 380.17 sec
2014-09-11 18:57:26,261 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 380.17 sec
2014-09-11 18:57:27,264 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 380.17 sec
MapReduce Total cumulative CPU time: 6 minutes 20 seconds 170 msec
Ended Job = job_201409072150_0032
MapReduce Jobs Launched: 
Job 0: Map: 13  Reduce: 1   Cumulative CPU: 380.17 sec   HDFS Read: 3415495752 HDFS Write: 1935 SUCCESS
Total MapReduce CPU Time Spent: 6 minutes 20 seconds 170 msec
OK
16600911        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.4910.0.3.2        NULL    NULL    58200   1993    1993_tcp        (empty) (empty) (empty) 22767000        4553400 758900  379450
16600930        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.4910.0.3.2        NULL    NULL    58200   13817   13817_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600920        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Info    Teardown        ASA-6-302014    TCP     10.6.6.7    172.10.0.2      NULL    NULL    42706   3389    3389_tcp        inbound SYN Timeout     (empty) 22767000        4553400 758900      379450
16600913        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.4910.0.3.2        NULL    NULL    58199   32179   32179_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600912        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.710.0.2.2        NULL    NULL    42465   556     remotefs        (empty) (empty) (empty) 22767000        4553400 758900  379450
16600918        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.4910.0.3.2        NULL    NULL    58199   25087   25087_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600919        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.710.0.2.2        NULL    NULL    42465   38300   38300_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600931        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.710.0.2.2        NULL    NULL    42464   33004   33004_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600915        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.13.77.4910.0.3.2        NULL    NULL    58200   32179   32179_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
16600916        2013-04-15 10:00:00     15/Apr/2013 10:00:00    1366020000      Warning Deny    ASA-4-106023    TCP     10.6.6.710.0.2.2        NULL    NULL    42465   52372   52372_tcp       (empty) (empty) (empty) 22767000        4553400 758900  379450
Time taken: 233.592 seconds, Fetched: 10 row(s)
hive>


收录于
Hive

当前文章被收录于：

Hive
19 篇文章 0 人学习
订阅专栏 查看详情
标签
#hive #Hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
【面试经验】京东-技术产品经理一面：从紧张到反思的全流程复盘 本文详细复盘了京东技术产品经理一面的全流程，从面试前的简历打磨、业务理解和技术基础准备，到面试中的高压对话和突发状况应对，再到面试后的深度反思与改进方案。文章特别强调了技术产品经理需要具备的技术理解力、数据敏感度和商业思维，并提供了实用的面试建议和准备技巧。 阅读详情

相关推荐

别再为reg2icg的setup违例头疼了！手把手教你用ICC2/Innovus这3招搞定（附实战数据对比）

本文详细介绍了在ICC2/Innovus中解决reg2icg的setup违例的三种高效方法，包括早期时钟树预估、精准控制时钟延迟与门控检查以及ICG专用优化选项组合。通过实战数据对比，展示了不同策略的效果差异，帮助工程师快速优化时序路径，提升芯片性能。

weixin_42569329的博客 408

Hive如何建表+导入数据+导出大文件

如果数据量很大，达到几百万，而且需要传输，一般将数据文件处理成一个大文件，方便传输后，对方进行数据处理INSERTOVERWRITEDIRECTORY表示重写目录，重写后源数据被覆盖。partitioned表示分区，数据量不大、无区域数据调整的情况下，一般不用分区。对于1万行以内的数据，我们常可以通过select直接获取，用excel进行分析。（1）注意数据的格式，最常见的为将含中文的文件保存为utf-8格式；都能将对应的partition数据文件删除。3.2将临时表数据存入一个特定的目录下。...

lz_N_one的博客 1857

使用下一代的嵌入式IDE-PlatformIO 教程

1.生成工程 1.1 打开STM32CubeMX： 选择你的MCU，这里我用的是STM32F103RC RCC中开启外部HSE时钟，外部时钟比HSI更稳定些 开启DEBUG 4线 打开串口1用于打印调试信息，波特率115200，校验位0，停止位1 打开串口2用于MODBUS协议，波特率9600，校验位0，停止位1 接入RT-thread：参考官方文档 基于 CubeMX 移植 RT-Thread Nano 根据官方文档，需要1.取消生成HardFault_Handler、PendSV_Handler、Sy

jiladahe1997的博客 3万+

hive大小文件合并及Hive配置参数

hive文件合并及Hive配置参数

KANGBboy的博客 4710

LabVIEW实战：基于DAQmx的实时正弦波生成、采集与波形图动态显示

本文详细介绍了如何使用LabVIEW和DAQmx驱动实现实时正弦波的生成、采集与动态显示。通过USB-6008数据采集卡，结合队列缓冲机制和动态波形图显示技术，构建了一个完整的信号处理系统。文章包含硬件连接、软件配置、任务设置及常见问题排查，适合LabVIEW初学者和硬件控制开发者参考。

weixin_30872337的博客 124

别再混淆了！5分钟搞懂机器人导纳控制与阻抗控制的本质区别（附Simulink对比仿真）

本文深入解析机器人柔顺控制中的导纳控制与阻抗控制的本质区别，通过信号流方向、架构对比和应用场景三个关键视角进行详细阐述。文章还提供Simulink对比仿真案例，帮助读者快速理解两种控制策略的差异及其适用场景，特别适合机器人控制领域的研究者和工程师参考。

weixin_42543046的博客 605

大数据开发之Hive案例篇3-sqoop导入到hive的大文件

大数据开发 hive 小文件

只是甲的博客 1138

NCCL使用指南-进阶篇

前文介绍了NCCL的常用环境变量，本文介绍了NCCL-test性能基准测试，以及一些大规模集群测试的经验数据。

father_yingying的博客 1407

13.107.最全的Hive 优化方案汇总：临时参数的作用域、切分大文件、合并小文件、设置 Map 和 Reduce 的内存大小、设置 Map 和 Reduce 的堆大小设置 等

13.107.最全的Hive 优化方案汇总：1.1.8. 临时参数的作用域：1.1.10.切分大文件1.1.11.合并小文件1.1.12.设置 Map 和 Reduce 的内存大小1.1.13.设置 Map 和 Reduce 的堆大小设置1.1.14.开启Combiner功能：在map端预聚合1.1.15.拓扑图，优化并行执行1.1.16. 万能方法1：一个MR，拆成多个（即纵向拆分），为了降低数据倾斜的压力。【比较少用】

涂作权的博客 1659

我爬取了某非法网贷后台系统数据，终于知道什么叫做利滚利滚利滚利了

我爬取了非法贷款的后台网站！

weixin_44042821的博客 2880

分析攻击流量样本

利用wireshark分析.pcap攻击样本的攻击流程，利用漏洞等。

一段旅途 6844

大数据Hive DML-Load加载数据

目录1 背景2 Load语法2.1 filepath2.2 LOCAL2.3 OVERWRITE3 案例：load加载数据到Hive表4 Hive3.0 Load新特性 1 背景 回想一下，当在Hive中创建好表之后，默认就会在HDFS上创建一个与之对应的文件夹，默认路径是由参数hive.metastore.warehouse.dir控制，默认值是/user/hive/warehouse。 要想让hive的表和结构化的数据文件产生映射，就需要把文件移到到表对应的文件夹下面，当然，可以在建

赵广陆 1337

containerd镜像加速配置全解析：从host.toml到config.toml的演进与实践

本文全面解析了containerd镜像加速的配置方法，重点对比了传统的config.toml集中式配置与现代的host.toml模块化配置的演进与优劣。文章通过实战演示，详细指导如何为docker.io、registry.k8s.io等常见仓库配置host.toml加速，并提供了生产环境下的最佳实践、故障排查与性能优化建议，帮助用户显著提升镜像拉取效率与系统稳定性。

jenkins8butler的博客 698

Hive合并小文件参数总结

hive merge小文件 一:为什么要合并小文件 当Hive输入由很多个小文件组成，由于每个小文件都会启动一个map任务，如果文件过小，以至于map任务启动和初始化的时间大于逻辑处理的时间，会造成资源浪费，甚至OOM。 为此，当我们启动一个任务，发现输入数据量小但任务数量多时，需要注意在Map前端进行输入合并 当然，在我们向一个表写数据时，也需要注意输出文件大小 1. Map输入合并小文件 对应...

qq_43688472的博客 3691

霍尼韦尔SoftMaster使用说明

霍尼韦尔SoftMaster使用说明

TMS_WebGMaps_1.9.8.0_Full_Source

TMS_WebGMaps_1.9.8.0_Full_Source 全源码 delphi cbuilder

Hive小文件合并

Hive的后端存储是HDFS，它对大文件的处理是非常高效的，如果合理配置文件系统的块大小，NameNode可以支持很大的数据量。但是在数据仓库中，越是上层的表其汇总程度就越高，数据量也就越小。而且这些表通常会按日期进行分区，随着时间的推移，HDFS的文件数目就会逐渐增加。 小文件带来的问题 关于这个问题的阐述可以读一读Cloudera的这篇文章。简单来说，HDFS的文件元信息，

yycdaizi的专栏 6万+

SIMATIC S7-1200, S7-1500 PID 控制[手册]

SIMATIC S7-1200, S7-1500 PID 控制[手册]

Hive新功能 Cube, Rollup介绍

说明：Hive之cube、rollup，还有窗口函数，在传统关系型数据（Oracle、sqlserver）中都是有的，用法都很相似。 GROUPING SETS GROUPING SETS作为GROUP BY的子句，允许开发人员在GROUP BY语句后面指定多个统计选项，可以简单理解为多条group by语句通过union all把查询结果聚合起来结合起来，下面是几个实例可以帮助我们了解，...

weixin_33862993的博客 350

廖雪峰Python教程[完整版]

廖雪峰的python教程整理的，已经排好版了，而且还是完整的，适合小白

spark sql and hive 3g数据测试

1：上传文件到hdfs [jifeng@feng01 hadoop]$ hadoop fs -mkdir /user/jifeng/fire 15/03/05 13:29:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java class

风行天下 1833

hive tez调优（3）

根据。方案最右侧一栏是一个8G VM的分配方案，方案预留1-2G的内存给操作系统，分配4G给Yarn/MapReduce，当然也包括了HIVE，剩余的2-3G是在需要使用HBase时预留给HBase的。 内存调优 一、AM、Container大小设置1、tez.am.resource.memory.mb　　#设置 tez AM容器内存 　　默认值：1024　　 　　配置文件：tez...

weixin_34112181的博客 2829

hive中contain

我整理的一些关于【SQL】的项目学习资料（附讲解～～）和大家一起分享、学习一下：https://edu.51cto.com/surl=QDW3g3使用 Hive 实现“Contain”功能的指南 在数据处理与分析的世界中，Apache Hive 是一个重要的工具，特别是在处理大数据时。若你想在 Hive 中实现“cont...

weixin_42094592的博客 212

pyspark读取hive数据写入到redis

1、首先把redis包引入工程，这样就不需要在集群里每台机器上安装redis客户端。 $pip install redis $cd /usr/local/lib/python3.6/dist-packages/ 找到自己环境的路径 $zip -r redis.zip redis/* $hadoop fs -put redis.zip /user/data/ 2、在代码里使用 addPyFile加...

TURING.DT 4000

hive新功能 Cube, Rollup介绍

说明：hive之cube、rollup，还有窗口函数，在传统关系型数据（oracle、sqlserver）中都是有的，用法都很相似。 GROUPING SETS GROUPING SETS作为GROUP BY的子句，允许开发人员在GROUP BY语句后面指定多个统计选项，可以简单理解为多条group by语句通过union all把查询结果聚合起来结合起来，下面是几个实例可以帮助我们了

扫大街的程序员 2万+

AI那些趣事系列121：智能问数场景中使用ClickHouse处理离线3G大文件

之前在做智能问数智能体项目，输入是用户的问题和各类不同的数据源，这个数据源可能是JDBC这一类(包括PostGreSql、Mysql、Oracle、SQLServer)等，利用大模型的能力来生成SQL语句进行查询获得相关数据，输出就是用户问题对应的SQL语句、执行SQL语句查询的结果还有对应的推荐的图表。对于算法测来说，我重点关注的是如何如何获取离线大文件的元数据信息，将用户问题和元数据信息等通过prompt工程调用大模型来生成可执行的SQL语句，从而进行查询。帆软等主流的方案是，对于3G 大文件，

abc50319的专栏 1080

hive调优

hive中数据倾斜问题的处理(hive调优) 数据倾斜的概况 hive在处理大数据经常会遇到数据倾斜的问题,表现为在map100%,reduce99%…的界面一直卡着,一般情况下就是遇到了数据倾斜的问题 主要原因是大车拉小货,小车拉大货,计算能力强的节点处理数据少,计算能力弱的节点需要计算的数据比较多,有的节点完成任务休息了,有的节点还在边卡顿边干活,拖长执行的时间,导致工作效率的低下. 问题实例...

weixin_45896475的博客 194

(二)presto集成Hive

一、集成之前 在presto集成Hive之前，要先启动Hive;由于Hive依赖HDFS,并且这里环境其元数据存储在mysql中，故在启动Hive之前要先启动HDFS和Mysql; 1、主节点启动hdfs [root@master sbin]# pwd /opt/softWare/hadoop/hadoop-2.7.3/sbin [root@master sbin]# ./start-all.sh 2、启动Mysql [root@master ~]# mysql -u root -p 3、启

hyj 1926

hive小文件合并机制_hive小文件合并

hive仓库表数据最终是存储在HDFS上，由于Hadoop的特性，对大文件的处理非常高效。而且大文件可以减少文件元数据信息，减轻NameNode的存储压力。但是在数据仓库中，越是上层的表汇总程度就越高，数据量也就越小，而且这些表通常会有日期分区，随着时间的推移，HDFS的文件数目就会逐步增加。一、小文件带来的问题HDFS的文件包好数据块和元信息，其中元信息包括位置、大小、分块等信息，都保存在Nam...

weixin_34008445的博客 750
上一篇： hive中删除表的错误Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException
下一篇： Hive 0.13 数据类型
点赞 0
评论 0
书签 书签 1
wind520
wind520
博客等级 码龄25年 303粉丝 264原创
+ 关注 专栏目录 订阅专栏
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值