Hive Multi Insert 引起的GC overhead limit exceeded

yjgithub yjgithub 博客等级 墨衍会员 · AI 创作全网分发 墨衍智能分发 本文由作者通过墨衍一键同步至各平台 1分发平台 1.8k累计阅读 1.9k平均单平台 已同步平台 优快云 微信公众号 微博 知乎 掘金 百家号 博客园 抖音 小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年，写一次发全网 我也要 码龄11年
关注
转载 最新推荐文章于 2026-04-14 09:02:22 发布 1.8k 阅读 AI 写同款· GEO 优化 ›
数据工程师的ML工程化实战框架：可部署、可监控、可迭代 机器学习工程化（MLOps）本质是将模型生命周期纳入数据平台统一治理，其核心在于特征管理、训练编排与推理服务的标准化封装。本文聚焦数据工程师视角，深入解析如何基于现有Airflow+Spark+Delta Lake技术栈，构建轻量、嵌入式、可观测的ML流水线——不依赖Kubeflow等重型平台，而是通过feature_catalog特征注册、DVC模型版本控制、Triton热切换等关键技术，实现特征新鲜度SLA保障、GPU资源隔离调度、分钟级模型回滚等生产级能力。适用于金融风控、电商推荐、IoT时序分析等需 阅读详情

转载于该大兄弟:
http://ju.outofmemory.cn/entry/224490

当你有这么个需求从某张hive表里做各类统计，完了之后落到各个分类的统计表里存储。自然而然我们会想到使用hive的Multi Insert 语句来实现。因为使用Multi Insert 语句可以避免多次扫描同一份原始表数据。本文记录一次使用Multi Insert 语句出现的GC overhead limit exceeded问题。

问题描述

我有这么个需求从某个域名相关的表里统计各个维度的数据落到相应的表里面。下面是我的SQL实例代码：

FROM qbox_bi_gold.domain_info INPUT
             INSERT OVERWRITE TABLE 5min PARTITION (day="20151130")
                  SELECT cast(time/300000 as bigint)*300000 AS time  , SUM(flow) AS flow,SUM(hits) AS hits
                  WHERE day="20151130"
                  GROUP BY cast(time/300000 as bigint)*300000
             INSERT OVERWRITE TABLE prov_5min PARTITION (day="20151130")
                   SELECT cast(time/300000 as bigint)*300000 AS time ,prov, SUM(flow) AS flow,SUM(hits) AS hits
                   WHERE day="20151130"
                   GROUP BY cast(time/300000 as bigint)*300000 ,prov
             INSERT OVERWRITE TABLE prov_uid_5min PARTITION (day="20151130")
                    SELECT cast(time/300000 as bigint)*300000 AS time ,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                    WHERE day="20151130"
                    GROUP BY cast(time/300000 as bigint)*300000 ,prov,uid
              INSERT OVERWRITE TABLE bucket_prov_uid_5min PARTITION (day="20151130")
                    SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                    WHERE day="20151130"
                    GROUP BY cast(time/300000 as bigint)*300000 ,bucket,prov,uid
              INSERT OVERWRITE TABLE bucket_domain_prov_uid_5min PARTITION (day="20151130")
                    SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,domain,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                    WHERE day="20151130"
                    GROUP BY cast(time/300000 as bigint)*300000 ,bucket,domain,prov,uid
              INSERT OVERWRITE TABLE bucket_city_domain_prov_uid_5min PARTITION (day="20151130")
                    SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,city,domain,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                    WHERE day="20151130"
                    GROUP BY cast(time/300000 as bigint)*300000 ,bucket,city,domain,prov,uid

上述语句会产生6个Job，你可以使用explain hsql来查看执行解析流程：


STAGE DEPENDENCIES:
  Stage-6 is a root stage
  Stage-0 depends on stages: Stage-6
  Stage-7 depends on stages: Stage-0
  Stage-8 depends on stages: Stage-6
  Stage-1 depends on stages: Stage-8
  Stage-9 depends on stages: Stage-1
  Stage-10 depends on stages: Stage-6
  Stage-2 depends on stages: Stage-10
  Stage-11 depends on stages: Stage-2
  Stage-12 depends on stages: Stage-6
  Stage-3 depends on stages: Stage-12
  Stage-13 depends on stages: Stage-3
  Stage-14 depends on stages: Stage-6
  Stage-4 depends on stages: Stage-14
  Stage-15 depends on stages: Stage-4
  Stage-16 depends on stages: Stage-6
  Stage-5 depends on stages: Stage-16
  Stage-17 depends on stages: Stage-5

STAGE PLANS:
  Stage: Stage-6
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: input
            Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: time (type: bigint), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint)
                mode: hash
                outputColumnNames: _col0, _col1, _col2
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                Reduce Output Operator
                  key expressions: _col0 (type: bigint)
                  sort order: +
                  Map-reduce partition columns: _col0 (type: bigint)
                  Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                  value expressions: _col1 (type: bigint), _col2 (type: bigint)
            Select Operator
              expressions: time (type: bigint), prov (type: string), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, prov, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint), prov (type: string)
                mode: hash
                outputColumnNames: _col0, _col1, _col2, _col3
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
            Select Operator
              expressions: time (type: bigint), prov (type: string), uid (type: int), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, prov, uid, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint), prov (type: string), uid (type: int)
                mode: hash
                outputColumnNames: _col0, _col1, _col2, _col3, _col4
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
            Select Operator
              expressions: time (type: bigint), bucket (type: string), prov (type: string), uid (type: int), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, bucket, prov, uid, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint), bucket (type: string), prov (type: string), uid (type: int)
                mode: hash
                outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
            Select Operator
              expressions: time (type: bigint), bucket (type: string), domain (type: string), prov (type: string), uid (type: int), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, bucket, domain, prov, uid, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint), bucket (type: string), domain (type: string), prov (type: string), uid (type: int)
                mode: hash
                outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
            Select Operator
              expressions: time (type: bigint), bucket (type: string), city (type: string), domain (type: string), prov (type: string), uid (type: int), flow (type: bigint), hits (type: bigint)
              outputColumnNames: time, bucket, city, domain, prov, uid, flow, hits
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              Group By Operator
                aggregations: sum(flow), sum(hits)
                keys: (UDFToLong((time / 300000)) * 300000) (type: bigint), bucket (type: string), city (type: string), domain (type: string), prov (type: string), uid (type: int)
                mode: hash
                outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
                Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: bigint), _col2 (type: bigint)
            outputColumnNames: _col0, _col1, _col2
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.5min

  Stage: Stage-0
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.5min

  Stage: Stage-7
    Stats-Aggr Operator

  Stage: Stage-8
    Map Reduce
      Map Operator Tree:
          TableScan
            Reduce Output Operator
              key expressions: _col0 (type: bigint), _col1 (type: string)
              sort order: ++
              Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string)
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              value expressions: _col2 (type: bigint), _col3 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint), KEY._col1 (type: string)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: bigint), _col3 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.prov_5min

  Stage: Stage-1
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.prov_5min

  Stage: Stage-9
    Stats-Aggr Operator

  Stage: Stage-10
    Map Reduce
      Map Operator Tree:
          TableScan
            Reduce Output Operator
              key expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: int)
              sort order: +++
              Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string), _col2 (type: int)
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              value expressions: _col3 (type: bigint), _col4 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint), KEY._col1 (type: string), KEY._col2 (type: int)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3, _col4
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: int), _col3 (type: bigint), _col4 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3, _col4
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.prov_uid_5min

  Stage: Stage-2
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.prov_uid_5min

  Stage: Stage-11
    Stats-Aggr Operator

  Stage: Stage-12
    Map Reduce
      Map Operator Tree:
          TableScan
            Reduce Output Operator
              key expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: int)
              sort order: ++++
              Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: int)
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              value expressions: _col4 (type: bigint), _col5 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: int)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: int), _col4 (type: bigint), _col5 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.bucket_prov_uid_5min

  Stage: Stage-3
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.bucket_prov_uid_5min

  Stage: Stage-13
    Stats-Aggr Operator

  Stage: Stage-14
    Map Reduce
      Map Operator Tree:
          TableScan
            Reduce Output Operator
              key expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: int)
              sort order: +++++
              Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: int)
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              value expressions: _col5 (type: bigint), _col6 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string), KEY._col4 (type: int)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: int), _col5 (type: bigint), _col6 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.bucket_domain_prov_uid_5min

  Stage: Stage-4
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.bucket_domain_prov_uid_5min

  Stage: Stage-15
    Stats-Aggr Operator

  Stage: Stage-16
    Map Reduce
      Map Operator Tree:
          TableScan
            Reduce Output Operator
              key expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int)
              sort order: ++++++
              Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int)
              Statistics: Num rows: 248368802 Data size: 106301849426 Basic stats: COMPLETE Column stats: NONE
              value expressions: _col6 (type: bigint), _col7 (type: bigint)
      Reduce Operator Tree:
        Group By Operator
          aggregations: sum(VALUE._col0), sum(VALUE._col1)
          keys: KEY._col0 (type: bigint), KEY._col1 (type: string), KEY._col2 (type: string), KEY._col3 (type: string), KEY._col4 (type: string), KEY._col5 (type: int)
          mode: mergepartial
          outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
          Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: _col0 (type: bigint), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int), _col6 (type: bigint), _col7 (type: bigint)
            outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
            Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
            File Output Operator
              compressed: false
              Statistics: Num rows: 124184401 Data size: 53150924713 Basic stats: COMPLETE Column stats: NONE
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: domain_areav1.bucket_city_domain_prov_uid_5min

  Stage: Stage-5
    Move Operator
      tables:
          partition:
            day 20151130
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: domain_areav1.bucket_city_domain_prov_uid_5min

  Stage: Stage-17
    Stats-Aggr Operator

从上面可以看到Stage-6 is a root stage。Stage-6是第一个需要完成的job，然而问题就出现在这里。GC overhead limit exceeded ！！！
从失败的jobhistory里可以看到失败是发生在map阶段。

...
map = 99%,  reduce = 33%, Cumulative CPU 9676.12 sec
map = 100%,  reduce = 100%, Cumulative CPU 9686.12 sec

也就是发现在map阶段。我们先看看错误堆栈：

-12-01 18:21:02,424 INFO [communication thread] org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at sun.nio.cs.StreamDecoder.(StreamDecoder.java:250)
    at sun.nio.cs.StreamDecoder.(StreamDecoder.java:230)
    at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:69)
    at java.io.InputStreamReader.(InputStreamReader.java:74)
    at java.io.FileReader.(FileReader.java:72)
    at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:381)
    at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsBasedProcessTree.java:162)
    at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:839)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:978)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:77)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:727)
    at java.lang.Thread.run(Thread.java:745)

map阶段OutOfMemoryError: GC overhead limit exceeded。

问题分析

OMM的通用原因大家都知道。加内存嘛！呵呵，咱不是土豪，而且OMM的原因加内存不一定能解决，还是找找内因。那么解决OMM该怎么做呢？首先我们得清楚OMM的原因的可能。1. 内存确实不够程序使用。2. 程序存在内存泄露或者程序的不够高效。作为立志成为资深程序猿的人应该从第二个入手。好，我们先分析分析：

hive程序运行环境 ：

系统46台 Ubuntu12.04， 8核心，32G Mem。Hadoop版本2.2.0 ，hive 0.12。数据100G+ Text。使用的队列最大大概是总体的40%。上述hive程序启动map数380左右，reduce数120左右。按理说这样的数量应该不算大。但问题是它确实OMM了。应为使用的时hive程序，不是自己写的。应该不大可能存在内存泄露的代码。那么应该是hive sql不合理，首先想到的是Multi Insert的效率问题。测试：分别跑单个Insert语句，即删掉一些Insert语句。
实例代码如下：


FROM qbox_bi_gold.domain_info INPUT
                      INSERT OVERWRITE TABLE 5min PARTITION (day="20151130")
                            SELECT cast(time/300000 as bigint)*300000 AS time  , SUM(flow) AS flow,SUM(hits) AS hits
                            WHERE day="20151130"
                            GROUP BY cast(time/300000 as bigint)*300000

FROM qbox_bi_gold.domain_info INPUT
           INSERT OVERWRITE TABLE prov_5min PARTITION (day="20151130")
                 SELECT cast(time/300000 as bigint)*300000 AS time ,prov, SUM(flow) AS flow,SUM(hits) AS hits
                 WHERE day="20151130"
                 GROUP BY cast(time/300000 as bigint)*300000 ,prov

FROM qbox_bi_gold.domain_info INPUT
         INSERT OVERWRITE TABLE prov_uid_5min PARTITION (day="20151130")
                            SELECT cast(time/300000 as bigint)*300000 AS time ,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                            WHERE day="20151130"
                            GROUP BY cast(time/300000 as bigint)*300000 ,prov,uid

FROM qbox_bi_gold.domain_info INPUT
          INSERT OVERWRITE TABLE bucket_prov_uid_5min PARTITION (day="20151130")
                            SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                            WHERE day="20151130"
                            GROUP BY cast(time/300000 as bigint)*300000 ,bucket,prov,uid

FROM qbox_bi_gold.domain_info INPUT
          INSERT OVERWRITE TABLE bucket_domain_prov_uid_5min PARTITION (day="20151130")
                            SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,domain,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                            WHERE day="20151130"
                            GROUP BY cast(time/300000 as bigint)*300000 ,bucket,domain,prov,uid
FROM qbox_bi_gold.domain_info INPUT
          INSERT OVERWRITE TABLE bucket_city_domain_prov_uid_5min PARTITION (day="20151130")
                            SELECT cast(time/300000 as bigint)*300000 AS time ,bucket,city,domain,prov,uid, SUM(flow) AS flow,SUM(hits) AS hits
                            WHERE day="20151130"
                            GROUP BY cast(time/300000 as bigint)*300000 ,bucket,city,domain,prov,uid

结果都是能够跑出来的。也就是说Multi Insert是比较耗费内存导致OMM，并不是sql程序的问题。那么最大的原因是我们给程序（mapreduce）的内存过小。那么先看下我们到底配置了多大的内存。
在hive cli里执行下面命令：

hive> set mapreduce.map.java.opts;
mapreduce.map.java.opts=-Xmx1500m

hive> set mapreduce.reduce.java.opts;
mapreduce.reduce.java.opts=-Xmx2048m

hive> set mapreduce.map.memory.mb;
mapreduce.map.memory.mb=2048

hive> set mapreduce.reduce.memory.mb;
mapreduce.reduce.memory.mb=3072

我们的程序问题出现在map阶段OMM，所以应该是map的内存设置小了（mapreduce.map.java.opts=1.5g）。也是设置大点，但是不能操作map允许的最大值 mapreduce.map.memory.mb（这里为2g）。

总结：

对于内存问题导致的OMM我们需要从两点入手：

程序是否有内存泄露
内存是否确实设置过小

对于第一个首先排查程序问题。在上面案例中我们使用了Multi Insert导致内存不够Gc。这里你就会问了什么是GC overhead limit exceeded 而不是Java heap space？

GC overhead limit exceeded的解释： 
一、异常描述： 
Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded

二、解释： 
JDK6新增错误类型。当GC为释放很小空间占用大量时间时抛出。 一般是因为堆太小。 
导致异常的原因：没有足够的内存。

三、解决方案： 
1、查看系统是否有使用大内存的代码或死循环。 
2、可以添加JVM的启动参数来限制使用内存：-XX:-UseGCOverheadLimit

所以对于本案例来说我的优化如下：

set mapreduce.map.java.opts=-Xmx1800m -XX:-UseGCOverheadLimit
收录于
Hive

当前文章被收录于：

Hive
19 篇文章 0 人学习
订阅专栏 查看详情
标签
#hive
Hadoop运行离线任务，Hive跑MR任务时候，出现Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 一、首先抛出异常问题 二、引发问题原因： 表连接，左表数据100W，右表数据 300W，进行左右表进行表关联，进行Count进行数据统计，等待运行结果！！！ 分析问题，查看上述运行流程，即可了解上述异常，查看详细日志信息，URL：即可见到首页出现的问题 三、查看页面可见，有如下图片 四、由上图和图一可知，此MR程序为 Map端出错，出错程序问题问 内存不足，初步怀疑为 Ma... 阅读详情

相关推荐

HIVE SQL运行内存溢出OOM（MR/TEZ）：java.lang.OutOfMemoryError: GC overhead limit exceeded

【现象】 今天用户反馈运行HQL作业失败，报OOM，于是来排查该问题 查看tez，intialmap大量失败，重试失败。 进入具体一个task，报错指向OOM，溢出原因是在单个map中产生大量对象导致。 HQL语句查询借据号——逾期金额信息，借据号重复率低，导致map维护key（借据号）-value（逾期金额信息）的内存超过了JVM配置的map进程内存大小限制，从而内存溢出。 【原因】 对于group by语句，在MR中，当数据量很大，且没有map聚合时，所有的记录将会被..

wangkuangood3200的专栏 4559

hive报错Error: GC overhead limit exceeded

事故缘由 根据yarn上资源使用情况,发现集群的内存,cpu,核数都正常,但是数据导入任务执行 88%后,就出现 GC overhead limit exceeded , 根据生成的执行日志,发现数据导入只有map 任务,数据量级别在几千万的数据量 设置jvm参数 set mapred.child.java.opts=-Xmx8000m; set mapreduce.map.java.opts=-Xmx8096m; set mapreduce.reduce.java.opts=-Xmx8096m; set

张不帅 3709

Keil MDK安装NRF52832开发环境全攻略（附系统时间错误解决方案）

本文详细介绍了如何使用Keil MDK搭建NRF52832开发环境，包括基础软件安装、项目配置关键步骤以及常见问题解决方案。特别针对系统时间错误导致的编译失败问题提供了实用解决方法，帮助开发者快速上手NRF52832开发。

weixin_29168153的博客 505

hive sql性能调优参数设置

set hive.map.aggr=true; set hive.optimize.skewjoin=true; set hive.groupby.skewindata=true; set hive.optimize.skewjoin=true; set hive.map.aggr=true; set mapred.child.java.opts=-Xmx8000m; set mapreduce...

飞翔的牛 1888

多模态数据集对比：MicroLens、PixelRec和Amazon在电商推荐中的表现差异

本文深入对比了三大主流多模态数据集MicroLens、PixelRec和Amazon在电商推荐中的表现差异。通过实际测试数据，分析了这些数据集在冷启动处理、模态融合、稀疏性挑战等关键场景下的真实表现，为算法工程师和研究员提供实用指南。特别关注多模态数据集的选择对推荐系统效果的影响，帮助开发者优化模型性能。

weixin_28717939的博客 418

Vue3集成百度实时语音识别

Vue3集成百度实时语音识别

weixin_61769998的博客 548

hive任务报Error: GC overhead limit exceeded

hive的调度任务突然报Error: GC overhead limit exceeded。-XX:+UseConcMarkSweepGC：启用并发标记清除垃圾回收器。-Xmx：设置JVM的最大堆大小。1. 改写SQL代码 ，不在讨论范围。2.优化Java虚拟机。

weixin_48336761的博客 1381

ANSYS Fluent热边界条件避坑指南：为什么你的自然对流计算结果总出错？

本文深入剖析了在ANSYS Fluent中进行自然对流仿真时，导致计算结果出错或发散的核心原因与解决方案。重点解析了重力项与操作密度设置、Boussinesq与不可压缩理想气体模型的抉择、以及压力离散格式（推荐Body Force Weighted）这三大关键雷区，并结合电子机箱散热等实战案例，提供了一套稳定可靠的仿真设置流程与调试技巧。

weixin_28396629的博客 741

hive Error: GC overhead limit exceeded

+ hive -e 'use trainschain;insert overwrite table dwd_acct_settlecore_clear_mer_detail_h_d partition (dp,end_date) select id,mer_clear_no,merchant_no,account_evidence_no,product,transaction_no,paymen...

柔情岁月的博客 4552

华为OD机考双机位C卷 - 打印机队列 （Java & Python& JS & C/C++ & GO ）

华为OD机考双机位C卷 - 打印机队列

算法大师！ 4619

Python 爬虫实战：爬取国家统计局公开数据

国家统计局作为权威的宏观经济数据发布机构，汇聚了人口、GDP、就业、产业经济等多维度的核心统计数据，是开展经济研究、政策分析、行业洞察的重要数据支撑。相较于商业数据平台，国家统计局的公开数据具备权威性、全面性和时效性等特点，但部分数据分散在不同栏目且缺乏结构化下载渠道。本文基于 Python 爬虫技术，系统讲解国家统计局公开数据的爬取方法，重点解决静态表格解析、分页数据抓取、数据结构化转换等问题，同时严格遵循政府网站数据使用规范，为科研、教学等场景提供合规的数据获取方案。本文以国家统计局官网（

2503_91057718的博客 1818

Hive 动态分区向表插入数据报错 GC overhead limit exceeded （分区过多场景）

hive 报错 动态分区 数据量很小，但是分区比较多的情况可以适用下面参数–启用自动排序参数，强制产生reduce任务。

weixin_43864336的博客 1702

第一部分：VTK基础入门（第1章：VTK简介与概述）

VTK简介与核心特性概述 VTK（Visualization Toolkit）是一个开源的跨平台C++类库，专用于3D计算机图形学、图像处理和科学可视化。由Kitware公司开发，VTK自1993年起源以来，经历了开源化、成熟化和现代化多个发展阶段，现已成为科学可视化的核心工具之一。 VTK的核心特性包括： 管道架构：模块化设计实现数据流管理 丰富的数据支持：处理多边形数据、图像数据等 强大的渲染能力：集成OpenGL，支持多种渲染技术 最新版本VTK 9.4.2增强了现代C++支持，优化了内存管理，并提升

仰望星空的博客 2360

解决 Hive 日志报错 OOM GC 堆内存问题：java.lang.OutOfMemoryError: GC overhead limit exceeded

解决 Hive 日志报错 OOM GC 堆内存问题：java.lang.OutOfMemoryError: GC overhead limit exceeded

想练武，就得下功夫 1202

实战交易策略 篇二十二：情绪流龙头交易策略

在股票市场中，机构主导趋势，游资驱动热点，散户易被情绪裹挟，形成“金字塔”式生态链，并存在情绪传递、强者恒强、物极必反的规律特点。股市是资金驱动的情绪博弈场，价格波动由群体心理、资金博弈和预期差共同驱动，核心是资本逐利性与人性贪婪恐惧的交织，兼具资源配置、财富再分配与风险定价功能。涨停加速与逃逸缺口（主升强化）凹字板/平台突破（趋势延续）天量弱转强（分歧买点）不参与首波（二波策略）情绪周期（道术合一）主线思维（聚焦核心）开盘抢筹（竞价博弈）一字助攻（情绪强化）分批进出（滚动操作）

生活在别处 1499

Data-Structures-and-Algorithm-Analysis-in-Java.pdf

作者: Weiss, Mark A. 出版社: Addison Wesley Data Structures and Algorithm Analysis in Java is an “advanced algorithms” book that fits between traditional CS2 and Algorithms Analysis courses. In the old ACM Curriculum Guidelines, this course was known as CS7. This text is for readers who want to learn good programming and algorithm analysis skills simultaneously so that they can develop such programs with the maximum amo...

Hive解决数据倾斜问题（GC overhead limit exceeded）的有效办法。

原因分析：在执行hive shell的时候map任务阶段执行到99%，而reduce节点只执行到33%人后就出现了上面的错误。 自我解释： （1）上面的情况说明，在map阶段执行结束的时候，当垃圾回收器在回收map阶段所产生的对象，因为数据倾斜的原因所导致对象过大，所以不能...

1750

Statistics and Machine Learning Toolbox

Statistics and Machine Learning Toolbox（带书签），MATLAB统计学习工具

Hive使用动态分区向表插入数据报错java heap space或者GC overhead limit exceeded

问题描述 今天使用hive动台分区向表插入数据时，一直报错java heap space或者GC overhead limit exceeded。排查了好久，发现以下问题。 1.表使用的是parquet存储格式（或者ORC）。 2.虽然数据量很小，但是分区比较多。 hive在进行多分区操作时，只启动map端任务，不启动reduce，再加上parquet格式的表会为每一个分区创建一个FileWriter，分区太多就会导致内存溢出。 解决方案 1.首先确定开启了动态分区和非严格模式。 set hive.exec

FlatTiger的博客 3261

大中型超市收银系统(可直接运行)

光步前台收银系统系统功能：商品销售、商品退货、挂单、收银员权限、会员销售、单品取消、整单取消、商品查询、脱机销售、脱机数据上传，基本资料下载，开钱箱、小票打印、收款结算，支持多币种付款方式（现金、储值卡、礼券、微信等）,支持多种促销。运行环境：.net 4.0登陆信息用户名:1001 密码:123456内置商品信息，商品编码可输入00001000 - 00010000,测试效果相关功能键F5 结算 F9挂单 F11商品查询 F8退货 Insert单品取消 Del整单取消 F12开钱箱F2 会员 Ctrl+Z退出系统 Home 锁屏 相关源码待公开

win10 跳过U8环境检测的IE WebControls.rar

win10 安装u8的时候 提示IE WebControls 不通过，这个工具可以欺骗u8的检查。 跳过U8环境检测的IE WebControls。提示导入失败后，重新检测通过。

多路输出（MULTI INSERT）

MaxCompute SQL支持在一个语句中插入不同的结果表或者分区实现多路输出。 命令格式如下： FROM from_statement INSERT OVERWRITE | INTO TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...)] select_statement1 [FROM from_statement]...

u011500419的专栏 3518

hive：记录一次因 inner join引起的GC overhead limit exceeded

文章目录一、问题现象二、分析思路三、排查验证1、排查范围2、验证数据质量问题3、猜想验证四、得出结论五、未完待续 一、问题现象 1、多表关联，其中主表与从表1为：inner join 关系； 2、该 SQL 语句在 MySQL 上能正常执行； 3、在Hive上执行时，则会出现GC overhead limit exceeded 导致无法正常执行； SELECT 字段1 ,字段2 ... ,字段N FROM -- 511298 INNER JO

qq_38294275的博客 745

Hive union all 与 multi insert

Hive union all 与 multi insert执行测试   A  insert overwrite table temp_test  select accountbin,accounttype from s_tbl_cardbininfo where accounttype = 'C' union all select accountbin,accounttype from s_t...

wflh323的专栏 2105

Hive的Multi-Inser

对于同一份数据导入两张不同的表，一般方法 insert overwrite table info1 select * from info where sex='male'; insert overwrite table info2 select * from info where sex='female'; 上述方法会对同一张表全表扫描两次，而使用下面的方法，只需要扫描一次即可将数据导入两张不用的表，结果和上述方法一致 from info insert overwrite table info

LSB19930706的博客 980

HIVE 内存溢出常见问题

MapReduce作业运行过程中内存溢出错误分类 1、Mapper/Reducer阶段JVM内存溢出（一般都是堆） 1)JVM堆(Heap)内存溢出：堆内存不足时，一般会抛出如下异常： 第一种：“java.lang.OutOfMemoryError:” GC overhead limit exceeded； 第二种：“Error: Java heapspace”异常信息； 第三种：“ru...

yisun123456的博客 3万+

Hive 优化之 union all 与 multi insert

对同一张表的union all 要比多重insert快的多， 原因是hive本身对这种union all做过优化，即只扫描一次源表；而多重insert也只扫描一次，但应为要insert到多个分区，所以做了很多其他的事情，导致消耗的时间非常长； 希望大家在开发的时候多测，多试！ lxw_test3 12亿左右记录数 Union all : 耗时7分钟左右 USE dm...

LouisLee 的博客 3663

Hive优化

优化手段 合理控制Map和Reduce数 合并小文件 避免数据倾斜，解决数据倾斜 减少job数（合并Job、大Job分拆……）   一、  Map数和Reduce数 Hive官方： https://cwiki.apache.org/confluence/display/Hive/Home   1.1、Map数 Map数过大     Map阶段输出文件太小，产生大量小文件 ...

qq_36864672的博客 495

MaxCompute SQL中insert语句使用详情

MaxCompute insert语句使用 本文介绍使用INSERT OVERWRITE和INSERT INTO两种命令更新表数据，主要内容包括： insert into table ... values ...语句 insert into/overwrite ... select ...语句 Insert多路输出（MULTI INSERT） 输出到动态分区（DYNAMIC PARTIT...

Blossom 8997
上一篇： union和union all区别
下一篇： map和reduce数量生成
点赞 0
评论 0
书签 书签 2
yjgithub
yjgithub
博客等级 码龄11年 53粉丝 105原创
+ 关注 专栏目录 订阅专栏
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值