bug
关注
分享
扫一扫
robles的心路
这个作者很懒，什么都没留下…
展开
-
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object
spark程序报错：java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)原因为打包时本地所用的scala版本是2.12 ，而集群中的scala版本是2.11，将本地的scala版本改成2.11后在打包运行， 运行成功...原创 2022-03-09 20:28:15 · 5211 阅读 · 0 评论 -
SparkException: Dynamic allocation of executors requires the external shuffle service解决办法
详细报错： ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled....ERROR ApplicationMaster: U...原创 2022-03-09 16:38:46 · 2540 阅读 · 1 评论 -
sqoop ERROR tool.ExportTool: Error during export: Export job failed!
sqoop导入数据报错：21/06/09 17:31:19 ERROR tool.ExportTool: Error during export:Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:442) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)...原创 2021-06-09 18:00:17 · 1060 阅读 · 0 评论 -
FAILED: SemanticException 0:0 Error creating temporary folder on: Error encountered near token
执行hive sql任务时报错：FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ShareSdkHadoop/user/hive/warehouse/dm_di_master.db. Error encountered near token 'TOK_TMP_FILE'报错原因：操作线上库，当前的测试用户没有线上库的写入权限解决办法：调用shell脚本时，在hive前加上HADOOP_USER.原创 2021-06-08 12:26:07 · 4165 阅读 · 0 评论 -
Application application_id failed 2 times due to ApplicationMaster for attempt xx timed out解决办法
spark-submit 提交任务失败，报错详情：Application application_1609129417468_1602993 failed 2 times due to ApplicationMaster for attempt appattempt_1609129417468_1602993_000002 timed out. Failing the application.排查为ApplicationMaster 为子任务尝试次数超时，于是尝试添加以下参数：--conf s.原创 2021-05-25 09:56:22 · 6743 阅读 · 2 评论 -
shell 中 报错 integer expression expected 解决办法
执行shell 报错：排查为 -ge -le 支持判断整数，不支持判断小数大小解决方法：使用bc命令比较小数大小，再用 -eq 1 判断，返回1 比较true 返回0表示false 如：echo "0.1>= 0.05" | bca=`echo "0.1 >= 0.05" | bc`if [ $a -eq 1 ];then echo trueelse echo falsefi...原创 2021-04-16 21:45:15 · 9267 阅读 · 0 评论 -
FAILED: SemanticException The abstract syntax tree is null 解决办法
hive中sql报错：FAILED: SemanticException The abstract syntax tree is null执行sql为：with t as(select 1 id,'k1' key,123 value union allselect 1 id,'k2' key,124 value union allselect 2 ,'k1' key,234 value )select id, max(case when key='k1' then ...原创 2021-04-14 12:08:41 · 1186 阅读 · 0 评论 -
Cannot create a table having a column whose name contains commas in Hive metastore. 解决办法
spark 报错：Cannot create a table having a column whose name contains commas in Hive metastore. Table: `xxx`.`xxx`; Column: get_geohash(43.834269, 87.605044, 8);解决办法：sql 中方法后写别名即可，如下：get_geohash(43.834269, 87.605044, 8) as b...原创 2021-03-26 20:13:33 · 1977 阅读 · 0 评论 -
Detected yarn cluster mode, but isn‘t running on a cluster. Deployment to YARN is not supported 解决办法
spark-submit报如下错：org.apache.spark.SparkException: Detected yarn cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.排查为 将--deploy-mode cluster 配置写入jar包时，使用spark-submit 提...原创 2021-03-16 17:43:34 · 1411 阅读 · 0 评论