org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 K

最新推荐文章于 2023-06-27 22:50:35 发布
原创 最新推荐文章于 2023-06-27 22:50:35 发布 · 1.7k 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#spark #hadoop #java

在Spark SQL中遇到一个异常：GeneratedClass$SpecificUnsafeProjection由于表达式长度过长导致编译错误。该问题发生在使用多个自定义UDF并进行嵌套时。为了解决这个问题，可以通过缩短UDF的表达式参数长度来避免编译报错。

sparksql Exception :GeneratedClass$SpecificUnsafeProjection

异常

2020-03-30 15:18:47,161 WARN TaskSetManager Lost task 13.0 in stage 3.0 (TID 35, hdp-dn12.cluster01.xxxxx, executor 2): 
java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: 
Code of method "apply_341$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;
Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB
  1. 使用了多个自定义UDF，多个UDF嵌套，表达式长度很长
  2. Spark优化生成的代码超过64KB 导致编译错误，一直刷新调试异常
  3. 解决：减少UDF的表达式参数长度，直到编译不报错
org.apache.doris.spark.exception.DorisException: Spark type is DATEV2, but arrow type is DATEDAY
SunnyRivers
08-08 557
例如，在不同的上下文中，日期的格式可能不同（如“YYYY-MM-DD”、“DD/MM/YYYY”等），这可能导致解析错误或比较时的问题。类型来存储日期信息是一种常见的做法，尤其是在需要存储非标准日期格式或需要保留额外信息（如时区信息）的情况下。date和datetime类型可以全部用varchar替代，但要从根本解决问题，依然得用第一种方式，因为date类型本身就是常用的日期类型。类型的长度是固定的，即使存储较短的日期字符串，也会占用预先定义的空间。因为字符串索引的查找效率通常低于日期类型的索引。
sparksql执行报错grows beyond 64 KB
weixin_43015677的博客
03-29 617
报错内容Code of method “apply_2(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClassSpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst
GeneratedClass$GeneratedIteratorForCodegenStage1“ grows beyond 64 KB
qq_65061974的博客
06-27 437
如果列"industry"的值满足某个条件，则替换为指定的新值；在上述示例中，使用表达式CASE WHEN industry = 'Energy' THEN '能源' ELSE industry END将"industry"列中所有值为"Energy"的行替换为"能源"。出现这个错误的原因通常是在使用Spark SQL进行复杂计算时，生成的代码量太大，超过了Janino编译器的限制。这是由于数据量过大、复杂的逻辑或者有大量列引起的。这个错误是由于编译器生成的代码超出了Janino编译器的限制导致的。
Spark报错：grows beyond 64 KB
小白白的博客
07-22 3788
（作者：陈玓玏 data-master） spark脚本执行中，碰到错误： 2020-06-24 16:59:53,246 [Thread-44] ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of meth
Spark源代码编译生成错误解决方案
Snowball
09-14 2625
最近项目需要对Spark进行二次开发，在对Spark二次开发完成之后，需要对其进行打包成exe文件的操作。互联网真是一个宝贵的知识库，千辛万苦之下找到了Spark开发记录(windows环境)和install4j打包Spark详解，前者介绍了两种版本spark的打包方式，而后者的打包方式适用于较早的2.5.x版本的Spark，对于2.6.x版本的Spark可以按照第一篇文章介绍的内容进行打包操作。但是按照操作进行下去之后发现项目无法ant打包成功。错误代码如下：
FlinkSQL:source端字段太多导致——64KB问题
周佳豪的博客
09-08 1988
FlinkSQL:source端字段太多导致"64KB 问题"
Caused by: java.lang.NullPointerException at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.mapelements_doConsume_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.deserializetoobject_doConsume_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:645) at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.next(InMemoryRelation.scala:93) at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.next(InMemoryRelati
08-30
import org.apache.spark.sql.functions._ val result = data.select( (0 until data.columns.length).map(i => coalesce(col(data.columns(i)).cast("string"), lit("")).as(data.columns(i)) ): _* ) ``` ##...
Caused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (array<double>, array<double>) => double) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Round1$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.CaseWhen$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.subExpr$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:217) at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:279) at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:250) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException
12-31
首先，用户的问题是关于解决Spark中的错误："org.apache.spark.SparkException: Failed to execute user defined function" 由于 "java.lang.NullPointerException"。用户提供了几个引用，但这些引用主要涉及类找不...
使用scala编写spark任务，计算余弦相似度和欧式距离报错 2025-12-30 19:43:11,060 | WARN | [task-result-getter-1] | Lost task 0.0 in stage 345.0 (TID 37861) (node-group-9Ddam0002 executor 2): java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException: Null value appeared in non-nullable field: - array element class: "scala.Double" - root class: "scala.collection.Seq" If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int). mapobjects(lambdavariable(MapObject, DoubleType, true, -1), assertnotnull(lambdavariable(MapObject, DoubleType, true, -1)), input[0, array<double>, true], Some(interface scala.collection.Seq)) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:186) at org.apache.spark.sql.catalyst.expressions.ScalaUDF.$anonfun$scalaConverter$2(ScalaUDF.scala:159) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.ScalaUDF_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.subExpr_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:209) at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:343) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:897) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:897) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:516) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1606) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:519) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field: - array element class: "scala.Double" - root class: "scala.collection.Seq" If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int). at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.MapObjects_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:182) ... 26 more
最新发布
12-31
import org.apache.spark.sql.DataFrame // 假设df是原始DataFrame df.printSchema() // 查看schema，注意非可空字段 // 删除任何包含空值的行（或者只删除关键列的空值） val cleanedDF = df.na.drop() // 或者...
使用scala编写spark任务报错 Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field: - array element class: "scala.Double" - root class: "scala.collection.Seq" If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int). at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.MapObjects_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:182) ... 26 more
12-31
import org.apache.spark.sql.types._ val arrayType = ArrayType(DoubleType, containsNull = false) val schema = StructType(Seq( StructField("COLLECT_ID", StringType, nullable = false), StructField(...
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write spark查询oracle异常
u010791030的博客
05-07 1647
本来oracle字段是VARCHAR2，然后查询的时候又带了个to_char(xx)导致下面错误 2019/05/07 15:23:03.198 +0800 WARN [TaskSetManager] [Azkaban] Lost task 0.0 in stage 1.0 (TID 5, localhost, executor driver): java.lang.RuntimeExce...
spark sql 中 java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.Gener
zhanghytc的博客
06-23 8704
最近在做推荐系统的项目，使用了spark 2.0，需要给每个userid 推荐出一个list ，使用了udf函数 udf ()在每行执行的时候 得到 org.apache.spark.SparkException: Failed to execute user defined function($anonfun$calcMaxSimilarity$2$1: (string, array) =>
org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to scala.Tuple2
热门推荐
乄浅醉的专栏
09-07 1万+
今天在写spark 提数的时候遇到一个异常，如下Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to scala.Tuple2 at $anonfun$1$$anonfun$apply$1.apply(<co
SparkSQL代码生成 - 基本表达式代码生成源码解析
qq_41775852的博客
04-12 1925
文章目录代码生成JavaCode和BlockExprValueExprCodeCodegenContextmutableStatespartitionlnitializationStatementsreferencesaddedFunctions其他方法CodeGeneratorgenerate表达式代码生成实例GeneratePredicatebind（GeneratePredicate）cre...
structured streaming java.lang.NullPointerException
sunyang098的博客
08-09 629
问题描述：structured streaming任务报空指针，但报的不是自己的代码，是spark框架的代码，报错日志如下 Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 39, 10.2.6.21, executor .
Sparkthrift-sql执行报错-File does not exist: hdfs://xxx/t_bd_materialgroup/xxx.parquet
weixin_42151880的博客
11-26 2125
spark
Spark数据格式UnsafeRow
1032851561的博客
01-14 8824
1. 简介 UnsafeRow是InternalRow的子类，它表示一个可变的基于原始内存(raw-memory)的二进制行格式，简单来说UnsafeRow代表一行记录，用于替代java对象（属于Tungsten计划的一部分，可以减少内存使用以及GC开销） InternalRow：spark sql内部使用的表示行的抽象类，对应表示输出的行有org.apache.spark.sql.Row/GenericRow/GenericRowWithSchema 2. 类属性 private Object base
Spark sql 中 自定义实体类无法识别其中的 Date 字段
H.king的博客
09-05 903
在使用过程中 创建了实体类 使用反射的方式将 rdd 转换为 df 但是在使用过程中 即使没有在 查询中使用该字段，依然会报错 主要提示为 `scala.MatchError: Thu Sep 05 10:16:25 CST 2019 (of class java.util.Date) ` 具体内容如下 19/09/05 10:16:30 ERROR Executor: ...
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值