How-to: resolve "java.io.NotSerializableException" issue during spark reading hbase table

本文介绍了一个在使用Spark Scala代码读取HBase表时遇到的非序列化异常问题,并给出了通过配置Kryo序列化器来解决此问题的方法。
During reading htable via spark scala code, the following error happened:

15/10/28 16:39:00 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 10.0 (TID 2536, slave14.dc.tj): java.lang.RuntimeException: java.io.NotSerializableException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
Serialization stack:
        - object not serializable (class: org.apache.hadoop.hbase.io.ImmutableBytesWritable, value: 30 30 5f 39 39 39 38 38 33)
        - field (class: scala.Tuple2, name: _1, type: class java.lang.Object)
        - object (class scala.Tuple2, (30 30 5f 39 39 39 38 38 33,keyval......
        at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
        at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
        at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1190)
        ......

The solution is turn spark to use KryoSerializer:
sparkconf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

Cannot resolve com.google.code.gson:gson:2.9.1 Cannot resolve org.apache.tika:tika-core:2.9.0 Cannot resolve org.springframework:spring-context-support:unknown Cannot resolve pro.fessional:kaptcha:2.3.3 Cannot resolve commons-io:commons-io:2.13.0 Cannot resolve com.baomidou:mybatis-plus-boot-starter:3.5.5 Cannot resolve io.minio:minio:8.2.2 Cannot resolve io.awspring.cloud:spring-cloud-aws-starter-sqs:3.0.0 Cannot resolve org.springframework.cloud:spring-cloud-starter-loadbalancer:unknown Cannot resolve com.fasterxml.jackson.core:jackson-databind:unknown Cannot resolve org.springframework.security:spring-security-core:5.7.12 Cannot resolve org.quartz-scheduler:quartz:unknown Cannot resolve org.xerial:sqlite-jdbc:3.42.0.0 Cannot resolve com.alibaba:transmittable-thread-local:2.14.4 Cannot resolve org.springframework.cloud:spring-cloud-loadbalancer:unknown Cannot resolve javax.servlet:javax.servlet-api:unknown Cannot resolve org.springframework.cloud:spring-cloud-starter-gateway:unknown Cannot resolve org.tukaani:xz:1.8 Cannot resolve cn.hutool:hutool-all:5.8.28 Cannot resolve software.amazon.awssdk:netty-nio-client:2.29.45 Cannot resolve org.springframework.boot:spring-boot-starter-actuator:unknown Cannot resolve com.mysql:mysql-connector-j:unknown Cannot resolve de.codecentric:spring-boot-admin-starter-server:2.7.16 Cannot resolve com.alibaba.cloud:spring-cloud-starter-alibaba-nacos-discovery:unknown Cannot resolve software.amazon.awssdk:s3-transfer-manager:2.29.45 Cannot resolve org.springframework.cloud:spring-cloud-starter-bootstrap:unknown Cannot resolve org.apache.velocity:velocity-engine-core:2.3 Cannot resolve com.alibaba.cloud:spring-cloud-alibaba-sentinel-gateway:unknown Cannot resolve org.springframework:spring-test:6.0.13 Cannot resolve org.springframework.cloud:spring-cloud-starter-bootstrap:unknown Cannot resolve com.alibaba.fastjson2:fastjson2:2.0.53 Cannot resolve software.amazon.awssdk:aws-sdk-java:2.29.45 Cannot resolve com.baomidou:dynamic-datasource-spring-boot-starter:4.3.1 Cannot resolve software.amazon.awssdk:sqs:2.29.45 Cannot resolve org.springframework.boot:spring-boot-starter-security:unknown Cannot resolve org.apache.poi:poi-ooxml:4.1.2 Cannot resolve org.lionsoul:ip2region:2.6.5 Cannot resolve org.apache.commons:commons-lang3:unknown Cannot resolve com.alibaba:druid-spring-boot-starter:1.2.23 Cannot resolve org.springframework.boot:spring-boot-starter-web:unknown Cannot resolve com.alibaba.cloud:spring-cloud-starter-alibaba-sentinel:unknown Cannot resolve org.springframework.cloud:spring-cloud-starter-openfeign:unknown Cannot resolve javax.xml.bind:jaxb-api:unknown Cannot resolve org.springframework:spring-framework-bom:5.3.39 Cannot resolve org.springframework.boot:spring-boot-starter-data-redis:unknown Cannot resolve eu.bitwalker:UserAgentUtils:1.21 Cannot resolve org.springframework.boot:spring-boot-starter-actuator:unknown Cannot resolve com.github.tobato:fastdfs-client:1.27.2 Cannot resolve com.alibaba.cloud:spring-cloud-starter-alibaba-nacos-config:unknown Cannot resolve org.springframework:spring-web:unknown Cannot resolve com.github.pagehelper:pagehelper-spring-boot-starter:2.0.0 Cannot resolve io.jsonwebtoken:jjwt:0.9.1 Cannot resolve org.springframework:spring-webmvc:unknown Cannot resolve org.apache.commons:commons-compress:1.21 Cannot resolve com.alibaba.csp:sentinel-datasource-nacos:unknown Cannot resolve com.alibaba.cloud:spring-cloud-starter-alibaba-seata:
最新发布
06-04
### 无法解析的依赖项解决方案 在Maven项目中,当某些依赖项无法解析时,通常是因为以下原因之一:依赖项版本不存在、依赖项未发布到中央仓库、本地Maven配置问题或网络连接问题。以下是针对每个依赖项的具体解决方案。 #### 1. **检查依赖项版本是否存在** 确保所使用的依赖项版本确实存在。可以通过访问[Maven Central Repository](https://search.maven.org/)来验证依赖项及其版本是否可用。例如: - `gson` 2.9.1[^1] - `tika-core` 2.9.0[^1] - `spring-context-support`[^1] - `kaptcha` 2.3.3[^1] - `commons-io` 2.13.0 - `mybatis-plus-boot-starter` 3.5.5[^1] - `minio` 8.2.2 - `spring-cloud-aws-starter-sqs` 3.0.0 如果某个版本不存在,则需要选择正确的版本号。 #### 2. **添加正确的仓库地址** 某些依赖项可能不在Maven中央仓库中,而是托管在其他仓库(如阿里巴巴云仓库、Spring官方仓库等)。需要在`pom.xml`中添加相应的仓库地址。例如: ```xml <repositories> <repository> <id>alibaba</id> <url>https://maven.aliyun.com/repository/public</url> </repository> <repository> <id>spring-milestones</id> <url>https://repo.spring.io/milestone</url> </repository> </repositories> ``` #### 3. **解决特定依赖项问题** 以下是一些常见依赖项的问题及解决方案: - **`spring-security-core` 5.7.12** 确保与`spring-boot-starter-parent`版本兼容。如果版本冲突,可以使用`dependencyManagement`显式声明版本[^3]。 - **`spring-cloud-starter-alibaba-nacos-discovery`** 需要添加阿里云的Maven仓库,并确保版本与`spring-cloud-alibaba`的BOM兼容[^4]。 - **`spring-cloud-starter-alibaba-sentinel`** 同样需要添加阿里云仓库,并确保版本与`spring-cloud-alibaba`的BOM兼容[^4]。 - **`spring-cloud-starter-openfeign`** 确保与`spring-cloud-dependencies`版本兼容。如果版本冲突,可以使用`dependencyManagement`显式声明版本[^3]。 - **`dynamic-datasource-spring-boot-starter` 4.3.1** 需要添加阿里云仓库,并确保版本与`spring-boot-starter-parent`兼容。 - **`aws-sdk-java` 2.29.45** 如果无法解析,可能是由于网络问题或仓库配置不正确。建议添加AWS官方仓库[^4]。 #### 4. **检查本地Maven配置** 确保本地`settings.xml`文件中没有错误配置。例如,代理设置可能导致某些依赖项无法下载。可以通过以下命令检查Maven配置: ```bash mvn help:effective-settings ``` #### 5. **清理并重新构建项目** 有时缓存问题会导致依赖项无法解析。可以尝试以下命令清理并重新构建项目: ```bash mvn clean install -U ``` #### 6. **代码示例:完整的`pom.xml`片段** 以下是一个完整的`pom.xml`片段,展示了如何正确配置依赖项和仓库: ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.3</version> </parent> <dependencies> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.9.1</version> </dependency> <dependency> <groupId>org.apache.tika</groupId> <artifactId>tika-core</artifactId> <version>2.9.0</version> </dependency> <!-- 其他依赖项 --> </dependencies> <repositories> <repository> <id>alibaba</id> <url>https://maven.aliyun.com/repository/public</url> </repository> <repository> <id>spring-milestones</id> <url>https://repo.spring.io/milestone</url> </repository> </repositories> </project> ``` ###
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值