Cannot inspect org.apache.hadoop.hive.serde2.io.DoubleWritable 可能原因_cannot inspect org.apache.hadoop.hive.serde2.io.hi-优快云博客

在尝试将Spark数据写入Hive表时遇到`java.io.IOException: HiveException: UnsupportedOperationException: Cannot inspect DoubleWritable`的问题。原因是Spark SQL结果中的double类型字段与Hive表中string类型字段不匹配。解决方案包括修改Hive表的duration字段为double类型或在Spark SQL中使用cast函数将double转换为string类型。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.hive.serde2.io.DoubleWritable

1.这是我要写入的目的表，可以看到duration字段为string类型

CREATE EXTERNAL TABLE IF NOT EXISTS dwi_m.dwi_staypoint_msk_d (
    mdn string
    ,grid_longi string 
    ,grid_lati string 
    ,grid_id string 
    ,county_id string 
    ,duration string
    ,grid_first_time string
    ,grid_last_time string 
)
PARTITIONED BY (
    day_id string
) 
ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY '\t' 
STORED AS PARQUET 
location '/daas/motl/dwi/dwi_staypoint_msk_d';

2.spark中的运行sql

SELECT mdn,longi AS grid_longi,lati AS grid_lati,grid_id,county_id,
(unix_timestamp(substring(start_time,0,14),'yyyyMMddHHmmss')-unix_timestamp(substring(start_time,16,29),'yyyyMMddHHmmss'))/60 AS duration,
SUBSTRING(start_time,16,29) AS grid_first_time,
SUBSTRING(start_time,0,14) AS grid_last_time 
FROM tablename;

3.spark 代码

    sqlContext
      .sql(sql)
      .write.mode(SaveMode.Overwrite)
      .parquet(path)

写入后在hive中读取新表的数据报错

Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.hive.serde2.io.DoubleWritable

原因很明显：sql运行的结果中duration字段是double类型的，而目的表中是string类型的。