spark sql 数据类型转换,通过类型的大小写匹配根据SparkSQL中的类型转换值

博客探讨了在Scala中使用模式匹配进行Spark SQL数据类型转换时,针对DecimalType类型转换遇到的问题。使用return语句会导致类型不匹配等错误,解决方案是避免在Scala代码中使用return,函数体最后计算的值会自动返回。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Is it possible to match a parametric type in Scala? Lets say I have a function that receives two parameters: a value and a type. I would like to use pattern matching to do a type conversion.

Something like this:

datatype match {

case IntegerType => return value.toInt

case FloatType => return value.toFloat

case StringType => return value

case DecimalType(_,_) => return BigDecimal(value) // this is not working

case _ => return strrepr

}

Here DecimalType accepts two parameters to specify precision the required precision. It can be for example:

org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

I have tried several options and nothing seems to be working:

For case DecimalType => return BigDecimal(value) I get:

error: pattern type is incompatible with expected type;

found : org.apache.spark.sql.types.DecimalType.type

required: org.apache.spark.sql.types.DataType

Note: if you intended to match against the class, try `case DecimalType(_,_)`

For case DecimalType(_,_) => return BigDecimal(value) I get:

error: result type Boolean of unapply defined in method unapply in object DecimalType does not conform to Option[_] or Boolean

For case DecimalType[_,_] => return BigDecimal(value) I get:

error: org.apache.spark.sql.types.DecimalType does not take type parameters

解决方案

The problem is the use of the return in your code. You said you use this code snippet in a function somewhere. What is the return type of that function? Obviously, you intend that sometimes it is Integer, sometimes String, sometimes BigDecimal; but if you use return, it will look to the type of the returned object to determine the return type of the function. In general, you should strongly avoid using return in Scala code. The last evaluated value in the function body is returned. The only case for using a return is when you want to force returning a value somewhere else in the function body. But still, a better way would be to save the return object in a variable and just evaluate that variable in the last line of your function body. And never use return!

Without return it works

scala> val datatype = DecimalType(10, 2)

datatype: org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

scala> val value = BigDecimal(10)

value: scala.math.BigDecimal = 10

scala> datatype match {case DecimalType(_,_) => value}

res150: scala.math.BigDecimal = 10

** Problems with return **

scala> def test = {datatype match {case DecimalType(_,_) => return value}}

:138: error: method test has return statement; needs result type

def test = {datatype match {case DecimalType(_,_) => return value}}

scala> def test:BigDecimal = {datatype match {case DecimalType(_,_) => return value}}

test: BigDecimal

scala> def test:DataType = {datatype match {case DecimalType(_,_) => return value}}

:138: error: type mismatch;

found : scala.math.BigDecimal

required: org.apache.spark.sql.types.DataType

def test:DataType = {datatype match {case DecimalType(_,_) => return value}}

scala> def test3 = {datatype match {case DecimalType(_,_) => value}}

test3: scala.math.BigDecimal

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值