Spark ML 正则化标准化归一化 ---- spark 中的归一化

本文链接：https://blog.youkuaiyun.com/wangyaninglm/article/details/119975312

本文详细介绍了Spark中用于数据预处理的归一化方法，包括MaxAbsScaler和MinMaxScaler。MaxAbsScaler将每个特征的绝对最大值作为缩放因子，而MinMaxScaler则将每个特征缩放到指定范围。文章通过代码示例展示了如何使用这两个方法，并讨论了处理单值情况的方法。同时，提供了相关参考文献和链接，帮助读者深入理解Spark中的特征归一化过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark 中的归一化

MaxAbsScaler

http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/feature/MaxAbsScaler.html

MinMaxScaler

http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/feature/MinMaxScaler.html

Rescale each feature individually to a common range [min, max] linearly using column summary
statistics, which is also known as min-max normalization or Rescaling. The rescaled value for
feature E is calculated as:
$Rescaled(e_i) = \frac{e_i - E_{min}}{E_{max} - E_{min}} * (max - min) + min$

For the case (E_{max} == E_{min}), (Rescaled(e_i) = 0.5 * (max + min)).
note ：

Since zero values will probably be transformed to non-zero values, output of the
transformer will be DenseVector even for sparse input.

核心代码：主要就是计算最大最小值

override def fit(dataset: Dataset[_]): MinMaxScalerModel = {
    transformSchema(dataset.schema, logging = true)

    val Row(max: Vector, min: Vector) = dataset
      .select(Summarizer.metrics("max", "min").summary(col($(inputCol))).as("summary"))
      .select("summary.max", "summary.min")
      .first()

    copyValues(new MinMaxScalerModel(uid, min.compressed, max.compressed).setParent(this))
  }

注意：上面的计算方式，我们发现只能支持Vector的形式，那么对于但一值的情况如何转换呢？

 val temp_mean = df_num.select(functions.mean(df_num.col("features"))).collect()(0)
        println(temp_mean.getDouble(0))
       val Row(mean2: Vector) =Row(Vectors.dense(temp_mean.getDouble(0)))

   val df_num = spark.createDataFrame(Seq(
            (0, 0.5, -1.0),
            (1, 1.0, 1.0),
            (2, 10.0, 2.0),
            (3, 10.0, 0.0)
        )).toDF("id", "features","result")
        df.show()