IsolationForest算法spark实现

最新推荐文章于 2025-07-08 14:16:54 发布

weixin_33778778

最新推荐文章于 2025-07-08 14:16:54 发布

阅读量915

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 scala python

原文链接：https://my.oschina.net/kyo4321/blog/2994487

本文介绍了一种基于IsolationForest算法的异常检测方法，并详细展示了如何使用Apache Spark MLlib库中的IsolationForest实现该算法。通过Wisconsin乳腺癌数据集进行实验，训练模型并评估其准确性，最终模型的AUC值达到0.931。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2019独角兽企业重金招聘Python工程师标准>>>

/*
Notice:
需要事先将IsolationForest算法源码利用mvn方式jar包，才可以使用import org.apache.spark.ml.iforest.IForest
scala源代码地址：https://github.com/titicaca/spark-iforest

python库sklearn.ensemble.IsolationForest官方文档地址：
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html
*/

import org.apache.spark.ml.feature.StringIndexer
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.iforest.IForest
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator


// Wisconsin Breast Cancer Dataset
val dataset = (spark.read.option("inferSchema", "true")
              .csv("/anomaly-detection/breastw.csv"))



// Index label values: 2 -> 0, 4 -> 1
val indexer = (new StringIndexer()
               .setInputCol("_c10")
               .setOutputCol("label"))

val assembler = (new VectorAssembler()
                    .setInputCols(dataset.columns.filter(!_.contains("label")))
                    .setOutputCol("features"))

val iForest = (new IForest()
                   .setNumTrees(100)
                   .setMaxSamples(256)
                   .setContamination(0.35)
                   .setBootstrap(false)
                   .setMaxDepth(100)
                   .setSeed(123456L))

val pipeline = new Pipeline().setStages(Array(indexer, assembler, iForest))


// let's split the dataset into a training and test dataframe
val Array(trainDF, testDF) = dataset.randomSplit(Array(0.8, 0.2),seed = 123456L)

val model = pipeline.fit(trainDF)
val predictions = model.transform(testDF)


// What was the overall accuracy of the model, using AUC
val evaluator = (new BinaryClassificationEvaluator()
   .setLabelCol("label")
   .setRawPredictionCol("prediction")
   .setMetricName("areaUnderROC"))

val auc = evaluator.evaluate(predictions)
println(s"The model's auc: $auc")

/*

scala> val auc = evaluator.evaluate(predictions)
auc: Double = 0.9311653116531164

scala> println(s"The model's auc: $auc")
The model's auc: 0.9311653116531164

*/

转载于:https://my.oschina.net/kyo4321/blog/2994487