利用spark ml 进行协同过滤推荐

最新推荐文章于 2025-07-09 17:43:31 发布

原创

最新推荐文章于 2025-07-09 17:43:31 发布 · 2.9k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#推荐 #spark #协同过滤 #spark ml

本文介绍了如何在Spark环境下搭建Spark ML并实现协同过滤推荐系统。首先讲解了Spark环境的配置，包括JDK8的安装和Spark的下载与环境变量设置。接着详细阐述了如何在Spark Shell中运行协同过滤的示例代码，讨论了输入数据格式和case class的使用。最后，分析了协同过滤算法在Spark ML中的参数，如并行度、隐因子个数、最大迭代数等，并指出其在推荐系统中的应用。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark环境搭建
协同过滤实践

本文面向spark以及入门推荐系统的新手（包括我自己）

spark环境搭建

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.3.1 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).

自己项目开发程序

Scala and Java users can include Spark in their projects using its Maven coordinates and in the future Python users can also install Spark from PyPI

本地spark

Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). It’s easy to run locally on one machine — all you need is to have java installed on your system PATH, or the JAVA_HOME environment variable pointing to a Java installation.

安装jdk8

mac java_home: https://www.mkyong.com/java/how-to-set-java_home-environment-variable-on-mac-os-x/

spark

下载spark：https://spark.apache.org/downloads.html
解压： spark-2.3.1-bin-hadoop2.7.tgz

环境变量

export JAVA_HOME=$(/usr/libexec/java_home)
export SPARK_HOME={YOUR_SPARK_HOME}
export PATH=$SPARK_HOME/bin:$PATH

测试spark demo

run-example SparkPi 10

如果设置正确，会出现以下日志：

2018-09-01 13:30:09 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-01 13:30:09 INFO  DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 0.785 s
2018-09-01 13:30:09 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 0.888303 s
Pi is roughly 3.142015142015142
2018-09-01 13:30:09 INFO  AbstractConnect