MLib -Dimensionality（I）

最新推荐文章于 2022-11-13 08:47:24 发布
翻译 最新推荐文章于 2022-11-13 08:47:24 发布 · 3.3k 阅读 · 0
文章标签：

#MLlib #spark-1.0.0 #SVD #PCA

本文介绍了Apache Spark MLlib中的两种主要降维方法：奇异值分解（SVD）和主成分分析（PCA）。通过实例演示了如何使用Spark对高瘦矩阵进行SVD分解和PCA分析，提取关键特征并降低数据维度。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

MLlib-Dimensionality Reduction

* Singular value decomposition (SVD)

* Principal component analysis(PCA)

      降维可以理解为在一定情况下一个减少变量的过程。 它经常被用于从原始和含有噪声的特征中提取潜在的有用特征，
或者用在在保留结构的情况下的数据压缩。spark-1.0.0这个版本，提供了针对tall and skinny矩阵的降维。

SVD

      具体不在多做介绍，这篇文章写的非常好。需要指出的是，对于大的矩阵，通常我们不需要做全部的因子分解，只需要头几个比较大的奇异值以及对应的奇异向量。这样就比较有利于存储，而且更重要的是，可以降维，看到矩阵的低秩结构。

     在spark-1.0.0这个版本中，提供了针对tall-and-skinny矩阵，即行数多，列数少（小于1000列）的SVD分解。 这个矩阵采用用RowMatrix这种分布式存储，对它进行SVD奇异值分解。

import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.SingularValueDecomposition

val mat: RowMatrix = ...

// Compute the top 20 singular values and corresponding singular vectors.
val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(20, computeU = true)
val U: RowMatrix = svd.U // The U factor is a RowMatrix.
val s: Vector = svd.s // The singular values are stored in a local dense vector.
val V: Matrix = svd.V // The V factor is a local dense matrix.
借助 博客(II）中生成的RowMatrix, 实验SVD分解。具体每一步过程如下：

scala> val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(2, computeU = true)
14/07/11 03:26:49 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:211
14/07/11 03:26:49 INFO DAGScheduler: Got job 4 (aggregate at RowMatrix.scala:211) with 2 output partitions (allowLocal=false)
14/07/11 03:26:49 INFO DAGScheduler: Final stage: Stage 4(aggregate at RowMatrix.scala:211)
14/07/11 03:26:49 INFO DAGScheduler: Parents of final stage: List()
14/07/11 03:26:49 INFO DAGScheduler: Missing parents: List()
14/07/11 03:26:49 INFO DAGScheduler: Submitting Stage 4 (MappedRDD[18] at map at <console>:29), which has no missing parents
14/07/11 03:26:49 INFO DAGScheduler: Submitting 2 missing tasks from Stage 4 (MappedRDD[18] at map at <console>:29)
14/07/11 03:26:49 INFO TaskSchedulerImpl: Adding task set 4.0 with 2 tasks
14/07/11 03:26:49 INFO TaskSetManager: Starting task 4.0:0 as TID 6 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:26:49 INFO TaskSetManager: Serialized task 4.0:0 as 3057 bytes in 0 ms
14/07/11 03:26:49 INFO TaskSetManager: Starting task 4.0:1 as TID 7 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:26:49 INFO TaskSetManager: Serialized task 4.0:1 as 3057 bytes in 0 ms
14/07/11 03:26:49 INFO Executor: Running task ID 6
14/07/11 03:26:49 INFO Executor: Running task ID 7
14/07/11 03:26:49 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:26:49 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:26:49 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9
14/07/11 03:26:49 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9
14/07/11 03:26:49 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
14/07/11 03:26:49 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
14/07/11 03:26:49 INFO Executor: Serialized size of result for 7 is 1037
14/07/11 03:26:49 INFO Executor: Serialized size of result for 6 is 1037
14/07/11 03:26:49 INFO Executor: Sending result for 7 directly to driver
14/07/11 03:26:49 INFO Executor: Sending result for 6 directly to driver
14/07/11 03:26:49 INFO Executor: Finished task ID 7
14/07/11 03:26:49 INFO Executor: Finished task ID 6
14/07/11 03:26:49 INFO DAGScheduler: Completed ResultTask(4, 1)
14/07/11 03:26:49 INFO TaskSetManager: Finished TID 7 in 39 ms on localhost (progress: 1/2)
14/07/11 03:26:49 INFO TaskSetManager: Finished TID 6 in 42 ms on localhost (progress: 2/2)
14/07/11 03:26:49 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
14/07/11 03:26:49 INFO DAGScheduler: Completed ResultTask(4, 0)
14/07/11 03:26:49 INFO DAGScheduler: Stage 4 (aggregate at RowMatrix.scala:211) finished in 0.043 s
14/07/11 03:26:49 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:211, took 0.049573705 s
14/07/11 03:26:49 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
14/07/11 03:26:49 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
14/07/11 03:26:49 INFO MemoryStore: ensureFreeSpace(200) called with curMem=332040, maxMem=309225062
14/07/11 03:26:49 INFO MemoryStore: Block broadcast_3 stored as values to memory (estimated size 200.0 B, free 294.6 MB)
svd: org.apache.spark.mllib.linalg.SingularValueDecomposition[org.apache.spark.mllib.linalg.distributed.RowMatrix,org.apache.spark.mllib.linalg.Matrix] =
SingularValueDecomposition(org.apache.spark.mllib.linalg.distributed.RowMatrix@2be207c2,[12.014542582220963,0.7307604855863146],-0.45395279953839474  0.6207943097808744   
-0.5636972053725028   0.35545611740775257  
-0.6900524012323158   -0.6987598826071044  )

scala> val U: RowMatrix = svd.U
U: org.apache.spark.mllib.linalg.distributed.RowMatrix = org.apache.spark.mllib.linalg.distributed.RowMatrix@2be207c2

scala> val s: Vector = svd.s
s: org.apache.spark.mllib.linalg.Vector = [12.014542582220963,0.7307604855863146]

scala> val V: Matrix = svd.V
V: org.apache.spark.mllib.linalg.Matrix =
-0.45395279953839474  0.6207943097808744   
-0.5636972053725028   0.35545611740775257  
-0.6900524012323158   -0.6987598826071044  

PCA
    

           主成份分析在降维中用的非常广泛，可以理解成用于找到一个旋转使得第一个坐标尽量有最大的变量，并且依次的坐标都尽量有较大的变量。旋转矩阵的列就是主成份。主成份分析在这个spark版本中也是针对tall-and-skinny矩阵。下面是主成份分解法实验：

import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix

val mat: RowMatrix = ...

// Compute the top 10 principal components.
val pc: Matrix = mat.computePrincipalComponents(10) // Principal components are stored in a local dense matrix.

// Project the rows to the linear space spanned by the top 10 principal components.
val projected: RowMatrix = mat.multiply(pc)
 具体每一步

scala> val pc: Matrix = mat.computePrincipalComponents(1)
14/07/11 03:32:17 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:312
14/07/11 03:32:17 INFO DAGScheduler: Got job 5 (aggregate at RowMatrix.scala:312) with 2 output partitions (allowLocal=false)
14/07/11 03:32:17 INFO DAGScheduler: Final stage: Stage 5(aggregate at RowMatrix.scala:312)
14/07/11 03:32:17 INFO DAGScheduler: Parents of final stage: List()
14/07/11 03:32:17 INFO DAGScheduler: Missing parents: List()
14/07/11 03:32:17 INFO DAGScheduler: Submitting Stage 5 (MappedRDD[18] at map at <console>:29), which has no missing parents
14/07/11 03:32:17 INFO DAGScheduler: Submitting 2 missing tasks from Stage 5 (MappedRDD[18] at map at <console>:29)
14/07/11 03:32:17 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks
14/07/11 03:32:17 INFO TaskSetManager: Starting task 5.0:0 as TID 8 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:32:17 INFO TaskSetManager: Serialized task 5.0:0 as 3105 bytes in 1 ms
14/07/11 03:32:17 INFO TaskSetManager: Starting task 5.0:1 as TID 9 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:32:17 INFO TaskSetManager: Serialized task 5.0:1 as 3105 bytes in 1 ms
14/07/11 03:32:17 INFO Executor: Running task ID 8
14/07/11 03:32:17 INFO Executor: Running task ID 9
14/07/11 03:32:17 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:32:17 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:32:17 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9
14/07/11 03:32:17 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9
14/07/11 03:32:17 INFO Executor: Serialized size of result for 9 is 1140
14/07/11 03:32:17 INFO Executor: Serialized size of result for 8 is 1140
14/07/11 03:32:17 INFO Executor: Sending result for 9 directly to driver
14/07/11 03:32:17 INFO Executor: Sending result for 8 directly to driver
14/07/11 03:32:17 INFO Executor: Finished task ID 9
14/07/11 03:32:17 INFO Executor: Finished task ID 8
14/07/11 03:32:17 INFO DAGScheduler: Completed ResultTask(5, 1)
14/07/11 03:32:17 INFO TaskSetManager: Finished TID 9 in 31 ms on localhost (progress: 1/2)
14/07/11 03:32:17 INFO DAGScheduler: Completed ResultTask(5, 0)
14/07/11 03:32:17 INFO TaskSetManager: Finished TID 8 in 33 ms on localhost (progress: 2/2)
14/07/11 03:32:17 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
14/07/11 03:32:17 INFO DAGScheduler: Stage 5 (aggregate at RowMatrix.scala:312) finished in 0.034 s
14/07/11 03:32:17 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:312, took 0.045160171 s
14/07/11 03:32:17 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:211
14/07/11 03:32:17 INFO DAGScheduler: Got job 6 (aggregate at RowMatrix.scala:211) with 2 output partitions (allowLocal=false)
14/07/11 03:32:17 INFO DAGScheduler: Final stage: Stage 6(aggregate at RowMatrix.scala:211)
14/07/11 03:32:17 INFO DAGScheduler: Parents of final stage: List()
14/07/11 03:32:17 INFO DAGScheduler: Missing parents: List()
14/07/11 03:32:17 INFO DAGScheduler: Submitting Stage 6 (MappedRDD[18] at map at <console>:29), which has no missing parents
14/07/11 03:32:17 INFO DAGScheduler: Submitting 2 missing tasks from Stage 6 (MappedRDD[18] at map at <console>:29)
14/07/11 03:32:17 INFO TaskSchedulerImpl: Adding task set 6.0 with 2 tasks
14/07/11 03:32:17 INFO TaskSetManager: Starting task 6.0:0 as TID 10 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:32:17 INFO TaskSetManager: Serialized task 6.0:0 as 3057 bytes in 0 ms
14/07/11 03:32:17 INFO TaskSetManager: Starting task 6.0:1 as TID 11 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 03:32:17 INFO TaskSetManager: Serialized task 6.0:1 as 3057 bytes in 0 ms
14/07/11 03:32:17 INFO Executor: Running task ID 10
14/07/11 03:32:17 INFO Executor: Running task ID 11
14/07/11 03:32:17 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:32:17 INFO BlockManager: Found block broadcast_2 locally
14/07/11 03:32:17 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9
14/07/11 03:32:17 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9
14/07/11 03:32:17 INFO Executor: Serialized size of result for 10 is 1037
14/07/11 03:32:17 INFO Executor: Serialized size of result for 11 is 1037
14/07/11 03:32:17 INFO Executor: Sending result for 10 directly to driver
14/07/11 03:32:17 INFO Executor: Sending result for 11 directly to driver
14/07/11 03:32:17 INFO Executor: Finished task ID 10
14/07/11 03:32:17 INFO Executor: Finished task ID 11
14/07/11 03:32:17 INFO DAGScheduler: Completed ResultTask(6, 0)
14/07/11 03:32:17 INFO TaskSetManager: Finished TID 10 in 15 ms on localhost (progress: 1/2)
14/07/11 03:32:17 INFO DAGScheduler: Completed ResultTask(6, 1)
14/07/11 03:32:17 INFO TaskSetManager: Finished TID 11 in 14 ms on localhost (progress: 2/2)
14/07/11 03:32:17 INFO DAGScheduler: Stage 6 (aggregate at RowMatrix.scala:211) finished in 0.016 s
14/07/11 03:32:17 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
14/07/11 03:32:17 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:211, took 0.024784268 s
pc: org.apache.spark.mllib.linalg.Matrix =
-0.5023120302337961  
-0.6114256390937501  
-0.6114256390937501  


scala> val projected: RowMatrix = mat.multiply(pc）

14/07/11 03:34:21 INFO MemoryStore: ensureFreeSpace(176) called with curMem=332240, maxMem=309225062

14/07/11 03:34:21 INFO MemoryStore: Block broadcast_4 stored as values to memory (estimated size 176.0 B, free 294.6 MB)
projected: org.apache.spark.mllib.linalg.distributed.RowMatrix = org.apache.spark.mllib.linalg.distributed.RowMatrix@4d8fde01
 

麦兜NG
关注
spark1.2 使用breeze 的 netlib.BLAS 异常
小刀的专栏
07-29 6644
使用spark1.2 的 standalone 模式运行使用breeze的任务，报如下警告： 15/07/28 02:49:32 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 15/07/28 02:49:32 WARN netlib.BLAS: F
Spark ALS应用BLAS加速
fansy1990的专栏
02-27 2246
文章目录Spark ALS应用BLAS加速1. 环境2. 问题引入3. 参考：4. 思路：4.1 简单测试：4.2 使用Native BLAS需要添加的Jar包方式1：在Intellij IDEA 中添加依赖找到方式2： 自行指定参数编译Spark源码4.3 使用新编译的Spark测试是否加载Native BLAS5. 修改官网提供的安装包，使其加载BLAS5.1 使用 --jars 参数5.2 ...
SparrowRec实战：运行特征工程和Embedding的scala文件报错
weixin_45744256的博客
11-13 376
SparrowRec实战中项目运行的一点hadoop报错
spark 环境搭建坑
12-08 777
spark的新人会有什么坑 spark是一个以java为基础的，以Scala实现的，所以在你在安装指定版本的spark，需要检查你用的是对应spark使用什么版本的scala,可以通过spark-shel --version进行查看，也需要看一下scala使用的什么版本的jdk运行，在scala官方下载页面会有说明jdk版本，然后在编译时使用指定版本的jdk。并且jdk8是jdk1...
从源码看DL4J中Native BLAS的加载,以及配置
along
11-27 7262
最近在用DeepLearning4J(DL4J)尝试语音识别的深度学习,但是报Native Blas找不到的警告,于是从源码中找下方法,把过程和经验记录与此.
MLib - MGtk-开源
04-24
MLib是一个以GLib为基础的库，它为开发者提供了一系列的工具和功能，以简化和增强在开发过程中的编程体验。GLib是GObject系统的核心组件，广泛用于GNOME桌面环境和其他跨平台的应用程序开发。GLib提供了基础的数据...
Spark-Mlib-机器学习算法调研@zyj
03-21
Spark-Mlib-机器学习算法调研@zyj spark Scala 介绍 教程 入门 手册 调研
spark-mlib-master分类算法
06-27
在这个"spark-mlib-master"压缩包中，我们重点关注的是分类算法的实现，这是一个在大数据分析中广泛使用的任务，例如预测用户行为、识别垃圾邮件等。 分类算法是机器学习中的一个重要分支，其目标是根据输入特征将...
MLIB - 分析尖峰数据的工具箱：用于神经生理学实验尖峰数据基本分析的函数集-matlab开发
05-30
MLIB是用于分析峰值数据（即细胞外记录的动作电位模式）的软件包。 特别是，MLIB 包含以下功能：a) 评估尖峰分类质量/单元隔离，b) 构建各种周围刺激时间直方图以及光栅显示和使用各种滤波器内核构建的尖峰密度函数...
利用matlab生成dsp运行代码-mlib_devel:mlib_devel
05-27
Toolflow存储库mlib_devel ！ 什么是mlib_devel？ mlib_devel存储库包含一组由FPGA开发和维护的FPGA DSP库和编程工具。 在合作中，该软件集合被亲切地称为The Toolflow。 CASPER工具流允许您使用MATLAB的图形编程...
netlib-native_system-win-x86_64全套
12-14
libblas3.dll libgcc_s_seh-1.dll libgfortran-3.dll liblapack3.dll libopenblas.a libopenblas.dll libopenblas.dll.a libquadmath-0.dll netlib-native_system-win-x86_64.dll
在 linux 下做矩阵计算用到的全套 netlib-native_system-win-x86_64
04-12
在 linux 下做矩阵计算用到的全套 netlib-native_system-win-x86_64。 libblas3.dll libgcc_s_seh-1.dll libgfortran-3.dll liblapack3.dll libopenblas.a libopenblas.dll libopenblas.dll.a libquadmath-0.dll netlib-native_system-win-x86_64.dll netlib
人工智能之Python人脸识别技术--face_recognition模块
kwame211的博客
05-07 1108
一、环境搭建１．系统环境Ubuntu 17.04 Python 2.7.14 pycharm 开发工具1232.开发环境，安装各种系统包人脸检测基于dlib，dlib依赖Boost和cmake$ sudo apt-get install build-essential cmake $ sudo apt-get install libgtk-3-dev $ sudo apt-get install ...
numpy是python标准库吗_python – 找出Numpy是否使用了哪个BLAS库
weixin_39581716的博客
12-04 786
numpy.show_config()并不总是提供可靠的信息.例如,如果我在Ubuntu 14.04上apt-get install python-numpy,则np.show_config()的输出如下所示：blas_info:libraries = ['blas']library_dirs = ['/usr/lib']language = f77lapack_info:libraries = ...
SparkML之特征提取（一）主成分分析(PCA)
legotime的博客
07-06 8037
主成分分析(Principal Component Analysis，PCA)， 将多个变量通过线性变换以选出较少个数重要变量的一种多 元统计分析方法. --------------------------------------------目录-------------------------------------------------------- 理论和数据见附录 Spark 源
基于MATLAB Simulink的六轴机器人阻抗力控制算法仿真与应用 · 机器人技术
最新发布
08-14
六轴机器人阻抗力控制算法的实现方法及其在MATLAB Simscape平台上的仿真应用。文章首先解释了六轴机器人和阻抗力控制算法的基本概念，然后展示了如何在Simscape环境中构建虚拟机器人模型，并通过M文件设置Simulink参数，实现对机器人运动轨迹和阻抗参数的精确控制。文中还提供了视频演示，直观展示了期望轨迹与实际轨迹的对比，验证了算法的有效性。最后，强调了一键运行的功能，简化了工程实践的操作流程，提升了效率。 适合人群：对机器人技术和控制算法感兴趣的科研人员、工程师和技术爱好者。 使用场景及目标：适用于需要深入了解六轴机器人阻抗力控制算法的工作原理及其实现方法的人群，旨在提高他们对该领域的理论认知和实际操作能力。 其他说明：通过本项目的实践，读者不仅可以掌握机器人阻抗力控制算法的关键技术点，还能学会利用MATLAB工具进行高效建模和仿真的方法。这对于后续的研究和开发工作具有重要的指导意义。
spark-spark mlib简介
03-16
Spark MLlib是Apache Spark的机器学习库，提供了一系列的机器学习算法和工具，包括分类、回归、聚类、协同过滤、降维等。它支持分布式计算，可以在大规模数据集上进行高效的机器学习。Spark MLlib还提供了数据预处理、特征提取、模型评估等功能，使得机器学习的整个流程都可以在Spark上完成。Spark MLlib的API简单易用，可以方便地与Spark的其他组件集成，如Spark SQL、Spark Streaming等。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值