mahout源码分析之DistributedLanczosSolver（一）实战

最新推荐文章于 2025-09-12 20:46:57 发布
最新推荐文章于 2025-09-12 20:46:57 发布
阅读量183 收藏
点赞数
文章标签： 大数据 java
本文介绍了使用Apache Mahout 0.7版本进行SVD（奇异值分解）的实战过程，详细记录了通过DistributedLanczosSolver算法进行矩阵分解的具体步骤及参数设置，并展示了Hadoop集群上运行SVD算法的日志输出。

Mahout版本：0.7，hadoop版本：1.0.4，jdk：1.7.0_25 64bit。

本篇开始系列svd，即降维。这个在mahout中可以直接运行MAHOUT_HOME/mahout/svd -h 即可看到该算法的调用参数，或者在官网相应页面也可以看到，本次实战使用的svd的调用参数如下：

package mahout.fansy.svd;

import org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver;

public class RunSvd {

	/**
	 * 调用svd算法
	 * @throws Exception 
	 */
	public static void main(String[] args) throws Exception {
		String[] arg=new String[]{"-jt","ubuntu:9001","-fs","ubuntu:9000",
				"-i","hdfs://ubuntu:9000/svd/input/wind",
				"-o","hdfs://ubuntu:9000/svd/output",
				"-nr","178","-nc","14",
				"-r","3",
				"-sym","square",
				"--cleansvd","true",
				"--tempDir","hdfs://ubuntu:9000/svd/temp"
		};
		DistributedLanczosSolver.main(arg);
	}

}
在运行这个算法之前，需要转换输入数据，比如使用下面的输入数据：


可以使用下面的代码把上面的文件转换为序列文件，同时value的格式为VectorWritable类型：

package mahout.fansy.utils.svd;

import java.io.IOException;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.mahout.common.HadoopUtil;
import org.apache.mahout.math.RandomAccessSparseVector;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;

public class ReadAndWriteSeq {

	/*
	 * 私有化 构造函数，只能使用方法调用
	 */
	private ReadAndWriteSeq(){}
	
	/**
	 * 读取文本数据，写入HDFS文件系统格式为<Writable,VectorWritable>
	 * @param input 输入文本文件
	 * @param output 输出HDFS文件（seq格式）
	 * @param jobtracker 使用的jobtracker地址
	 * @param regex  使用的解析类，用于解析文本文件
	 * @return 任务是否成功
	 * @throws IOException
	 */
	public static boolean readAndWriteSeq(String input,String output,String jobtracker) throws IOException{
		boolean flag=true;
		
		Configuration conf=new Configuration();
		conf.set("mapred.job.tracker", jobtracker);
		
		FileSystem fsIn = FileSystem.get(URI.create(input), conf);
		FileSystem fsOut = FileSystem.get(URI.create(output), conf);
		HadoopUtil.delete(conf, new Path(output));
	    Path pathIn = new Path(input);
	    Path pathOut = new Path(output);
	    
	    SequenceFile.Writer writer = null;
	    FSDataInputStream in = fsIn.open(pathIn);
	    IntWritable key=new IntWritable(0);
	    VectorWritable value=new VectorWritable();
	    try {
	      writer = SequenceFile.createWriter(fsOut, conf, pathOut,
	              key.getClass(), value.getClass());
	      String line=null;
	      int length=0;
	      int row=1;
	      while ((line=in.readLine())!=null) {
	    	  String[] vs=line.split(",");
	    	  length=vs.length;
	    	  Vector vector=new RandomAccessSparseVector(length);
	    	  for(int i=0;i<length;i++){
	    		  vector.set(i, Double.parseDouble(vs[i]));
	    	  }
	    	  value.set(vector);
	    	  key.set(row++);
	    	  writer.append(key, value);
	      }
	    } catch(IOException e){
	    	flag=false;
	    }finally {
	       IOUtils.closeStream(writer);
	       in.close();
	    }
		return flag;
	}
}
运行后的在HDFS中一共有三个输出，其中一个是temp的，即临时的，还有两个一个是 rawEigenvectors，如下：


一个是：cleanEigenvectors（最后的输出）：


终端的显示信息如下：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/workspase/mahout/lib/mahout-examples-0.7-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/workspase/mahout/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
13/10/28 00:14:13 WARN fs.FileSystem: "ubuntu:9000" is a deprecated filesystem name. Use "hdfs://ubuntu:9000/" instead.
13/10/28 00:14:13 INFO common.AbstractJob: Command line arguments: {--cleansvd=[true], --endPhase=[2147483647], --inMemory=[false], --input=[hdfs://ubuntu:9000/svd/input/wind], --maxError=[0.05], --minEigenvalue=[0.0], --numCols=[14], --numRows=[178], --output=[hdfs://ubuntu:9000/svd/output], --rank=[3], --startPhase=[0], --symmetric=[square], --tempDir=[hdfs://ubuntu:9000/svd/temp]}
13/10/28 00:15:43 INFO lanczos.LanczosSolver: Finding 3 singular vectors of matrix with 178 rows, via Lanczos
13/10/28 00:15:45 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/28 00:15:49 INFO mapred.JobClient: Running job: job_201310220012_0023
13/10/28 00:15:50 INFO mapred.JobClient:  map 0% reduce 0%
13/10/28 00:18:40 INFO mapred.JobClient:  map 100% reduce 0%
13/10/28 00:19:11 INFO mapred.JobClient:  map 100% reduce 100%
13/10/28 00:19:16 INFO mapred.JobClient: Job complete: job_201310220012_0023
13/10/28 00:19:16 INFO mapred.JobClient: Counters: 30
13/10/28 00:19:16 INFO mapred.JobClient:   Job Counters 
13/10/28 00:19:16 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 00:19:16 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=241868
13/10/28 00:19:16 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 00:19:16 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 00:19:16 INFO mapred.JobClient:     Launched map tasks=2
13/10/28 00:19:16 INFO mapred.JobClient:     Data-local map tasks=2
13/10/28 00:19:16 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=28198
13/10/28 00:19:16 INFO mapred.JobClient:   File Input Format Counters 
13/10/28 00:19:16 INFO mapred.JobClient:     Bytes Read=23216
13/10/28 00:19:16 INFO mapred.JobClient:   File Output Format Counters 
13/10/28 00:19:16 INFO mapred.JobClient:     Bytes Written=220
13/10/28 00:19:16 INFO mapred.JobClient:   FileSystemCounters
13/10/28 00:19:16 INFO mapred.JobClient:     FILE_BYTES_READ=238
13/10/28 00:19:16 INFO mapred.JobClient:     HDFS_BYTES_READ=23967
13/10/28 00:19:16 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70111
13/10/28 00:19:16 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=220
13/10/28 00:19:16 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 00:19:16 INFO mapred.JobClient:     Map output materialized bytes=244
13/10/28 00:19:16 INFO mapred.JobClient:     Map input records=178
13/10/28 00:19:16 INFO mapred.JobClient:     Reduce shuffle bytes=244
13/10/28 00:19:16 INFO mapred.JobClient:     Spilled Records=4
13/10/28 00:19:16 INFO mapred.JobClient:     Map output bytes=228
13/10/28 00:19:16 INFO mapred.JobClient:     Total committed heap usage (bytes)=246685696
13/10/28 00:19:16 INFO mapred.JobClient:     CPU time spent (ms)=53940
13/10/28 00:19:16 INFO mapred.JobClient:     Map input bytes=21094
13/10/28 00:19:16 INFO mapred.JobClient:     SPLIT_RAW_BYTES=172
13/10/28 00:19:16 INFO mapred.JobClient:     Combine input records=2
13/10/28 00:19:16 INFO mapred.JobClient:     Reduce input records=2
13/10/28 00:19:16 INFO mapred.JobClient:     Reduce input groups=1
13/10/28 00:19:16 INFO mapred.JobClient:     Combine output records=2
13/10/28 00:19:16 INFO mapred.JobClient:     Physical memory (bytes) snapshot=414433280
13/10/28 00:19:16 INFO mapred.JobClient:     Reduce output records=1
13/10/28 00:19:16 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=2934288384
13/10/28 00:19:16 INFO mapred.JobClient:     Map output records=2
13/10/28 00:19:16 INFO lanczos.LanczosSolver: 1 passes through the corpus so far...
13/10/28 00:19:17 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/28 00:19:18 INFO mapred.JobClient: Running job: job_201310220012_0024
13/10/28 00:19:19 INFO mapred.JobClient:  map 0% reduce 0%
13/10/28 00:20:06 INFO mapred.JobClient:  map 100% reduce 0%
13/10/28 00:20:25 INFO mapred.JobClient:  map 100% reduce 100%
13/10/28 00:20:30 INFO mapred.JobClient: Job complete: job_201310220012_0024
13/10/28 00:20:30 INFO mapred.JobClient: Counters: 30
13/10/28 00:20:30 INFO mapred.JobClient:   Job Counters 
13/10/28 00:20:30 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 00:20:30 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=79120
13/10/28 00:20:30 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 00:20:30 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 00:20:30 INFO mapred.JobClient:     Launched map tasks=2
13/10/28 00:20:30 INFO mapred.JobClient:     Data-local map tasks=2
13/10/28 00:20:30 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=15191
13/10/28 00:20:30 INFO mapred.JobClient:   File Input Format Counters 
13/10/28 00:20:30 INFO mapred.JobClient:     Bytes Read=23216
13/10/28 00:20:30 INFO mapred.JobClient:   File Output Format Counters 
13/10/28 00:20:30 INFO mapred.JobClient:     Bytes Written=220
13/10/28 00:20:30 INFO mapred.JobClient:   FileSystemCounters
13/10/28 00:20:30 INFO mapred.JobClient:     FILE_BYTES_READ=238
13/10/28 00:20:30 INFO mapred.JobClient:     HDFS_BYTES_READ=23967
13/10/28 00:20:30 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70105
13/10/28 00:20:30 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=220
13/10/28 00:20:30 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 00:20:30 INFO mapred.JobClient:     Map output materialized bytes=244
13/10/28 00:20:30 INFO mapred.JobClient:     Map input records=178
13/10/28 00:20:30 INFO mapred.JobClient:     Reduce shuffle bytes=244
13/10/28 00:20:30 INFO mapred.JobClient:     Spilled Records=4
13/10/28 00:20:30 INFO mapred.JobClient:     Map output bytes=228
13/10/28 00:20:30 INFO mapred.JobClient:     Total committed heap usage (bytes)=246685696
13/10/28 00:20:30 INFO mapred.JobClient:     CPU time spent (ms)=18830
13/10/28 00:20:30 INFO mapred.JobClient:     Map input bytes=21094
13/10/28 00:20:30 INFO mapred.JobClient:     SPLIT_RAW_BYTES=172
13/10/28 00:20:30 INFO mapred.JobClient:     Combine input records=2
13/10/28 00:20:30 INFO mapred.JobClient:     Reduce input records=2
13/10/28 00:20:30 INFO mapred.JobClient:     Reduce input groups=1
13/10/28 00:20:30 INFO mapred.JobClient:     Combine output records=2
13/10/28 00:20:30 INFO mapred.JobClient:     Physical memory (bytes) snapshot=406294528
13/10/28 00:20:30 INFO mapred.JobClient:     Reduce output records=1
13/10/28 00:20:30 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=2934288384
13/10/28 00:20:30 INFO mapred.JobClient:     Map output records=2
13/10/28 00:20:30 INFO lanczos.LanczosSolver: 2 passes through the corpus so far...
13/10/28 00:20:30 INFO lanczos.LanczosSolver: Lanczos iteration complete - now to diagonalize the tri-diagonal auxiliary matrix.
13/10/28 00:20:30 INFO lanczos.LanczosSolver: Eigenvector 0 found with eigenvalue 0.0
13/10/28 00:20:30 INFO lanczos.LanczosSolver: Eigenvector 1 found with eigenvalue 190.66366814935847
13/10/28 00:20:30 INFO lanczos.LanczosSolver: Eigenvector 2 found with eigenvalue 10886.664434372891
13/10/28 00:20:30 INFO lanczos.LanczosSolver: LanczosSolver finished.
13/10/28 00:20:30 INFO decomposer.DistributedLanczosSolver: Persisting 3 eigenVectors and eigenValues to: hdfs://ubuntu:9000/svd/output/rawEigenvectors
13/10/28 00:20:31 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/28 00:20:31 INFO mapred.JobClient: Running job: job_201310220012_0025
13/10/28 00:20:32 INFO mapred.JobClient:  map 0% reduce 0%
13/10/28 00:21:06 INFO mapred.JobClient:  map 50% reduce 0%
13/10/28 00:21:12 INFO mapred.JobClient:  map 100% reduce 0%
13/10/28 00:21:30 INFO mapred.JobClient:  map 100% reduce 100%
13/10/28 00:21:35 INFO mapred.JobClient: Job complete: job_201310220012_0025
13/10/28 00:21:35 INFO mapred.JobClient: Counters: 30
13/10/28 00:21:35 INFO mapred.JobClient:   Job Counters 
13/10/28 00:21:35 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 00:21:35 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=61438
13/10/28 00:21:35 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 00:21:35 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 00:21:35 INFO mapred.JobClient:     Launched map tasks=2
13/10/28 00:21:35 INFO mapred.JobClient:     Data-local map tasks=2
13/10/28 00:21:35 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=17342
13/10/28 00:21:35 INFO mapred.JobClient:   File Input Format Counters 
13/10/28 00:21:35 INFO mapred.JobClient:     Bytes Read=23216
13/10/28 00:21:35 INFO mapred.JobClient:   File Output Format Counters 
13/10/28 00:21:35 INFO mapred.JobClient:     Bytes Written=220
13/10/28 00:21:35 INFO mapred.JobClient:   FileSystemCounters
13/10/28 00:21:35 INFO mapred.JobClient:     FILE_BYTES_READ=238
13/10/28 00:21:35 INFO mapred.JobClient:     HDFS_BYTES_READ=24061
13/10/28 00:21:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70108
13/10/28 00:21:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=220
13/10/28 00:21:35 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 00:21:35 INFO mapred.JobClient:     Map output materialized bytes=244
13/10/28 00:21:35 INFO mapred.JobClient:     Map input records=178
13/10/28 00:21:35 INFO mapred.JobClient:     Reduce shuffle bytes=244
13/10/28 00:21:35 INFO mapred.JobClient:     Spilled Records=4
13/10/28 00:21:35 INFO mapred.JobClient:     Map output bytes=228
13/10/28 00:21:35 INFO mapred.JobClient:     Total committed heap usage (bytes)=291512320
13/10/28 00:21:35 INFO mapred.JobClient:     CPU time spent (ms)=13100
13/10/28 00:21:35 INFO mapred.JobClient:     Map input bytes=21094
13/10/28 00:21:35 INFO mapred.JobClient:     SPLIT_RAW_BYTES=172
13/10/28 00:21:35 INFO mapred.JobClient:     Combine input records=2
13/10/28 00:21:35 INFO mapred.JobClient:     Reduce input records=2
13/10/28 00:21:35 INFO mapred.JobClient:     Reduce input groups=1
13/10/28 00:21:35 INFO mapred.JobClient:     Combine output records=2
13/10/28 00:21:35 INFO mapred.JobClient:     Physical memory (bytes) snapshot=417210368
13/10/28 00:21:35 INFO mapred.JobClient:     Reduce output records=1
13/10/28 00:21:35 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=2934288384
13/10/28 00:21:35 INFO mapred.JobClient:     Map output records=2
13/10/28 00:21:36 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/28 00:21:36 INFO mapred.JobClient: Running job: job_201310220012_0026
13/10/28 00:21:37 INFO mapred.JobClient:  map 0% reduce 0%
13/10/28 00:22:11 INFO mapred.JobClient:  map 100% reduce 0%
13/10/28 00:22:29 INFO mapred.JobClient:  map 100% reduce 100%
13/10/28 00:22:34 INFO mapred.JobClient: Job complete: job_201310220012_0026
13/10/28 00:22:34 INFO mapred.JobClient: Counters: 30
13/10/28 00:22:34 INFO mapred.JobClient:   Job Counters 
13/10/28 00:22:34 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 00:22:34 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=46990
13/10/28 00:22:34 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 00:22:34 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 00:22:34 INFO mapred.JobClient:     Launched map tasks=2
13/10/28 00:22:34 INFO mapred.JobClient:     Data-local map tasks=2
13/10/28 00:22:34 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=15142
13/10/28 00:22:34 INFO mapred.JobClient:   File Input Format Counters 
13/10/28 00:22:34 INFO mapred.JobClient:     Bytes Read=23216
13/10/28 00:22:34 INFO mapred.JobClient:   File Output Format Counters 
13/10/28 00:22:34 INFO mapred.JobClient:     Bytes Written=220
13/10/28 00:22:34 INFO mapred.JobClient:   FileSystemCounters
13/10/28 00:22:34 INFO mapred.JobClient:     FILE_BYTES_READ=238
13/10/28 00:22:34 INFO mapred.JobClient:     HDFS_BYTES_READ=24061
13/10/28 00:22:34 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70105
13/10/28 00:22:34 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=220
13/10/28 00:22:34 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 00:22:34 INFO mapred.JobClient:     Map output materialized bytes=244
13/10/28 00:22:34 INFO mapred.JobClient:     Map input records=178
13/10/28 00:22:34 INFO mapred.JobClient:     Reduce shuffle bytes=244
13/10/28 00:22:34 INFO mapred.JobClient:     Spilled Records=4
13/10/28 00:22:34 INFO mapred.JobClient:     Map output bytes=228
13/10/28 00:22:34 INFO mapred.JobClient:     Total committed heap usage (bytes)=336338944
13/10/28 00:22:34 INFO mapred.JobClient:     CPU time spent (ms)=4670
13/10/28 00:22:34 INFO mapred.JobClient:     Map input bytes=21094
13/10/28 00:22:34 INFO mapred.JobClient:     SPLIT_RAW_BYTES=172
13/10/28 00:22:34 INFO mapred.JobClient:     Combine input records=2
13/10/28 00:22:34 INFO mapred.JobClient:     Reduce input records=2
13/10/28 00:22:34 INFO mapred.JobClient:     Reduce input groups=1
13/10/28 00:22:34 INFO mapred.JobClient:     Combine output records=2
13/10/28 00:22:34 INFO mapred.JobClient:     Physical memory (bytes) snapshot=426409984
13/10/28 00:22:34 INFO mapred.JobClient:     Reduce output records=1
13/10/28 00:22:34 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=2934288384
13/10/28 00:22:34 INFO mapred.JobClient:     Map output records=2
13/10/28 00:22:34 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/28 00:22:35 INFO mapred.JobClient: Running job: job_201310220012_0027
13/10/28 00:22:36 INFO mapred.JobClient:  map 0% reduce 0%
13/10/28 00:22:54 INFO mapred.JobClient:  map 100% reduce 0%
13/10/28 00:23:03 INFO mapred.JobClient:  map 100% reduce 33%
13/10/28 00:23:09 INFO mapred.JobClient:  map 100% reduce 100%
13/10/28 00:23:14 INFO mapred.JobClient: Job complete: job_201310220012_0027
13/10/28 00:23:14 INFO mapred.JobClient: Counters: 30
13/10/28 00:23:14 INFO mapred.JobClient:   Job Counters 
13/10/28 00:23:14 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 00:23:14 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=25276
13/10/28 00:23:14 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 00:23:14 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 00:23:14 INFO mapred.JobClient:     Launched map tasks=2
13/10/28 00:23:14 INFO mapred.JobClient:     Data-local map tasks=2
13/10/28 00:23:14 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=14956
13/10/28 00:23:14 INFO mapred.JobClient:   File Input Format Counters 
13/10/28 00:23:14 INFO mapred.JobClient:     Bytes Read=23216
13/10/28 00:23:14 INFO mapred.JobClient:   File Output Format Counters 
13/10/28 00:23:14 INFO mapred.JobClient:     Bytes Written=220
13/10/28 00:23:14 INFO mapred.JobClient:   FileSystemCounters
13/10/28 00:23:14 INFO mapred.JobClient:     FILE_BYTES_READ=238
13/10/28 00:23:14 INFO mapred.JobClient:     HDFS_BYTES_READ=24031
13/10/28 00:23:14 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=70105
13/10/28 00:23:14 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=220
13/10/28 00:23:14 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 00:23:14 INFO mapred.JobClient:     Map output materialized bytes=244
13/10/28 00:23:14 INFO mapred.JobClient:     Map input records=178
13/10/28 00:23:14 INFO mapred.JobClient:     Reduce shuffle bytes=244
13/10/28 00:23:14 INFO mapred.JobClient:     Spilled Records=4
13/10/28 00:23:14 INFO mapred.JobClient:     Map output bytes=228
13/10/28 00:23:14 INFO mapred.JobClient:     Total committed heap usage (bytes)=336338944
13/10/28 00:23:14 INFO mapred.JobClient:     CPU time spent (ms)=2680
13/10/28 00:23:14 INFO mapred.JobClient:     Map input bytes=21094
13/10/28 00:23:14 INFO mapred.JobClient:     SPLIT_RAW_BYTES=172
13/10/28 00:23:14 INFO mapred.JobClient:     Combine input records=2
13/10/28 00:23:14 INFO mapred.JobClient:     Reduce input records=2
13/10/28 00:23:14 INFO mapred.JobClient:     Reduce input groups=1
13/10/28 00:23:14 INFO mapred.JobClient:     Combine output records=2
13/10/28 00:23:14 INFO mapred.JobClient:     Physical memory (bytes) snapshot=423587840
13/10/28 00:23:14 INFO mapred.JobClient:     Reduce output records=1
13/10/28 00:23:14 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=2934288384
13/10/28 00:23:14 INFO mapred.JobClient:     Map output records=2

具体明天再分析吧


分享，成长，快乐

转载请注明blog地址：http://blog.youkuaiyun.com/fansy1990


iteye_21202
关注
Mahout源码
03-15
**Apache Mahout 源码解析** Apache Mahout 是一个基于Java的开源机器学习库，旨在简化大规模数据集上的机器学习算法实现。它为开发者提供了一系列预构建的、可扩展的机器学习算法，包括分类、聚类、推荐系统以及...
mahout Algorithms源码分析
08-05
樊哲是Mahout的积极学习者和实践者，他在优快云上分享了关于Mahout算法的解析与案例实战的博客，获得了“优快云2013博客之星”的荣誉。樊哲的经验表明，虽然Hadoop平台上算法开发一般需要耗费很长时间，但Mahout已经实现...
mahout源码分析之DistributedLanczosSolver（七）总结篇
445822357
10-31 147
Mahout版本：0.7，hadoop版本：1.0.4，jdk：1.7.0_25 64bit。 看svd算法官网上面使用的是亚马逊的云平台计算的，不过给出了svd算法的调用方式，当算出了eigenVectors后，应该怎么做呢？比如原始数据是600*60（600行，60列）的数据，计算得到的eigenVectors是24*60（其中的24是不大于rank的一个值），那么最后得到的结果应该是ori...
mahout源码分析之DistributedLanczosSolver（四）rawEigen是什么
445822357
10-30 164
Mahout版本：0.7，hadoop版本：1.0.4，jdk：1.7.0_25 64bit。 接上篇，eigen分解，额，太复杂了，人太浮躁了，静不下来分析（说java对矩阵操作支持度不足，额，好吧是外部原因）。 1. 前奏： eigen分解的是triDiag矩阵，这个矩阵，上篇求得的结果是： [[0.315642761491587, 0.9488780991876485, 0.0]...
lanczos学习
gushuizerotoone的博客
07-07 1253
1. 《matrix computation》P470: Equally important, information about A's extremal eigenvalues tends to emerge long before the tridiagonalization is complete. This makes the Lanczos algorithm particularly...
mahout 实战
mach_learn的专栏
06-08 5282
本文整理了mahout比较常用的算法，部分章节理论知识进行了引用，小节后有引用网址，对mahout中的多种算法，使用中文实例进行讲解，像k-means中文聚类，朴素贝叶斯中文分类等，最后讲解了推荐算法应用的一般流程。
mahout源码
12-11
Apache Mahout是一款基于Java开发的机器学习库，旨在提供可扩展的、高效的算法，用于数据挖掘和模式识别。在大数据时代，Mahout已经成为数据科学家和工程师们的重要工具，尤其在文本分析、推荐系统和分类任务中扮演...
掌握最新Mahout源码：Mahout in Action源码解析
从“压缩包子文件的文件名称列表”中给出的“mahout-distribution-0.9”文件名来看，这可能是某一个版本的Mahout源码包的名称。文件名中的“0.9”意味着这是0.9版本的Mahout。在软件开发中，版本号非常重要，它标识...
mahout实现的算法集（一）
时刻保持危机感&对知识的渴望
02-26 8844
mahout实现的算法：（翻译自mahout官方文档：点击打开链接） (一)Classification（分类算法） 完全支持： 1.Logistic Regression（Logistic回归） 2.Naive Bayes/ Complementary Naive Bayes(朴素贝叶斯/互补的朴素贝叶斯) 3.Random Forests(随机森林) 4.Hidden Markov
mahout算法源码分析之Collaborative Filtering with ALS-WR 并行思路
445822357
10-10 264
Mahout版本：0.7，hadoop版本：1.0.4，jdk：1.7.0_25 64bit。 mahout算法源码分析之Collaborative Filtering with ALS-WR 这个算法的并行主要就应该是ParallelALSFactorizationJob这里的并行了，下图是这个Job的大部分操作： 这里分析并行就是看每个job任务是否可以出现多个map或者reduce即...
TDengine 选择函数 TOP() 用户手册
最新发布
TDengine(老段）专注时序数据库领域
09-12 533
其中，TDengine TSDB 是一款高性能、分布式的时序数据库（Time Series Database），同时它还带有内建的缓存、流式计算、数据订阅等系统功能；TDengine IDMP 是一款AI原生工业数据管理平台，它通过树状层次结构建立数据目录，对数据进行标准化、情景化，并通过 AI 提供实时分析、可视化、事件管理与报警等功能。函数返回指定列中最大的 k 个非 NULL 值及其对应的记录。如果多条数据取值相同，且全部取用会超出 k 条限制时，系统会从相同值中随机选取符合要求的数量返回。
AI-调查研究-74-具身智能 机器人学习新突破：元学习与仿真到现实迁移的挑战与机遇
永远好奇，无限进步！
09-10 964
具身智能体要实现高效学习与技能迁移，核心在于提升少样本学习与跨任务泛化能力。当前机器人学习面临多重挑战：一是任务间的迁移难度大，已学技能难以快速应用到相似任务；二是不同机器人形态和传感系统的异构性增加了适配难度；三是现有强化学习与模仿学习在样本效率和泛化能力上存在局限。为突破瓶颈，研究重点聚焦于元学习、少样本模仿学习和多任务预训练策略。例如，MAML等元学习框架可显著提升新任务适应速度，大规模多任务预训练则能为下游
还在重启应用改 Topic？Spring Boot 动态 Kafka 消费的“终极形态”
郑龙飞
09-10 709
/ 用于存储 @KafkaListener 的“蓝图”// ... 可按需添加 concurrency, autoStartup 等其他属性(元数据采集与注册)@Component@OverrideClass<?= null &&!通过巧妙地结合和动态配置中心，我们实现了一个功能极其强大的动态 Kafka 消费管理方案。
内存大（巨）页
fpcc的专栏
09-12 406
文章摘要： 大页（巨页）是比标准4KB页更大的内存页（如2MB或1GB），用于优化内存管理。随着内存容量增长，传统小页导致页表膨胀、TLB命中率下降等问题。大页通过减少页表层级、提升连续内存访问效率，显著提高性能，尤其适用于数据库、大数据处理、高性能计算等场景。但其分配复杂，可能加剧内存碎片。技术发展推动内存管理演进，未来可能出现更优方案。开发者需持续关注前沿技术动态。 （150字）
想拿到ip地址库的离线包
Nothing
09-09 684
如果你只做“中国境内日志 -> 省市”离线解析，纯真社区版无论体积还是精度都够用；想要街道级或风险标签，再考虑商业数据源；任何 IP 库都有边界漂移，记得留好“未知”兜底与灰度观测。
当建筑工程遇上“AI+虚拟仿真”：打开未来工程人才培养的创新之门
恒点虚拟仿真官方博客
09-09 733
虚拟仿真技术助力国际工程管理人才培养。随着一带一路建设推进，海外工程需求激增，传统教学面临实践成本高、文化差异大等瓶颈。AI+虚拟仿真技术构建沉浸式实训场景，通过虚拟仿真技术，模拟海外项目全过程，涵盖招投标、施工管理等环节，同时整合各国法规标准数据库，有效提升学生的跨文化适应能力和实操技能。该模式突破时空限制，为培养复合型国际工程人才提供创新解决方案，精准对接政策要求和市场需求。
Mac本地docker安装Kibana+ElasticSearch
liliang199的专栏
09-11 511
Kibana是和ElasticSearch协同的经典工具，这里尝试在Mac以Docker方式快速搭建Kibana+ElasticSearch。假设Docker已安装，细节参考Docker若不能正常pull，可能需修改配置，参考。
【Ambari监控】Ambari-Metrics 的演进过程
TTBIGDATA的博客
09-11 392
本文解析了Ambari-Metrics组件在不同版本中的源码位置变化。在2.7.9及之前版本，该组件直接集成在Ambari主仓库中；从2.8.0版本开始，被移出主仓成为独立仓库。作者推测这一变化源于架构解耦、生态适配和社区治理的需求。独立后的Ambari-Metrics具有更灵活的发布节奏和维护方式，便于与外部监控系统集成。文章通过版本对比表清晰展示了这一演进过程。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值