spark 矩阵转置 java,矩阵在Spark中的RowMatrix上转置

Suppose I have a RowMatrix.

How can I transpose it. The API documentation does not seem to have a transpose method.

The Matrix has the transpose() method. But it is not distributed. If I have a large matrix greater that the memory how can I transpose it?

I have converted a RowMatrix to DenseMatrix as follows

DenseMatrix Mat = new DenseMatrix(m,n,MatArr);

which requires converting the RowMatrix to JavaRDD and converting JavaRDD to an array.

Is there any other convenient way to do the conversion?

Thanks in advance

解决方案

You are correct: there is no

RowMatrix.transpose()

method. You will need to do this operation manually.

Here is the non-distributed/local matrix versions:

def transpose(m: Array[Array[Double]]): Array[Array[Double]] = {

(for {

c

} yield m.map(_(c)) ).toArray

}

The distributed version would be along the following lines:

origMatRdd.rows.zipWithIndex.map{ case (rvect, i) =>

rvect.zipWithIndex.map{ case (ax, j) => ((j,(i,ax))

}.groupByKey

.sortBy{ case (i, ax) => i }

.foldByKey(new DenseVector(origMatRdd.numRows())) { case (dv, (ix,ax)) =>

dv(ix) = ax

}

Caveat: I have not tested the above: it will have bugs. But the basic approach is valid - and similar to work I had done in the past for a small LinAlg library for spark.