GraphX增加BFS测试用例

最新推荐文章于 2024-01-28 14:30:53 发布

飞火流云

最新推荐文章于 2024-01-28 14:30:53 发布

阅读量782

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/cloudeagle_bupt/article/details/78043380

本文介绍了一种基于Apache Spark的广度优先搜索(BFS)算法实现，并详细展示了如何通过Scala编程来完成图数据的处理与分析。该算法适用于大规模图数据的遍历问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

基于Analytics.scala,在其中增加：

      case "bfs" =>
        val outFname = options.remove("output").getOrElse("")
        options.foreach {
          case (opt, _) => throw new IllegalArgumentException("Invalid option: " + opt)
        }

        println("======================================")
        println("|      BFS          |")
        println("======================================")

        val sc = new SparkContext(conf.setAppName("ConnectedComponents(" + fname + ")"))
        val unpartitionedGraph = GraphLoader.edgeListFile(sc, fname,
          numEdgePartitions = numEPart,
          edgeStorageLevel = edgeStorageLevel,
          vertexStorageLevel = vertexStorageLevel).cache()
        
        val graph = partitionStrategy.foldLeft(unpartitionedGraph)(_.partitionBy(_)) //unpartitionedGraph是初始值
        val bfs = BFS.run(graph)
//        println("BFS: " + cc.vertices.map { case (vid, data) => data }.distinct())
        if (!outFname.isEmpty) {  //修改使输出结果
          logWarning("Saving pageranks of pages to " + outFname)
          bfs.vertices.map { case (vid, data) => vid + "\t" + data }.saveAsTextFile(outFname)
        }
        sc.stop()

新建bfs.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.graphx.lib

import scala.reflect.ClassTag
import org.apache.spark.graphx.PartitionStrategy._

import org.apache.spark.graphx._
import org.apache.spark._

object BFS {  
  
    def run[VD: ClassTag, ED: ClassTag](graph: Graph[VD, ED]): Graph[Double, ED] = {
      val srcVertex =  1
     val bfsGraph = graph.mapVertices((id, _) => if (id == srcVertex) 0.0 else Double.PositiveInfinity)  
    
    Pregel(bfsGraph, Double.PositiveInfinity, activeDirection = EdgeDirection.Either)(
      vprog = (id, attr, msg) => math.min(attr, msg),  
      sendMsg = edge => {
        if (edge.srcAttr != Double.PositiveInfinity) {  
              Iterator((edge.dstId, edge.srcAttr+1))  
          } else {  
              Iterator.empty  
          }
      },
      mergeMsg = (a,b) => math.min(a,b))
  } // end of bfs
}