如何更改Spark程序的HDFS复制因子？

冽夫

于 2020-11-23 18:06:52 发布

阅读量274

点赞数

CC 4.0 BY-SA版权

分类专栏： scala Spark

本文链接：https://blog.youkuaiyun.com/helloworld0906/article/details/110003830

Spark 同时被 2 个专栏收录

49 篇文章

订阅专栏

scala

23 篇文章

订阅专栏

将Spark程序的HDFS复制因子从3更改为1

    val sparkConf = new SparkConf()
    sparkConf.set("spark.driver.maxResultSize", "10g")//.set("autoBroadcastJoinThresho1d","8388608000").set("spark.sql.broadcastTimeout","")

    val spark =SparkSession.builder().config(sparkConf).config("spark.sql.crossJoin.enabled","true").appName("indexSimilarity").getOrCreate()

    val hconf: org.apache.hadoop.conf.Configuration = spark.sparkContext.hadoopConfiguration
    hconf.setInt("dfs.replication", 1)