Spark项目连接Hadoop HA配置文件编写方案

最新推荐文章于 2025-01-06 16:10:11 发布

原创最新推荐文章于 2025-01-06 16:10:11 发布 · 637 阅读

1 ·

CC 4.0 BY-SA版权

hadoop 专栏收录该内容

20 篇文章

订阅专栏

博客围绕Spark项目连接Hadoop HA展开，重点介绍文件编写方案，涉及信息技术领域中大数据开发相关内容，为Spark与Hadoop HA的连接提供有效解决办法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

val sparkConf = new SparkConf().setAppName("sparkRealTimeServer")
      .setMaster("yarn").setSparkHome("/app/data1/spark").set("spark.submit.deployMode", "client").
      set("spark.testing.memory", "2147480000").
      set("spark.sql.hive.verifyPartitionPath", "true").
      set("spark.yarn.executor.memoryOverhead", "2048m").
      set("spark.dynamicAllocation.enabled", "true").
      set("spark.shuffle.service.enabled", "true").
      set("spark.dynamicAllocation.executorIdleTimeout", "60").
      set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "18000").
      set("spark.dynamicAllocation.initialExecutors", "15").
      set("spark.dynamicAllocation.maxExecutors", "15").
      set("spark.dynamicAllocation.minExecutors", "15").
      set("spark.dynamicAllocation.schedulerBacklogTimeout", "10").
      set("spark.eventLog.enabled", "true").
      set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").
      set("spark.hadoop.dfs.nameservices","iptvcluster").
      set("spark.hadoop.dfs.client.failover.proxy.provider.iptvcluster","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider").
      set("spark.hadoop.dfs.ha.namenodes.iptvcluster", "nn1,nn2").
      set("spark.hadoop.dfs.namenode.rpc-address.iptvcluster.nn1", "namenode1:8020").
      set("spark.hadoop.dfs.namenode.rpc-address.iptvcluster.nn2", "namenode2:8020").
      set("spark.hadoop.yarn.resourcemanager.hostname","namenode1").
      set("spark.hadoop.yarn.resourcemanager.address","namenode1:8032").
      set("spark.yarn.access.namenodes", "hdfs://namenode1:8020,hdfs://namenode2:8020").
      set("spark.yarn.stagingDir", "hdfs://iptvcluster/user/hadoop/").
      set("spark.driver.allowMultipleContexts", "true").
      set("spark.scheduler.listenerbus.eventqueue.size", "100000").
      set("spark.rdd.compress","true")