问题

最新推荐文章于 2024-06-30 22:31:02 发布

原创最新推荐文章于 2024-06-30 22:31:02 发布 · 589 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #java #hadoop #大数据 #hdfs

本文深入探讨Spark大数据处理中常见的问题及解决策略，包括处理大规模数据集的限制、优化内存使用、提升并行处理效率，以及如何正确配置Spark参数以应对不同场景的挑战。

问题

RowMatrix 65535
- This cannot be computed on matrices with more than 65535 columns.
- 只能处理65535维的数据
vituralbox 启动spark集群，当result过大时，driver需要通过blockmanager 获取，由于端口不是固定的所以无法配置端口映射
- 17/11/28 09:49:24 ERROR RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks java.io.IOException: Failed to connect to /172.17.0.2:45442
- 解决方案 https://stackoverflow.com/questions/44935477/spark-2-broadcast-inside-docker-uses-random-port

docker run -ti --rm \
    --name sparkmaster \
    --hostname=host.a \
    --add-host host.a:xx.xx.xx.xx \
    --add-host host.b:xx.xx.xx.xx \
    -p 18080:18080 \

    .config("spark.fileserver.port", "7002") \
    .config("spark.broadcast.port", "7003") \
    .config("spark.replClassServer.port", "7004") \
    .config("spark.blockManager.port", "7005") \
    .config("spark.broadcast.blockSize", "4096") \
    .getOrCreate()

    //spark.driver.maxResultSize
    // spark.task.maxDirectResultSize

     conf.getSizeAsBytes("spark.task.maxDirectResultSize", 1L << 20),
    RpcUtils.maxMessageSizeBytes(conf))
     private val MAX_MESSAGE_SIZE_IN_MB = Int.MaxValue / 1024 / 1024
      ==>
          .config("spark.task.maxDirectResultSize", "2147483648") \
    .config("spark.driver.maxResultSize","3g") \

python pandas datafram语法问题：
- group_by_post_colums_sum = data_T.apply(lambda x:sum(x))

  File "/Users/seki/git/dyd/data_foundation/python/com/dyd/data/test/TestRecommend.py", line 158, in <lambda>
    group_by_post_colums_sum = data_T.apply(lambda x:sum(x))
  File "/Users/seki/git/learn/spark/python/lib/pyspark.zip/pyspark/sql/functions.py", line 40, in _
    jc = getattr(sc._jvm.functions, name)(col._jc if isinstance(col, Column) else col)
AttributeError: ("'NoneType' object has no attribute '_jvm'", u'occurred at index 6293615542250297450')

spark on yarn spark ui

7/11/29 11:25:34 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> uhadoop-adarbt-master1,uhadoop-adarbt-master2, PROXY_URI_BASES -> http://uhadoop-adarbt-master1:23188/proxy/application_1511859350454_0038,http://uhadoop-adarbt-master2:23188/proxy/application_1511859350454_0038), /proxy/application_1511859350454_0038 17/11/29 11:25:34 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter

通过这个访问
- https://hadoop.diyidan.net:23188/proxy/application_1511859350454_0038

17/05/03 17:58:02 ERROR client.TransportClient: Failed to send RPC 7785784597803174149 to /172.26.159.91:56630: java.nio.channels.ClosedChannelException

http://duguyiren3476.iteye.com/blog/2372479
单个executor 使用的 cpu 不能超过5个（dyd ucloud集群的情况）

spark on yarn cdh 5.49 Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.

startshell --driver-class-path $CONF_DIR:$LIB_JARS == 》export SPARK_CLASSPATH="$SPARK_CLASSPATH:$CONF_DIR:$LIB_JARS"

spark broadcast Failed to get broadcast_8_piece0 of broadcast_8

spark 初始化不能是成员变量

breeze Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS

https://github.com/fommil/netlib-java
https://stackoverflow.com/questions/37848216/how-to-configure-high-performance-blas-lapack-for-breeze-on-amazon-emr-ec2
update-alternatives: using /usr/lib/atlas-base/atlas/libblas.so.3 to provide /usr/lib/libblas.so.3 (libblas.so.3) in auto mode Setting up libopenblas-base (0.2.8-6ubuntu1) ... update-alternatives: using /usr/lib/openblas-base/libblas.so.3 to provide /usr/lib/libblas.so.3 (libblas.so.3) in auto mode
sudo ln -s /usr/lib/openblas-base/liblapack.so.3 /usr/lib/liblapack.so.3

　Spark SQL从2.0开始已经不再支持ALTER TABLE

https://www.iteblog.com/archives/2031.html

spark sql 无法读取 hive acid delta 文件

shell 脚本定期执行compact major ,压缩后的 base文件可以读取到，

Caused by: java.lang.NullPointerException: Cannot suppress a null exception.at java.lang.Throwable.addSuppressed(Throwable.java:1046)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:243)

mysql 账号没有写入权限

java.io.IOException: No space left on deviceat java.io.FileOutputStream.writeBytes(Native Method)at java.io.FileOutputStream.write(FileOutputStream.java:326)at org.apache.spark.storage.TimeTrackingOutputStream.write(TimeTrackingOutputStream.java:58)

https://developer.ibm.com/hadoop/2016/07/18/troubleshooting-and-tuning-spark-for-heavy-workloads/

使用hbase-spark Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

用1.6.0 logging打成jar
添加到driver运行机器的spark_home的lib
driver运行时，add-jar添加

org.apache.spark.shuffle.FetchFailedException: /data/yarn/local/usercache/root/appcache/application_1514973560079_1031/blockmgr-b8eb9d9c-e6e2-4c4c-a5ac-5120a52b1e32/1d/shuffle_9_1_0.index (No such file or directory)

spark als 推荐慢问题

blockify -》 blockSize 调整

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 222.0 failed 4 times, most recent failure: Lost task 1.3 in stage 222.0 (TID 9505, uhadoop-adarbt-core1): java.lang.ArrayIndexOutOfBoundsException at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)

spark2.3.1 调试问题，java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = -3720498261147521051, local class serialVersionUID = -6655865447853211720 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)

core 编译target 目录 original-spark-core_2.11-2.3.1-SNAPSHOT.jar 替换 assembly 项目下 scala-2.11/jars对应的spark core jar

spark dataframe保存到hdfs 数据量较小11m,保存只用了1个分区，spark repartion无用，集中在一个节点，任务计算时也只用一个节点

问题一：小数据加载后cache缓存在一个节点的一个executor
问题二：缓存在一个节点的一个executor后，并行处理的task只分配这个executor ,其他executor无任务，并行度下降

spark.locality.wait和spark.locality.wait.process，spark.locality.wait.node, spark.locality.wait.rack这几个参数影响了任务分配时的本地性策略的相关细节。

sparkConf.set("spark.locality.wait","0")，不等待，直接分配

笛卡尔积优化策略：大表物品特征表：25万 180维度的数据，小表用户特征： 1000 用户，180维度,

2个exeutor , 每个2个cpu, 总共4个cpu
postFactorsDf,物品特征表四个分区,每个分区数据量： 88.0 MB / 23

postFactorsDf加载后调用cache缓存，cache block为4mb ，所以显示为 23条记录
每个分区实际记录条数 63250 , 每一块缓存数据为 63250/ 23 = 2750 ~ 4mb
0.0013913043 mb/g

用户数据 18.3 KB / 1

 def recommendPosts4LongIdSmallSrcFactors(rank: Int,
                                           srcFactors: DataFrame,
                                           dstFactors: DataFrame,
                                           srcOutputColumn: String,
                                           dstOutputColumn: String,
                                           num: Int
                                           ): DataFrame = {
       import srcFactors.sparkSession.implicits._

    val srcFactorsBlocked = blockify4Long(srcFactors.as[(Long, Array[Float])],srcSize)
    val dstFactorsBlocked = blockify4Long(dstFactors.as[(Long, Array[Float])],desSize)

    val ratings = broadcast(srcFactorsBlocked).join(dstFactorsBlocked)
      .as[(Seq[(Long, Array[Float])], Seq[(Long, Array[Float])])]
      .flatMap { case (srcIter, dstIter) =>
        val m = srcIter.size
        val n = math.min(dstIter.size, num)
        val output = new Array[(Long, Long, Float)](m * n)
        var i = 0
        val pq = new BoundedPriorityQueue[(Long, Float)](num)(Ordering.by(_._2))
        srcIter.foreach { case (srcId, srcFactor) =>
          dstIter.foreach { case (dstId, dstFactor) =>
            val score = f2jBLAS.sdot(rank, srcFactor, 1, dstFactor, 1)
            pq += dstId -> score
          }
          pq.foreach { case (dstId, score) =>
            output(i) = (srcId, dstId, score)
            i += 1
          }
          pq.clear()
        }
        output.filter(_ != null).toSeq
      }
    val topKAggregator = new TopByKeyAggregator[Long, Long, Float](num * 10, Ordering.by(_._2))
    val recs = ratings.as[(Long, Long, Float)].groupByKey(_._1).agg(topKAggregator.toColumn)
      .toDF("id", "recommendations")
    recs
  }

doTest(userDf,1000,3000)

Peak Execution Memory 160.0 MB
一个分区的一个dstIter = 3000 条 4mb ，23 块dstIter
Shuffle Write Size / Records = 10.9 MB / 1000

doTest(userDf,1000,50000)

Peak Execution Memory 24.0 MB
Shuffle Write Size / Records = 2.2 MB / 1000
一个分区的一个dstIter = 50000 条 70m ，2 块dstIter

doTest(userDf,1000,30000)

Peak Execution Memory 24.0 MB
Shuffle Write Size / Records 3.2 MB / 1000
3 块dstIter

doTest(userDf,1000,10000)

Peak Execution Memory 64.0 MB
Shuffle Write Size / Records 7.6 MB / 1000
7 块dstIter

doTest(userDf,1000,100000)

Peak Execution Memory 20.0 MB
Shuffle Write Size / Records 1.1 mb / 1000

总结

group数决定了有多少块dstIter，每一块dstIter 会维持 srcIter.size 也就用户数 1000 * 100 的 top 100 的记录在 output
- 划分多少块 dstIter 就会有多少个1000 * 100 的 top 100的记录数
- 影响 Peak Execution Memory 和 Shuffle Write Size
- 单块dstIter的 Shuffle Write Size = 1.1mb ，