wufabao的博客

报错内容java.lang.IllegalStateException: No current assignment for partition topicBC-3 at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:259) at org.apache.kafka.clients.consumer.internals.SubscriptionStat

错误一：Unable to locate hive jars to connect to metastore. Please set spark.sql.hive.metastore.jars.解决方案：将JDK11切换为JDK8，问题解决。JDK11和Spark/Hive应该存在版本兼容性问题错误二：连接成功后只显示一个default数据库解决方案：把HIVE_HOME/conf目录下的hive-site.xml文件拷贝到本地IDEA的maven项目的resources目录下完美解决

在Standalone模式下，RDD操作所在的机器未必是提交任务的机器，所以输出RDD元素在提交任务的机器未必可以显示，解决方案：先把数据collect收集起来，然后打印收集的数据（生产环境不建议使用collect算子），可以正常打印：...

scala> val lines = sc.textFile("file:///root/wc.txt")lines: org.apache.spark.rdd.RDD[String] = file:///root/wc.txt MapPartitionsRDD[11] at textFile at <console>:24scala> lines.count[Stage 7:>

22/01/06 22:10:05 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy22/01/06 22:10:05 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, linux121, 34370, None)22/01/06 22:10:0