hadoop 3.x大数据集群搭建系列10-配置Spark Shell及Hive on Spark

一. Spark shell配置

Spark shell默认就是可以访问的

spark-shell
spark.sql("select count(*) from test.t2").show()

二. Hive on Spark配置

2.1 问题描述

set hive.execution.engine=mr;
select count(*) from test.t2;
set hive.execution.engine=spark;
select count(*) from test.t2;

报错:

FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.

2.2 解决方案

在hive中创建spark配置文件

cd /home/apache-hive-3.1.3-bin/conf
vim spark-defaults.conf 
添加如下内容（在执行任务时，会根据如下参数执行）
spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs://hp5:8020/spark-history
spark.executor.memory                    1g
spark.driver.memory					   1g

vim /home/apache-hive-3.1.3-bin/conf/hive-site.xml
<!--Spark依赖位置（注意：端口号9000必须和namenode的端口号一致）-->
<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://hp5:8020/spark-jars/*</value>
</property>
  
<!--Hive执行引擎-->
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

拷贝spark jar包到到hive的lib目录:

cd /home/spark-3.2.2-bin-hadoop3.2/jars
cp ./scala-library-2.12.15.jar /home/apache-hive-3.1.3-bin/lib/
cp ./spark-core_2.12-3.2.2.jar /home/apache-hive-3.1.3-bin/lib/
cp ./spark-network-common_2.12-3.2.2.jar /home/apache-hive-3.1.3-bin/lib/

Spark和Hive的新版本不兼容:
需要编译安装
https://blog.youkuaiyun.com/rfdjds/article/details/125389450

和我这边看到的报错一样，报的的

java.lang.NoSuchMethodError: org.apache.spark.api.java.JavaSparkContext.accumulator(Ljava/lang/Object;Ljava/lang/String;Lorg/apache/spark/AccumulatorParam;)Lorg/apache/spark/Accumulator;