Spark Standalone模式集群对TIDB/Mysql支持

最新推荐文章于 2023-07-24 00:00:00 发布

CHINA华军

最新推荐文章于 2023-07-24 00:00:00 发布

阅读量513

点赞数

分类专栏： MySQL Other 文章标签： TiDB mysql spark

本文链接：https://blog.youkuaiyun.com/chinahuanghuajun/article/details/79281649

版权

Spark让TIDB、MySQL一些大的查询速度提升多倍

集群环境：

CentOS 7.4
Master：192.168.106.42
Slave ： 192.168.106.43 192.168.106.46 192.168.106.47

安装java1.8
hostnamectl set-hostname master-42
mkdir /usr/java
cp /mnt/jdk-8u111-linux-x64.gz /usr/java/
cd /usr/java
tar -zxvf jdk-8u111-linux-x64.gz
source /etc/profile
java -version

master节点192.168.106.42安装spark
cd /opt/
tar -xzvf spark-2.1.1-bin-hadoop2.7.tgz
cd spark-2.1.1-bin-hadoop2.7
./sbin/start-master.sh -h 192.168.106.42
支持TIDB
复制tispark-SNAPSHOT-jar-with-dependencies.jar 到/opt/spark-2.1.1-bin-hadoop2.7/jars
支持MySql
复制mysql-connector-java-5.1.44-bin.jar到/opt/spark-2.1.1-bin-hadoop2.7/jars
./sbin/stop-master.sh -h 192.168.106.42
./sbin/start-master.sh -h 192.168.106.42
注意关闭防火墙
iptables -F

slave节点192.168.106.43安装spark
cd /opt/
tar -xzvf spark-2.1.1-bin-hadoop2.7.tgz
cd /opt/spark-2.1.1-bin-hadoop2.7
支持TIDB
复制tispark-SNAPSHOT-jar-with-dependencies.jar 到/opt/spark-2.1.1-bin-hadoop2.7/jars
支持MySql
复制mysql-connector-java-5.1.44-bin.jar到/opt/spark-2.1.1-bin-hadoop2.7/jars.
/sbin/start-slave.sh spark://192.168.106.42:7077
分别安装slave节点192.168.106.46 和192.168.106.47

通过web监控可以要看到各节点
http://192.168.106.42:8080/
Spark Master at spark://192.168.106.42:7077
URL: spark://192.168.106.42:7077
REST URL: spark://192.168.106.42:6066 (cluster mode)
Alive Workers: 3
Cores in use: 12 Total, 12 Used
Memory in use: 19.8 GB Total, 3.0 GB Used
Applications: 1 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE
Workers
Worker Id Address State Cores Memory
worker-20180205145117-192.168.106.43-35632 192.168.106.43:35632 ALIVE 4 (4 Used) 6.6 GB (1024.0 MB Used)
worker-20180205145204-192.168.106.46-36288 192.168.106.46:36288 ALIVE 4 (4 Used) 6.6 GB (1024.0 MB Used)
worker-20180205145314-192.168.106.47-33544