Spark连接Redis数据库

Spark连接Redis数据库

环境

spark-redis 2.5.0
java 12.0.1
Spark version 3.0.0-preview2
Scala 2.12.10
Apache Maven 3.6.3
阿里云ECS
CentOS Linux release 8.1.1911 (Core)
Redis server v=6.0.9
java version "12.0.1" 2019-04-16
IntelliJ IDEA 2020.2.3 (Ultimate Edition)

Maven配置文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.demo.spark.sql</groupId>
    <artifactId>spark-sql</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.21</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>com.redislabs</groupId>
            <artifactId>spark-redis_2.12</artifactId>
            <version>2.5.0</version>
        </dependency>


    </dependencies>
</project>

实现代码

• [hostname] : 主机名（本地为localhost）
• [port] : 端口号（默认端口号为6379）
• [password] : 密码

package com.demo.spark.sql

import org.apache.spark.sql.SparkSession
import redis.clients.jedis.Jedis

object Redis {
  def main(args: Array[String]): Unit = {
    val sparkSession = SparkSession.builder().master("local[*]").appName("Redis").getOrCreate();
    val jedis = new Jedis("[host]", [port]);
    jedis.auth("[password]");
    jedis.set("spark", "spark-redis");
    println(jedis.get("spark"));
  }
}

测试结果

redis

参考

Maven spark-redis

最后

• 由于博主水平有限，不免有疏漏之处，欢迎读者随时批评指正，以免造成不必要的误解！