Spark-Doris-Connecter By pySpark

最新推荐文章于 2025-09-07 03:26:12 发布

原创

最新推荐文章于 2025-09-07 03:26:12 发布 · 1.1k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据 #big data

本文档展示了如何配置Spark-Doris-Connector环境，并通过pyspark从Doris数据库中读取数据。首先，按照指定链接的步骤设置环境，然后使用`spark.read.format()`方法指定数据源为Doris，设置相关连接参数如表名、节点地址、用户名和密码，最后加载数据并展示前5行记录。

准备Spark-Doris-Connecter环境，具体参照：

https://blog.youkuaiyun.com/wangleigiser/article/details/124724390?spm=1001.2014.3001.5501

输入如下命令进入pyspark

[root@xxx ~]# pyspark
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/05/12 10:29:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/05/12 10:29:26 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Welcome to
      ____