Doris整合Iceberg+Spark联邦查询实战

最新推荐文章于 2025-10-20 20:52:12 发布

原创

最新推荐文章于 2025-10-20 20:52:12 发布 · 2.3k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #spark #hdfs

本文档详述了如何在Hadoop 3.3、Hive 3.1.3、Doris 1.1-pr2、Spark 3.1.2和Iceberg 0.13.2环境下配置和使用Doris进行Iceberg表的联邦查询。步骤包括Hadoop、Hive、Doris和Spark的安装、配置，以及Hive元数据的初始化，最后展示了Doris如何读取Iceberg数据。

Doris整合Iceberg+Spark联邦查询实战

环境介绍：

Soft	Verison
Hadoop	3.3
Hive	3.1.3
Doris	1.1-pr2
Spark	3.1.2
Iceberg	0.13.2

Hadoop HDFS 环境安装

Hadoop3.3 下载

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.3/hadoop-3.3.3.tar.gz

Lacalhost SSH 免密登录

  ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  chmod 0600 ~/.ssh/authorized_keys
  ssh localhost

执行 ssh localhost，会弹出安全提示，填写yes即可。

如果是docker环境下，需要手动启动sshd服务。

 /usr/sbin/sshd

core-site.xml配置

[root@17a5da45700b hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

<property>
  <name>hadoop.proxyuser.root.hosts</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.proxyuser.root.groups</name>
  <value>*</value>
</property>

</configuration>

hdfs-site.xml配置

[root@17a5da45700b hadoop]# cat  hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>