Doris整合Iceberg+Spark联邦查询实战

最新推荐文章于 2025-11-05 21:15:15 发布

原创

最新推荐文章于 2025-11-05 21:15:15 发布 · 1.4k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #hdfs

本文详细介绍了如何在Hadoop 3.3、Hive 3.1.3、Doris 1.1-pr2、Spark 3.1.2和Iceberg 0.13.2的环境中，进行HDFS配置、Hive安装、Doris部署、Spark设置以及建立Iceberg表和Doris外表，实现跨系统联邦查询。通过这些步骤，可以有效地整合大数据处理工具，提升数据操作的灵活性和效率。

Doris整合Iceberg+Spark联邦查询实战

环境介绍：

Soft	Verison
Hadoop	3.3
Hive	3.1.3
Doris	1.1-pr2
Spark	3.1.2
Iceberg	0.13.2

Hadoop HDFS 环境安装

Hadoop3.3 下载

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.3/hadoop-3.3.3.tar.gz

Lacalhost SSH 免密登录

  ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  chmod 0600 ~/.ssh/authorized_keys
  ssh localhost

执行 ssh localhost，会弹出安全提示，填写yes即可。

如果是docker环境下，需要手动启动sshd服务。

 /usr/sbin/sshd

core-site.xml配置

[root@17a5da45700b hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

<property>
  <name>hadoop.proxyuser.root.hosts</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.proxyuser.root.groups</name>
  <value>*</value>
</property>

</configuration>

hdfs-site.xml配置

[root@17a5da45700b hadoop]# cat  hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>