Doris安装

已于 2024-03-13 16:23:14 修改 · 1.2k 阅读

文章标签：

#数据库 #大数据 #数据仓库

于 2024-03-12 12:27:57 首次发布

1.安装docker

如果对docker感兴趣，关注 docker容器总结_服务器中,容器创建时所产生的文件有什么-优快云博客

1）Docker 要求 CentOS 系统的内核版本高于 3.10 ，首先查看系统内核版本是否满足

uname -r

2）使用 root 权限登录系统，确保 yum 包更新到最新

sudo yum update -y

3）假如安装过旧版本，先卸载旧版本

sudo yum remove docker docker-common docker-selinux docker-engine

4）安装 yum-util 工具包和 devicemapper 驱动依赖

sudo yum install -y yum-utils device-mapper-persistent-data lvm2

5）设置 yum 源（加速 yum 下载速度）

sudo  yum-config-manager  --add-repo  http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

6）查看所有仓库中所有 docker 版本，并选择特定版本安装，一般可直接安装最新版

yum list docker-ce --showduplicates | sort -r

7）安装 docker

（1）安装最新稳定版本的方式：

sudo yum install docker-ce -y #安装的是最新稳定版本，因为 repo 中默认只

8）启动并加入开机启动

sudo systemctl start docker #启动 docker
sudo systemctl enable docker #加入开机自启动

9）查看 Version，验证是否安装成功

docker version

若出现 Client 和 Server 两部分内容，则证明安装成功。

2.使用docker开发镜像编译

1.下载源码包

wget https://dist.apache.org/repos/dist/dev/incubator/doris/0.15/0.15.0-rc04/apache-doris-0.15.0-incubating-src.tar.gz

注意：这个包路径现在已经过期了，可以在官网去下载：Index of /dist/doris/0.15.0-incubating

2.解压

tar -zxvf apache-doris-0.15.0-incubating-src.tar.gz -C  /opt/software

3.下载docker镜像（编译doris的镜像）

[root@hadoop102 software]# docker pull apache/incubator-doris:build-env-for-0.15.0

4.启动镜像

[root@hadoop102 software]# 
docker run -it \
-v /opt/software/.m2:/root/.m2 \
-v /opt/software/apache-doris-0.15.0-incubating-src/:/root/apache-doris-0.15.0-incubating-src/ \
apache/incubator-doris:build-env-for-0.15.0

5.切换jdk环境

前提：服务器要安装好jdk8

查看一下服务器的jdk

[atguigu@hadoop102 ~]$ java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)

注意：因为第四步启动了镜像，所以现在是进入了镜像的，现在查看镜像的jdk

[root@ca4d9b90497c ~]# java -version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

看见没有jdk的11版本，所以要换成8版本

[root@ca4d9b90497c ~]# alternatives --set java java-1.8.0-openjdk.x86_64
[root@ca4d9b90497c ~]# alternatives --set javac java-1.8.0-openjdk.x86_64
[root@ca4d9b90497c ~]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0

再次查看镜像的jdk版本

[root@ca4d9b90497c ~]# java -version
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)

6.上传doris-repo.tar.gz到 /opt/software中

7.解压

[root@hadoop102 software]#tar -zxvf doris-repo.tar.gz -C /opt/software

8.编译

第一次编译用这个

[root@90095eb69c8a apachedoris-0.15.0-incubating-src]# sh build.sh --clean --be --fe --ui

第二次编译用这个

[root@90095eb69c8a apachedoris-0.15.0-incubating-src]#sh build.sh

9.查看编译是否成功

[atguigu@hadoop102 apache-doris-0.15.0-incubating-src]$ cd output/
[atguigu@hadoop102 output]$ ll
total 12
drwxr-xr-x 8 root root 4096 Mar  8 10:31 be
drwxr-xr-x 9 root root 4096 Mar  8 10:31 fe
drwxr-xr-x 4 root root 4096 Mar  8 10:31 udf

如果出现上面的目录，说明成功

3.安装Doris

架构：

hadoop102	hadoop103	hadoop104
FE(LEADER)	FE(FOLLOWER)	FE(OBSERVER)
BE	BE	BE
BROKER	BROKER	BROKER

注意：如果要搭建上面表格的架构，请完成下面安装Doris部分和Doris扩容缩容中的扩容FE部分即可

1.拷贝目录

[atguigu@hadoop102 output]$ mkdir /opt/module/apache-doris-0.15.0

[atguigu@hadoop102 output]$ cp -r /opt/software/apache-doris-0.15.0-incubating-src/output/*  /opt/module/apache-doris-0.15.0

2.修改打开的文件数

sudo vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
* soft nproc 131072
* hard nproc 131072

注意：修改参数后，请重启服务器

3.部署FE

目前只部署一台hadoop102上

只在hadoop102上操作

vim /opt/module/apache-doris-0.15.0/fe/conf/fe.conf

#配置文件中指定元数据路径：
meta_dir = /opt/module/apache-doris-0.15.0/doris-meta
#修改绑定 ip（每台机器修改成自己的 ip）（ifconfig查看一下）
priority_networks = 192.168.8.101/24

mkdir /opt/module/apache-doris-0.15.0/doris-meta

4.启动

[atguigu@hadoop102 fe]$ bin/start_fe.sh --daemon

[atguigu@hadoop102 fe]$ jps -l
2545 org.apache.doris.PaloFe
2593 sun.tools.jps.Jps

5.访问

http://hadoop102:8030/

注意：如果是服务器，请在安全组打开 8030 端口

6.部署BE

分发

[atguigu@hadoop102 apache-doris-0.15.0]$ xsync be

在hadoop102，hadoop103，hadoop104创建目录

mkdir /opt/module/apache-doris-0.15.0/doris-storage1
mkdir /opt/module/apache-doris-0.15.0/doris-storage2

在hadoop102，hadoop103，hadoop104修改配置文件

vim /opt/module/apache-doris-0.15.0/be/conf/be.conf

#配置文件中指定数据存放路径：
storage_root_path = /opt/module/apache-doris-0.15.0/doris-storage1;/opt/module/apache-doris-0.15.0/doris-storage2
#修改绑定 ip（每台机器修改成自己的 ip）（ifconfig查看一下）
priority_networks = 192.168.8.101/24

7.FE中添加所有BE节点

1.创建mysql客户端（hadoop102上操作）（在你在没有mysql的服务器上操作）

[atguigu@hadoop102 software]$ mkdir mysql-client

2.检查服务器是否有mysql

[atguigu@hadoop102 mysql-client]$ sudo rpm -qa|grep mariadb
#如果存在，先卸载
[atguigu@hadoop102 mysql-client]$ sudo rpm -e --nodeps mariadb mariadb-libs mariadb-server

3.上传包

[atguigu@hadoop102 mysql-client]$ ll
total 48640
-rw-r--r-- 1 atguigu atguigu 45109364 Apr 28  2022 mysql-community-client-5.7.28-1.el7.x86_64.rpm
-rw-r--r-- 1 atguigu atguigu   318768 Apr 28  2022 mysql-community-common-5.7.28-1.el7.x86_64.rpm
-rw-r--r-- 1 atguigu atguigu  4374364 Apr 28  2022 mysql-community-libs-5.7.28-1.el7.x86_64.rpm

4.安装

[atguigu@hadoop102 mysql-client]$ sudo rpm -ivh /opt/software/mysql-client/*

5.登录

[atguigu@hadoop102 apache-doris-0.15.0]$ mysql -h hadoop102 -P 9030 -uroot

6.设置密码

ysql> SET PASSWORD FOR 'root' = PASSWORD('000000');

7.退出

mysql> quit

8.重新进入

[atguigu@hadoop102 apache-doris-0.15.0]$ mysql -h hadoop102 -P 9030 -uroot -p000000

9.添加BE

mysql>ALTER SYSTEM ADD BACKEND "hadoop102:9050";
mysql>ALTER SYSTEM ADD BACKEND "hadoop103:9050";
mysql>ALTER SYSTEM ADD BACKEND "hadoop104:9050";

10.查看 BE 状态

SHOW PROC '/backends';

8.启动BE

/opt/module/apache-doris-0.15.0/be/bin/start_be.sh --daemon

查看状态

mysql -h hadoop1 -P 9030 -uroot -p
SHOW PROC '/backends';

9.部署FS_Broker（可选）

1.编译

[atguigu@hadoop102 bin]$ 
sudo docker run -it \
-v /opt/software/.m2:/root/.m2 \
-v /opt/software/apache-doris-0.15.0-incubating-src/:/root/apache-doris-0.15.0-incubating-src/ \
apache/incubator-doris:build-env-for-0.15.0

2.找编译包

[root@36c5d05df12d apache_hdfs_broker]# pwd
/root/apache-doris-0.15.0-incubating-src/fs_brokers/apache_hdfs_broker
[root@36c5d05df12d apache_hdfs_broker]# ll
total 36
drwxrwxr-x 2 1000 1000  4096 Nov 16  2021 bin
-rwxrwxr-x 1 1000 1000  1773 Nov 16  2021 build.sh
drwxrwxr-x 2 1000 1000  4096 Nov 16  2021 conf
-rw-rw-r-- 1 1000 1000 19478 Nov 16  2021 pom.xml
drwxrwxr-x 4 1000 1000  4096 Nov 16  2021 src

3.查看容器的java版本

[root@36c5d05df12d apache_hdfs_broker]# java -version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)
[root@36c5d05df12d apache_hdfs_broker]# echo $JAVA_HOME
/usr/lib/jvm/java-11

4.更换java版本

[root@36c5d05df12d apache_hdfs_broker]# alternatives --set java java-1.8.0-openjdk.x86_64
[root@36c5d05df12d apache_hdfs_broker]# alternatives --set javac java-1.8.0-openjdk.x86_64
[root@36c5d05df12d apache_hdfs_broker]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0

5.编译

[root@36c5d05df12d apache_hdfs_broker]# sh build.sh

6.检查是否成功

容器里面：

[root@36c5d05df12d apache_hdfs_broker]# pwd
/root/apache-doris-0.15.0-incubating-src/fs_brokers/apache_hdfs_broker/output/apache_hdfs_broker
[root@36c5d05df12d apache_hdfs_broker]# ll
total 12
drwxr-xr-x 2 root root 4096 Mar 12 02:40 bin
drwxr-xr-x 2 root root 4096 Mar 12 02:40 conf
drwxr-xr-x 2 root root 4096 Mar 12 02:40 lib

服务器上：

[atguigu@hadoop102 apache_hdfs_broker]$ pwd
/opt/software/apache-doris-0.15.0-incubating-src/fs_brokers/apache_hdfs_broker/output/apache_hdfs_broker
[atguigu@hadoop102 apache_hdfs_broker]$ ll
total 12
drwxr-xr-x 2 root root 4096 Mar 12 10:40 bin
drwxr-xr-x 2 root root 4096 Mar 12 10:40 conf
drwxr-xr-x 2 root root 4096 Mar 12 10:40 lib

7.拷贝文件

[atguigu@hadoop102 output]$ pwd
/opt/software/apache-doris-0.15.0-incubating-src/fs_brokers/apache_hdfs_broker/output

[atguigu@hadoop102 output]$ sudo cp -r apache_hdfs_broker/ /opt/module/apache-doris-0.15.0/

8.更改权限

[atguigu@hadoop102 apache-doris-0.15.0]$ cd /opt/module/apache-doris-0.15.0/
[atguigu@hadoop102 apache-doris-0.15.0]$ sudo chown -R atguigu:atguigu

9.启动FS_Broker

[atguigu@hadoop102 apache-doris-0.15.0]$/opt/module/apache-doris-0.15.0/apache_hdfs_broker/bin/start_broker.sh --daemon

10.添加Broker

[atguigu@hadoop102 bin]$ mysql -h hadoop102 -P 9030 -uroot -p000000

mysql> ALTER SYSTEM ADD BROKER broker_name "hadoop102:8000","hadoop103:8000","hadoop104:8000";

11.查看状态

SHOW PROC "/brokers";

4.Doris扩容缩容

Doris 可以很方便的扩容和缩容 FE、BE、Broker 实例。

1.扩容FE

1.查看FE

mysql> show proc '/frontends'\G
*************************** 1. row ***************************
             Name: 172.28.19.250_9010_1710149955809
               IP: 172.28.19.250
         HostName: hadoop102
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: true
        ClusterId: 551010137
             Join: true
            Alive: true
ReplayedJournalId: 2061
    LastHeartbeat: 2024-03-12 11:00:51
         IsHelper: true
           ErrMsg: 
          Version: 0.15.0-rc04-Unknown
 CurrentConnected: Yes
1 row in set (0.03 sec)

目前只要hadoop102是一台FE

2.分发FE

[atguigu@hadoop102 apache-doris-0.15.0]$ pwd
/opt/module/apache-doris-0.15.0
[atguigu@hadoop102 apache-doris-0.15.0]$ xsync fe

3.修改配置

在hadoop103修改配置文件（注意：这里写自己的内网ip，ifconfig查看一下）

[atguigu@hadoop103 conf]$ mkdir /opt/module/apache-doris-0.15.0/doris-meta
[atguigu@hadoop103 conf]$ cd /opt/module/apache-doris-0.15.0/fe/conf
[atguigu@hadoop103 conf]$ vim fe.conf
priority_networks = 172.28.19.251/24

在hadoop104修改配置文件（注意：这里写自己的内网ip，ifconfig查看一下）

[atguigu@hadoop104 conf]$ mkdir /opt/module/apache-doris-0.15.0/doris-meta
[atguigu@hadoop104 conf]$ cd /opt/module/apache-doris-0.15.0/fe/conf
[atguigu@hadoop104 conf]$ vim fe.conf
priority_networks = 172.28.19.252/24

4.添加FE

mysql> ALTER SYSTEM ADD FOLLOWER "hadoop103:9010";
mysql> ALTER SYSTEM ADD OBSERVER "hadoop104:9010";

5.启动

hadoop103上启动（注意：第一次启动需要添加 --helper，后面就不需要了）

[atguigu@hadoop103 conf]$ /opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh --helper hadoop102:9010 --daemon

hadoop104上启动（注意：第一次启动需要添加 --helper，后面就不需要了）

[atguigu@hadoop104 conf]$ /opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh --helper hadoop102:9010 --daemon

6.查看是否成功

hadoop102：

mysql> show proc '/frontends'\G
*************************** 1. row ***************************
             Name: 172.28.19.250_9010_1710149955809
               IP: 172.28.19.250
         HostName: hadoop102
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: true
        ClusterId: 551010137
             Join: true
            Alive: true
ReplayedJournalId: 3382
    LastHeartbeat: 2024-03-12 12:13:52
         IsHelper: true
           ErrMsg: 
          Version: 0.15.0-rc04-Unknown
 CurrentConnected: Yes
*************************** 2. row ***************************
             Name: 172.28.19.251_9010_1710216190385
               IP: 172.28.19.251
         HostName: hadoop104
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: OBSERVER
         IsMaster: false
        ClusterId: 551010137
             Join: true
            Alive: true
ReplayedJournalId: 3381
    LastHeartbeat: 2024-03-12 12:13:52
         IsHelper: false
           ErrMsg: 
          Version: 0.15.0-rc04-Unknown
 CurrentConnected: No
*************************** 3. row ***************************
             Name: 172.28.19.252_9010_1710213630618
               IP: 172.28.19.252
         HostName: hadoop103
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
         IsMaster: false
        ClusterId: 551010137
             Join: true
            Alive: true
ReplayedJournalId: 3381
    LastHeartbeat: 2024-03-12 12:13:52
         IsHelper: true
           ErrMsg: 
          Version: 0.15.0-rc04-Unknown
 CurrentConnected: No
3 rows in set (0.02 sec)

注意：IsMaster代表是否为LEADER，Role代表是FOLLOWER或者OBSERVER，Alive代表通信，必须是true

访问页面：http://hadoop102:8030/System?path=//frontends

2..缩容FE

不建议操作

模板：

ALTER SYSTEM DROP FOLLOWER[OBSERVER] "fe_host:edit_log_port";

注意: FOLLOWER[OBSERVER]是可选，想选择缩容的那台机器是FOLLOWER就写FOLLOWER，是OBSERVER就写OBSERVER

例如：

mysql> ALTER SYSTEM DROP FOLLOWER "hadoop103:9010";

[atguigu@hadoop103 bin]$ ./stop_fe.sh --daemon

3.BE的扩容和缩容

运维负责，这里不写了

4.Broker扩容和缩容

运维负责，这里不写了

5.启停脚本

#!/bin/bash
case $1 in
"start"){
  echo ---------- Doris hadoop102  FE 启动 ------------
  ssh hadoop102 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh --daemon"
  echo ---------- Doris hadoop103  FE 启动 ------------
  ssh hadoop103 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh hadoop102:9010 --daemon"
  echo ---------- Doris hadoop104  FE 启动 ------------
  ssh hadoop104 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh hadoop102:9010 --daemon"
for i in hadoop102 hadoop103 hadoop104
do
  echo ---------- Doris $i  BE 启动 ------------
  ssh $i "/opt/module/apache-doris-0.15.0/be/bin/start_be.sh --daemon"
done

for i in hadoop102 hadoop103 hadoop104
do
  echo ---------- Doris $i  Broker 启动 ------------
  ssh $i "/opt/module/apache-doris-0.15.0/apache_hdfs_broker/bin/start_broker.sh --daemon"
done
};;
"stop"){
for i in hadoop102 hadoop103 hadoop104
do
echo ---------- Doris $i  Broker 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/apache_hdfs_broker/bin/stop_broker.sh --daemon"
echo ---------- Doris $i  BE 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/be/bin/stop_be.sh --daemon"
echo ---------- Doris $i  FE 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/fe/bin/stop_fe.sh --daemon"
done
};;
"status"){
for i in hadoop102 hadoop103 hadoop104
do
echo ---------- Doris $i  Broker 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/apache_hdfs_broker/bin/stop_broker.sh --daemon"
echo ---------- Doris $i  BE 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/be/bin/stop_be.sh --daemon"
echo ---------- Doris $i  FE 停止 ------------
ssh $i "/opt/module/apache-doris-0.15.0/fe/bin/stop_fe.sh --daemon"
done

"start"){
  echo ---------- Doris hadoop102  FE 启动 ------------
  ssh hadoop102 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh --daemon"
  echo ---------- Doris hadoop103  FE 启动 ------------
  ssh hadoop103 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh hadoop102:9010 --daemon"
  echo ---------- Doris hadoop104  FE 启动 ------------
  ssh hadoop104 "/opt/module/apache-doris-0.15.0/fe/bin/start_fe.sh hadoop102:9010 --daemon"
for i in hadoop102 hadoop103 hadoop104
do
  echo ---------- Doris $i  BE 启动 ------------
  ssh $i "/opt/module/apache-doris-0.15.0/be/bin/start_be.sh --daemon"
done

for i in hadoop102 hadoop103 hadoop104
do
  echo ---------- Doris $i  Broker 启动 ------------
  ssh $i "/opt/module/apache-doris-0.15.0/apache_hdfs_broker/bin/start_broker.sh --daemon"
done

};;
esac