Ambari 2.7.5+HDP 3.1.5.0部署

最新推荐文章于 2023-09-15 18:18:41 发布

原创最新推荐文章于 2023-09-15 18:18:41 发布 · 541 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#ambari #hadoop #大数据

本文档详细介绍了在HDP 3.1.5和Ambari 2.7.5环境下进行大数据平台部署的步骤，包括环境准备，如配置主机名、hosts文件，禁用selinux firewalld，设置umask，主机互信，配置ulimit，时间同步，设置swap，安装jdk1.8，以及Ambari的安装，涉及httpd配置和repo设置。

HDP（Hortonworks Data Platform）3.1.5和Ambari（Apache Ambari）2.7.5是开源的大数据平台软件。它们本身没有特别限制使用的限制，可以在符合其运行要求的计算机上自由安装和使用。

但是需要注意的是，使用这些软件可能涉及到一些许可证和法律条款方面的问题。例如，如果您在商业环境中使用HDP和Ambari，则需要了解相关的商业许可证费用和条件。此外，在某些国家或地区，可能有禁止或限制使用此类软件的法规。

因此，建议在使用HDP和Ambari之前，仔细阅读相应的许可证和法律条款，并遵守相关规定。

环境准备

yum update

配置主机名和/etc/hosts文件

cat >> /etc/hosts << EOF
192.168.110.134 hdp0.itcast.cn hdp0
192.168.110.135 hdp1.itcast.cn hdp1
192.168.110.136 hdp2.itcast.cn hdp2
EOF

使用 hostname -f 检查是否能够映射FQDN主机名

禁用selinux firewalld

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
systemctl disable --now firewalld

设置umask

为当前登录session设置umask:
umask 0022
检查设置:
umask
为每个用户设置:
echo umask 0022 >> /etc/profile

UMASK（用户掩码或用户文件创建MASK）设置在Linux计算机上创建新文件或文件夹时授予的默认权限或基本权限。
大多数Linux发行版（发行版）都将022设置为默认的umask值。
umask值为022，为新文件或文件夹授予755的读取，写入和执行权限。
umask值027为新文件或文件夹授予750读取，写入和执行权限。
Ambari，HDP和HDF支持umask值022（0022在功能上等效），027（0027在功能上等效）。
必须在所有主机上设置这些值。

主机互信（可选）

yum install -y sshpass
ssh-keygen -f /root/.ssh/id_rsa -P ''
export IP="192.168.110.134 192.168.110.135 192.168.110.136"
export SSHPASS=Unique#2022
for HOST in $IP;do
     sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST
done

cat ssh_key.sh

#!/bin/bash

COLOR="echo -e \\033[01;31m"
END='\033[0m'

NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`
export SSHPASS=Unique#2022
HOSTS="
192.168.10.134
192.168.10.135
192.168.10.136"

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' /etc/os-release`
}

ssh_key_push(){
    rm -f ~/.ssh/id_rsa*
    ssh-keygen -f /root/.ssh/id_rsa -P '' &> /dev/null
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;then
        rpm -q sshpass &> /dev/null || { ${COLOR}"安装sshpass软件包"${END};yum -y install sshpass &> /dev/null; }
    else
        dpkg -S sshpass &> /dev/null || { ${COLOR}"安装sshpass软件包"${END};apt -y install sshpass &> /dev/null; }
    fi
    sshpass -e ssh-copy-id -o StrictHostKeyChecking=no ${IP} &> /dev/null
    [ $? -eq 0 ] && echo ${IP} is finished || echo ${IP} is false

    for i in ${HOSTS};do
        sshpass -e scp -o StrictHostKeyChecking=no -r /root/.ssh root@${i}: &> /dev/null
        [ $? -eq 0 ] && echo ${i} is finished || echo ${i} is false
    done

    for i in ${HOSTS};do
        scp /root/.ssh/known_hosts ${i}:.ssh/ &> /dev/null
        [ $? -eq 0 ] && echo ${i} is finished || echo ${i} is false
    done
}

main(){
    os
    ssh_key_push
}

main

配置ulimt

ulimit -SHn 65535

时间同步

yum -y install chrony tree
systemctl enable --now chronyd
chronyc sources

设置swap

echo vm.swappiness = 1 >> /etc/sysctl.conf
sysctl vm.swappiness=1
sysctl -p

安装jdk1.8

tar xf jdk-8u341-linux-x64.tar.gz -C /opt/
cat >> /etc/profile.d/java.sh << EOF
JAVA_HOME=/opt/jdk1.8.0_341
PATH=\$PATH:\$JAVA_HOME/bin:\$JAVA_HOME/sbin
EOF
source /etc/profile
java -version

安装ambari

配置httpd

yum -y install httpd
tar xf ambari-2.7.5.0-centos7.tar.gz -C /var/www/html/
tar xf HDP-3.1.5.0-centos7-rpm.tar.gz -C /var/www/html/
tar xf HDP-GPL-3.1.5.0-centos7-gpl.tar.gz -C /var/www/html/
tar xf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/

mkdir /var/www/html/libtirpc
mv libtirpc-0.2.4-0.16.el7.x86_64.rpm libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm /var/www/html/libtirpc/
yum -y install createrepo
createrepo /var/www/html/libtirpc/

systemctl enable --now httpd

http://192.168.10.134/ambari/
http://192.168.10.134/HDP/
http://192.168.10.134/HDP-GPL/
http://192.168.10.134/HDP-UTILS/
http://192.168.10.134/libtirpc/

配置repo

rm -f /etc/yum.repos.d/{ambari.repo,HDP.repo,libtirpc.repo}

cat >> /etc/yum.repos.d/ambari.repo << EOF
[Ambari-2.7.5.0]
name=Ambari-2.7.5.0
baseurl=http://192.168.10.134/ambari/centos7/2.7.5.0-72/
gpgcheck=0
enabled=1
priority=1
EOF

cat >> /etc/yum.repos.d/HDP.repo << EOF
[HDP-3.1.5.0]
name=HDP Version - HDP-3.1.5.0
baseurl=http://192.168.10.134/HDP/centos7/3.1.5.0-152/
gpgcheck=0
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.10.134/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
enabled=1
priority=1

[HDP-GPL-3.1.5.0]
name=HDP-GPL Version - HDP-GPL-3.1.5.0
baseurl=http://192.168.10.134/HDP-GPL/centos7/3.1.5.0-152
gpgcheck=0
enabled=1
priority=1
EOF

cat >> /etc/yum.repos.d/libtirpc.repo << EOF
[libtirpc_repo]
name=libtirpc-0.2.4-0.16
baseurl=http://192.168.10.134/libtirpc/
gpgcheck=0
enabled=1
priority=1
EOF

yum makecache
yum repolist

安装ambari-server

yum -y install ambari-server
ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2                  
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/jdk1.8.0_341
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? 
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK


About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.7.5.0.72.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

ambari-server start