presto适配arm环境

最新推荐文章于 2025-04-14 22:26:17 发布

何亚告

最新推荐文章于 2025-04-14 22:26:17 发布

阅读量1k

点赞数 1

分类专栏：菜鸟日常文章标签： hive hadoop arm开发

本文链接：https://blog.youkuaiyun.com/qq_44229840/article/details/130015897

版权

菜鸟日常专栏收录该内容

30 篇文章

订阅专栏

Presto适配arm环境

安装jdk1.8

Presto在java17及更高版本中编译会出现如下报错，需要安装jdk1.8

链接: jdk1.8

[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:3.1.10:spotbugs (spotbugs) on project hadoop-apache2: Execution spotbugs of goal com.github.spotbugs:spotbugs-maven-plugin:3.1.10:spotbugs failed: Unable to load the mojo 'spotbugs' in the plugin 'com.github.spotbugs:spotbugs-maven-plugin:3.1.10'. A required class is missing: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7

cd /usr/local
tar -zxvf jdk-8u361-linux-aarch64.tar.gz

#vim /etc/profile

export JAVA_HOME=/usr/local/jdk1.8
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile

java -version

在这里插入图片描述

安装maven3

链接: apache-maven-3.8.6-bin.tar.gz

cd /usr/local
tar -zxvf apache-maven-3.8.6-bin.tar.gz

vim /etc/profile

export MAVEN_HOME=/usr/local/apache-maven-3.8.6
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"
export PATH=$MAVEN_HOME/bin:$PATH

source /etc/profile

mvn -version

在这里插入图片描述

安装protobuf 2.5

presto适配arm环境需要presto-hadoop-apache2，presto-hadoop-apache2的编译需要在protobuf 2.5版本中执行。protobuf2.5版本官方没有适配arm环境，需要通过第三方patch打补丁后安装。

参考: ARM64 系统中编译安装Hadoop · Issue #12 · liusheng/liusheng.github.io · GitHub

protobuf 2.5
ARM64 系统中编译安装Hadoop · Issue #12 · liusheng/liusheng.github.io · GitHub
yum install -y build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libsasl2-dev
wget -O protobuf-2.5.0-arm64.patch https://git.io/JeObp
wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2

#因为缺少bzip包
yum install -y bzip2
tar xvfj protobuf-2.5.0.tar.bz2

#因为缺少patch包
cd protobuf-2.5.0
yum -y install patch
patch -p1 < ../protobuf-2.5.0-arm64.patch

./autogen.sh
./configure --prefix=/usr
#报错configure: error: C++ preprocessor "/lib/cpp" fails sanity check
yum install gcc-c++
make
make check
make install
protoc --version

在这里插入图片描述

编译presto-hadoop-apache2

git clone https://github.com/prestodb/presto-hadoop-apache2.git

需要最新版：presto-hadoop-apache2:2.7.4-12-SNAPSHOT

mvn install

编译报错需要检查jdk版本是否过高或者跳过测试：

mvn install -DskipTests

编译后可以在本地仓库找到对应的包：

#pwd
/root/.m2/repository/com/facebook/presto/hadoop/hadoop-apache2

在这里插入图片描述

编译presto

参考: Support ARM in Presto by ChunxuTang · Pull Request #18960 · prestodb/presto (github.com)
下载presto源代码压缩包解压:

#https://github.com/prestodb/presto
tar -zxvf presto-0.280.tar.gz

pom.xml添加

            <dependency>
                <groupId>org.fusesource.jansi</groupId>
                <artifactId>jansi</artifactId>
                <version>1.8</version>
            </dependency>

            <plugin>
                    <groupId>pl.project13.maven</groupId>
                    <artifactId>git-commit-id-plugin</artifactId>
                    <configuration>
                        <failOnNoGitDirectory>false</failOnNoGitDirectory>
                        <runOnlyOnce>true</runOnlyOnce>
                        <injectAllReactorProjects>true</injectAllReactorProjects>
                        <offline>true</offline>
                        <useNativeGit>true</useNativeGit>
                    </configuration>
            </plugin>

修改：版本与presto-hadoop-apache2对应

            <dependency>
                <groupId>com.facebook.presto.hadoop</groupId>
                <artifactId>hadoop-apache2</artifactId>
                <version>2.7.4-12-SNAPSHOT</version>
            </dependency>

presto-cli/pom.xml添加

        <dependency>
            <groupId>org.fusesource.jansi</groupId>
            <artifactId>jansi</artifactId>
        </dependency>

            <plugin>
                <groupId>org.basepom.maven</groupId>
                <artifactId>duplicate-finder-maven-plugin</artifactId>
                <configuration>
                    <ignoredDependencies>
                        <dependency>
                            <groupId>org.fusesource.jansi</groupId>
                            <artifactId>jansi</artifactId>
                        </dependency>
                    </ignoredDependencies>
                </configuration>
            </plugin>

修改：presto-main/src/main/java/com/facebook/presto/server/PrestoSystemRequirements.java

            if (!"amd64".equals(osArch) && !"ppc64le".equals(osArch) && !"aarch64".equals(osArch)) {
                failRequirement("Presto requires amd64 or ppc64le or aarch64 on Linux (found %s)", osArch);

           if (!"x86_64".equals(osArch) && !"aarch64".equals(osArch)) {
                failRequirement("Presto requires x86_64 or aarch64 on Mac OS X (found %s)", osArch);

编译：

mvn clean package -DskipTests -Dmaven.javadoc.skip=true  -pl '!:presto-docs'