greenplum访问hdfs配置

最新推荐文章于 2023-05-22 09:41:26 发布

原创最新推荐文章于 2023-05-22 09:41:26 发布 · 2.1k 阅读

0 ·

CC 4.0 BY-SA版权

greenplum 专栏收录该内容

26 篇文章

订阅专栏

本文介绍如何在gpadmin用户环境下安装Greenplum，并完成Hadoop客户端、Java的配置，实现Greenplum与Hadoop环境的集成，包括设置环境变量、配置权限及创建外部表进行测试。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

#安装greenplum环境(gpadmin用户)
软件路径：/apps/greenplum
实例路径：/export/gpdata/

#安装hadoop客户端：
路径：/opt/cloudera/parcels/CDH-5.7.1-1

#安装java
/usr/java/jdk1.7.0_80

#切换至gpadmin用户
(1) 添加环境变量至.bash_profile
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/client

(2) 添加环境变量至greenplum的配置文件hadoop_env.sh中
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/client
source /apps/greenplum/greenplum_path.sh

(3)修改greenplum中的hadoop相差参数，修改后并重启动hadoop的权限
gpconfig -c gp_hadoop_target_version -v "cdh5"
gpconfig -c gp_hadoop_home -v "'/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/client'"
gpstop -u

以上操作是针对所有greenplum所有节点信息

#重启动greenplum集群环境信息
gpstop -a
gpstart -a

#为HDFS protocol赋权限
grant insert on protocol gphdfs to gpadmin;
grant select on protocol gphdfs to gpadmin;
grant all on protocol gphdfs to gpadmin;

#创建外部表进行测试
drop external table ext_devinfo;
CREATE EXTERNAL TABLE ext_devinfo(
devid varchar(100),
appid varchar(50)
)
LOCATION ('gphdfs://perf044:8020/user/gpadmin/devinfo.txt') format 'text' (delimiter '|');

CREATE table deviceinfo(
devid varchar(100),
appid varchar(50)
) distributed by (devid);

drop external table if exists ext_amb_mac_cnt;
CREATE external TABLE ext_amb_mac_cnt (
stat_date character varying(8),
macid character varying(64),
dtype character varying(64),
name character varying(32),
wifitype character varying(100),
devicetype character varying(32),
total bigint
) LOCATION ('gphdfs://perf044:8020/user/gpadmin/amb_mac_cnt_bak.txt') format 'text' (delimiter '|');