Sqoop部署

最新推荐文章于 2023-04-27 11:20:30 发布

原创最新推荐文章于 2023-04-27 11:20:30 发布 · 7.1k 阅读

0 ·

CC 4.0 BY-SA版权

Sqoop 专栏收录该内容

1 篇文章

订阅专栏

本文档详细介绍了在CDH5环境下配置Sqoop连接Hive的步骤，包括解压并创建软链接，将json和mysqljdbcjar包放入指定目录，拷贝hive相关jar包到Sqoop的lib目录，复制hive配置文件到Sqoop的conf目录，以及更新环境变量等关键操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

文章预览：

放一个安装包的地址： https://archive.cloudera.com/cdh5/cdh/5/

1.解压，做软连接

[hadoop@ruozedata001 app]$ ll
lrwxrwxrwx 1 hadoop hadoop 21 Dec 21 13:28 sqoop -> sqoop-1.4.6-cdh5.16.2
drwxr-xr-x 10 hadoop hadoop 4096 Jun 3 2019 sqoop-1.4.6-cdh5.16.2

2.放入json & mysql jdbc jar包

[hadoop@ruozedata001 app]$ cd sqoop/lib
[hadoop@ruozedata001 lib]$ ll
total 62120
-rwxr-xr-x@ 1 jepson staff 84697 Sep 25 20:21 java-json.jar
-rw-r--r-- 1 jepson staff 1007502 Sep 25 20:18 mysql-connector-java-5.1.47.jar

3.放入hive jar包

[hadoop@ruozedata001 lib]$ cp $HIVE_HOME/lib/hive-common-1.1.0-cdh5.16.2.jar $SQOO
P_HOME/lib
[hadoop@ruozedata001 lib]$ cp $HIVE_HOME/lib/hive-shims* $SQOOP_HOME/lib

4.hive配置⽂件

[hadoop@ruozedata001 lib]$ cp $HIVE_HOME/conf/hive-site.xml 
$SQOOP_HOME/conf
[hadoop@ruozedata001 lib]$ cd ../conf
[hadoop@ruozedata001 conf]$ cat sqoop-env.sh
#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/app/hive

5.配置环境变量

[hadoop@ruozedata001 ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
 . /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
export JAVA_HOME=/usr/java/jdk1.8.0_45
export HIVE_HOME=/home/hadoop/app/hive
export HIVE_CONF_DIR=$HIVE_HOME/conf
export HADOOP_HOME=/home/hadoop/app/hadoop
export HADOOP_INSTALL=/home/hadoop/app/hadoop
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export HBASE_HOME=/home/hadoop/app/hbase
export ZOOKEEPER_HOME=/home/hadoop/app/zookeeper
export PHOENIX_HOME=/home/hadoop/app/phoenix
export ELASTICSEARCH_HOME=/home/hadoop/app/elasticsearch
export KIBANA_HOME=/home/hadoop/app/kibana
export SQOOP_HOME=/home/hadoop/app/sqoop
export RDECK_BASE=/home/hadoop/app/rundeck
export PATH=$JAVA_HOME/bin:$SQOOP_HOME/bin:$ELASTICSEARCH_HOME/bin:$KIBANA_HOME/bi
n:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin:$PHOENIX_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOM
E/bin:$HADOOP_HOME/sbin:$PATH

[hadoop@ruozedata001 ~]$ source .bashrc