sqoop 1.4.7部署及报错解决

部署

# 下载
https://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

# 解压
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/

# 设置环境变量，并应用生效
echo 'export SQOOP_HOME=/opt/sqoop-1.4.7.bin__hadoop-2.6.0' >> /etc/profile
echo 'export PATH="$SQOOP_HOME/bin:$PATH"' >> /etc/profile
echo 'export CLASSPATH="$SQOOP_HOME/lib:$CLASSPATH"' >> /etc/profile
source /etc/profile

# 复制配置模板文件，并编辑
cp $SQOOP_HOME/conf/sqoop-env-template.sh $SQOOP_HOME/conf/sqoop-env.sh
vi $SQOOP_HOME/conf/sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop-3.3.0

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop-3.3.0

#set the path to where bin/hbase is available
export HBASE_HOME=/opt/hbase-2.5.6

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive-3.1.3

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/zookeeper-3.7.2

# 测试sqoop是否能够正常打开，会输出版本
sqoop version

# 复制hive内的mysql连接jar包到sqoop内
cp /opt/hive-3.1.3/lib/mysql-connector-j-8.0.33.jar  $SQOOP_HOME/lib/

# 
sqoop list-databases --connect jdbc:mysql://192.168.10.75:3306/ --username root -password HoMf@123

报错

java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils

• 原因：
  Sqoop1.4.7默认只加载了commons-lang3-3.4.jar的jar包，里面的StringUtils类的package为：org/apache/commons/lang3/StringUtils，所以直接使用sqoop命令时报上述错误。

[root@linux-master ~]# sqoop list-databases --connect jdbc:mysql://192.168.10.75:3306/ --username root -password HoMf@123
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2024-11-21 10:30:16,518 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2024-11-21 10:30:16,630 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2024-11-21 10:30:16,711 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
        at org.apache.sqoop.manager.MySQLManager.initOptionDefaults(MySQLManager.java:73)
        at org.apache.sqoop.manager.SqlManager.<init>(SqlManager.java:89)
        at com.cloudera.sqoop.manager.SqlManager.<init>(SqlManager.java:33)
        at org.apache.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:51)
        at com.cloudera.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:30)
        at org.apache.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:46)
        at com.cloudera.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:31)
        at org.apache.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:38)
        at com.cloudera.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:31)
        at org.apache.sqoop.manager.MySQLManager.<init>(MySQLManager.java:65)
        at org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)
        at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
        at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)
        at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:359)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 20 more

• 解决:
  将压缩包内的commons-lang-2.6.jar导入到sqoop目录下的lib目录下即可
  下载地址：
  http://mirrors.tuna.tsinghua.edu.cn/apache//commons/lang/binaries/commons-lang-2.6-bin.zip

Warning警告

Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

• 解决：
  进入$SQOOP_HOME/bin下，修改configure-sqoop文件，将下面的内容注释掉

vi $SQOOP_HOME/bin/configure-sqoop

## Moved to be a runtime check in sqoop.
##
if [ ! -d "${HCAT_HOME}" ]; then
  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
fi

if [ ! -d "${ACCUMULO_HOME}" ]; then
  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
fi
if [ ! -d "${ZOOKEEPER_HOME}" ]; then
  echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
  echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
fi
##

转换数据过程中问题

2024-11-21 14:23:20,771 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
2024-11-21 14:23:20,772 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

• 原因：
  sqoop缺少hive jar包

• 解决：
  将hive/lib 包下的hive-common-x.x.x.jar和 hive-exec-x.x.x.jar 拷贝到sqoop/lib包下
  x.x.x代表具体的版本，hive版本不同jar包版本也就会不同

[root@linux-master ~]# cp /opt/hive-3.1.3/lib/hive-common-3.1.3.jar  /opt/sqoop-1.4.7.bin__hadoop-2.6.0/lib/
[root@linux-master ~]# cp /opt/hive-3.1.3/lib/hive-exec-3.1.3.jar  /opt/sqoop-1.4.7.bin__hadoop-2.6.0/lib/