Sqoop

最新推荐文章于 2025-03-29 21:18:45 发布

chenxiaokang97

最新推荐文章于 2025-03-29 21:18:45 发布

阅读量795

点赞数 1

分类专栏： Hadoop

本文链接：https://blog.youkuaiyun.com/chen45682kang/article/details/81479220

版权

本文介绍了Sqoop的安装过程，包括下载、解压和环境变量配置。针对在导入MySQL数据时遇到的问题，如Streaming result set异常，解决方案是替换特定的JDBC连接jar包。文章详细阐述了Sqoop的导入原理，包括MapReduce作业的角色、数据类型映射、反序列化与序列化过程，并提到了水平切分的重要性。此外，还讨论了如何通过指定列、WHERE子句或自定义SQL进行更灵活的导入，并提供了将导入、建表和加载数据合并为一步的操作。最后，简单介绍了 Sqoop 的导出过程，强调了导出前需要在目标数据库创建接收表。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Sqoop

安装
- 下载，解压，配置环境变量
- conf里的配置不需要动，如果没有安装ZooKeeper和Hbase，就把configure-sqoop里有关zk和hbase的脚本全部注释掉；如果安装了zk和hbase，就不需要改。
导入，一个mysql的坑

我们导入hive表的DBS表

➜  sqoop git:(master) ✗ sqoop import --connect jdbc:mysql://localhost:3306/hive --table DBS --username root -password root

java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@3901d134 is still active.

Warning: /Users/chenxiaokang/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /Users/chenxiaokang/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/08/07 10:52:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/08/07 10:52:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/08/07 10:52:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/08/07 10:52:24 INFO tool.CodeGenTool: Beginning code generation
18/08/07 10:52:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `DBS` AS t LIMIT 1
18/08/07 10:52:25 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@3901d134 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@3901d134 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.

这是MySQL的一个bug，把（lib目录下）mysql的连接jar包mysql-connector-java-5.1.13-bin.jar换成mysql-connector-java-5.1.32.jar就好了。

➜  lib git:(master) ✗ sqoop import --connect jdbc:mysql://localhost:3306/hive --table DBS --username root -password root

Warning: /Users/chenxiaokang/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /Users/chenxiaokang/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/08/07 11:01:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/08/07 11:01:47 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/08/07 11:01:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/08/07 11:01:47 INFO tool.CodeGenTool: Beginning code generation
18/08/07 11:01:48 INFO

最低0.47元/天解锁文章