尝试用sqoop将mysql数据导入hdfs中失败==

在尝试使用 Sqoop 命令行工具将 MySQL 数据导入到 HDFS 中时遇到错误,警告提示 HBase、HCatalog、Accumulo 和 Zookeeper 的安装路径不存在。此外,还出现了 SSL 连接警告和连接超时问题。尽管 Sqoop 版本为 1.4.7,但任务在 MapReduce 阶段失败,导致通信链接故障。问题尚未解决,对初学者来说,处理 Linux 环境中的这类问题颇具挑战性。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

使用命令行:sqoop import --connect jdbc:mysql://*.*.*.*:3306/test?characterEncoding=UTF-8 --username aaa--password 'bbb' -table lll

尝试将mysql的数据导入到hdfs中,结果一直导入失败,并显示:

Warning: /usr/local/sqoop/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop/sqoop/../hcatalog does not exist! HCatalog jobs will fa                                                                                                                          il.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/sqoop/../accumulo does not exist! Accumulo imports will                                                                                                                           fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop/sqoop/../zookeeper does not exist! Accumulo imports wil                                                                                                                          l fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2019-06-11 02:41:42,287 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2019-06-11 02:41:42,500 WARN tool.BaseSqoopTool: Setting your password on the com                                                                                                                          mand-line is insecure. Consider using -P instead.
2019-06-11 02:41:42,753 INFO manager.MySQLManager: Preparing to use a MySQL strea                                                                                                                          ming resultset.
2019-06-11 02:41:42,753 INFO tool.CodeGenTool: Beginning code generation
Tue Jun 11 02:41:43 EDT 2019 WARN: Establishing SSL connection without server's i                                                                                                                          dentity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and                                                                                                                           5.7.6+ requirements SSL connection must be established by default if explicit opt                                                                                                                          ion isn't set. For compliance with existing applications not using SSL the verify                                                                                                                          ServerCertificate property is set to 'false'. You need either to explicitly disab                                                                                                                          le SSL by setting useSSL=false, or set useSSL=true and provide truststore for ser                                                                                                                          ver certificate verification.
2019-06-11 02:41:45,892 INFO manager.SqlManager: Executing SQL statement: SELECT                                                                                                                           t.* FROM `data_weilai_pressure` AS t LIMIT 1
2019-06-11 02:41:47,326 INFO manager.SqlManager: Executing SQL statement: SELECT                                                                                                                           t.* FROM `data_weilai_pressure` AS t LIMIT 1
2019-06-11 02:41:48,010 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/l                                                                                                                          ocal/hadoop
Note: /tmp/sqoop-myy/compile/abd370f29650c545e0e82607b93e8cf0/data_weilai_pressur                                                                                                                          e.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2019-06-11 02:41:51,300 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop                                                                                                                          -myy/compile/abd370f29650c545e0e82607b93e8cf0/data_weilai_pressure.jar
2019-06-11 02:41:51,434 WARN manager.MySQLManager: It looks like you are importin                                                                                                                          g from mysql.
2019-06-11 02:41:51,440 WARN manager.MySQLManager: This transfer can be faster! U                                                                                                                          se the --direct
2019-06-11 02:41:51,440 WARN manager.MySQLManager: option to exercise a MySQL-spe                                                                                                                          cific fast path.
2019-06-11 02:41:51,440 INFO manager.MySQLManager: Setting zero DATETIME behavior                                                                                                                           to convertToNull (mysql)
2019-06-11 02:41:52,127 INFO mapreduce.ImportJobBase: Beginning import of data_we                                                                                                                          ilai_pressure
2019-06-11 02:41:52,129 INFO Configuration.deprecation: mapred.job.tracker is dep                                                                                                                          recated. Instead, use mapreduce.jobtracker.address
2019-06-11 02:41:52,386 INFO Configuration.deprecation: mapred.jar is deprecated.                                                                                                                           Instead, use mapreduce.job.jar
2019-06-11 02:41:53,971 INFO Configuration.deprecation: mapred.map.tasks is depre                                                                                                                          cated. Instead, use mapreduce.job.maps
2019-06-11 02:41:54,129 INFO client.RMProxy: Connecting to ResourceManager at mas                                                                                                                          ter/10.141.200.75:8032
2019-06-11 02:41:55,024 INFO mapreduce.JobResourceUploader: Disabling Erasure Cod                                                                                                                          ing for path: /tmp/hadoop-yarn/staging/myy/.staging/job_1560165070962_0011
Tue Jun 11 02:42:06 EDT 2019 WARN: Establishing SSL connection without server's i                                                                                                                          dentity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and                                                                                                                           5.7.6+ requirements SSL connection must be established by default if explicit opt                                                                                                                          ion isn't set. For compliance with existing applications not using SSL the verify                                                                                                                          ServerCertificate property is set to 'false'. You need either to explicitly disab                                                                                                                          le SSL by setting useSSL=false, or set useSSL=true and provide truststore for ser                                                                                                                          ver certificate verification.
2019-06-11 02:42:07,808 INFO db.DBInputFormat: Using read commited transaction is                                                                                                                          olation
2019-06-11 02:42:08,425 INFO mapreduce.JobSubmitter: number of splits:1
2019-06-11 02:42:08,530 INFO Configuration.deprecation: yarn.resourcemanager.syst                                                                                                                          em-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publ                                                                                                                          isher.enabled
2019-06-11 02:42:08,879 INFO mapreduce.JobSubmitter: Submitting tokens for job: j                                                                                                                          ob_1560165070962_0011
2019-06-11 02:42:08,882 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-06-11 02:42:09,245 INFO conf.Configuration: resource-types.xml not found
2019-06-11 02:42:09,246 INFO resource.ResourceUtils: Unable to find 'resource-typ                                                                                                                          es.xml'.
2019-06-11 02:42:09,391 INFO impl.YarnClientImpl: Submitted application applicati                                                                                                                          on_1560165070962_0011
2019-06-11 02:42:09,507 INFO mapreduce.Job: The url to track the job: http://mast                                                                                                                          er:8088/proxy/application_1560165070962_0011/
2019-06-11 02:42:09,509 INFO mapreduce.Job: Running job: job_1560165070962_0011
2019-06-11 02:42:21,686 INFO mapreduce.Job: Job job_1560165070962_0011 running in                                                                                                                           uber mode : false
2019-06-11 02:42:21,688 INFO mapreduce.Job:  map 0% reduce 0%
2019-06-11 02:44:35,691 INFO mapreduce.Job: Task Id : attempt_1560165070962_0011_                                                                                                                          m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exc                                                                                                                          eptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The drive                                                                                                                          r has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.ja                                                                                                                          va:170)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java                                                                                                                          :161)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77                                                                                                                          )
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.jav                                                                                                                          a:137)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat                                                                                                                          ion.java:1730)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communicat                                                                                                                          ionsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The drive                                                                                                                          r has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputForma                                                                                                                          t.java:223)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.ja                                                                                                                          va:168)
        ... 10 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communication                                                                                                                          s link failure

The last packet sent successfully to the server was 0 milliseconds ago. The drive                                                                                                                          r has not received any packets from the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo                                                                                                                          rAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo                                                                                                                          nstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:99                                                                                                                          0)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:22                                                                                                                          30)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo                                                                                                                          rAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo                                                                                                                          nstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:                                                                                                                          330)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfigur                                                                                                                          ation.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputForma                                                                                                                          t.java:216)
        ... 11 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav                                                                                                                          a:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI                                                                                                                          mpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:                                                                                                                          188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.jav                                                                                                                          a:211)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
        ... 27 more

2019-06-11 02:46:50,403 INFO mapreduce.Job: Task Id : attempt_1560165070962_0011_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:170)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:223)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168)
        ... 10 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:216)
        ... 11 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
        ... 27 more

2019-06-11 02:49:02,941 INFO mapreduce.Job: Task Id : attempt_1560165070962_0011_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:170)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:223)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168)
        ... 10 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:216)
        ... 11 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
        ... 27 more

 

目前还没有找到原因,好气!!!!初学者用linux简直像是在炼丹!!

 

评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值