Warning: $HADOOP_HOME is deprecated.的原因以及解决方法

本文探讨了Hadoop从0.20.2版本升级到1.2.1版本后出现的警告:“Warning:$HADOOP_HOMEisdeprecated.”。解释了Apache官方为何弃用HADOOP_HOME环境变量,并介绍了新的环境变量HADOOP_PREFIX。提供了两种消除警告的方法。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

今天把hadoop从0.20.2版本升级到1.2.1版本之后,在很多时候都会报如下警告:Warning: $HADOOP_HOME is deprecated.

        其实我刚看到这个警告的时候,第一反应并不是怎么屏蔽掉它,而是想知道Apache为什么准备废弃掉HADOOP_HOME这个环境变量,因为我们知道,后面好多地方都会用到这个环境变量。

        在Google上搜到的结果都是关于如何屏蔽掉这个警告的,据说这个警告会影响到后面Hive程序的执行,没有人解释官方为什么会给这么一个警告,所以就专门写一篇博客来替官方解释一下。

        下面是我在官方网站上搜索到的一封邮件,有人关于这个警告提问,官方人员给出的回复如下:邮件

         Due to a new packaging format, the ApacheHadoop 1.x hasdeprecated
         the HADOOP_HOME env-var in favor of a new env-var called
         'HADOOP_PREFIX'. You can setHADOOP_PREFIX, or set
         HADOOP_HOME_WARN_SUPPRESS in your environment to a non-empty value to
         suppress the warning.

        里面提到了从1.x版本起,HADOOP_HOME环境变量由HADOOP_PREFIX这个环境变量替代。在1.2.1版本的所有配置文件里我搜索“HADOOP_PREFIX”的结果如下:

        由这些结果可以看出来,新版本已经用HADOOP_PREFIX这个环境变量替代HADOOP_HOME了。

解决办法:

        1、在conf/hadoop-env.sh中加入(本人测试只有在hadoop-env.sh中加入才有效,在/etc/profile、~/.bashrc等中加入均无效,不知什么原因,如果哪位朋友知道希望不吝赐教):

              export HADOOP_HOME_WARN_SUPPRESS=1

              还有其他人选择将bin/hadoop-config.sh中的相关提示代码注释掉,但本人不建议这样做。

        2、用环境变量HADOOP_PREFIX替代HADOOP_HOME,但本人建议保留HADOOP_HOME,因为Pig和Hive程序中或许会用到这个环境变量。

[root@localhost sqoop-1.4.7.bin__hadoop-2.6.0]# bin/sqoop export --connect jdbc:mysql://localhost:3306/dbtaobao --username root --password 123456 --table user_log --export-dir '/user/hive/warehouse/dbtaobao.db/inner_user_log' --fields-terminated-by ','; Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 2025-06-06 00:44:19,902 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-06-06 00:44:20,015 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-06-06 00:44:20,281 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-06-06 00:44:20,287 INFO tool.CodeGenTool: Beginning code generation Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-06-06 00:44:22,137 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_log` AS t LIMIT 1 2025-06-06 00:44:22,270 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_log` AS t LIMIT 1 2025-06-06 00:44:22,293 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hadoop/hadoop-3.1.3 注: /tmp/sqoop-root/compile/2647ebe9a3777fcaa95ca65a919294ec/user_log.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 2025-06-06 00:44:26,610 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/2647ebe9a3777fcaa95ca65a919294ec/user_log.jar 2025-06-06 00:44:26,629 INFO mapreduce.ExportJobBase: Beginning export of user_log 2025-06-06 00:44:26,629 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-06-06 00:44:27,018 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-06-06 00:44:28,689 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:28,961 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-06-06 00:44:28,966 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-06-06 00:44:28,966 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-06-06 00:44:29,494 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2025-06-06 00:44:29,762 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2025-06-06 00:44:29,762 INFO impl.MetricsSystemImpl: JobTracker metrics system started 2025-06-06 00:44:30,031 INFO input.FileInputFormat: Total input files to process : 1 2025-06-06 00:44:30,053 INFO input.FileInputFormat: Total input files to process : 1 2025-06-06 00:44:30,136 INFO mapreduce.JobSubmitter: number of splits:4 2025-06-06 00:44:30,284 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-06-06 00:44:30,660 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local583218486_0001 2025-06-06 00:44:30,660 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-06-06 00:44:31,271 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/local/1749141870870/libjars <- /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/libjars/* 2025-06-06 00:44:31,278 WARN fs.FileUtil: Command 'ln -s /tmp/hadoop-root/mapred/local/1749141870870/libjars /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/libjars/*' failed 1 with: ln: 无法创建符号链接"/usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/libjars/*": 没有那个文件或目录 2025-06-06 00:44:31,278 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /tmp/hadoop-root/mapred/local/1749141870870/libjars <- /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/libjars/* 2025-06-06 00:44:31,278 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/root583218486/.staging/job_local583218486_0001/libjars as file:/tmp/hadoop-root/mapred/local/1749141870870/libjars 2025-06-06 00:44:31,569 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 2025-06-06 00:44:31,571 INFO mapreduce.Job: Running job: job_local583218486_0001 2025-06-06 00:44:31,614 INFO mapred.LocalJobRunner: OutputCommitter set in config null 2025-06-06 00:44:31,631 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.sqoop.mapreduce.NullOutputCommitter 2025-06-06 00:44:31,798 INFO mapred.LocalJobRunner: Waiting for map tasks 2025-06-06 00:44:31,802 INFO mapred.LocalJobRunner: Starting task: attempt_local583218486_0001_m_000000_0 2025-06-06 00:44:32,000 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-06 00:44:32,006 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:355065+59179,/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:414244+59179 2025-06-06 00:44:32,012 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2025-06-06 00:44:32,012 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2025-06-06 00:44:32,012 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2025-06-06 00:44:32,050 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:32,260 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:32,299 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-06 00:44:32,328 INFO mapred.LocalJobRunner: Starting task: attempt_local583218486_0001_m_000001_0 2025-06-06 00:44:32,357 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-06 00:44:32,359 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:0+118355 2025-06-06 00:44:32,387 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:32,524 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-06 00:44:32,553 INFO mapred.LocalJobRunner: Starting task: attempt_local583218486_0001_m_000002_0 2025-06-06 00:44:32,566 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-06 00:44:32,567 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:118355+118355 2025-06-06 00:44:32,585 INFO mapreduce.Job: Job job_local583218486_0001 running in uber mode : false 2025-06-06 00:44:32,587 INFO mapreduce.Job: map 0% reduce 0% 2025-06-06 00:44:32,616 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:32,745 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:32,799 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-06 00:44:32,866 INFO mapred.LocalJobRunner: Starting task: attempt_local583218486_0001_m_000003_0 2025-06-06 00:44:32,901 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-06 00:44:32,903 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:236710+118355 2025-06-06 00:44:32,930 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:33,004 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-06 00:44:33,034 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-06 00:44:33,049 INFO mapred.LocalJobRunner: map task executor complete. 2025-06-06 00:44:33,050 WARN mapred.LocalJobRunner: job_local583218486_0001 java.lang.Exception: java.io.IOException: java.lang.ClassNotFoundException: user_log at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) Caused by: java.io.IOException: java.lang.ClassNotFoundException: user_log at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:74) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: user_log at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:70) ... 10 more 2025-06-06 00:44:33,590 INFO mapreduce.Job: Job job_local583218486_0001 failed with state FAILED due to: NA 2025-06-06 00:44:33,599 INFO mapreduce.Job: Counters: 0 2025-06-06 00:44:33,623 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-06-06 00:44:33,625 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 4.6274 seconds (0 bytes/sec) 2025-06-06 00:44:33,626 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-06-06 00:44:33,626 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-06-06 00:44:33,626 ERROR mapreduce.ExportJobBase: Export job failed! 2025-06-06 00:44:33,626 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [root@localhost sqoop-1.4.7.bin__hadoop-2.6.0]# 给我解决方案
06-07
数据从 hive 复制到 mysql 中报错hadoop@t3l-VirtualBox:/usr/local/sqoop$ ./bin/sqoop export --connect jdbc:mysql://localhost:3306/dblab --username root --password '*-+' --table user_action --export-dir '/user/hive/warehouse/dblab.db/user_action' --fields-terminated-by '\t'; Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /usr/local/sqoop/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.GetJavaProperty 2025-08-08 16:00:57,770 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 2025-08-08 16:00:57,835 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-08-08 16:00:58,042 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-08-08 16:00:58,042 INFO tool.CodeGenTool: Beginning code generation Fri Aug 08 16:00:58 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-08-08 16:00:59,101 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_action` AS t LIMIT 1 2025-08-08 16:00:59,171 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_action` AS t LIMIT 1 2025-08-08 16:00:59,210 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop 注: /tmp/sqoop-hadoop/compile/e5381a60ac3daedca9188e7ac7d80584/user_action.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 2025-08-08 16:01:01,847 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/e5381a60ac3daedca9188e7ac7d80584/user_action.jar 2025-08-08 16:01:01,853 INFO mapreduce.ExportJobBase: Beginning export of user_action 2025-08-08 16:01:01,854 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-08-08 16:01:02,095 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-08-08 16:01:03,097 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-08-08 16:01:03,334 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-08-08 16:01:03,337 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-08-08 16:01:03,342 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-08-08 16:01:03,692 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2025-08-08 16:01:03,859 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2025-08-08 16:01:03,859 INFO impl.MetricsSystemImpl: JobTracker metrics system started 2025-08-08 16:01:04,058 INFO input.FileInputFormat: Total input files to process : 1 2025-08-08 16:01:04,065 INFO input.FileInputFormat: Total input files to process : 1 2025-08-08 16:01:04,168 INFO mapreduce.JobSubmitter: number of splits:4 2025-08-08 16:01:04,275 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-08-08 16:01:04,481 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1968709802_0001 2025-08-08 16:01:04,485 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-08-08 16:01:04,820 INFO mapred.LocalDistributedCacheManager: Creating symlink: /usr/local/hadoop/tmp/mapred/local/1754640064639/libjars <- /usr/local/sqoop/libjars/* 2025-08-08 16:01:04,827 WARN fs.FileUtil: Command 'ln -s /usr/local/hadoop/tmp/mapred/local/1754640064639/libjars /usr/local/sqoop/libjars/*' failed 1 with: ln: 无法创建符号链接'/usr/local/sqoop/libjars/*': 没有那个文件或目录 2025-08-08 16:01:04,827 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /usr/local/hadoop/tmp/mapred/local/1754640064639/libjars <- /usr/local/sqoop/libjars/* 2025-08-08 16:01:04,827 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hadoop1968709802/.staging/job_local1968709802_0001/libjars as file:/usr/local/hadoop/tmp/mapred/local/1754640064639/libjars 2025-08-08 16:01:04,958 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 2025-08-08 16:01:04,959 INFO mapreduce.Job: Running job: job_local1968709802_0001 2025-08-08 16:01:04,962 INFO mapred.LocalJobRunner: OutputCommitter set in config null 2025-08-08 16:01:04,990 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.sqoop.mapreduce.NullOutputCommitter 2025-08-08 16:01:05,123 INFO mapred.LocalJobRunner: Waiting for map tasks 2025-08-08 16:01:05,130 INFO mapred.LocalJobRunner: Starting task: attempt_local1968709802_0001_m_000000_0 2025-08-08 16:01:05,245 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-08-08 16:01:05,253 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:0+3897281 2025-08-08 16:01:05,259 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2025-08-08 16:01:05,259 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2025-08-08 16:01:05,259 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2025-08-08 16:01:05,277 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Fri Aug 08 16:01:05 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-08-08 16:01:05,392 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-08-08 16:01:05,396 INFO mapred.LocalJobRunner: Starting task: attempt_local1968709802_0001_m_000001_0 2025-08-08 16:01:05,398 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-08-08 16:01:05,398 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:3897281+3897281 2025-08-08 16:01:05,422 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Fri Aug 08 16:01:05 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-08-08 16:01:05,509 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-08-08 16:01:05,515 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-08-08 16:01:05,523 INFO mapred.LocalJobRunner: Starting task: attempt_local1968709802_0001_m_000002_0 2025-08-08 16:01:05,527 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-08-08 16:01:05,544 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:7794562+3897281 2025-08-08 16:01:05,557 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Fri Aug 08 16:01:05 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-08-08 16:01:05,635 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-08-08 16:01:05,648 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-08-08 16:01:05,652 INFO mapred.LocalJobRunner: Starting task: attempt_local1968709802_0001_m_000003_0 2025-08-08 16:01:05,655 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-08-08 16:01:05,674 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:11691843+3897281 2025-08-08 16:01:05,697 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Fri Aug 08 16:01:05 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-08-08 16:01:05,768 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-08-08 16:01:05,782 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-08-08 16:01:05,790 INFO mapred.LocalJobRunner: map task executor complete. 2025-08-08 16:01:05,792 WARN mapred.LocalJobRunner: job_local1968709802_0001 java.lang.Exception: java.io.IOException: java.lang.ClassNotFoundException: user_action at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) Caused by: java.io.IOException: java.lang.ClassNotFoundException: user_action at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:70) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: user_action at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:66) ... 10 more 2025-08-08 16:01:05,962 INFO mapreduce.Job: Job job_local1968709802_0001 running in uber mode : false 2025-08-08 16:01:05,965 INFO mapreduce.Job: map 0% reduce 0% 2025-08-08 16:01:05,967 INFO mapreduce.Job: Job job_local1968709802_0001 failed with state FAILED due to: NA 2025-08-08 16:01:05,973 INFO mapreduce.Job: Counters: 0 2025-08-08 16:01:06,018 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-08-08 16:01:06,022 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 2.6143 seconds (0 bytes/sec) 2025-08-08 16:01:06,036 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-08-08 16:01:06,036 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-08-08 16:01:06,036 ERROR tool.ExportTool: Error during export: Export job failed!
最新发布
08-09
从hdfs到mysql报错Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /usr/local/sqoop/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.GetJavaProperty 2025-06-05 17:41:55,556 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 2025-06-05 17:41:55,583 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-06-05 17:41:55,664 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-06-05 17:41:55,664 INFO tool.CodeGenTool: Using existing jar: /tmp/sqoop-hadoop/compile/8b77c7c2bee6c4dd7f3856618f5f8556/user_action.jar 2025-06-05 17:41:55,668 INFO mapreduce.ExportJobBase: Beginning export of user_action 2025-06-05 17:41:55,668 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-06-05 17:41:55,827 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-06-05 17:41:56,366 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Thu Jun 05 17:41:56 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-06-05 17:41:56,900 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-06-05 17:41:56,902 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-06-05 17:41:56,904 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-06-05 17:41:56,967 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2025-06-05 17:41:57,035 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2025-06-05 17:41:57,035 INFO impl.MetricsSystemImpl: JobTracker metrics system started 2025-06-05 17:41:57,104 INFO input.FileInputFormat: Total input files to process : 1 2025-06-05 17:41:57,111 INFO input.FileInputFormat: Total input files to process : 1 2025-06-05 17:41:57,139 INFO mapreduce.JobSubmitter: number of splits:4 2025-06-05 17:41:57,159 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-06-05 17:41:57,236 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1705881514_0001 2025-06-05 17:41:57,236 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-06-05 17:41:57,374 INFO mapred.LocalDistributedCacheManager: Creating symlink: /usr/local/hadoop/tmp/mapred/local/1749116517294/libjars <- /usr/local/sqoop/libjars/* 2025-06-05 17:41:57,376 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hadoop1705881514/.staging/job_local1705881514_0001/libjars as file:/usr/local/hadoop/tmp/mapred/local/1749116517294/libjars 2025-06-05 17:41:57,408 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 2025-06-05 17:41:57,409 INFO mapreduce.Job: Running job: job_local1705881514_0001 2025-06-05 17:41:57,413 INFO mapred.LocalJobRunner: OutputCommitter set in config null 2025-06-05 17:41:57,416 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.sqoop.mapreduce.NullOutputCommitter 2025-06-05 17:41:57,455 INFO mapred.LocalJobRunner: Waiting for map tasks 2025-06-05 17:41:57,456 INFO mapred.LocalJobRunner: Starting task: attempt_local1705881514_0001_m_000000_0 2025-06-05 17:41:57,492 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-05 17:41:57,499 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:11692413+1948736,/user/hive/warehouse/dblab.db/user_action/000000_0:13641149+1948737 2025-06-05 17:41:57,507 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2025-06-05 17:41:57,507 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2025-06-05 17:41:57,507 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2025-06-05 17:41:57,515 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Thu Jun 05 17:41:57 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-06-05 17:41:57,543 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-05 17:41:57,548 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-05 17:41:57,551 INFO mapred.LocalJobRunner: Starting task: attempt_local1705881514_0001_m_000001_0 2025-06-05 17:41:57,552 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-05 17:41:57,553 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:0+3897471 2025-06-05 17:41:57,562 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Thu Jun 05 17:41:57 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-06-05 17:41:57,574 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-05 17:41:57,577 INFO mapred.LocalJobRunner: Starting task: attempt_local1705881514_0001_m_000002_0 2025-06-05 17:41:57,578 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-05 17:41:57,579 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:3897471+3897471 2025-06-05 17:41:57,583 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Thu Jun 05 17:41:57 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-06-05 17:41:57,603 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-05 17:41:57,607 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-05 17:41:57,614 INFO mapred.LocalJobRunner: Starting task: attempt_local1705881514_0001_m_000003_0 2025-06-05 17:41:57,616 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2025-06-05 17:41:57,617 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dblab.db/user_action/000000_0:7794942+3897471 2025-06-05 17:41:57,623 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false Thu Jun 05 17:41:57 CST 2025 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2025-06-05 17:41:57,638 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2025-06-05 17:41:57,640 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false 2025-06-05 17:41:57,641 INFO mapred.LocalJobRunner: map task executor complete. 2025-06-05 17:41:57,642 WARN mapred.LocalJobRunner: job_local1705881514_0001 java.lang.Exception: java.io.IOException: java.lang.ClassNotFoundException: user_action at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) Caused by: java.io.IOException: java.lang.ClassNotFoundException: user_action at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:70) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: user_action at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:66) ... 10 more 2025-06-05 17:41:58,413 INFO mapreduce.Job: Job job_local1705881514_0001 running in uber mode : false 2025-06-05 17:41:58,414 INFO mapreduce.Job: map 0% reduce 0% 2025-06-05 17:41:58,416 INFO mapreduce.Job: Job job_local1705881514_0001 failed with state FAILED due to: NA 2025-06-05 17:41:58,422 INFO mapreduce.Job: Counters: 0 2025-06-05 17:41:58,432 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-06-05 17:41:58,434 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 1.5124 seconds (0 bytes/sec) 2025-06-05 17:41:58,436 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-06-05 17:41:58,436 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-06-05 17:41:58,437 ERROR tool.ExportTool: Error during export: Export job failed!
06-06
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值