Hadoop学习-错误记录：TokenizerMapper not found

最新推荐文章于 2024-08-08 17:12:36 发布

摘果子的人

最新推荐文章于 2024-08-08 17:12:36 发布

阅读量1.6k

点赞数 3

CC 4.0 BY-SA版权

分类专栏：大数据文章标签： hadoop 解决方案

本文链接：https://blog.youkuaiyun.com/csdn_lzl/article/details/77985561

大数据专栏收录该内容

2 篇文章

订阅专栏

本文解决了WordCount程序运行时报出的ClassNotFoundException问题，错误原因是未能找到Mapper类。通过调整Eclipse项目的配置，移除多余的配置文件，成功解决了该问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

在运行WordCount程序时，报如题错误信息，在网上苦寻解决方案无果后，终于自己排查出了错误。发表在此给需要的人一个参考。
报错信息：

17/09/14 06:17:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/14 06:17:12 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.137.129:8032
17/09/14 06:17:13 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/09/14 06:17:13 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
17/09/14 06:17:13 INFO input.FileInputFormat: Total input paths to process : 9
17/09/14 06:17:13 INFO mapreduce.JobSubmitter: number of splits:9
17/09/14 06:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505394411892_0002
17/09/14 06:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
17/09/14 06:17:13 INFO impl.YarnClientImpl: Submitted application application_1505394411892_0002
17/09/14 06:17:13 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1505394411892_0002/
17/09/14 06:17:13 INFO mapreduce.Job: Running job: job_1505394411892_0002
17/09/14 06:17:22 INFO mapreduce.Job: Job job_1505394411892_0002 running in uber mode : false
17/09/14 06:17:22 INFO mapreduce.Job:  map 0% reduce 0%
17/09/14 06:17:26 INFO mapreduce.Job: Task Id : attempt_1505394411892_0002_m_000008_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class wordCount.WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class wordCount.WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
    ... 8 more

17/09/14 06:17:30 INFO mapreduce.Job: Task Id : attempt_1505394411892_0002_m_000008_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class wordCount.WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class wordCount.WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
    ... 8 more

报错原因及修正方法：在配置eclipse项目时，按照网上的教程提示，将所有的修改过的/etc/hadoop中的文件放到了eclipse工程src目录下面。
由于我撘的环境是全分布式集群，修改了core-site.xml,hdfs-site.xml,mapred-site.xml,slaves,yarn-site.xml这五个文件，所以我将这五个文件以及log4j.properties全部放到了src目录下，但是实际上只需要core-site.xml和hdfs-site.xml两个文件以及log4j.properties。
在删除多余的三个文件后，再次运行程序不再报错。

ps：具体为什么这三个文件会导致报TokenizerMapper not found错误目前尚不清楚，如果有大牛知道还请赐教！
pps：多种原因可能导致程序报这个错误，细心排查最重要。