Hadoop学习过程遇到的问题总结

最新推荐文章于 2023-04-10 15:05:45 发布

原创

最新推荐文章于 2023-04-10 15:05:45 发布 · 4.7k 阅读

3 ·

CC 4.0 BY-SA版权

这篇博客总结了在学习Hadoop过程中遇到的三个主要问题。问题一涉及mapred-site.xml配置，解决方法是在配置文件中添加相应设置。问题二是由于缺少htrace-core4-4.1.0-incubating.jar，通过下载并添加该jar解决了问题。问题三是运行mahout程序时遇到的类找不到问题，解决方案是避免将Hadoop配置文件放在src目录下，以免在eclipse中引起不必要的类加载和jar打包问题。

问题一

问题描述：

8/01/22 16:52:16 WARN Client: Failed to connect to server: master/192.168.217.128:10020: retries get failed due to exceeded maximum allowed retries number: 10
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:682)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:778)
	at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1544)
	at org.apache.hadoop.ipc.Client.call(Client.java:1375)
	at org.apache.hadoop.ipc.Client.call(Client.java:1339)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy15.getCounters(Unknown Source)
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.had