MapReduce 动态设置 namenode HDFS 的URI

最新推荐文章于 2022-01-11 19:19:50 发布

原创最新推荐文章于 2022-01-11 19:19:50 发布 · 366 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#java #大数据 #开发工具

HBase 同时被 3 个专栏收录

11 篇文章

订阅专栏

hadoop

11 篇文章

订阅专栏

zookeeper

5 篇文章

订阅专栏

本文讨论了在使用Eclipse编译MapReduce程序时遇到的错误，并提供了通过动态设置DFS URI来解决该问题的方法。

开发MapReduce程序一直都是在namenode 那台pc上用eclipse 直接运行，没有任何问题

今天在一台namenode上用eclipse 编译一M/R小程序，结果死活都是

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: Unknown protocol to job tracker: org.apache.hadoop.hdfs.protocol.ClientProtocol
	at org.apache.hadoop.mapred.JobTracker.getProtocolVersion(JobTracker.java:344)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1383)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1379)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1377)

	at org.apache.hadoop.ipc.Client.call(Client.java:1030)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
	at $Proxy1.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1310)
	at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:210)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372)
	at gucas.xiaoxia.InvertedIndex.main(InvertedIndex.java:110)

baidu google了半天也没找出如何在MapReduce之前动态设置 DFS的URI

无奈之下只能求助与 M/R 源代码发现 org.apache.hadoop.fs.FileSystem

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.fs.FileSystem

setDefaultUri

public static void setDefaultUri(Configuration conf,
                                 String uri)
Set the default filesystem URI in a configuration.
Parameters:
conf - the configuration to alter
uri - the new default filesystem uri

setDefaultUri 就是他了。

Configuration conf = new Configuration();


		
		String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
		System.out.println(FileSystem.getDefaultUri(conf)); 
		FileSystem.setDefaultUri(conf, new URI("hdfs://gucas-s2:9000")); //动态设置 dfs uri
		System.out.println(FileSystem.getDefaultUri(conf));
		if(otherArgs.length != 2){
		    System.err.println("Usage: invertedindex <in> <out>");
		    System.exit(2);
		}
		
		Job job = new Job(conf, "InvertedIndex");
		job.setJarByClass(InvertedIndex.class);

一定要在Job job = new Job(conf, "InvertedIndex"); 之前设置
FileSystem.setDefaultUri(conf, new URI("hdfs://gucas-s2:9000")); //动态设置 dfs uri

否则无效