代码
public class HDFSApp {
public static final String HDFS_PATH="hdfs://hadoop001:9000";
Configuration configuration;
FileSystem fileSystem;
@Before
public void setUp() throws Exception{
BasicConfigurator.configure();
configuration = new Configuration();
configuration.set("dfs.replication","1");
fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration,"hadoop");
}
@After
public void tearDown() throws Exception{
fileSystem.close();
}
@Test
public void mkdir() throws Exception{
boolean isSucess = fileSystem.mkdirs(new Path("/ruozedata/hdfsapi"));
Assert.assertEquals(true,isSucess);
}
@Test
public void mkdir2() throws Exception{
//BasicConfigurator.configure();
boolean isSucess = fileSystem.mkdirs(new Path("/ruozedata/hdfsapi"));
Assert.assertEquals(true,isSucess);
}
@Test
public void copyFromLocal() throws Exception {
Path localPath = new Path("f:/yt.txt");
Path desPath = new Path("/ruozedata/hdfsapi");
fileSystem.copyFromLocalFile(localPath, desPath);
}
@Test
public void testReplication() throws Exception{
System.out.println(configuration.get("dfs.replication"));
}
@Test
public void copyToLocal() throws Exception{
Path srcPath = new Path("hdfs://hadoop001:9000/ruozedata/hdfsapi/test");
Path destPath = new Path("f:/");
fileSystem.copyToLocalFile(false,srcPath,destPath,true);
}
采坑1:
执行copyFromLocal() 方法时,报错:
解决方案1.可参考地址:https://stackoverflow.com/questions/14288453/writing-to-hdfs-from-java-getting-could-only-be-replicated-to-0-nodes-instead?tdsourcetag=s_pcqq_aiomsg
注意:云主机hosts文件配内网ip,本地windows电脑配置主机名对应公网ip
做好上面三部,然后再将配置好的文件hdfs-site.xml到本地代码的resource目录下
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop001:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address</name>
<value>hadoop001:50091</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>true</value>
</property>
</configuration>
采坑2:此时再运行copyToLocal() 方法依然空指针错误:
是因为我之前fileSystem.copyToLocalFile(false,srcPath,destPath,true);
代码前面没加false和后面没加ture.
盘点MapReudce在Windows上运行的坑
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries
解决办法:
下载winutils.exe
创建一个目录,比如C:\winutils\bin
将winutils.exe放入上述创建好的目录下
设置HADOOP_HOME=C:\winutils环境变量,并将其放入PATH变量中,例如%HADOOP_HOME%\bin
或者直接在程序中加入System.setProperty(“hadoop.home.dir”, “full path to the folder with winutils”);
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
解决办法:
下载winutils.exe和hadoop.dll
放这两个在%HADOOP_HOME%\bin下,同时hadoop.dll也放在C:\Windows\System32目录下
注意
:如果注意设置任然继续报错
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
这时候不要怀疑你上面的配置的步骤没得用,很有可能是你的代码中job.set map和reduce
输出设置是否有问题,仔细检查。