(转)HDFS的JAVA接口API操作实例

来源：

http://blog.sina.com.cn/s/blog_466678e80100p823.html

1.Copy a file from the local file system to HDFS

The srcFile variable needs to contain the full name (path + file name) of the file in the local file system.

The dstFile variable needs to contain the desired full name of the file in the Hadoop file system.

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path srcPath=newPath(srcFile); Path dstPath=newPath(dstFile); hdfs.copyFromLocalFile(srcPath,dstPath);

2.Create HDFS file

The fileName variable contains the file name and path in the Hadoop file system.

The content of the file is the buff variable which is an array of bytes.

//byte[] buff - The content of the file Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); FSDataOutputStreamoutputStream=hdfs.create(path); outputStream.write(buff,0,buff.length);

3.Rename HDFS file

In order to rename a file in Hadoop file system, we need the full name (path + name) of

the file we want to rename. The rename method returns true if the file was renamed, otherwise false.

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path fromPath=newPath(fromFileName); Path toPath=newPath(toFileName); booleanisRenamed=hdfs.rename(fromPath,toPath);

4.Delete HDFS file

In order to delete a file in Hadoop file system, we need the full name (path + name)

of the file we want to delete. The delete method returns true if the file was deleted, otherwise false

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); booleanisDeleted=hdfs.delete(path,false); //Recursivedelete: Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); booleanisDeleted=hdfs.delete(path,true);

5.Get HDFS file last modification time

In order to get the last modification time of a file in Hadoop file system,

we need the full name (path + name) of the file.

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); FileStatus fileStatus=hdfs.getFileStatus(path); longmodificationTime=fileStatus.getModificationTime

6.Check if a file exists in HDFS

In order to check the existance of a file in Hadoop file system,

we need the full name (path + name) of the file we want to check.

The exists methods returns true if the file exists, otherwise false.

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); booleanisExists=hdfs.exists(path);

7.Get the locations of a file in the HDFS cluster

A file can exist on more than one node in the Hadoop file system cluster for two reasons:

Based on the HDFS cluster configuration, Hadoop saves parts of files on different nodes in the cluster.

Based on the HDFS cluster configuration, Hadoop saves more than one copy of each file on different nodes for redundancy (The default is three).

Configurationconfig=newConfiguration(); FileSystem hdfs=FileSystem.get(config); Path path=newPath(fileName); FileStatus fileStatus=hdfs.getFileStatus(path);BlockLocation[]blkLocations=hdfs.getFileBlockLocations(path,0,fileStatus.getLen()); BlockLocation[]blkLocations=hdfs.getFileBlockLocations(fileStatus,0,fileStatus.getLen()); //这个地方，作者写错了，需要把path改为fileStatus intblkCount=blkLocations.length; for(inti=0;i<blkCount;i++){ String[]hosts=blkLocations[i].getHosts(); // Do something with the block hosts }

8. Get a list of all the nodes host names in the HDFS cluster

his method casts the FileSystem Object to a DistributedFileSystem Object.

This method will work only when Hadoop is configured as a cluster.

Running Hadoop on the local machine only, in a non cluster configuration will

cause this method to throw an Exception.

Configurationconfig=newConfiguration(); FileSystem fs=FileSystem.get(config); DistributedFileSystem hdfs=(DistributedFileSystem)fs; DatanodeInfo[]dataNodeStats=hdfs.getDataNodeStats(); String[]names=newString[dataNodeStats.length]; for(inti=0;i<dataNodeStats.length;i++){ names[i]=dataNodeStats[i].getHostName(); }

程序实例

importorg.apache.hadoop.conf.*; importorg.apache.hadoop.fs.*; importorg.apache.hadoop.hdfs.*; importorg.apache.hadoop.hdfs.protocol.*; importjava.util.Date; publicclassDFSOperater{ publicstaticvoidmain(String[]args){ Configurationconf=newConfiguration(); try{ // Get a list of all the nodes host names in the HDFS cluster FileSystem fs=FileSystem.get(conf); DistributedFileSystem hdfs=(DistributedFileSystem)fs; DatanodeInfo[]dataNodeStats=hdfs.getDataNodeStats(); String[]names=newString[dataNodeStats.length]; System.out.println("list of all the nodes in HDFS cluster:");//print info for(inti=0;i<dataNodeStats.length;i++){ names[i]=dataNodeStats[i].getHostName(); System.out.println(names[i]);//print info } Path f=newPath("/user/cluster/dfs.txt"); //check if a file exists in HDFS booleanisExists=fs.exists(f); System.out.println("The file exists? ["+isExists+"]"); //if the file exist, delete it if(isExists){ booleanisDeleted=hdfs.delete(f,false);//fase : not recursive if(isDeleted)System.out.println("now delete "+f.getName()); } //create and write System.out.println("create and write ["+f.getName()+"] to hdfs:"); FSDataOutputStream os=fs.create(f,true,0); for(inti=0;i<10;i++){ os.writeChars("test hdfs "); } os.writeChars("\n"); os.close(); //get the locations of a file in HDFS System.out.println("locations of file in HDFS:"); FileStatus filestatus=fs.getFileStatus(f); BlockLocation[]blkLocations=fs.getFileBlockLocations(filestatus,0,filestatus.getLen()); intblkCount=blkLocations.length; for(inti=0;i<blkCount;i++){ String[]hosts=blkLocations[i].getHosts(); //Do sth with the block hosts System.out.println(hosts); } //get HDFS file last modification time longmodificationTime=filestatus.getModificationTime();// measured in milliseconds since the epoch Dated=newDate(modificationTime); System.out.println(d); //reading from HDFS System.out.println("read ["+f.getName()+"] from hdfs:"); FSDataInputStream dis=fs.open(f); System.out.println(dis.readUTF()); dis.close(); }catch(Exceptione){ // TODO: handle exception e.printStackTrace(); } } }