hadoop hdfs的一些用法

本文介绍了使用Java进行Hadoop文件系统操作的四个示例,包括通过URL读取文件、直接使用FileSystem API读取文件、重复读取文件以及将本地文件复制到Hadoop文件系统并显示进度。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Example 3-1. Displaying files from a Hadoop filesystem on standard output using a
URLStreamHandler

Java代码
  1. //Reading Data from a Hadoop URL  
  2.   
  3. public class URLCat {  
  4.     static {  
  5.         URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());  
  6.     }  
  7.   
  8.     public static void main(String[] args) throws Exception {  
  9.         InputStream in = null;  
  10.         try {  
  11.             in = new URL(args[0]).openStream();  
  12.             IOUtils.copyBytes(in, System.out, 4096, false);  
  13.         } finally {  
  14.             IOUtils.closeStream(in);  
  15.         }  
  16.     }  
  17. }  
  18. -----------------------------------------  
  19. result:  
  20. Here’s a sample run:  
  21. % hadoop URLCat hdfs://localhost/user/tom/quangle.txt  
  22. On the top of the Crumpetty Tree  
  23. The Quangle Wangle sat,  
  24. But his face you could not see,  
  25. On account of his Beaver Hat.  
//Reading Data from a Hadoop URL

public class URLCat {
	static {
		URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
	}

	public static void main(String[] args) throws Exception {
		InputStream in = null;
		try {
			in = new URL(args[0]).openStream();
			IOUtils.copyBytes(in, System.out, 4096, false);
		} finally {
			IOUtils.closeStream(in);
		}
	}
}
-----------------------------------------
result:
Here’s a sample run:
% hadoop URLCat hdfs://localhost/user/tom/quangle.txt
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.



Example 3-2. Displaying files from a Hadoop filesystem on standard output by using the FileSystem
directly

Java代码
  1. public class FileSystemCat {  
  2.     public static void main(String[] args) throws Exception {  
  3.         String uri = args[0];  
  4.         Configuration conf = new Configuration();  
  5.         FileSystem fs = FileSystem.get(URI.create(uri), conf);  
  6.         InputStream in = null;  
  7.         try {  
  8.             in = fs.open(new Path(uri));  
  9.             IOUtils.copyBytes(in, System.out, 4096, false);  
  10.         } finally {  
  11.             IOUtils.closeStream(in);  
  12.         }  
  13.     }  
  14. }  
  15.   
  16. ------------------------------------------  
  17. The program runs as follows:  
  18. % hadoop FileSystemCat hdfs://localhost/user/tom/quangle.txt  
  19. On the top of the Crumpetty Tree  
  20. The Quangle Wangle sat,  
  21. But his face you could not see,  
  22. On account of his Beaver Hat.  
  23. The  
public class FileSystemCat {
	public static void main(String[] args) throws Exception {
		String uri = args[0];
		Configuration conf = new Configuration();
		FileSystem fs = FileSystem.get(URI.create(uri), conf);
		InputStream in = null;
		try {
			in = fs.open(new Path(uri));
			IOUtils.copyBytes(in, System.out, 4096, false);
		} finally {
			IOUtils.closeStream(in);
		}
	}
}

------------------------------------------
The program runs as follows:
% hadoop FileSystemCat hdfs://localhost/user/tom/quangle.txt
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.
The



Example 3-3 is a simple extension of Example 3-2 that writes a file to standard out
twice: after writing it once, it seeks to the start of the file and streams through it once
again.

Java代码
  1. //Example 3-3. Displaying files from a Hadoop filesystem on standard output twice, by using seek  
  2.   
  3. public class FileSystemDoubleCat {  
  4.     public static void main(String[] args) throws Exception {  
  5.         String uri = args[0];  
  6.         Configuration conf = new Configuration();  
  7.         FileSystem fs = FileSystem.get(URI.create(uri), conf); //通过get()方法获得一个FileSystem流  
  8.         FSDataInputStream in = null;  
  9.         try {  
  10.             in = fs.open(new Path(uri)); //通过open()方法打开一个FSDataInputStream流  
  11.             IOUtils.copyBytes(in, System.out, 4096, false);  
  12.             in.seek(0); // go back to the start of the file  
  13.             IOUtils.copyBytes(in, System.out, 4096, false);  
  14.         } finally {  
  15.             IOUtils.closeStream(in);  
  16.         }  
  17.     }  
  18. }  
  19. ----------------------------------------------------  
  20. Here’s the result of running it on a small file:  
  21. % hadoop FileSystemDoubleCat hdfs://localhost/user/tom/quangle.txt  
  22. On the top of the Crumpetty Tree  
  23. The Quangle Wangle sat,  
  24. But his face you could not see,  
  25. On account of his Beaver Hat.  
  26. On the top of the Crumpetty Tree  
  27. The Quangle Wangle sat,  
  28. But his face you could not see,  
  29. On account of his Beaver Hat.  
//Example 3-3. Displaying files from a Hadoop filesystem on standard output twice, by using seek

public class FileSystemDoubleCat {
	public static void main(String[] args) throws Exception {
		String uri = args[0];
		Configuration conf = new Configuration();
		FileSystem fs = FileSystem.get(URI.create(uri), conf); //通过get()方法获得一个FileSystem流
		FSDataInputStream in = null;
		try {
			in = fs.open(new Path(uri)); //通过open()方法打开一个FSDataInputStream流
			IOUtils.copyBytes(in, System.out, 4096, false);
			in.seek(0); // go back to the start of the file
			IOUtils.copyBytes(in, System.out, 4096, false);
		} finally {
			IOUtils.closeStream(in);
		}
	}
}
----------------------------------------------------
Here’s the result of running it on a small file:
% hadoop FileSystemDoubleCat hdfs://localhost/user/tom/quangle.txt
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.



Example 3-4 shows how to copy a local file to a Hadoop filesystem. We illustrate progress
by printing a period every time the progress() method is called by Hadoop, which
is after each 64 K packet of data is written to the datanode pipeline. (Note that this
particular behavior is not specified by the API, so it is subject to change in later versions
of Hadoop. The API merely allows you to infer that “something is happening.”)

Java代码
  1. //Example 3-4. Copying a local file to a Hadoop filesystem, and shows progress  
  2. public class FileCopyWithProgress {  
  3.     public static void main(String[] args) throws Exception {  
  4.         String localSrc = args[0];  
  5.         String dst = args[1];  
  6.         InputStream in = new BufferedInputStream(new FileInputStream(localSrc));  
  7.         Configuration conf = new Configuration();  
  8.         FileSystem fs = FileSystem.get(URI.create(dst), conf);  
  9.         OutputStream out = fs.create(new Path(dst), new Progressable() {  
  10.             public void progress() {  
  11.                 System.out.print(".");  
  12.             }  
  13.         });  
  14.         IOUtils.copyBytes(in, out, 4096, true);  
  15.     }  
  16. }  
  17.   
  18. Typical usage:  
  19. % hadoop FileCopyWithProgress input/docs/1400-8.txt hdfs://localhost/user/tom/1400-8.txt 
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值