Hadoop example program code_haoop example-优快云博客

本文详细介绍了使用Hadoop创建配置对象、将文件添加到HDFS以及部署到Linux系统的全过程，包括配置文件路径设置、文件上传流程及Linux部署步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Hadoop example code:

1. Creatinga configuration object: to be able to read from or write to hdfs, you need tocreate a configuration object and pass configuration parameter to it usinghadoop configuration files.

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

public class Main {

public static void main(String[] args){

Configurationconf = newConfiguration();

conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\core-site.xml"));

conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml"));

System.out.println(conf);

}

Output:

Configuration:core-default.xml, core-site.xml, C:/Documents andSettings/mz50947/Desktop/hadoop-1.2.1/conf/core-site.xml, C:/Documents andSettings/mz50947/Desktop/hadoop-1.2.1/conf/hdfs-site.xml

2. Addingfile to hdfs: create a filesystem object and use a file stream to add a file.

import java.io.BufferedInputStream;

import java.io.File;

import java.io.FileInputStream;

import java.io.IOException;

import java.io.InputStream;

import org.apache.hadoop.conf.Configuration;

importorg.apache.hadoop.fs.FSDataOutputStream;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

public class Main {

public static void main(String[] args) throws IOException{

Configurationconf = newConfiguration();

conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\core-site.xml"));

conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml"));

System.out.println(conf);

FileSystem fileSystem = FileSystem.get(conf);

// Check ifthe file already exists

Path path = new Path("C:\\Documents andSettings\\mz50947\\Desktop\\output\\file.ext");

if (fileSystem.exists(path)){

System.out.println("File" + path + "already exists");

fileSystem.delete(path, true);

}

// Create anew file and write data to it.

FSDataOutputStream out =fileSystem.create(path);

InputStream in = new BufferedInputStream(new FileInputStream(new File("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml")));

byte[] b = new byte[1024];

int numBytes = 0;

while ((numBytes = in.read(b)) > 0) {

out.write(b, 0, numBytes);

}

// Closeall the file descripters

in.close();

out.close();

fileSystem.close();

}

Will create file .file.ext and file under folder output

3. Deployto linux,