hadoop2

## hadoop客户端开发

 

• 这一段操作简直疯狂受苦，记录一下当前摸索的正确步骤

• Idea+maven+hadoop2.9.2 ，连接虚拟机集群

• 没有编译hadoop 源码，也没有使用share文件中的包，但HADOOP_HOME 和PATH 是必须配置的

• maven切换到阿里云镜像

1. /usr/.m2/settings.xml

''' ~xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0

https://maven.apache.org/xsd/settings-1.0.0.xsd">

      <mirrors>

        <mirror>

            <id>alimaven</id>

            <name>aliyun maven</name>

            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>

            <mirrorOf>central</mirrorOf>         

        </mirror>

      </mirrors>

</settings>

'''

 

1. 在Idea>Settings>Build...>Build Tools>Maven中确定User Settings /Local rep.... 确定override勾选

 

• pom中添加dependencies，没有hadoop-core...,而是替换为common, hdfs, client 三个模块。版本号这种东西有没有省心一点的方法？

 

'''    ~xml

<dependencies>

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>4.13-beta-2</version>

<scope>test</scope>

</dependency>

 

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-common</artifactId>

<version>2.9.2</version>

</dependency>

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>2.9.2</version>

</dependency>

<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>2.9.2</version>

</dependency>

 

</dependencies>

'''

 

• 现在可以新建一个测试类，没有拷贝core-site.xml到项目中

'''    ~java

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

import org.junit.Test;

 

import java.io.IOException;

import java.net.URI;

import java.net.URISyntaxException;

 

public class TestHdfsClient {

 

@Test

public void TestHdfs() throws IOException, URISyntaxException, InterruptedException {

System.out.println("ends");

 

// 1 创建配置信息对象

Configuration configuration = new Configuration();

 

FileSystem fs = FileSystem.get(new URI("hdfs://had002:9000"),configuration, "lee");

 

// 2 创建要上传文件所在的本地路径

Path src = new Path("D:\\Test\\dat.txt");

 

// 3 创建要上传到hdfs的目标路径

Path dst = new Path("hdfs://had002:9000/user/lee/dat.txt");

 

// 4 拷贝文件

fs.copyFromLocalFile(src, dst);

fs.close();

}

 

}

 

'''

 

• 在http://had002:50070/explorer.html#/user/lee 能够看到上传的文件了。 ok，搞定