windows+Hadoop+Spark+eclipse开发环境配置

配置文件

core-site.xml

<configuration>
	<property>       
		<name>fs.defaultFS</name>       
		<value>hdfs://localhost:9000</value>   
	</property>
	
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/D:/hadoop/bigdata/hadoopdata/temp</value>
        <--!每个人的缓存位置可能不同根据实际情况修改目录-->
    </property>
</configuration>

mapred-site.xml

<configuration>
	<property>       
		<name>mapreduce.framework.name</name>       
		<value>yarn</value>   
	</property>
	
    <property>
       <name>mapred.job.tracker</name>
       <value>hdfs://localhost:9001</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
	<property>       
		<name>dfs.replication</name>       
		<value>1</value>   
	</property> 
	  
	<property>       
		<name>dfs.namenode.name.dir</name>       
		<value>/D:/hadoop/bigdata/hadoopdata/namenode</value>   
		<--!每个人的namenode位置可能不同根据实际情况修改目录-->
	</property>  
	 
	<property>       
		<name>dfs.datanode.data.dir</name>     
		<value>/D:/hadoop/bigdata/hadoopdata/datanode</value>  
		<--!每个人的datanode位置可能不同根据实际情况修改目录--> 
	</property>
</configuration>

yarn-site.xml

	<property>       
		<name>yarn.nodemanager.aux-services</name>       
		<value>mapreduce_shuffle</value>   
	</property>
	  
	<property>       
		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>       
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>   
	</property>

验证结果

运行Hadoop

1.打开cmd，执行‘hdfs namenode -format’,执行没有报错，说明成功

#重要提示：只能执行一次，如果执行多次会出现开启不了DataNode
PS D:\hadoop\hadoop-2.7.1\sbin>hdfs namenode -format

2.然后进入hadoop目录下sbin文件夹，运行start-all.cmd,得到如下图所示开启4个进程：

cd  D:\hadoop\hadoop-2.7.1\sbin
PS D:\hadoop\hadoop-2.7.1\sbin>  .\start-all.cmd

开启四个进程 NodeManager、NameNode、ResourceManager、DataNode

jps进程结果

正常结果有四个进程，
17456 NodeManager
17872 NameNode
13668 ResourceManager
16636 DataNode

如果结果如图下所示没有DataNode，则参考[13]

上传测试

1.1创建目录

C:\WINDOWS\system32>hadoop fs -mkdir hdfs://localhost:9000/user/ 
C:\WINDOWS\system32>hadoop fs -mkdir hdfs://localhost:9000/user/wcinput

1.2查看目录

2.1上传数据

PS D:\hadoop\bigdata\localdata> hadoop fs -put file.txt hdfs://localhost:9000/user/wcinput
PS D:\hadoop\bigdata\localdata> hadoop fs -put test.txt hdfs://localhost:9000/user/wcinput

2.2查看文件

#完整命令
C:\WINDOWS\system32>hadoop fs -ls hdfs://localhost:9000/user/wcinput
#简化命令
PS D:\hadoop\bigdata\localdata> hadoop fs -ls /user/wcinput

hadoop web控制台GUI

YARN-GUI:http://localhost:8088

HDFS-GUI:http://localhost:50070

相关文件

[1]: eclipse运行MapReduce架包和eclipse-MapReduce-plugins
[2]:hadooponwindows-master.zip
[3]:hadooponwindows2.71/bin/etc
[4]:JDK1.8
[5]:Scala2.1.x
[6]:Hadoop2.7x
[7]:Spark2.3x

参考文

[1]: https://blog.youkuaiyun.com/qq_30993409/article/details/81185025
[2]:https://blog.youkuaiyun.com/u012103898/article/details/77616398
[3]: https://blog.youkuaiyun.com/wangaz521/article/details/79717177
[4]: https://blog.youkuaiyun.com/csdn_fzs/article/details/78985586
[5]: https://blog.youkuaiyun.com/u013077468/article/details/77334541
[6]: https://www.cnblogs.com/huligong1234/p/4137133.html
[7]: https://blog.youkuaiyun.com/u010185220/article/details/79095179
[8]:https://www.cnblogs.com/supiaopiao/p/7240308.html
[9]:https://blog.youkuaiyun.com/u010523770/article/details/56482172
[10]:https://www.cnblogs.com/bornteam/p/6518480.html
[11]:https://www.cnblogs.com/baixl/p/4154429.html
[12]:https://www.cnblogs.com/yifengjianbai/p/8258898.html
[13]:https://blog.youkuaiyun.com/islotus/article/details/78357857