centos7安装hadoop3.03和jdk1.8.0_172的本地安装方法

最新推荐文章于 2024-10-16 10:19:20 发布
最新推荐文章于 2024-10-16 10:19:20 发布
阅读量929 收藏 1
点赞数
CC 4.0 BY-SA版权
分类专栏： 操作系统 Linux/Unix hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/hsg77/article/details/80901956
本文介绍在CentOS7系统中安装Hadoop3.03和JDK1.8.0_172的具体步骤，并演示如何进行本地模式验证，包括配置环境变量、设置路径及运行WordCount示例。

centos7安装hadoop3.03和jdk1.8.0_172的本地安装方法
先关闭防火墙
systemctl stop firewalld
systemctl disalbe firewalld

解压jdk1.8.0_172并安装到目录:/opt/java/jdk1.8

解压hadoop3.03并安装到目录:/usr/hadoop/hadoop303

配置/etc/profile文件

#set java path
export JAVA_HOME=/opt/java/jdk1.8
export PATH=$JAVA_HOME/bin:$PATH


#set hadoop path
export HADOOP_HOME=/usr/hadoop/hadoop303
export PATH=$HADOOP_HOME:$PATH

启用配置文件
source /etc/profile

验证环境

[root@hadoop001 /]# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
[root@hadoop001 /]# echo ${JAVA_HOME}
/opt/java/jdk1.8
[root@hadoop001 /]# echo ${HADOOP_HOME}
/usr/hadoop/hadoop303

添加/opt/data/wc.input文件 内容如下：

hadoop mapreduce hive
hbase spark storm
sqoop hadoop hive
spark hadoop

验证hadoop本地模式

[root@hadoop001 hadoop303]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /opt/data/wc.input output2

[root@hadoop001 hadoop303]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /opt/data/wc.input output2
2018-07-03 17:20:32,010 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2018-07-03 17:20:44,103 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2018-07-03 17:20:44,104 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2018-07-03 17:21:10,621 INFO input.FileInputFormat: Total input files to process : 1
2018-07-03 17:21:10,647 INFO mapreduce.JobSubmitter: number of splits:1
2018-07-03 17:21:10,970 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1631897872_0001
2018-07-03 17:21:10,971 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-07-03 17:21:11,134 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2018-07-03 17:21:11,134 INFO mapreduce.Job: Running job: job_local1631897872_0001
2018-07-03 17:21:11,145 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2018-07-03 17:21:11,149 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2018-07-03 17:21:11,149 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2018-07-03 17:21:11,150 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2018-07-03 17:21:11,192 INFO mapred.LocalJobRunner: Waiting for map tasks
2018-07-03 17:21:11,204 INFO mapred.LocalJobRunner: Starting task: attempt_local1631897872_0001_m_000000_0
2018-07-03 17:21:11,246 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2018-07-03 17:21:11,254 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2018-07-03 17:21:11,338 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2018-07-03 17:21:11,340 INFO mapred.MapTask: Processing split: file:/opt/data/wc.input:0+73
2018-07-03 17:21:11,551 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2018-07-03 17:21:11,551 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2018-07-03 17:21:11,551 INFO mapred.MapTask: soft limit at 83886080
2018-07-03 17:21:11,551 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2018-07-03 17:21:11,551 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2018-07-03 17:21:11,580 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2018-07-03 17:21:11,584 INFO mapred.LocalJobRunner: 
2018-07-03 17:21:11,588 INFO mapred.MapTask: Starting flush of map output
2018-07-03 17:21:11,588 INFO mapred.MapTask: Spilling map output
2018-07-03 17:21:11,588 INFO mapred.MapTask: bufstart = 0; bufend = 115; bufvoid = 104857600
2018-07-03 17:21:11,588 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2018-07-03 17:21:11,601 INFO mapred.MapTask: Finished spill 0
2018-07-03 17:21:11,619 INFO mapred.Task: Task:attempt_local1631897872_0001_m_000000_0 is done. And is in the process of committing
2018-07-03 17:21:11,634 INFO mapred.LocalJobRunner: map
2018-07-03 17:21:11,634 INFO mapred.Task: Task 'attempt_local1631897872_0001_m_000000_0' done.
2018-07-03 17:21:11,638 INFO mapred.Task: Final Counters for attempt_local1631897872_0001_m_000000_0: Counters: 18
    File System Counters
        FILE: Number of bytes read=316211
        FILE: Number of bytes written=782460
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=4
        Map output records=11
        Map output bytes=115
        Map output materialized bytes=94
        Input split bytes=88
        Combine input records=11
        Combine output records=7
        Spilled Records=7
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=17
        Total committed heap usage (bytes)=135335936
    File Input Format Counters 
        Bytes Read=73
2018-07-03 17:21:11,638 INFO mapred.LocalJobRunner: Finishing task: attempt_local1631897872_0001_m_000000_0
2018-07-03 17:21:11,639 INFO mapred.LocalJobRunner: map task executor complete.
2018-07-03 17:21:11,651 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2018-07-03 17:21:11,652 INFO mapred.LocalJobRunner: Starting task: attempt_local1631897872_0001_r_000000_0
2018-07-03 17:21:11,670 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2018-07-03 17:21:11,670 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2018-07-03 17:21:11,670 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2018-07-03 17:21:11,685 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@49dbd8b3
2018-07-03 17:21:11,686 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2018-07-03 17:21:11,711 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=323557792, maxSingleShuffleLimit=80889448, mergeThreshold=213548144, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2018-07-03 17:21:11,753 INFO reduce.EventFetcher: attempt_local1631897872_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2018-07-03 17:21:11,779 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1631897872_0001_m_000000_0 decomp: 90 len: 94 to MEMORY
2018-07-03 17:21:11,793 INFO reduce.InMemoryMapOutput: Read 90 bytes from map-output for attempt_local1631897872_0001_m_000000_0
2018-07-03 17:21:11,794 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 90, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->90
2018-07-03 17:21:11,806 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2018-07-03 17:21:11,806 INFO mapred.LocalJobRunner: 1 / 1 copied.
2018-07-03 17:21:11,806 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2018-07-03 17:21:11,813 INFO mapred.Merger: Merging 1 sorted segments
2018-07-03 17:21:11,813 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 81 bytes
2018-07-03 17:21:11,818 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
    at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2018-07-03 17:21:11,820 INFO reduce.MergeManagerImpl: Merged 1 segments, 90 bytes to disk to satisfy reduce memory limit
2018-07-03 17:21:11,820 INFO reduce.MergeManagerImpl: Merging 1 files, 94 bytes from disk
2018-07-03 17:21:11,821 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2018-07-03 17:21:11,821 INFO mapred.Merger: Merging 1 sorted segments
2018-07-03 17:21:11,834 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 81 bytes
2018-07-03 17:21:11,834 INFO mapred.LocalJobRunner: 1 / 1 copied.
2018-07-03 17:21:11,836 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2018-07-03 17:21:11,837 INFO mapred.Task: Task:attempt_local1631897872_0001_r_000000_0 is done. And is in the process of committing
2018-07-03 17:21:11,837 INFO mapred.LocalJobRunner: 1 / 1 copied.
2018-07-03 17:21:11,837 INFO mapred.Task: Task attempt_local1631897872_0001_r_000000_0 is allowed to commit now
2018-07-03 17:21:11,838 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1631897872_0001_r_000000_0' to file:/usr/hadoop/hadoop303/output2
2018-07-03 17:21:11,852 INFO mapred.LocalJobRunner: reduce > reduce
2018-07-03 17:21:11,852 INFO mapred.Task: Task 'attempt_local1631897872_0001_r_000000_0' done.
2018-07-03 17:21:11,852 INFO mapred.Task: Final Counters for attempt_local1631897872_0001_r_000000_0: Counters: 24
    File System Counters
        FILE: Number of bytes read=316431
        FILE: Number of bytes written=782626
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Combine input records=0
        Combine output records=0
        Reduce input groups=7
        Reduce shuffle bytes=94
        Reduce input records=7
        Reduce output records=7
        Spilled Records=7
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=1
        Total committed heap usage (bytes)=135335936
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Output Format Counters 
        Bytes Written=72
2018-07-03 17:21:11,852 INFO mapred.LocalJobRunner: Finishing task: attempt_local1631897872_0001_r_000000_0
2018-07-03 17:21:11,852 INFO mapred.LocalJobRunner: reduce task executor complete.
2018-07-03 17:21:12,138 INFO mapreduce.Job: Job job_local1631897872_0001 running in uber mode : false
2018-07-03 17:21:12,141 INFO mapreduce.Job:  map 100% reduce 100%
2018-07-03 17:21:12,143 INFO mapreduce.Job: Job job_local1631897872_0001 completed successfully
2018-07-03 17:21:12,177 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=632642
        FILE: Number of bytes written=1565086
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=4
        Map output records=11
        Map output bytes=115
        Map output materialized bytes=94
        Input split bytes=88
        Combine input records=11
        Combine output records=7
        Reduce input groups=7
        Reduce shuffle bytes=94
        Reduce input records=7
        Reduce output records=7
        Spilled Records=14
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=18
        Total committed heap usage (bytes)=270671872
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=73
    File Output Format Counters 
        Bytes Written=72
[root@hadoop001 hadoop303]#

执行完后输出结果为：
/usr/hadoop/hadoop303/output2目录下有两个文件
_SUCCESS
part-r-00000

part-r-00000内容如下:
hadoop 3
hbase 1
hive 2
mapreduce 1
spark 2
sqoop 1
storm 1

—the–end

Centos7 Hadoop 单机版安装教程（图文）
猿小白的博客
09-11 955
本章教程，主要记录如何在Centos7中安装Hadoop单机版。
在Centos7中安装Hadoop详细教程
weixin_42732268的博客
07-16 2033
在末端添加以下代码，如果跟我路径一致，可直接复制，不一样的话，要注意自己安装路径，即上面上传解压的路径。第一个property中的Value:hdfs://zhang:9000/，修改主机名，比如我的主机名为zhang,端口号9000是固定的，特别注意下 主机名，可以使用。第二个property中的Value:hadoop的安装路径，opt/server/hadoop-2.8.0/data/，数据存储在data文件夹里。在配置文件夹/opt/server/hadoop-2.8.0/etc/hadoop/
jdk1.8.0_172
06-04
JDK是 Java 语言的软件开发工具包，主要用于移动设备、嵌入式设备上的java应用程序。JDK是整个java开发的核心，它包含了JAVA的运行环境（JVM+Java系统类库）和JAVA工具。
jdk1.8.0_172免安装版
05-21
jdk1.8.0_172免安装版 解压后配置环境变量 JAVA_HOME=D:/Java/jdk1.8.0_172 CLASSPATH=.;%JAVA_HOME%\lib;%JAVA_HOME%\lib\dt.jar;%JAVA_HOME%\lib\tools.jar PATH=;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin 即可
JDK1.8安装教程及环境变量的配置（附安装包下载）
nankechxl的博客
04-13 1145
一.下载JDK1.8安装包： 链接：https://pan.baidu.com/s/1DcQz5VafcRnHPocp2lXRVA 提取码：qwer 二.安装教程： 1.双击解压后的exe文件 2.点击下一步 3.更改安装路径 4.根据各自习惯选择安装路径，建议最内层的文件夹名是JDK的版本号，图中是楼主的安装路径，大家可以参照我的命名方式去选择安装路径。(建议不要选择默认路径，该路径中有空格，导致楼主后来安装其他软件时报错)，选完之后，然后按确定。 5.点击下一步 6.等待安装 7.安装 jr
jdk1.8.0_172_windows-x64.rar
10-30
jdk1.8.0_172_windows-x64.rar jdk1.8.0_172_windows-x64.rar
linux 安装jdk1.8
攻防人生3722的博客
06-28 333
下载相应版本的jdk. 复制jdk到安装目录。cp jdk-8u172-linux-x64.tar.gz /oot/java/ 进入oot/java 解压到本目录 tar -zxvf jdk-8u172-linux-x64.tar.gz 编辑配置文件 vi /etc/profile,加入如下内容 JAVA_HOME=/oot/java/jdk1.8.0_172 CLASSPATH=$J...
CentOS下jdk1.8.0_161和kylin麒麟配置文件.zip
07-26
这里我们关注的是在 CentOS 操作系统上配置 JDK 1.8.0_161 和 Kylin（一款基于Hadoop的数据仓库与分析平台）的详细过程。这个压缩包包含了必要的文件，如 MySQL 的驱动包和其他辅助工具，帮助用户顺利完成配置。 ...
Linux(centOS7) 安装Hadoop+JDK(详细教程)
FlyJune的博客
09-13 3131
Linux(centOS7) 安装Hadoop+JDK(详细教程)
基于Centos7安装Hadoop
m0_72839702的博客
04-22 510
其中192.168.43.133为你的master节点ip地址，master节点名称在。此时我的JDK路径为(记住，后面要用)这里只显示5核，按照规则应该显示6核。文件，主动添加映射关系。
jdk1.8.0_172-win-32.rar
07-26
jdk1.8.0_172-win-32.rar
jdk1.8.0_172.rar
08-19
jdk1.8.0_172.rar
jdk1.8.0_172.zip
11-12
jdk1.8
jdk_1.8.0_172.zip
06-05
从ORACLE官网下载的，研究1.8源码使用，因为官网下载很慢，无偿提供压缩包，解压后可以直接使用。
linux中配置centos8,linux（centos8）下安装jdk1.8
weixin_36403956的博客
05-16 259
安装前彻底卸载jdk1、判断jdk是否安装(显示版本号等信息，说明已经安装)java -version2、查看JDK的安装路径which java3、rm -rf JDK地址(卸载JDK)rm -rf /usr/java/jdk/jdk1.8.0_172/4、删除环境变量信息vim /etc/profile #打开的记事本文件里将JAVA_HOME配置的信息删除5、删除环境中保存的java、j...
【免费下载】 JDK 1.8.0_172 下载
最新发布
gitblog_09732的博客
10-16 918
JDK 1.8.0_172 下载 【下载地址】JDK1.8.0_172下载 本仓库提供了 Java Development Kit (JDK) 1.8.0_172 版本的直接下载资源。JDK 是开发人员编写Java应用程序必不可少的工具包，它包含了Java运行环境（JRE）、编译器以及其他工具，使得开发者能够创建、编译、...
CentOS7.5.1804 Minimal 安装JDK1.8.0_172
lc1010078424的博客
07-03 3380
一、安装前检查 安装之前先检查一下系统有没有自带open-jdk 命令： rpm -qa |grep java rpm -qa |grep jdk rpm -qa |grep gcj 如果没有输出信息表示没有安装。 如果有输出信息，表示安装了，检查是否是自带的open-jdk，那就通过 rpm -e –nodeps 后面跟系统自带的jdk名 这个命令来删除系统自带的j...
jdk1.8.0_172_64位
03-14
jdk1.8.0_172_64位绿色版解压就可以用，有需要的可以直接下载使用。java8开发必备。。。。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值