jdk,Python,Scala,hadoop,mysql,spark安装

最新推荐文章于 2024-09-18 20:32:47 发布

jrymos001

最新推荐文章于 2024-09-18 20:32:47 发布

阅读量560

点赞数

CC 4.0 BY-SA版权

分类专栏： spark一千篇旅游日记文章标签： spark hadoop python mysql scala

本文链接：https://blog.youkuaiyun.com/m0_37681914/article/details/71467694

spark一千篇旅游日记专栏收录该内容

10 篇文章

订阅专栏

本文详细介绍了一台CentOS 64位虚拟机从初始安装到配置完成的过程，包括网络配置、安装必要的软件如JDK、SSH、Python、Scala、Spark、MySQL、Hadoop等，并设置了相应的环境变量。

所有的安装包都放置在/opt/

这里写图片描述
假设这是一台刚刚安装好的最简单版的centOS64位虚拟机:
一. 配置网络
vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
M_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.227.200
NETMASK=255.255.255.0
GATEWAY=192.168.227.2
DNS1=192.168.227.2
DNS2=192.168.227.3

service network restart
二. 准备yum的环境
yum install wget
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
三.安装jdk
yum -y install java-1.8.0-openjdk*
java -version
四.安装ssh
chkconfig iptables off#开机关闭防火墙
yum -y install gcc gcc-c++ libstdc++-devel
yum -y install openssh-server
yum groupinstall ‘Development Tools’
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel
五.安装Python
wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz
mv Python-3.6.1.tar.xz /opt/
cd /opt/
tar Jxvf Python-3.6.1.tar.xz
cd Python-3.6.1
./configure –prefix=/usr/local/python3
make && make install
六. 安装Scala
wget https://downloads.lightbend.com/scala/2.12.2/scala-2.12.2.tgz
tar zxvf scala-2.12.2.tgz
mv scala-2.12.2 scala
#vi /etc/profile
scala
source /etc/profile
scala
七.安装spark
wget http://www-eu.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-hadoop2.7.tgz
cd spark-2.1.1-bin-hadoop2.7.tgz
tar zxvf spark-2.1.1-bin-hadoop2.7.tgz
mv spark-2.1.1-bin-hadoop2.7 spark
八.安装mysql
yum -y install mysql-server
service mysqld start
chkconfig mysqld on
九.安装hadoop
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
tar zxvf hadoop-2.7.3.tar.gz
mv hadoop-2.7.3 hadoop
十.配置环境变量
vi /etc/profile
source /etc/profile

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL
export PATH=/opt/scala/bin:$PATH
export SPARK_HOME=/opt/spark
export PYTHONPATH=$SPARK_HOME/python
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$SPARK_HOME/sbin