Apache Slider编译安装

最新推荐文章于 2024-11-05 18:23:01 发布

weixin_34250709

最新推荐文章于 2024-11-05 18:23:01 发布

阅读量347

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 json python

原文链接：https://my.oschina.net/yulongblog/blog/744255

本文介绍Apache Slider项目的安装及使用方法，包括系统要求、编译安装步骤、环境变量设置及基本命令测试等内容。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2019独角兽企业重金招聘Python工程师标准>>>

Apache Slider

简介

Apache Slider目前是Apache二级孵化项目，主要用途是在YARN上部署用户已有的服务或者组件，如:hbase,storm,kafka.

下载地址:Apache Slider Download,目前最新0.91.0版本

系统要求

需要安装软件

Apache Hadoop Hadoop 2.6+
Apache Zookeeper 3.4+
Oracle JDK 1.7 (64-bit)
Python 2.6+
openssl

需要启动服务

Zookeeper

提供slider应用所需的zookeeper数据存放目录，如hbase的zookeeper.znode.parent

/services/slider/users/{username}/{application_name}
#for example:
/services/slider/users/hadoop/cl1

将在slider中启动的应用所用container信息注册在此位置

/registry/users/{username}/services/org-apache-slider/{application_name}
#for example:
/registry/users/hadoop/services/org-apache-slider/cl1/components/container-1473130308719-0016-01-000002

HDFS

提供slider应用包所需的hdfs数据存放目录

 /user/{username}/.slider/package/{package_name}/slider-{package_name}-app-package-{version}.zip
 #for example:
 /user/hadoop/.slider/package/HBASE/slider-hbase-app-package-0.98.21-hadoop2.zip

提供slider应用所需的hdfs数据存放目录，包括一些配置以及hdfs数据

 /user/{username}/.slider/cluster/{application_name}/*
 #for example:
 /user/hadoop/.slider/cluster/cl1/app_config.json
 /user/hadoop/.slider/cluster/cl1/confdir
 /user/hadoop/.slider/cluster/cl1/database
 /user/hadoop/.slider/cluster/cl1/generated
 /user/hadoop/.slider/cluster/cl1/history
 /user/hadoop/.slider/cluster/cl1/internal.json
 /user/hadoop/.slider/cluster/cl1/resources.json
 /user/hadoop/.slider/cluster/cl1/snapshot
 /user/hadoop/.slider/cluster/cl1/tmp

slider应用的全局配置文件app_config.json，可以通过修改进行相关服务配置的修改

 /user/{username}/.slider/cluster/{application_name}/app_config.json
 #for example:
 /user/hadoop/.slider/cluster/cl1/app_config.json

Yarn

提供Slider启动时所需要的container.yarn-site.xml需要修改以下配置

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>256</value>
  <description>more than 256 (ensure that YARN can allocate sufficient number of containers)，general set to more than 1000 for production environment</description>
</property>
<property>
  <name>yarn.nodemanager.delete.debug-delay-sec</name>
  <value>3600</value>
  <description>Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below).</description>
</property>

##编译

下载源码包解压至某目录${BUILD_DIR}，maven项目，编译命令

mvn clean site:site site:stage package -DskipTests

执行包生成在

${BUILD_DIR}/apache-slider-0.91.0-incubating/slider-assembly/target/

压缩包为tar.gz和zip格式，任选其一

slider-0.91.0-incubating-all.tar.gz
slider-0.91.0-incubating-all.zip

##安装

选择安装目录${SLIDER_INSTALL_DIR}

cp slider-0.91.0-incubating-all.zip ${SLIDER_INSTALL_DIR}
cd ${SLIDER_INSTALL_DIR}
unzip slider-0.91.0-incubating-all.zip
ln -s slider-0.91.0-incubating slider

设置环境变量，根据你自己的需要修改/etc/profile或者~/.bash_profile

export SLIDER_HOME=${SLIDER_INSTALL_DIR}/slider
export PATH=$SLIDER_HOME/bin:$PATH

修改配置文件$SLIDER_HOME/conf/slider-env.sh: export JAVA_HOME=${JAVA_HOME} export HADOOP_HOME=${HADOOP_HOME} export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

##测试通过slider命令验证一些基本功能。 ###slider help 测试slider help命令查看是否正常，正常说明PATH等环境变量是没有问题的。

[hadoop[@breath](http://my.oschina.net/liu1994) ~]$ slider help
2016-09-08 14:29:58,200 [main] INFO  client.SliderClient - 
Usage: slider COMMAND [options]
where COMMAND is one of
	am-suicideTell the Slider Application Master to simulate a process failure by terminating itself
	build Build a Slider cluster specification, but do not start it
	clientInstall the application client in the specified directory or obtain a client keystore or truststore
	createCreate a live Slider application
	dependencySlider AM and agent dependency (libraries) management
	destroy   Destroy a stopped Slider application
	diagnostics   Diagnose the configuration of the running slider application and slider client
	existsProbe for an application running
	flex  Flex a Slider application
	stop  Stop a running application
	help  Print help information
	install-keytabInstall the Kerberos keytab. Deprecated, use 'keytab --install'.
	install-package   Install application package. Deprecated, use 'package --install'.
	kdiag Diagnose Kerberos problems
	keytabManage a Kerberos keytab file (install, delete, list) in the sub-folder 'keytabs' of the user's Slider base directory
	kill-containerKill a container in the application
	list  List running Slider applications
	lookuplook up a YARN application
	nodes List the node information for the YARN cluster or a running application
	package   Install/list/delete application packages and list app instances that use the packages
	registry  Query the registry of a YARN application
	resolve   Query the registry of a YARN application
	statusGet the status of an application
	start Start a stopped application
	tokenssave tokens to a file or list tokens in a file
	updateUpdate template for a Slider application
	upgrade   Rolling upgrade/downgrade the application to a newer/previous version
	version   Print the Slider version information
Most commands print help when invoked without parameters or with --help
2016-09-08 14:29:58,244 [main] INFO  util.ExitUtil - Exiting with status 0

###slider package --list 测试slider package --list命令查看是否正常，正常说明HDFS、YARN等服务配置是正常的。我这里上传了一个HBASE的app包，所以显示HBASE。

[hadoop[@breath](http://my.oschina.net/liu1994) conf]$ slider package --list
2016-09-08 14:28:49,187 [main] INFO  client.RMProxy - Connecting to ResourceManager at breath/192.168.246.136:23140
2016-09-08 14:28:50,102 [main] INFO  client.SliderClient - Package install path : hdfs://breath:9000/user/hadoop/.slider/package
List of installed packages:
	HBASE

2016-09-08 14:28:51,068 [main] INFO  util.ExitUtil - Exiting with status 0

###slider list 测试slider package --list命令查看是否正常，正常说明HDFS、YARN等服务配置是正常的。我这里已经通过HBASE已经启动了一个cluster名为cl1，所以显示cl1状态为RUNNING.

[hadoop[@breath](http://my.oschina.net/liu1994) conf]$ slider list
2016-09-08 14:28:24,088 [main] INFO  client.RMProxy - Connecting to ResourceManager at breath/192.168.246.136:23140
cl1RUNNING  application_1473130308719_0016  http://breath:23188/proxy/application_1473130308719_0016/
2016-09-08 14:28:26,187 [main] INFO  util.ExitUtil - Exiting with status 0

转载于:https://my.oschina.net/yulongblog/blog/744255