1:下载
[jifeng@feng02 storm]$ wget https://github.com/yahoo/storm-yarn/archive/master.zip
--2015-03-08 21:07:24-- https://github.com/yahoo/storm-yarn/archive/master.zip
正在解析主机 github.com... 192.30.252.130
正在连接 github.com|192.30.252.130|:443... 已连接。
已发出 HTTP 请求,正在等待回应... 302 Found
位置:https://codeload.github.com/yahoo/storm-yarn/zip/master [跟随至新的 URL]
--2015-03-08 21:07:27-- https://codeload.github.com/yahoo/storm-yarn/zip/master
正在解析主机 codeload.github.com... 192.30.252.145
正在连接 codeload.github.com|192.30.252.145|:443... 已连接。
已发出 HTTP 请求,正在等待回应... 200 OK
长度:未指定 [application/zip]
正在保存至: “master”
[ <=> ] 17,346,738 288K/s in 2m 0s
2015-03-08 21:09:30 (141 KB/s) - “master” 已保存 [17346738]
2:解压
[jifeng@feng02 storm]$ mv master storm-yarn.zip
[jifeng@feng02 storm]$ unzip storm-yarn.zip
Archive: storm-yarn.zip
e570e9665bac68396db09a9711875948cc63540e
creating: storm-yarn-master/
inflating: storm-yarn-master/CLA.pdf
inflating: storm-yarn-master/LICENSE.txt
inflating: storm-yarn-master/README.md
creating: storm-yarn-master/bin/
inflating: storm-yarn-master/bin/read-link
inflating: storm-yarn-master/bin/storm-yarn
inflating: storm-yarn-master/create-tarball.sh
creating: storm-yarn-master/lib/
inflating: storm-yarn-master/lib/storm-0.9.0-wip21.zip
inflating: storm-yarn-master/lib/storm-starter-0.0.1-SNAPSHOT.jar
linking: storm-yarn-master/lib/storm.zip -> storm-0.9.0-wip21.zip
inflating: storm-yarn-master/pom.xml
creating: storm-yarn-master/src/
creating: storm-yarn-master/src/main/
inflating: storm-yarn-master/src/main/genthrift.sh
creating: storm-yarn-master/src/main/java/
creating: storm-yarn-master/src/main/java/com/
creating: storm-yarn-master/src/main/java/com/yahoo/
creating: storm-yarn-master/src/main/java/com/yahoo/storm/
creating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Client.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Config.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/LaunchCommand.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/MasterClient.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/MasterServer.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormAMRMClient.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormMasterCommand.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormMasterServerHandler.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Util.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Version.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/VersionCommand.java
creating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/generated/
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/generated/StormMaster.java
creating: storm-yarn-master/src/main/resources/
inflating: storm-yarn-master/src/main/resources/logback.xml
inflating: storm-yarn-master/src/main/resources/master_defaults.yaml
inflating: storm-yarn-master/src/main/storm_master.thrift
creating: storm-yarn-master/src/test/
creating: storm-yarn-master/src/test/java/
creating: storm-yarn-master/src/test/java/com/
creating: storm-yarn-master/src/test/java/com/yahoo/
creating: storm-yarn-master/src/test/java/com/yahoo/storm/
creating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/EmbeddedZKServer.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestConfig.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestIntegration.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestStormCluster.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestStormMaster.java
creating: storm-yarn-master/src/test/resources/
inflating: storm-yarn-master/src/test/resources/log4j.properties
inflating: storm-yarn-master/src/test/resources/logback-test.xml
finishing deferred symbolic links:
storm-yarn-master/lib/storm.zip -> storm-0.9.0-wip21.zip
[jifeng@feng02 storm]$ ls
storm-yarn-master storm-yarn.zip
[jifeng@feng02 storm]$ cd storm-yarn-master/
[jifeng@feng02 storm-yarn-master]$ ls
bin CLA.pdf create-tarball.sh lib LICENSE.txt pom.xml README.md src
3:编译
[jifeng@feng02 storm-yarn-master]$ mvn package -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building storm-yarn 1.0-alpha
[INFO] ------------------------------------------------------------------------
Downloading: http://repo1.maven.org/maven2/storm/storm-core/0.9.0-wip21/storm-core-0.9.0-wip21.jar
Downloading: http://repo1.maven.org/maven2/org/clojure/clojure/1.4.0/clojure-1.4.0.jar
Downloading: http://repo1.maven.org/maven2/storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar
Downloading: http://repo1.maven.org/maven2/clj-time/clj-time/0.4.1/clj-time-0.4.1.jar
Downloading: http://repo1.maven.org/maven2/backtype/jzmq/2.1.0/jzmq-2.1.0.jar
Downloading: http://repo1.maven.org/maven2/compojure/compojure/1.1.3/compojure-1.1.3.jar
Downloading: http://repo1.maven.org/maven2/clout/clout/1.0.1/clout-1.0.1.jar
Downloading: http://repo1.maven.org/maven2/ring/ring-core/1.1.5/ring-core-1.1.5.jar
Downloading: http://repo1.maven.org/maven2/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar
Downloading: http://repo1.maven.org/maven2/ring/ring-devel/0.3.11/ring-devel-0.3.11.jar
Downloading: http://repo1.maven.org/maven2/clj-stacktrace/clj-stacktrace/0.2.2/clj-stacktrace-0.2.2.jar
Downloading: http://repo1.maven.org/maven2/ring/ring-jetty-adapter/0.3.11/ring-jetty-adapter-0.3.11.jar
Downloading: http://repo1.maven.org/maven2/ring/ring-servlet/0.3.11/ring-servlet-0.3.11.jar
Downloading: http://repo1.maven.org/maven2/storm/carbonite/1.5.0/carbonite-1.5.0.jar
Downloading: http://repo1.maven.org/maven2/storm/tools.cli/0.2.2/tools.cli-0.2.2.jar
Downloading: http://repo1.maven.org/maven2/storm/jgrapht/0.8.3/jgrapht-0.8.3.jar
Downloading: http://repo1.maven.org/maven2/com/google/guava/guava/13.0/guava-13.0.jar
Downloading: http://repo1.maven.org/maven2/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.1.0-beta/hadoop-yarn-server-nodemanager-2.1.0-beta.jar
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
Downloading: http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.1.0-beta/hadoop-yarn-server-nodemanager-2.1.0-beta.jar (452 KB at 46.8 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar
Downloaded: http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar (315 KB at 25.4 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar (258 KB at 18.0 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar (47 KB at 2.9 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.4/commons-compress-1.4.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.4/commons-compress-1.4.jar (234 KB at 9.4 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/hadoop-mapreduce-client-core-2.1.0-beta.jar
Downloaded: http://repo1.maven.org/maven2/com/google/guava/guava/13.0/guava-13.0.jar (1847 KB at 26.7 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/hadoop-mapreduce-client-core-2.1.0-beta.jar (1341 KB at 15.1 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/2.1.0-beta/hadoop-yarn-client-2.1.0-beta.jar
Downloaded: http://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar (973 KB at 7.8 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.1.0-beta/hadoop-yarn-api-2.1.0-beta.jar
Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar (1264 KB at 8.3 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.3.3/zookeeper-3.3.3.jar
Downloading: https://clojars.org/repo/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar
Downloaded: https://clojars.org/repo/storm/carbonite/1.5.0/carbonite-1.5.0.jar (58 KB at 8.9 KB/sec)
Downloaded: https://clojars.org/repo/storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar (296 KB at 33.4 KB/sec)
Downloaded: https://clojars.org/repo/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar (42 KB at 4.7 KB/sec)
Downloaded: https://clojars.org/repo/storm/jgrapht/0.8.3/jgrapht-0.8.3.jar (240 KB at 18.9 KB/sec)
Downloaded: https://clojars.org/repo/storm/storm-core/0.9.0-wip21/storm-core-0.9.0-wip21.jar (5197 KB at 19.1 KB/sec)
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 13 source files to /home/jifeng/hadoop/storm/storm-yarn-master/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ storm-yarn ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ storm-yarn ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 5 source files to /home/jifeng/hadoop/storm/storm-yarn-master/target/test-classes
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default) @ storm-yarn ---
[INFO] Building jar: /home/jifeng/hadoop/storm/storm-yarn-master/target/storm-yarn-1.0-alpha.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ storm-yarn ---
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom (3 KB at 0.8 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.pom
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.pom (3 KB at 3.9 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom (4 KB at 6.8 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.jar
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.jar
Downloading: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.1/plexus-utils-2.1.jar
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.jar (34 KB at 17.1 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar (158 KB at 64.6 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.jar (60 KB at 21.7 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.1/plexus-utils-2.1.jar (220 KB at 16.2 KB/sec)
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:18 min
[INFO] Finished at: 2015-03-08T22:13:01+08:00
[INFO] Final Memory: 13M/32M
[INFO] ------------------------------------------------------------------------
[jifeng@feng02 storm-yarn-master]$ ls
bin CLA.pdf create-tarball.sh lib LICENSE.txt pom.xml README.md src target
[jifeng@feng02 storm-yarn-master]$ cd target/
[jifeng@feng02 target]$ ls
classes generated-test-sources maven-status test-classes
generated-sources maven-archiver storm-yarn-1.0-alpha.jar
[jifeng@feng02 target]$ cd ..
4:storm-0.9.0解压并移到storm-yarn-master 同级
[jifeng@feng02 storm-yarn-master]$ cd lib
[jifeng@feng02 lib]$ unzip storm-0.9.0-wip21.zip
Archive: storm-0.9.0-wip21.zip
[jifeng@feng02 lib]$ mv storm-0.9.0-wip21 ../../
[jifeng@feng02 lib]$ cd ..
[jifeng@feng02 storm-yarn-master]$ cd ..
[jifeng@feng02 storm]$ ls
storm-0.9.0-wip21 storm-yarn-master storm-yarn.zip
5 :配置
[jifeng@feng02 storm-yarn-master]$ vi ~/.bash_profile
后面加上:
export STORM_WORK=$HOME/hadoop/storm
export STORM_HOME=$STORM_WORK
export PATH=$PATH:$STORM_WORK/storm-yarn-master/bin:$STORM_WORK/storm-0.9.0-wip21/bin
[jifeng@feng02 lib]$ source ~/.bash_profile
生效
6:上传storm.zip到hdfs
[jifeng@feng02 lib]$ hadoop fs -mkdir -p /lib/storm/0.9.0-wip21
[jifeng@feng02 lib]$ hadoop fs -moveFromLocal storm.zip /lib/storm/0.9.0-wip21
[jifeng@feng02 lib]$ hadoop fs -mkdir -p /user/storm
[jifeng@feng02 lib]$ hadoop fs -chown storm /user/storm
[jifeng@feng02 lib]$ cd ..
[jifeng@feng02 storm-yarn-master]$ cd ..
7:启动
storm-yarn launch /home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml
[jifeng@feng02 storm-0.9.0-wip21]$ cd ..
[jifeng@feng02 storm]$ storm-yarn launch /home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/14 17:35:23 INFO client.RMProxy: Connecting to ResourceManager at feng01/10.6.3.84:8032
15/03/14 17:35:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/14 17:35:25 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment
15/03/14 17:35:27 INFO yarn.StormOnYarn: Set the environment for the application master
15/03/14 17:35:27 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]]
15/03/14 17:35:27 INFO yarn.StormOnYarn: YARN CLASSPATH = [/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/*:/home/jifeng/hadoop/hadoop-2.4.1/contrib/capacity-scheduler/*.jar:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*]
15/03/14 17:35:27 INFO yarn.StormOnYarn: Using JAVA_HOME = [/home/jifeng/jdk1.7.0_45]
15/03/14 17:35:27 INFO yarn.StormOnYarn: Setting up app master command:[/home/jifeng/jdk1.7.0_45/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr]
15/03/14 17:35:28 INFO impl.YarnClientImpl: Submitted application application_1425798050526_0001
application_1425798050526_0001
[jifeng@feng02 storm]$
8:关闭
[jifeng@feng02 storm]$ storm-yarn shutdown -appId application_1425798050526_0001 -output ~/hadoop/storm/storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/15 21:13:27 INFO client.RMProxy: Connecting to ResourceManager at feng01/10.6.3.84:8032
15/03/15 21:13:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/15 21:13:28 INFO yarn.StormOnYarn: application report for application_1425798050526_0001 :feng03:9000
15/03/15 21:13:28 INFO yarn.StormOnYarn: Attaching to feng03:9000 to talk to app master application_1425798050526_0001
15/03/15 21:13:29 INFO yarn.StormMasterCommand: org.apache.thrift7.transport.TTransportException