Mac系统安装JStorm 2.1.0配置过程

从头安装jstorm

创建安装目录

当前在桌面创建work目录

# mkdir  /work/jstrom
# cd /work/jstrom

安装jdk(需要jdk1.5以上)

1. 下载：访问Oracle官网 http://www.oracle.com，在页面的底部Download Java for Develops，同时选择Accept Lisence Agreement，下载后点击即可以安装
   Mac下安装地址：/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home
   （当前安装版本为1.8.0_131）
2. 配置环境变量
   vi ~/.bash_profile

export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home"

3. 查看安装结果

java -version

显示结果为以下则说明安装成功

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

安装python（需要2.6以上）

Mac自带python，可以通过python命令来进行查看
如果需要重新安装
（1）brew install python
（2）官网下载安装 https://www.python.org/download
通过which python查看python的安装位置

安装ZooKeeper

1. 下载并解压：当前安装版本3.4.12
   （1）从Apache网站（http://zookeeper.apache.org/releases.html）上下载zookeeper
   （2）命令行下载

# wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.12/zookeeper-3.4.12.tar.gz
# sudo tar -zxvf zookeeper-3.4.12.tar.gz

2. 配置环境变量
   可以在~/.bash_profile或者 ~/.bashrc中进行配置
   terminal运行vi ~/.bash_profile

# export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home"
ZOOKEEPER_HOME=/Users/.../work/jstorm/zookeeper-3.4.12
# export PATH="$PATH:$ZOOKEEPER_HOME"

3. 关闭当前terminal或者运行一下命令使环境变量生效

source  ~/.bash_profile

4.可以运行cd和pwd检查环境变量是否配置正确

# cd $ZOOKEEPER_HOME
# pwd //查看当前目录

5. 添加zoo.cfg配置

# cd $ZOOKEEPER_HOME/conf/
# cp zoo_sample.cfg zoo.cfg
# vi zoo.cfg

在zoo.cfg中仅需要修改dataDir的配置

dataDir=/work/jstorm/zookeeper-3.4.12/data

6. zookeeper的启动和停止

# $ZOOKEEPER_HOME/bin/zkServer.sh start
# $ZOOKEEPER_HOME/bin/zkServer.sh stop

安装jstorm

1. 下载jstorm并解压 当前版本2.1.0（jstorm-2.1.0.tar.bz2）（因为后面用到mvn编写jar包，mvn只提供三个jstorm-core的版本支持）
   下载地址： http://jstorm.io/Downloads.html
   下载至/work/jstorm

# wget https://github.com/alibaba/jstorm/releases/download/2.1.0/jstorm-2.1.0.tar.bz2
# tar -zxvf jstorm-2.1.0.tar.bz2

2. 只在2.1.0下需要注意

mv deploy/jstorm ./jstorm-2.1.0

3. 配置环境变量

export JSTORM_HOME=/Users/.../work/jstorm/jstorm-2.1.0
export PATH="$PATH:$JSTORM_HOME"

4. 配置storm.yaml

# cd jstorm-2.1.0/conf
# cp storm.yaml.example storm.yaml
# vim storm.yaml

修改storm.yaml，配置nimbus

storm.zookeeper.servers:
     - "192.168...."
     - "192.168...."
     - "192.168..."
 storm.zookeeper.root: "/jstorm"
 nimbus.host: "....."   # 只能用ip
 storm.local.dir: "%JSTORM_HOME%/data"

5. 修改supervisor配置

# cd jstorm-2.1.0/conf
# cp storm.yaml.example storm.yaml
# vim storm.yaml

讲下面的内容前面的#去掉

supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803

6. 修改集群的配置，在jstorm安装目录下conf/storm.yaml
   去掉下面内容前面的#号

 ui.cluseters:
     - {
          name: "jstorm.share",
          zkRoot: "/jstorm",
          zkServers:
              [ "192.168......", "", ""],
          zkPort: 2181,
        }

7. 启动节点

# nohup jstorm nimbus &
# nohup jstorm supervisor &

启动Nimbus：进入JStorm目录，执行bin/jstorm nimbus
启动Supervisor：进入JStorm目录，执行bin/jstorm supervisor
启动之后可以通过jps命令查看结果

可能报错

报错

MaxTenuringThreshold of 20 is invalid; must be between 0 and 15
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

解决方案：在conf/storm.yaml内添加

nimbus.childopts: " -Xms2g -Xmx2g -Xmn768m -XX:PermSize=128m  -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=15 -XX:+UseConcMarkSweepGC  -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError  -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000 "

worker.gc.childopts: " -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=15 -XX:+UseConcMarkSweepGC  -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSFullGCsBeforeCompaction=5 -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=5000 "

安装JStorm UI

1. 下载Tomcat：https://tomcat.apache.org/download-80.cgi#8.5.35
   或者
   http://mirrors.cnnic.cn/apache/tomcat/tomcat-8/v8.0.15/bin/apache-tomcat-8.0.53.tar.gz
   当前安装版本为8.0.53
2. 安装8.0.53

# tar -xzf apache-tomcat-8.0.53.tar.gz 
# cd apache-tomcat-8.0.53
# cd webapps
# cp $JSTORM_HOME/jstorm-ui-2.1.0.war ./
# mv ROOT ROOT.old
# ln -s jstorm-ui-2.1.0 ROOT
# cd ../bin
# ./startup.sh 

3. 浏览器查看： http://localhost:8080查看JStorm的管理页面

提交Topology

1. 使用Hello进行测试
   github代码：https://github.com/ashrithr/storm-helloworld
   bin/jstorm jar hello-world-1.0-SNAPSHOT.jar storm.cookbook.HelloWorldTopology HelloWorld
   生成jar包，提交Storm集群
   mvn clean install
   mvn clean package
   mvn clean deploy
2. 自己的测试样例
   bin/jstorm jar storm-1.0.jar w…papercode.SimpleTopology
   bin/jstorm jar storm-1.0.jar w…papercode.MiddleTopology
   bin/jstorm jar storm-1.0.jar w…papercode.ComplexTopology

参考博文：https://blog.fish2bird.com/?p=383
https://www.cnblogs.com/chen110xi/p/5655050.html
https://blog.youkuaiyun.com/szzhaom/article/details/41778763