storm集群和单点安装

lnmz3156 9年前

来自： http://my.oschina.net/goudingcheng/blog/614236

Storm实时分析平台的起源，Storm分布式集群实施
流式处理场景：算法交易
storm组件
1.nimbus集群中心，控制节点
2.supervisor nodes taskernode
3.the zookeeper cluster 交换信息
storm数据模型：由许多记录组成，送入
环境准备
1.关闭防火墙
2.修改hosts
192.168.88.197 user5.hadoop.com
192.168.88.198 user6.hadoop.com
192.168.88.196 user7.hadoop.com
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
tar -zxvf zookeeper-3.3.6.tar.gz
cd zookeeper-3.3.6
mkdir data
cd conf/
cp -rp zoo_sample.cfg zoo.cfg
dataDir=/root/soft/zookeeper-3.3.6/data
前一个端口选举，后一个端口zookeeper进行通讯
server.1=user5.hadoop.com:2888:3888
server.2=user6.hadoop.com:2888:3888
server.3=user7.hadoop.com:2888:3888
cd ~
scp -rp zookeeper-3.3.6/ root@ip:/home/grid
[root@localhost data]# cd /root/soft/zookeeper-3.3.6/data

创建myid
在dataDir(/home/grid/zookeeper-3.4.6/data)中创建一个myid
因为server.1=user5.hadoop.com:2888:3888指定的是1，所以在user5.hadoop.com的
机器上echo “1” >myid
其余两台机器配置，user6.hadoop.com myid=2 user7.hadoop.com myid=3

/root/soft/zookeeper-3.3.6/bin
[root@localhost bin]#
[root@localhost bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: leader
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower
[root@localhost bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower

安装python
https://www.python.org/downloads/ftp/python2.6.6
python
[root@localhost Desktop]# tar -zxf Python-2.7.2.tgz
[root@localhost soft]# cd Python-2.7.2
./configure --prefix=/usr/local/python27
make
make install
[root@localhost Python-2.7.2]# ls /usr/local/python27/ -al
total 28
drwxr-xr-x 6 root root 4096 Feb 5 23:29 .
drwxr-xr-x 12 root root 4096 Feb 5 23:28 ..
drwxr-xr-x 2 root root 4096 Feb 5 23:29 bin
drwxr-xr-x 3 root root 4096 Feb 5 23:29 include
drwxr-xr-x 4 root root 4096 Feb 5 23:29 lib
drwxr-xr-x 3 root root 4096 Feb 5 23:29 share
[root@localhost Python-2.7.2]# mv /usr/bin/python /usr/bin/python_old
[root@localhost Python-2.7.2]# ln -s /usr/local/python27/bin/python /usr/bin/
[root@localhost Python-2.7.2]# python
Python 2.7.2 (default, Feb 5 2016, 23:28:47)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[root@localhost Python-2.7.2]# vi /usr/bin/yum
#!/usr/bin/python2.4

0.9之前的安装步骤
同样安装JDK和python，zookeeper集群
2.和前面一样，只是要多安装storm的两个依赖库，zeroMQ和JZMQ
1.搭建zookeper集群
2.安装storm依赖库【JDK和python，zeroMQ，JZMQ】
3.下载并且解压storm发布版本
4.修改storm.yaml文件
5.启动storm后台进程

[root@localhost soft]# wget http://mirror.bit.edu.cn/apache/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz
[root@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
t@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
@localhost soft]# mv apache-storm-0.9.1-incubating storm-0.9.1
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
storm.yaml
[root@localhost conf]# vi storm.yaml

storm.zookeeper.servers:
     - "192.168.88.197"
     - "192.168.88.198"
     - "192.168.88.196"
#
nimbus.host: "192.168.88.197"
storm.local.dir:"/root/soft/storm-0.9.1/data"
supervisor.slots.ports:
-6700
-6701
-6702
-6703

/root/soft/storm-0.9.1
[root@localhost soft]# scp -rp storm-0.9.1/ root@user6.hadoop.com:~/soft
[root@localhost soft]# scp -rp storm-0.9.1/ root@user7.hadoop.com:~/soft
.BASH_PROFILE
STORM_HOME=/root/soft/storm-0.9.1
PATH=$PATH:$HOME/bin:$STORM_HOME/bin
export STORM_HOME最下面

[root@localhost storm-0.9.1]# source /etc/profile
[root@localhost storm-0.9.1]# echo $PATH
/root/soft/storm-0.9.1/bin
[root@localhost ~]# jps确保三台都能够正常启动
5215 QuorumPeerMain
23327 Jps

[root@localhost conf]# storm nimbus &在nimbus节点
[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@user6 ~]# storm supervisor &
[root@user6 ~]# jps
\5165 QuorumPeerMain
24473 Jps
24354 supervisor

[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@localhost conf]# storm logviewer &
[root@localhost conf]# storm ui &网址8080端口
[root@localhost conf]# jps
5536 nimbus
5841 logviewer
6013 core
5431 QuorumPeerMain
6074 Jps

[root@user6 ~]# jps
5165 QuorumPeerMain
24678 Jps
24354 supervisor

安装git
yum install git
安装maven
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
解压配置环境变量
[root@localhost soft]# git clone git://github.com/nathanmarz/storm-starter.git
[root@localhost soft]# cd storm-starter
[root@localhost storm-starter]# ls
LICENSE     multilang    README.markdown storm-starter.iml
m2-pom.xml project.clj src              test
[root@localhost storm-starter]# mv m2-pom.xml pom.xml
[root@localhost storm-starter]# vi pom.xml
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>0.9.1-incubating</version>
      
      <scope>provided</scope>
    </dependency>

[root@localhost storm-starter]# mvn compile exec:java -Dstorm.topology=storm.starter.wordCountTopology

mvn package
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.RollingTopwords
远程模式
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology ExclamationTopology
storm kill ExclamationTopology
安装ZeroMQ
安装依赖包
yum install gcc-c++ libuuid-devel
对于libuuid-devel
先安装ncurses-5.9.tar
在下载ftp://ftp.kernel.org/pub/linux/utils/util-linux/v2.21/util-linux-2.21.1.tar.gz
编译安装
下载http://download.zeromq.org/zeromq-2.1.7.tar.gz
编译安装JZMQ
git clone git://github.com/nathanmarz/jzmq.git
cd jzmq
[root@localhost jzmq]# ./autogen.sh
[root@localhost jzmq]# ./configure
make
make install
下载并解压storm第三台
yum install unzip
wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
ocalhost soft]# unzip storm-0.8.1.zip
[root@localhost soft]# cd storm-0.8.1
cd storm-0.8.1
修改conf/storm.yaml文件在配置的每一项前加空格，冒号后也要加空格
storm.zookeeper.servers:
     - "192.168.88.197"
     - "192.168.88.198"
     - "192.168.88.199"
#
nimbus.host: "192.168.88.199"
storm.local.dir: "/root/soft/storm-0.8.1/data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
[root@localhost conf]# jps
5215 QuorumPeerMain
13883 Jps
修改环境变量

[root@localhost ~]# storm nimbus &
[1] 13974
[root@localhost ~]# jps
[root@localhost ~]# jps
14127 Jps
13974 nimbus
5215 QuorumPeerMain
14037 logviewer
14059 core

storm集群和单点安装

相关经验

目录