JStorm 2.1.1 发布,分布式计算系统

jopen 9年前

Storm 是一个类似Hadoop MapReduce的系统, 用户按照指定的接口实现一个任务,然后将这个任务递交给JStorm系统,Jstorm将这个任务跑起来,并且按7 * 24小时运行起来,一旦中间一个worker 发生意外故障, 调度器立即分配一个新的worker替换这个失效的worker。因此,从应用的角度,JStorm 应用是一种遵守某种编程规范的分布式应用。从系统角度, JStorm一套类似MapReduce的调度系统。 从数据的角度, 是一套基于流水线的消息处理机制。实时计算现在是大数据领域中最火爆的一个方向,因为人们对数据的要求越来越高,实时性要求也越来越快,传统的Hadoop Map Reduce,逐渐满足不了需求,因此在这个领域需求不断。

新特性:

  1. 1.5~6x performance boost from worst to best scenarios compared to JStorm-2.1.0

  2. Add application-level auto-batch

  3. Add independent control channel to separate control msgs from biz msgs to guarantee high priority for control msgs

  4. Dramatic performance boost in metrics, see "Improvements" section

  5. Support jdk1.8

  6. Add Nimbus hook and topology hook

  7. Metrics system:

    1. Support disable/enable metrics on the fly

    2. Add jstorm metrics design docs, see JSTORM-METRICS.md

  8. JStorm web UI:

    1. Add zookeeper viewer in web UI, thanks to @dingjun84

    2. Add log search and deep log search, support both backward search and forward search

    3. Support log file download

  9. Support changing log level on the fly

  10. Change error structure in zk, add errorLevel, errorCode and duration.

  11. Add supervisor health check

  12. Add -Dexclude.jars option to enable filtering jars manually

Bug 修复:

  1. Fix deadlock when starting workers

  2. Fix the bug that when localstate file is empty, supervisor can't start

  3. Fix kryo serialization for HeapByteBuffer in metrics

  4. Fix total memory usage calculation

  5. Fix the bug that empty worker is assigned when configured worker number is bigger than the actual number for user defined scheduler

  6. Fix UI log home directory

  7. Fix XSS security bug in web UI

  8. Don't start TopologyMetricsRunnable thread in local mode, thanks to @L-Donne

  9. Fix JSTORM-141, JSTORM-188 that TopologyMetricsRunnable consumes too much CPU

  10. Remove MaxTenuringThreshold JVM option support jdk1.8, thanks to @249550148

  11. Fix possible NPE in MkLocalShuffer

下载地址:


来自: http://www.oschina.net//news/71306/jstorm-2-1-1