分布式计算系统，Onyx 0.8.0 发布

jopen 9年前

Onyx 是什么？

  • Onyx 是一个无中心、支持云、容错的分布式计算系统

    </li>

  • 使用 Clojure 编写

  • 支持批处理和流处理混合

  • 提供信息模型用于描述和构建分布式工作流


  • 竞争对手： Storm, Cascading, Map/Reduce, Dryad, Apache Sqoop, 推ter Crane 等等

    • </ul>

    用途？

    • 实时的事件流处理

    • 连续计算

    • ETL

    • 在 MapReduce 进行数据转换

    • 数据清理

    • 数据摄入和存储媒介传输

    Onyx 0.8.0 发布，此版本最主要是增加了自动化状态管理功能，恢复，Windowing 和触发器。状态会被记录在 BookKeeper 日志记录，segments 可以使用嵌入式 RocksDB 删除处理。开箱支持多个类型的聚合。

    升级注意事项：

    • Onyx 0.8.0 不兼容之前的版本

    改进列表：

    • Breaking change :onyx.messaging/peer-port-range and :onyx.messaging/peer-ports are deprecated in favour of a single :onyx.messaging/peer-port port. The Aeron layer multiplexed all communication over a single port so multiple port selection is longer required.

    • Important change: default task batch timeout was reduced from 1000ms to 50ms.

    • New major feature: Windowing and Triggers.

    • New feature: Fixed windows

    • New feature: Sliding windows

    • New feature: Session windows

    • New feature: Global windows

    • New feature: Timer triggers

    • New feature: Segment-based triggers

    • New feature: Watermark triggers

    • New feature: Percentile Watermark triggers

    • New feature: Punctuation triggers

    • New feature: Accumulating refinement mode

    • New feature: Discarding refinement mode

    • New feature: BookKeeper & embedded RocksDB automatic "exactly once" filtering.

    • New feature: friendlier error messages. Added custom schema error handlers to print out documentation and required types in a more customized format.

    • New flux policy: Recover

    • New lifecycle: after-read-batch

    • New metric: peer-send-bytes

    • Bug fix: fixed an issue where the percentage job scheduler would misallocate.

    • Bug fix: fixed :data key being removed from exception messages in the logs.

    详细改进内容请看发行说明。