快如闪电的集群计算，Spark 0.6.0 发布

jopen 12年前

Spark 0.6.0是一个重要的版本，带来了一些新的功能，体系结构的变化，以及性能增强。最显着的增加是一个独立的部署模式，一个Java API，以及扩展的文档。在某些方面性能提升了 2 倍。

您也可以下载此版本的源码包（2 MB的tar.gz）或预编译包（48 MB的tar.gz）。

更简单的部署

In addition to running on Mesos, Spark now has a standalone deploy mode that lets you quickly launch a cluster without installing an external cluster manager. The standalone mode only needs Java installed on each machine, and Spark deployed to it.

In addition, there is experimental support for running on YARN (Hadoop NextGen), currently in a separate branch.

Java API

Java programmers can now use Spark through a new Java API layer. This layer makes available all of Spark's features, including parallel transformations, distributed datasets, broadcast variables, and accumulators, in a Java-friendly manner.

文档扩展

Spark's documentation has been expanded with a new quick start guide, additional deployment instructions, configuration guide, tuning guide, and improved Scaladoc API documentation.

引擎变化

Under the hood, Spark 0.6 has new, custom storage and communication layers brought in from the upcoming Spark Streaming project. These can improve performance over past versions by as much as 2x. Specifically:

  • A new communication manager using asynchronous Java NIO lets shuffle operations run faster, especially when sending large amounts of data or when jobs have many tasks.
  • A new storage manager supports per-dataset storage level settings (e.g. whether to keep the dataset in memory, deserialized, on disk, etc, or even replicated across nodes).
  • Spark's scheduler and control plane have been optimized to better support ultra-low-latency jobs (under 500ms) and high-throughput scheduling decisions.

新的APIs

  • This release adds the ability to control caching strategies on a per-RDD level, so that different RDDs may be stored in memory, on disk, as serialized bytes, etc. You can choose your storage level using the persist() method on RDD.
  • A new Accumulable class generalizes Accumulators for the case when the type being accumulated is not the same as the types of elements being added (e.g. you wish to accumulate a collection, such as a Set, by adding individual elements).
  • You can now dynamically add files or JARs that should be shipped to your workers with SparkContext.addFile/Jar.
  • More Spark operators (e.g. joins) support custom partitioners.

增强的调试

Spark's log now prints which operation in your program each RDD and job described in your logs belongs to, making it easier to tie back to which parts of your code experience problems.

Maven Artifacts

Spark is now available in Maven Central, making it easier to link into your programs without having to build it as a JAR. Use the following Maven identifiers to add it to a project:

  • groupId: org.spark-project
  • artifactId: spark-core_2.9.2
  • version: 0.6.0

兼容性

This release is source-compatible with Spark 0.5 programs, but you will need to recompile them against 0.6. In addition, the configuration for caching has changed: instead of having a spark.cache.class parameter that sets one caching strategy for all RDDs, you can now set a per-RDD storage level. Spark will warn if you try to set spark.cache.class.

Credits

Spark 0.6 was the work of a large set of new contributors from Berkeley and outside.

  • Tathagata Das contributed the new communication layer, and parts of the storage layer.
  • Haoyuan Li contributed the new storage manager.
  • Denny Britz contributed the YARN deploy mode, key aspects of the standalone one, and several other features.
  • Andy Konwinski contributed the revamped documentation site, Maven publishing, and several API docs.
  • Josh Rosen contributed the Java API, and several bug fixes.
  • Patrick Wendell contributed the enhanced debugging feature and helped with testing and documentation.
  • Reynold Xin contributed numerous bug and performance fixes.
  • Imran Rashid contributed the new Accumulable class.
  • Harvey Feng contributed improvements to shuffle operations.
  • Shivaram Venkataraman improved Spark's memory estimation and wrote a memory tuning guide.
  • Ravi Pandya contributed Spark run scripts for Windows.
  • Mosharaf Chowdhury provided several fixes to broadcast.
  • Henry Milner pointed out several bugs in sampling algorithms.
  • Ray Racine provided improvements to the EC2 scripts.
  • Paul Ruan and Bill Zhao helped with testing.

还要感谢所有Spark用户一直在努力建议功能和报告错误。