高效和分布式的通用数据处理平台：Apache Flink

jopen 10年前

Apache Flink 是高效和分布式的通用数据处理平台。

Apache Flink 声明式的数据分析开源系统，结合了分布式 MapReduce 类平台的高效，灵活的编程和扩展性。同时在并行数据库发现查询优化方案。

 DataSet<String> input = env.readTextFile(inputPath);    input.flatMap(new FlatMapFunction() {     public void flatMap(String value, Collector out) {         for (String s : value.split(" ")) {             out.collect(new Tuple2<String, Long>(s, 1L);         }     }  })  .groupBy(0)  .sum(1)  .writeAsText(outputPath);

System Stack

The Apache Flink stack consists of

  • Programming APIs for different languages (Java, Scala) and paradigms (record-oriented, graph-oriented).
  • A program optimizer that decides how to execute the program for good performance. It decides among other things about data movement and caching strategies.
  • A distributed runtime that executes programs in parallel distributed over many machines.

Flink runs independently from Hadoop, but integrates seamlessly with YARN (Hadoop's next-generation scheduler). Various file systems (including the Hadoop Distributed File System) can act as data sources.

高效和分布式的通用数据处理平台：Apache Flink

项目主页：http://www.open-open.com/lib/view/home/1409195666666