Apache Beam: Difference between revisions

From David's Wiki
Created page with "[https://beam.apache.org/ Apache Beam] is a library for building parallel data pipelines.<br> Such pipelines are executed on a runner such as Apache Spark. Apache Beam is orig..."
 
No edit summary
Line 1: Line 1:
[https://beam.apache.org/ Apache Beam] is a library for building parallel data pipelines.<br>
[https://beam.apache.org/ Apache Beam] is a library for building parallel data pipelines.<br>
Such pipelines are executed on a runner such as Apache Spark. Apache Beam is originally developed by Google.
Such pipelines are executed on a runner such as Apache Flink. Apache Beam is originally developed by Google.


==Usage==
==Usage==

Revision as of 15:44, 20 January 2022

Apache Beam is a library for building parallel data pipelines.
Such pipelines are executed on a runner such as Apache Flink. Apache Beam is originally developed by Google.

Usage

Programming guide, examples in Python.

Background

Data are referred to as PCollection

Create

Map

ParDo

Pardo allows you to pass in a function and generate multiple items.
If you are yielding many items though, you should do a beam.Reshuffle() afterwards to split and get more parallelism.