Apache Beam, Python and GCP: Deploying a Batch Pipeline on Google DataFlow

In this article, we will describe how to deploy a batch pipeline, created locally, to Google Dataflow, in a very simplified way.

In the previous article (here), we explored how to change a pipeline from batch to streaming with just a few extra lines. This shows us the versatility of using Apache Beam.

In this article, we will describe how to deploy a batch pipeline, created locally, to Google Dataflow, in a very simplified way. There are other methods to deploy, more or less complex. Complexity that depends on your level of knowledge in python, especially.

Let’s put your hand in the dough?

Create a Service Account

Go to IAM & Admin > Service Accounts > + Create > name your SA > Create:

Then give Dataflow Worker permission > Click Done