Apache Beam, Python and GCP: Deploying a Streaming Pipeline on Google DataFlow using PubSub

Following a sequence of articles on APACHE BEAM, here we will describe how to deploy a streaming pipeline, created locally, to Google Dataflow, in a very simplified way.

I already published 2 previous articles about apache beam. One talking about the simplicity of changing a batch pipe to streaming in this framework ( here ), and another about how to publish a batch pipeline on Google DataFlow ( here ). And in this third article, I want to share with you the publication of a streaming pipeline, in a simplified way, where you consume data from a PubSub subscription and write, another PubSub topic.

ready?

CREATE SERVICE ACCOUNT

Go to IAM & Admin > Service Accounts > + Create > name your SA > Create:

Then give Dataflow Worker permission > Click Done: