I am working on a usecase where I need to read from pubSub and create a streaming pipeline. I am using colab notebook, to write and test the pipeline. But, when I am reading from pubSub, its throwing a Keyerror.
KeyError: 'ref_PCollection_PCollection_6'
I dont know from where this error is coming. Can anyone help with this?
Below is my code in colab.
import apache_beam as beam
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
import apache_beam.runners.interactive.interactive_beam as ib
from apache_beam.options import pipeline_options
from apache_beam.options.pipeline_options import GoogleCloudOptions
import google.auth
ib.options.recording_duration = '10m'
ib.options.recording_size_limit = 1e9
options = pipeline_options.PipelineOptions()
options.view_as(pipeline_options.StandardOptions).streaming = True
with beam.Pipeline(InteractiveRunner(), options=options) as p:
words = p|"read" >> beam.io.ReadFromPubSub(topic="projects/PROJECT_ID/topics/Test")| beam.WindowInto(beam.window.FixedWindows(10)) | "count" >> beam.combiners.Count.PerElement()
ib.show(words, include_window_info=True)