GCP Pub Sub topics

Question

My application is based on 2 GCP cloud functions which consume messages from different topics.

The first cloud function consumes a message from the 1st topic about object creation. The second cloud function consumes a message from the 2nd topic about the status change of THE OBJECT which has been created in the connection of the first cloud function and the first topic.

So I have 2 different topics which are independent, but the 2nd topic generates messages which are the logical continuation of the messages of the 1st topic.

We do not have a guarantee that my application will consume firstly from the 1st topic and then from the 2nd topic.

How to design the architecture not to have data loss?

Answer 1

You can have an event driven pattern if:

• The events published in the 2nd topic (for the 2nd functions) are the result of the operation made by the 1st functions (for instance, the first function update a value in Firestore, the 2nd functions is triggered on value updated in Firestore)

However, if you need orchestration, use orchestration tool, like Cloud Workflow (preferably) or Cloud Composer. So, remove your 2nd PubSub topic, and orchestrate like that

• Catch the event in the PubSub topic
• Start the orchestration process with the PubSub message as parameter
• Process the message with the 1st functions.
• Get the result of the update in your orchestration tool
• Invoke the 2nd function