how to calculate the cost analysis of the composer and scheduler running time vs total cost per day in GCP?

Question

I have a requirement to cost optimization in my project, for this i need to do a check of how to calculate the cost analysis of the composer and scheduler running time vs total cost per day in GCP? I mean like how to cost optimize of the composer if we run the entire day vs we will restrict the time in non business hours?

Can someone suggest me how can we calculate this to optimize?

Thank you!

I tried by checking in the GCP console --> cost breakdown and checked the cost of my composer based on my project and downloaded the .csv file and this we gave demo with the client and they are not expected the same.

I am expecting that i need to do a check of how to calculate the cost analysis of the composer and scheduler running time vs total cost per day in GCP? I mean like how to cost optimize of the composer if we run the entire day vs we will restrict the time in non business hours?

Thanks!

Answer 1

Based on Cloud Composer environments which you use, the Google Cloud Pricing Calculator can estimate the cost of using Cloud Composer.

As per the official GCP doc on Optimize environment performance and costs guidance provided on this page might not accurately reflect specific configuration requirements of real-life applications because a number of workload-specific features affect your environment's performance. Use this guide as a starting point for further tuning of Cloud Composer environments.

Note : On March 25, 2024, Cloud Composer 1 will enter its post-maintenance mode. Google will not release any further updates to Cloud Composer 1, including updates that address security issues. We recommend migration to Cloud Composer 2.

So you can take advantage of Cloud Composer 2's auto-scaling feature to maximize resource usage as well as reduce cost. This minimizes resource utilization and expense by automatically scaling your pods up during high execution times and reducing them to zero during non-business hours.

The execution time of each DAG in your composer environment is represented by the composer running time, which can be found in the airflow user interface or logs. Keep a close eye on your composer expenses in the gcp billing interface and note when consumption is at its highest. This will assist you in defining your resource optimization and scheduling plans.

Refer to Economize blog1 and blog 2 by Manish Kumar Barnwal on how to optimize GCP Cloud Scheduler Costs: A Step-by-Step Guide and 6+ Strategies to Cut Costs on Google Cloud Composer.