We are moving towards separating our huge codepipeline to smaller codepipelines. But before doing that, I needed some feedback regarding the same.

For now we have one codepipeline for one of the projects, which deploys:

  • Network stack (VPC, Subnets etc)
  • Resource Stack (RDS, Redis, DynamoDB, S3 bucket)
  • Firehose-With-Transformation-Lambda-Stack (Sets up firehose with transformation lambda, stores data in the S3 )
  • Lambda stack (Project specific business logic)
  • API Gateway stack (Endpoints, api keys, basepath mapping)
  • Monitoring stack (Get metrics from all the resources from the previous stacks and then present a dashboard)

The codepipeline is triggered by codecommit, so that whenever we push code changes the pipeline gets triggered.

Now there are a couple of major reasons why we want to separate out the pipeline above.

  1. A small change in lambda takes really long time to go through the codepipeline. -> Quick solution to update the lambda directly from within a make file to just update the lambda directly but thats more of a hack.
  2. Some of the resources like the network and some databases like RDS and DynamoDB are being used by multiple projects who for now have their own small infrastructure.
  3. Arranging smaller and more codepipelines would enable us to perform separation of concern and not going through them again and again when changes are performed in different part of the infrastructure in a different stack.

Based on the above information, I have a few questions:

  1. What are the pros and cons of the approach?
  2. How can we perform triggering of one codepipeline when the parent one ends? (Anyway to do using CW events?) - This is needed in case something changed in a parent stack (in a different codepipeline) which involved replacement of the resources, and thus the values need to be propagated to the child stack (in a different codepipeline)

More Information:

During my research - this is what I have found out. Cloudwatch Events rule which can get triggered based on CodePipeline State change - https://docs.aws.amazon.com/codepipeline/latest/userguide/detect-state-changes-cloudwatch-events.html

Also, an AWS solutions architect pointed out that I can use Step functions for CodePipeline synchronization but I need more than 1 start state for different CodePipelines.

0 Answers