how to specify 'source' in module if I don't want to use local path

Question

I have two repos(it has to be two because of my pipeline setup), one repo contains the needed terraform scripts that will creates an S3 bucket (very simple), another repo contains the backend setup, and when I test locally, I can download the model in the first repo by using:

module "xxxxxx-sagemaker" {
  source                         = "../../the_first_repo_name/terraform"
 }

and this will create the s3 bucket for me by running 'terraform apply'.

But now if I don't want to use this local path ../../the_first_repo_name/terraform, I need Concourse pipeline to apply terraform for me, how can I modify this local path so that Concourse can find the terraform scripts and download it?

I had a look at this page: https://www.terraform.io/docs/modules/sources.html, it says I can also specify a github url, but it doesn't work because my terraform codes are not merged into develop branch yet, and it seems can't recognise the code, what should I do now? Hope this makes sense, thanks.

Answer 1

The general assumption in Terraform's design here is that if you have two modules which usually (or always) change together then you'll keep them in the same repository, because the usual convention is that separate code repositories denote software that is released independently.

If you expect to usually be changing both of your modules at the same time then by far the easiest path is to keep them together in the same repository so that a particular git commit always describes a particular combination of the two, and you can send your changes through code review together, etc.

If you wish to separate them, the assumption is that one of them will be considered an "upstream dependency" of the other one, and so you'll be working on changes to them separately. The downstream one will typically depend on a particular version of the upstream one, so you can make a change to the upstream without it immediately being adopted by the downstream, and then separately change the dependency version downstream when you're ready to change it to use the new version. If you're using Git module sources then you can specify a particular "version" of the upstream using the ref argument in the source URL, which you can set to a commit id, a branch name, or a tag name.

Treating the upstream module as a system in its own right does mean you'll probably need to do some extra work to make it self-contained as a maintainable system in its own right, such as writing special test configurations within the module's repository so that you are able to test it in isolation. Those test configurations are what will give you the confidence that your changes are correct before you merge them, so that you can test the upstream independently of any particular downstream module.

Answer 2

This is something you can do with Terragrunt through the --terragrunt-source option. Terragrunt is a wrapper around Terraform for extra features.

Terragrunt CLI options