I'm having trouble to know how to properly work with Terraform in a distributed team.
My team is using using git to work with terraform "*tf" files, using shared AWS accounts.
We keep a shared distributed Terraform state in an AWS S3 bucket and each team member is working on its own terraform resource/s on its own branch. For example some member add some resource to an API-gateway while another add a lambda.
The different git branches with different terraform *tf are not synchronized/merged for 1-3 days, so teams members do not see "*tf" resources created by other team members in the mean time, just the shared state and the local/non-shared resource declarations.
With this scenario Terraform is constantly breaking with "terraform apply" being locked while trying to free resources created by someone else.
What is the correct flow / procedure to sync shared state in this scenario?
I guess creating "micro-shared-states" is the easy solution, but then I'm wondering what's the final purpose of a shared state, other than being able to recreate all the infrastructure in a single "terraform apply" (vs multiple "terraform apply" to multiple subdirectories). Wouldn't it be better to keep everything on git and let git share the state? Is there anything wrong (any anti-pattern) with the idea of git-distributed micro-states?
I would advise against storing your .tfstate file in version control. In AWS storing state in s3 is your best option.
One issue with storing your .tfstate in git is it may contain sensitive data such as database keys or rsa keys that are not appropriate to expose in version control.
Another shortcoming of using git is if you or another team member forgets to push their changes to .tfstate to the remote repository in git, you may run into state conflicts anyway.
As you suggest in your last paragraph, It might be helpful to start segregating your terraform into smaller logical groupings and have a seperate state for each of them. You can use a seperate backend configuration for each logical grouping. They each use a different key of the same s3 bucket if you do not want many s3 buckets.
Here is an excellent Stack Overflow response about commiting state to git vs using remote state, I strongly reccomend reviewing it:
Should I commit .tfstate files to Git?
Here is Terraform's own documentation about remote state and how it applies to the the problem you are describing of conflict between teams with more than one dev working simultaneously:
https://www.terraform.io/language/state/remote