Reddit mentions of Terraform: Up and Running: Writing Infrastructure as Code

I think the approach you've described seems totally reasonable.

​

Setting aside the aspect of one vs multiple repos, there is a spectrum of component isolation. So for example, at one end would be a single terraform configuration (main.tf) that defines ALL your resources -- the lambdas, the queues, the dbs, etc. This has some nice properties -- for example less bookkeeping, avoidance of deployment ordering issues, and no need for a mechanism to share references to resources (such as the shared tfvar file you mentioned). And it is possible with this setup to deploy only specific resources by using the -target flag when doing terraform apply.

​

But to your point, every time you deploy a lambda you're putting at risk all these other resources if something goes sideways.

​

At the other end of the spectrum, with highly isolated components, split across several tf configs / state files, you mitigate that risk but it creates other issues.

​

You mentioned the one about needing to share resource names across tf configurations...for that, IMO, the remote_state data source would be a better mechanism to share outputs across tf configurations.

​

Another problem specific to lambda subscriptions that you might run into is that you have to define each lambda's event source, and the way you do this varies confusingly (cause AWS, not cause tf). Example: you'd create an aws_lambda_event_source_mapping to have the lambda subscribe to an SQS queue, and an aws_s3_bucket_notification to have it subscribe to bucket events. If components are split across configurations (and repos)...which tf config owns those mappings? The first I'd argue makes sense with the lambda, but the second might make more sense alongside the bucket.

​

So you have to share ARNs or names or whatever across tf configurations, but then you also have to decide how to split up these resources that are kind of on the fence, and this might create dependency ordering problems, like, you have to deploy the SQS queue before lambda #1, but lambda #2 before the S3 bucket notification, etc.

​

Zooming out: for this kind of architecture I think it's useful to group together stateful and stateless resources. TF is great for deploying the stateful resources like buckets, queues, dbs. There are arguably better tools for deploying the stateless resources, i.e. the lambda functions, such as the serverless framework or other language-specific frameworks. (Where tf is really painful is setting up API Gateways, lolsob, and serverless makes that part easy.)

​

But you still need to deal with this issue of communicating resource identifiers across the stateful/stateless resource definitions. And something like serverless framework provides a nice clean abstraction over "lambda events," but that abstraction becomes quite leaky when you're not able to do something like subscribe a lambda to an existing bucket that was created by tf because Cloudformation (which serverless uses under the hood) is like "nope, I don't control that bucket, and the subscription is owned by the bucket -- can't do it."

​

Sorry for the longwinded answer :D

​

TLDR: lambda subscriptions are confusing, and I would probably do something like what you're advocating but consider putting it all into separate terraform configs in the same repo if it makes sense, and use the remote_state data source. And if the lambda deployments / APIGW stuff becomes painful, swap in a better tool for that part at that time.

​

Terraform: up and running is a really great book and chapter 3 in particular has more advice that might be useful for your situation!