We have used Terraform to provision and manage infrastructure across more than fifty client deployments over the past six years. The technology has matured significantly in that time, but the most important lessons we have learned are not about syntax or features. They are about workflow, team discipline, and architectural decisions that compound over time.
State management is the foundation of a reliable Terraform workflow. We use S3 backends with DynamoDB locking for every project, without exception. Local state files are acceptable for experiments but never for shared infrastructure. We separate state into logical boundaries: networking, compute, data stores, and application configuration each get their own state file. This limits the blast radius of any single apply operation and allows different team members to work on different layers simultaneously.
The value of Terraform is not automation. It is that your infrastructure becomes reviewable, testable, and versioned like any other code.
Every module we write follows a consistent structure: a variables file with typed inputs and validation rules, a main configuration file, an outputs file that exposes only what consumers need, and a README generated from the variable descriptions. We version our modules using Git tags and reference them from a private module registry. This means a change to a module does not immediately propagate to all consumers; each project pins a specific version and upgrades deliberately.
Module Design That Scales
The mistake we see most often in client codebases is overuse of count and for_each to create dynamic infrastructure. These constructs are powerful but create implicit dependencies that make state manipulation difficult. When you need to remove one item from a list of resources created with count, Terraform may want to destroy and recreate several resources to maintain index ordering. We prefer explicit resource definitions with meaningful names, using for_each only when the resources are truly homogeneous and independently manageable.
Plan review is a non-negotiable part of our workflow. Every terraform plan output is reviewed by at least one person who did not write the change. We use Atlantis to automate plan generation on pull requests, which means reviewers see both the code diff and the infrastructure diff before approving. This has prevented numerous incidents where a seemingly minor code change would have triggered an unexpected resource replacement.
- Always use remote state with locking, never local state for shared infrastructure
- Separate state files by logical boundary to limit blast radius
- Version modules with Git tags and pin versions in consuming projects
- Review terraform plan output on every pull request before merging
- Use explicit resource names rather than relying on count indexes
- Implement automated testing with Terratest for critical modules
Testing Terraform configurations is an area where most teams invest too little. We use Terratest for integration testing of critical modules, spinning up real infrastructure, verifying its properties, and tearing it down. For faster feedback, we use terraform validate and tflint in CI. These tools catch the majority of configuration errors before any infrastructure is provisioned.
