keepcalculator

06 Numbers and Powers

Terraform Data Source: What It Is & How to Use

Card No. 06·75933 · Updated · Standard formula, shown below

A Terraform data source lets you fetch and use information that already exists outside your Terraform configuration—like an existing AWS instance, a DNS zone, or a local file—without creating or managing it. You define a data source with a data block, and Terraform reads the current state of that resource during planning and applies it to your configuration. This article explains how data sources work, how they differ from managed resources, and how to use them effectively.

What Is a Terraform Data Source?

A data source is a read-only query that Terraform makes to a provider (like AWS, Azure, or Google Cloud) to fetch information about resources that are not managed by your current configuration. For example, you might have an existing VPC created manually or by another team; you can use a data source to look up its ID and then reference that ID when creating a new subnet. The data source does not create or modify the VPC—it only reads its attributes.

Data sources are declared with the data block, followed by the provider’s data source type and a local name. Here’s a basic example:

data "aws_vpc" "existing" {
  filter {
    name   = "tag:Name"
    values = ["prod-vpc"]
  }
}

resource "aws_subnet" "new" {
  vpc_id = data.aws_vpc.existing.id
  cidr_block = "10.0.1.0/24"
}

In this snippet, the data source aws_vpc finds the VPC with the tag Name=prod-vpc, and the subnet uses its ID. The data source is only read; it does not create a new VPC.

Data Source vs. Resource

The key difference is lifecycle: a resource block creates, updates, and deletes infrastructure, while a data block only reads. If you use a data source for something that doesn’t exist, Terraform will fail with an error—it won’t create it. In contrast, a resource will create the missing object. This makes data sources ideal for referencing shared infrastructure, fetching the latest AMI ID, or pulling in a list of availability zones.

Another distinction is that data sources are refreshed on every plan and apply, so they always reflect the current state of the external object. Resources, on the other hand, track their own state and may drift if someone changes the resource outside Terraform.

How to Choose Between Data Source and Resource

Use a data source when:

  • The object is not managed by your Terraform configuration (e.g., created manually or by another team).
  • You need to read dynamic information like the latest AMI ID or a list of subnets.
  • You want to avoid importing an existing resource into your state.

Use a resource when you need to create, update, or delete the object. If you already have an object but want Terraform to manage it, you can import it as a resource instead of using a data source. Importing brings the object under Terraform’s management, while a data source keeps it external.

Common Problems and How to Solve Them

One frequent issue is that a data source returns no results, causing Terraform to fail. This usually happens when the filter or name doesn’t match anything. Check the provider’s documentation for the exact filter syntax and verify the object exists in your account. For example, an AWS AMI data source might fail if the owner ID or filter is wrong.

Another problem is that data sources can be slow if they query a large API. For instance, fetching all instances in an account just to get one ID is inefficient. Use specific filters to narrow the query. Also, remember that data sources are read on every plan, so if you have many data sources, plan time can increase.

A third common mistake is using a data source to get an attribute that doesn’t exist. Each data source has a defined set of attributes; check the documentation. For example, an aws_vpc data source has id, cidr_block, default_network_acl_id, etc., but not every possible attribute is exposed.

Data Sources in Modules and Remote State

Data sources are especially useful in modules. Instead of passing dozens of IDs as variables, a module can use data sources to look up the required information. For example, a module that creates an RDS instance might look up the default VPC and subnets itself, reducing the input variables needed.

However, this can make modules harder to test and less portable. A common alternative is to use remote state data sources, which read outputs from another Terraform state file. The terraform_remote_state data source lets you fetch outputs from a different configuration, which is cleaner for cross-team dependencies. For example, a network team’s configuration can output the VPC ID, and an application team can read it via terraform_remote_state.

Data Source Dependencies and Lifecycle

Terraform understands dependencies between data sources and resources. If a data source references a resource’s attribute, Terraform will wait until the resource is created before reading the data source. Conversely, if a resource uses a data source’s attribute, the data source is read first. This automatic dependency graph prevents ordering issues.

However, there is a subtlety: data sources are read during the refresh and plan phases, not during apply. If a resource is created in the same run and a data source depends on it, Terraform will handle it correctly because it knows the dependency. But if you try to use a data source to read an attribute that is only available after apply, you may need to use a depends_on or restructure your configuration.

Frequently Asked Questions

Can a data source create a resource?

No. Data sources are strictly read-only. They fetch information from the provider’s API but never create, modify, or delete anything. If you need to create a resource, use a resource block.

What happens if a data source returns multiple results?

Most data sources require that the query return exactly one result. If it returns zero or more than one, Terraform will error. To handle multiple results, you can use a data source that returns a list, like aws_subnets (plural), and then reference individual elements by index or use a for expression.

Can I use a data source to get the current AWS account ID?

Yes. The aws_caller_identity data source returns the account ID, ARN, and user ID of the credentials being used. This is commonly used to build ARNs or to conditionally apply policies.

Is a data source refreshed on every apply?

Yes, data sources are read during the refresh phase and again during plan. This ensures the data is current. However, if the data source depends on a resource that is being created in the same run, the data source will be read after the resource is created, not before.

Can I use a data source to read a local file?

Yes, the local_file data source reads a file from disk and exposes its content. This is useful for reading configuration files, templates, or SSH keys without embedding them in the configuration.

How do I debug a data source that returns an error?

Run terraform plan with TF_LOG=DEBUG to see detailed API requests and responses. Also verify that the data source name and filters are correct by checking the provider documentation. Often the error message includes the exact API call that failed, which can help identify the issue.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →