☁️ Azure Data Factory Dataset — Azure Blob Terraform Module

A typed Data Factory dataset describing a blob, or a folder of blobs, in Azure Blob Storage, targeting hashicorp/azurerm ~> 4.0.

🧩 Overview

🗂️ Creates one azurerm_data_factory_dataset_azure_blob — a named description of a blob or blob folder that pipelines read from and write to.
🔗 Points at a linked service by bare name — not a Resource ID, unlike the Azure SQL Table dataset in the same family. Nothing can verify a bare name.
🎚️ Renders path and filename alongside two booleans that change what those strings mean: dynamic_path_enabled and dynamic_filename_enabled decide literal versus Data Factory expression.
🚨 Compares each flag against whether its value actually looks like an expression, and emits four distinct mismatch flags plus a roll-up — because the provider cross-validates none of it.
🧾 Renders the same closed, case-sensitive fifteen-value schema_column block as its SQL sibling.

💡 Why it matters: a wrong dynamic flag produces no error at any stage. Terraform applies, the pipeline runs, and either an expression is used as a literal folder name or a literal is evaluated as an expression. The result is a successful run against the wrong data.

❤️ Support this project

If this module saved you time:

⭐ Star the repository — it is the cheapest signal that this work is worth continuing.
💼 Connect on LinkedIn: linkedin.com/in/microsoftexpert
☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

🗺️ Where this fits in the family

flowchart TB
  RG["terraform-azurerm-resource-group"]
  SA["terraform-azurerm-storage-account"]
  ADF["terraform-azurerm-data-factory"]
  LS["a Data Factory linked service"]
  THIS["terraform-azurerm-data-factory-dataset-azure-blob"]
  SQLT["terraform-azurerm-data-factory-dataset-azure-sql-table"]
  CUSTOM["terraform-azurerm-data-factory-custom-dataset"]
  PIPE["a Data Factory pipeline"]
  BLOBS["a container and its blobs"]

  RG -->|"name"| ADF
  RG -->|"name"| SA
  SA -->|"named inside path, never read here"| THIS
  ADF -->|"id"| THIS
  ADF -->|"id"| SQLT
  ADF -->|"id"| CUSTOM
  LS -->|"linked_service_name, a bare NAME"| THIS
  LS -->|"linked_service_id, a full Resource ID"| SQLT
  THIS -->|"name, referenced by"| PIPE
  PIPE -->|"reads only at run time, never here"| BLOBS

  classDef this fill:#0078D4,stroke:#004578,color:#ffffff,stroke-width:2px
  classDef keystone fill:#004578,stroke:#00243d,color:#ffffff,stroke-width:2px
  classDef sibling fill:#eef3f8,stroke:#b9c8d8,color:#1b2733
  class THIS this
  class ADF keystone
  class RG,SA,LS,SQLT,CUSTOM,PIPE,BLOBS sibling

The storage account is named inside path, never read. The edge labels carry the family's real difference: a name arrives here, an ID arrives at the SQL table dataset.

🧬 What this module builds

flowchart TB
  subgraph INPUTS["Inputs"]
    NAME["name (force-new)"]
    ADFID["data_factory_id (force-new)"]
    LSNAME["linked_service_name, a bare NAME"]
    PATHS["path and filename"]
    FLAGS["dynamic_path_enabled and dynamic_filename_enabled"]
    COLS["schema_column list"]
    PARAMS["parameters and additional_properties"]
    META["annotations, description, folder"]
  end

  THIS["azurerm_data_factory_dataset_azure_blob.this"]

  subgraph OUTPUTS["Outputs"]
    OID["id and name"]
    OMISMATCH["dynamic_flags_look_inconsistent, plus its four components"]
    ONONE["describes_no_specific_blob"]
    OUNTYPED["untyped_schema_column_names"]
    OFOLDER["folder_and_path_mean_different_things"]
    OFACTS["the constant facts, including the clone refutation"]
  end

  NAME --> THIS
  ADFID --> THIS
  LSNAME --> THIS
  PATHS --> THIS
  FLAGS --> THIS
  COLS --> THIS
  PARAMS --> THIS
  META --> THIS

  THIS --> OID
  PATHS --> OMISMATCH
  FLAGS --> OMISMATCH
  PATHS --> ONONE
  PARAMS --> ONONE
  COLS --> OUNTYPED
  META --> OFOLDER
  THIS --> OFACTS

  classDef this fill:#0078D4,stroke:#004578,color:#ffffff,stroke-width:2px
  classDef sibling fill:#eef3f8,stroke:#b9c8d8,color:#1b2733
  class THIS this
  class NAME,ADFID,LSNAME,PATHS,FLAGS,COLS,PARAMS,META,OID,OMISMATCH,ONONE,OUNTYPED,OFOLDER,OFACTS sibling

Resource	Count	Notes
`azurerm_data_factory_dataset_azure_blob.this`	1	The keystone. A real ARM child of the factory.
`schema_column`	`dynamic`, 0..n	Optional column definitions.
`timeouts`	`dynamic`, 0..1	All four keys exist on this resource.

✅ Provider / Versions

Requirement	Value
Terraform	`>= 1.12.0`
`hashicorp/azurerm`	`~> 4.0`
Provider block	None. The caller configures the provider, including the mandatory `features {}` block.

Schema notes that bite — verified against the live provider source, not inferred from the schema:

🔴 This resource takes linked_service_name, a bare name checked only for non-emptiness. Its sibling azurerm_data_factory_dataset_azure_sql_table takes linked_service_id. This resource's own documentation calls its argument set "common across all Azure Data Factory Datasets" and lists linked_service_name in it — and the sibling contradicts that.
🔴 dynamic_path_enabled and dynamic_filename_enabled reinterpret path and filename, and nothing cross-validates them. A wrong flag applies cleanly and fails — or silently succeeds against the wrong blob — at pipeline run time.
⚠️ folder is not part of path. folder is the portal's authoring tree; path is the storage container and prefix. The names invite exactly the wrong reading.
🔴 schema_column.type is CASE-SENSITIVE, fifteen values, identical to the SQL table dataset's. "String" is accepted; "string" is rejected. Byte[] carries brackets.
⚠️ path, filename and parameters are all optional. A dataset with none of them applies and resolves to nothing.
⚠️ Only name and data_factory_id force replacement. The linked service, path, filename and both dynamic flags update in place.
⚠️ Timeouts default to 30m / 5m / 30m / 30m.

🔑 Required Azure RBAC Roles / Permissions

Permission	Scope	Why
`Microsoft.DataFactory/factories/datasets/write`	the Data Factory	Create and update.
`Microsoft.DataFactory/factories/datasets/read`	the Data Factory	Refresh and plan.
`Microsoft.DataFactory/factories/datasets/delete`	the Data Factory	Destroy — see the warning below.
`Data Factory Contributor`	the Data Factory	The built-in role containing all three.

🔒 No permission on the storage account is required, requested or used. The dataset names a container and a path; it never reads them. Storage access belongs to the linked service, so whoever can write datasets can point one at any container the factory's identity can already reach — including data they have no direct permission on. Say so in a review: it is invisible from the storage side.

⚠️ Delete is the operation to review, not create. Creating a dataset changes nothing. Removing one breaks every pipeline that references it by name.

Azure Prerequisites

An existing Data Factory, and its Resource ID.
A linked service already configured in that factory, and its name. Nothing here verifies it exists, belongs to this factory, or points at Blob Storage.
A storage account and container for path to name — or parameters, if the path is chosen per run.
A dataset name meeting Microsoft's Data Factory naming rules. The provider checks only non-emptiness.
The Microsoft.DataFactory resource provider registered in the subscription.

📁 Module Structure

terraform-azurerm-data-factory-dataset-azure-blob/
├── providers.tf    # required_version + the pinned azurerm; no provider block
├── variables.tf    # 14 typed inputs, 12 validations
├── main.tf         # the keystone, the dynamic-flag analysis, two dynamic blocks
├── outputs.tf      # 57 outputs; id first
├── README.md       # this file
├── SCOPE.md        # the cross-module contract
├── LICENSE         # MIT
└── .gitignore

⚙️ Quick Start

provider "azurerm" {
  features {}
}

module "landing_blobs" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path     = "landing/orders"
  filename = "orders.csv"
}

ℹ️ The caller configures the provider, its authentication, and the mandatory features {} block. This module declares none of them.

🔌 Cross-Module Contract

Consumes

Input	Type	Typical source
`data_factory_id`	`string`	`terraform-azurerm-data-factory` → `id`
`linked_service_name`	`string`	a linked service's name — not a Resource ID
`path` / `filename`	`string`	the caller

Emits (selected — 57 in total)

Output	Description
`id`	The dataset's Resource ID.
`name`	How a pipeline refers to it.
`dynamic_flags_look_inconsistent`	The one output to assert on.
`describes_no_specific_blob`	Applies cleanly, resolves to nothing.
`folder_and_path_mean_different_things`	Constant. The confusion that produces wrong-data success.
`untyped_schema_column_names`	Columns given no type.
`the_dataset_family_is_not_a_clone_cluster`	Constant. Quotes the documentation that says otherwise.

📚 Example Library

1 · The minimum call

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name
}

⚠️ Legal, and it describes no blob — no path, no filename, no parameters. describes_no_specific_blob is true.

2 · A literal path and filename

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path     = "landing/orders"
  filename = "orders.csv"
}

💡 Both dynamic flags default to false, so both strings are used exactly as written.

3 · A dynamic path — the flag and the value must agree

module "daily" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_daily"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path                 = "@concat('landing/', formatDateTime(utcnow(), 'yyyy/MM/dd'))"
  dynamic_path_enabled = true

  filename = "orders.csv"
}

🔒 The value is a Data Factory expression and the flag is set. dynamic_flags_look_inconsistent is false. Get either half wrong and nothing errors.

4 · The mistake this module exists to catch

module "broken" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_broken"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path = "@concat('landing/', formatDateTime(utcnow(), 'yyyy/MM/dd'))"
  # dynamic_path_enabled is left at false
}

check "dynamic_flags_agree" {
  assert {
    condition     = !module.broken.dynamic_flags_look_inconsistent
    error_message = "path looks like an expression but dynamic_path_enabled is false; it will be used literally."
  }
}

⚠️ Without the check, this applies cleanly and Data Factory looks for a folder literally named @concat(...). path_expression_without_dynamic_flag is true.

5 · The opposite mismatch

module "also_wrong" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_also_wrong"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path                 = "landing/orders" # a plain literal
  dynamic_path_enabled = true             # ... marked dynamic
}

output "which_mismatch" {
  value = {
    expression_without_flag = module.also_wrong.path_expression_without_dynamic_flag # false
    marked_but_literal      = module.also_wrong.path_marked_dynamic_but_looks_literal # true
  }
}

💡 The two mismatches are emitted separately because the remedy differs: one means "set the flag", the other means "this value is not an expression". The roll-up exists for callers who only want one check.

6 · When the heuristic is wrong, and why it only reports

module "literal_at" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_literal_at"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path     = "@archive/2026" # a real folder that starts with @
  filename = "orders.csv"
}

ℹ️ path_expression_without_dynamic_flag is true here and the configuration is correct. A literal path may legitimately begin with @, so the module reports and never refuses — refusing would reject legal input, and a failing validation {} would also block terraform destroy.

7 · A dynamic filename with a literal path

module "hourly" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_hourly"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path = "landing/orders"

  filename                 = "@concat('orders_', formatDateTime(utcnow(), 'HH'), '.csv')"
  dynamic_filename_enabled = true
}

💡 The two flags are independent — a fixed folder with a rotating filename is the common shape.

8 · `folder` is not `path`

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  path   = "landing/orders" # WHERE THE DATA IS, in storage
  folder = "bronze/ingest"  # where the dataset APPEARS, in the portal's tree
}

⚠️ Confusing these produces a working apply pointing at the wrong storage location — no error, wrong data. folder_and_path_mean_different_things is emitted as a constant for exactly this reason.

9 · A typed column schema

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name
  path                = "landing/orders"
  filename            = "orders.csv"

  schema_column = [
    { name = "OrderId", type = "Int64", description = "Primary key." },
    { name = "Placed", type = "DateTimeOffset" },
    { name = "Total", type = "Decimal" },
  ]
}

🔒 The fifteen legal types are case-sensitive and identical to the SQL table dataset's: Byte, Byte[], Boolean, Date, DateTime, DateTimeOffset, Decimal, Double, Guid, Int16, Int32, Int64, Single, String, TimeSpan.

10 · A parameterised dataset

module "any_blob" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_any_blob"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name

  parameters = {
    containerPath = "landing"
    fileName      = ""
  }
}

💡 With parameters set, describes_no_specific_blob is false even with no path — each pipeline run supplies one. This is the normal shape for a generic copy pipeline.

11 · Many datasets from one map

locals {
  feeds = {
    orders    = "landing/orders"
    customers = "landing/customers"
    ledger    = "landing/ledger"
  }
}

module "feeds" {
  for_each = local.feeds

  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_${each.key}"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name
  path                = each.value
  filename            = "${each.key}.csv"
  folder              = "bronze/blob"
}

check "no_feed_has_a_flag_mismatch" {
  assert {
    condition     = alltrue([for m in module.feeds : !m.dynamic_flags_look_inconsistent])
    error_message = "At least one feed's dynamic flags disagree with its path or filename."
  }
}

💡 The provider takes no lock on the Data Factory for this resource, so these are created concurrently and safely.

12 · Annotations, description, and the escape hatch

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"
  data_factory_id     = var.data_factory_id
  linked_service_name = var.blob_linked_service_name
  path                = "landing/orders"

  description = "Raw order extracts, dropped hourly by the upstream vendor."
  annotations = ["bronze", "hourly", "vendor-feed"]

  additional_properties = {
    "structure" = "[]"
  }
}

⚠️ annotations is a list of strings, unrelated to Azure resource tags — which this resource does not support at all. additional_properties writes unvalidated top-level keys; a collision with a managed field is reported by nothing. uses_additional_properties flags that the hatch is open.

13 · Re-pointing a dataset is an in-place update

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing"           # changing this REPLACES
  data_factory_id     = var.data_factory_id    # changing this REPLACES
  linked_service_name = var.other_linked_svc   # changing this updates IN PLACE
  path                = "archive/orders"       # changing this updates IN PLACE
}

output "what_forces_replacement" {
  value = module.landing.force_new_fields # ["name", "data_factory_id"]
}

⚠️ Everything describing what the dataset reads — including which storage account, via the linked service — changes in place. A review scanning plans for destroys will not see it.

14 · 🏗️ End-to-end composition

provider "azurerm" {
  features {}
}

module "rg" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-analytics-eastus"
  location = "eastus"
}

module "storage" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-storage-account.git?ref=v1.0.0"

  name                = "stanalyticseastus01"
  resource_group_name = module.rg.name
  location            = module.rg.location

  containers = {
    landing = {}
  }
}

module "adf" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory.git?ref=v1.0.0"

  name                = "adf-analytics-eastus"
  resource_group_name = module.rg.name
  location            = module.rg.location

  identity = {
    type = "SystemAssigned"
  }
}

# No linked-service module exists in this suite yet, so the linked service is
# declared directly. It authenticates with the factory's managed identity, so
# no secret appears anywhere in this configuration.
resource "azurerm_data_factory_linked_service_azure_blob_storage" "landing" {
  name            = "ls_blob_landing"
  data_factory_id = module.adf.id

  service_endpoint     = module.storage.primary_blob_endpoint
  use_managed_identity = true
}

module "landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-dataset-azure-blob.git?ref=v1.0.0"

  name                = "ds_landing_orders"
  data_factory_id     = module.adf.id
  linked_service_name = azurerm_data_factory_linked_service_azure_blob_storage.landing.name

  path                 = "@concat('landing/', formatDateTime(utcnow(), 'yyyy/MM/dd'))"
  dynamic_path_enabled = true

  filename = "orders.csv"
  folder   = "bronze/blob"

  schema_column = [
    { name = "OrderId", type = "Int64" },
    { name = "Placed", type = "DateTimeOffset" },
    { name = "Total", type = "Decimal" },
  ]

  annotations = ["bronze", "daily"]
}

check "wiring_is_consistent" {
  assert {
    condition     = !module.landing.dynamic_flags_look_inconsistent
    error_message = "A dynamic flag disagrees with the value it governs."
  }
  assert {
    condition     = !module.landing.describes_no_specific_blob
    error_message = "The dataset resolves to nothing."
  }
}

output "dataset_id" {
  value = module.landing.id
}

output "reads_from" {
  value = module.landing.path
}

🔒 The factory's system-assigned identity authenticates to storage. Grant it Storage Blob Data Reader on the container out of band — no credential is stored in Terraform, and this module accepts none.

📥 Inputs

Required: name, data_factory_id, linked_service_name. What it points at: path, filename, parameters. How those are read: dynamic_path_enabled, dynamic_filename_enabled. Shape: schema_column, additional_properties. Metadata: description, folder, annotations. Tail: timeouts. There is no tags variable — the resource supports none.

Full input schemas

Name	Type	Default	Notes
`name`	`string`	—	Force-new. Checked only for non-emptiness.
`data_factory_id`	`string`	—	Force-new. Anchored Resource-ID validator.
`linked_service_name`	`string`	—	A bare name, not a Resource ID. Refuses a value beginning with `/`, which is the probable mistake; nothing else about it is checkable.
`path`	`string`	`null`	The container and prefix, in storage.
`filename`	`string`	`null`	The blob name.
`dynamic_path_enabled`	`bool`	`false`	Whether `path` is an expression.
`dynamic_filename_enabled`	`bool`	`false`	Whether `filename` is an expression.
`schema_column`	`list(object({ name, type, description }))`	`[]`	`type` is a closed, case-sensitive set of fifteen.
`parameters`	`map(string)`	`{}`	Values a pipeline supplies per run.
`additional_properties`	`map(string)`	`{}`	Unvalidated top-level keys.
`annotations`	`list(string)`	`[]`	Not tags.
`description`	`string`	`null`	The provider refuses an empty string while accepting omission.
`folder`	`string`	`null`	The authoring tree — not a storage path.
`timeouts`	`object({ create, read, update, delete })`	`null`	30m / 5m / 30m / 30m.

🧾 Outputs

Output	Description	Notes
`id`	The dataset's Resource ID.	First, by convention.
`name`	The dataset name.
`data_factory_id`, `data_factory_name`	The parent factory.	Name parsed from the end of the ID.
`resource_group_name`, `subscription_id`	Where the factory lives.
`linked_service_name`	The linked service, as supplied.	A name — nothing can verify it.
`path`, `filename`	What it points at.
`dynamic_path_enabled`, `dynamic_filename_enabled`	How those are read.
`path_looks_like_an_expression`, `filename_looks_like_an_expression`	Whether the value begins with `@`.	A heuristic.
`path_expression_without_dynamic_flag`, `filename_expression_without_dynamic_flag`	An expression that will be used literally.
`path_marked_dynamic_but_looks_literal`, `filename_marked_dynamic_but_looks_literal`	The opposite mismatch.
`dynamic_flags_look_inconsistent`	Any of the four.	Assert on this.
`has_path`, `has_filename`	Presence flags.
`describes_no_specific_blob`	No path, filename or parameters.	Assert on this.
`parameters`, `parameter_count`	The run-time inputs.
`schema_column_count`, `has_schema_columns`, `schema_column_names`	The column schema.
`untyped_schema_column_names`	Columns with no type.	Assert on this.
`uses_additional_properties`, `additional_property_count`	The escape hatch.
`annotations`, `annotation_count`, `has_annotations`	A list, not tags.
`description`, `has_description`, `folder`, `has_folder`	Metadata.
`folder_and_path_mean_different_things`	Constant.	The wrong-data confusion.
`the_dynamic_flags_change_the_meaning_of_their_neighbours`	Constant.	The trap, named.
`force_new_fields`, `fields_that_can_change_after_creation`, `fields_azure_returns_on_read`	The change surface.
`this_resource_takes_a_linked_service_NAME_not_an_ID`	Constant.	The family's real difference.
`the_dataset_family_is_not_a_clone_cluster`	Constant.	Quotes the contradicting documentation.
`the_linked_service_is_not_verified_to_exist`	Constant.	A bare name cannot be checked.
`the_schema_column_type_set_is_closed_and_case_sensitive`	Constant.
`the_schema_column_block_is_identical_on_the_sql_table_dataset`	Constant.	Why they look like clones.
`the_module_cannot_see_which_pipelines_use_this`	Constant.	Read before destroying.
`the_module_cannot_verify_the_container_or_blob_exists`	Constant.
`no_credential_is_configured_here`	Constant.
`destroying_this_does_not_touch_the_blobs`	Constant.
`this_resource_supports_no_azure_resource_tags`	Constant.	Why there is no `tags` variable.
`no_secret_is_accepted_or_emitted_by_this_module`	Constant.

No output is sensitive, and none can be — every argument is a name, path or column definition.

🧠 Architecture Notes

Two booleans change what their neighbours mean. dynamic_path_enabled and dynamic_filename_enabled decide whether path and filename are literals or Data Factory expressions, and the provider cross-validates none of it. Both failure directions are silent: an expression treated as a literal produces a folder named @concat(...); a literal marked dynamic is evaluated at run time. The module compares each flag against whether its value begins with @, emits all four mismatches individually and rolls them up.

The heuristic reports, and never refuses. A real folder can be named @archive. Refusing a leading @ would reject legal input, and — because a failing validation {} blocks terraform destroy as well as apply — would strand anyone who already owns such a dataset. The limits are stated in each output's own description rather than assumed away.

The family is not a clone cluster, and the documentation says it is. This resource's registry page opens its argument list with "common across all Azure Data Factory Datasets" and includes linked_service_name. The Azure SQL Table dataset takes linked_service_id — a Resource ID with a real validator. Eight arguments and the whole schema_column block genuinely are shared, which is exactly what makes templating the family so tempting and so wrong.

folder and path are unrelated, and their names conceal it. folder is the portal's authoring tree; path is the storage location. Swapping them applies cleanly and reads the wrong data.

Only identity forces replacement. Everything describing what the dataset reads — including which storage account, by way of the linked service name — updates in place.

The module cannot see downstream. Pipelines reference a dataset by name and nothing points back. Destroying one produces a clean plan and breaks every consumer.

lifecycle is not valid inside a module block, so a caller cannot add prevent_destroy. Where destroy protection matters, apply a CanNotDelete management lock — which prevents deletion and does not prevent the replacement that editing name would cause.

🧱 Design Principles

Concern	This module's default	Opt-out
Secrets	None accepted, none emitted.	Not available — by design.
Credentials	Live on the linked service, never here.	—
Storage access	No permission required or used.	—
Dynamic flags	Both default to `false` — literal, the interpretation that cannot surprise you.	Set either to `true`.
A flag disagreeing with its value	Reported four ways, plus a roll-up. Never refused.	Ignore the outputs.
A dataset that resolves to nothing	Reported via `describes_no_specific_blob`.	Ignore the output.
Column types	Enforced against the provider's case-sensitive fifteen.	None — the set is closed.
A Resource ID passed as `linked_service_name`	Refused — a leading `/` is the probable mistake.	None; pass a name.
`tags`	Not offered — the resource supports none.	Tag the factory.

🔒 There is no risky toggle to close here: the resource holds no credential and touches no data. The exposures are the delete and the silent flag mismatch, so the module documents the first and reports the second rather than pretending to gate either.

🚀 Runbook

terraform init -backend=false
terraform validate
terraform fmt -check

Pin the module with ?ref=v1.0.0 — never a branch. This library is plan-only: a human applies from CI.

🧪 Testing

terraform plan and the module's own validation {} blocks cover, offline and without credentials — note it is plan and not terraform validate, which through a module call evaluates no variable values and reports success:

the anchored Resource-ID shape of data_factory_id;
the refusal of a Resource ID passed where linked_service_name wants a name;
the case-sensitive schema_column.type set;
non-empty name, and the description's empty-string rule;
every derived flag — all four dynamic-flag mismatches, their roll-up, describes_no_specific_blob and untyped_schema_column_names — each exercised with one fixture per branch through terraform console, which does fire root-module variable validations.

Only a real plan or apply can tell you whether the linked service exists, whether it points at Blob Storage, whether the container or blob exists, or whether a pipeline still depends on the dataset. And nothing at any stage can tell you whether a dynamic flag is semantically right — that is why the mismatch flags exist.

💬 Example Output

dataset_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-analytics-eastus/providers/Microsoft.DataFactory/factories/adf-analytics-eastus/datasets/ds_landing_orders"
reads_from = "@concat('landing/', formatDateTime(utcnow(), 'yyyy/MM/dd'))"

linked_service_name                  = "ls_blob_landing"
dynamic_path_enabled                 = true
path_looks_like_an_expression        = true
path_expression_without_dynamic_flag = false
path_marked_dynamic_but_looks_literal = false
dynamic_flags_look_inconsistent      = false
describes_no_specific_blob           = false
schema_column_names                  = ["OrderId", "Placed", "Total"]
force_new_fields                     = ["name", "data_factory_id"]

🔍 Troubleshooting

Symptom	Cause	Fix
A pipeline looks for a folder literally named `@concat(...)`	`path` is an expression and `dynamic_path_enabled` is `false`.	Set the flag. `path_expression_without_dynamic_flag` reports this at plan.
A pipeline fails evaluating a plain folder name as an expression	`dynamic_path_enabled` is `true` and the value is a literal.	Unset the flag. `path_marked_dynamic_but_looks_literal` reports this.
`dynamic_flags_look_inconsistent` is `true` on a correct config	The `@` heuristic — a real folder may begin with `@`.	Read the four component outputs and ignore the roll-up if the value is genuinely literal.
`linked_service_name must be a bare linked service NAME, not a Resource ID.`	A Resource ID was passed. The SQL table dataset takes an ID; this one takes a name.	Pass `azurerm_data_factory_linked_service_azure_blob_storage.x.name`.
`every schema_column type must be one of Byte, Byte[], ...` on `"string"`	The set is case-sensitive.	Use `"String"`. Note `Byte[]` carries brackets.
The dataset reads from an unexpected location	`folder` was set where `path` was meant.	They are unrelated: `folder` is the portal tree, `path` is storage.
Apply fails: the linked service is not found	It does not exist, or is in another factory. A bare name cannot be checked at plan.	Confirm the linked service; see `the_linked_service_is_not_verified_to_exist`.
The dataset applies but a pipeline cannot use it	No path, filename or parameters.	Check `describes_no_specific_blob`.
A pipeline breaks after a clean destroy	Pipelines reference datasets by name; nothing points back.	Check consumers before destroying; consider a `CanNotDelete` lock.
A `timeouts` key seems to have no effect	Terraform silently discards an object key the type does not declare.	Compare against the declared `object({ create, read, update, delete })`.

🔗 Related Docs

azurerm_data_factory_dataset_azure_blob
azurerm_data_factory_dataset_azure_sql_table — the sibling that takes an ID
Azure Data Factory datasets
Data Factory expressions and functions
SCOPE.md — this module's cross-module contract
Sibling modules: terraform-azurerm-data-factory, terraform-azurerm-data-factory-dataset-azure-sql-table, terraform-azurerm-data-factory-custom-dataset

💙 "Infrastructure as Code should be standardized, consistent, and secure."

Navigation Menu

Repository navigation

About

Repository files navigation

☁️ Azure Data Factory Dataset — Azure Blob Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required Azure RBAC Roles / Permissions

Azure Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SCOPE.md		SCOPE.md
main.tf		main.tf
outputs.tf		outputs.tf
providers.tf		providers.tf
variables.tf		variables.tf

Latest commit

History

Folders and files

Repository files navigation

☁️ Azure Data Factory Dataset — Azure Blob Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required Azure RBAC Roles / Permissions

Azure Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages