☁️ Azure Kusto Cluster Managed Private Endpoint Terraform Module

Gives an Azure Data Explorer cluster a private route out to a data source (azurerm_kusto_cluster_managed_private_endpoint) — the outbound direction, which is the opposite of what the name suggests. Targets hashicorp/azurerm ~> 4.0.

• 🔌 Declares one managed private endpoint for a Kusto cluster, as a keystone resource named this.
• ➡️ This is private egress from the cluster. It lets ingestion read from a storage account or event hub whose firewall denies public access. It does not make the cluster privately reachable — that is terraform-azurerm-private-endpoint, a different resource in your own virtual network.
• ⏳ Creating it only requests the connection. Someone with rights on the target has to approve it, quite possibly a different team, and until they do ingestion from that target keeps failing. A clean apply is not a working private path.
• 💬 Recommends request_message — the only updatable field, and the only thing that lets an approver act on the request confidently.
• 🎯 Validates private_link_resource_id for shape but deliberately not group_id, whose legal set is target-specific and grows.
• 📛 Notes the provider inconsistency: this resource takes the cluster by name, while its customer-managed-key sibling takes cluster_id.

💡 Why it matters: Two failure modes here are quiet rather than loud. Confusing this with a private endpoint gives you a working resource that does nothing you wanted. And a connection left in Pending looks identical in Terraform to one that was approved — the apply succeeded either way, while the ingestion it was meant to enable keeps failing for a reason nothing in your state file mentions.

If this module saves you time, please consider supporting its continued development:

• ⭐ Star the repository on GitHub.
• 🤝 Connect on LinkedIn: linkedin.com/in/microsoftexpert
• ☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

flowchart LR
  cluster["terraform-azurerm-kusto-cluster"]
  db["terraform-azurerm-kusto-database"]
  adc["terraform-azurerm-kusto-attached-database-configuration"]
  script["terraform-azurerm-kusto-script"]
  cosmos["terraform-azurerm-kusto-cosmosdb-data-connection"]
  eg["terraform-azurerm-kusto-eventgrid-data-connection"]
  eh["terraform-azurerm-kusto-eventhub-data-connection"]
  iot["terraform-azurerm-kusto-iothub-data-connection"]
  cmk["terraform-azurerm-kusto-cluster-customer-managed-key"]
  mpe["terraform-azurerm-kusto-cluster-managed-private-endpoint"]
  cpa["terraform-azurerm-kusto-cluster-principal-assignment"]
  dpa["terraform-azurerm-kusto-database-principal-assignment"]
  cluster -->|"hosts"| db
  cluster -->|"attaches leader db"| adc
  db -->|"target of"| script
  db -->|"ingests via"| cosmos
  db -->|"ingests via"| eg
  db -->|"ingests via"| eh
  db -->|"ingests via"| iot
  cluster -->|"encrypted by, BY ID"| cmk
  cluster -->|"private egress via, BY NAME"| mpe
  cluster -->|"data-plane roles: ALL databases"| cpa
  db -->|"data-plane roles: ONE database, prefer this"| dpa
  mpe -->|"makes the source reachable"| eh
  classDef me fill:#0078D4,stroke:#004578,color:#ffffff;
  classDef ext fill:#f2f2f2,stroke:#c8c8c8,color:#111111;
  class mpe me;
  class cluster,db,adc,script,cosmos,eg,eh,iot,cmk,cpa,dpa ext;

flowchart TB
  dir["THE OUTBOUND DIRECTION, and the opposite of what the name suggests to most readers"]
  mpe["MANAGED private endpoint (this): the Kusto SERVICE operates it in ITS managed network, giving the CLUSTER private egress TO a data source"]
  pe["PRIVATE endpoint (terraform-azurerm-private-endpoint): lives in YOUR virtual network, giving clients private ingress INTO something"]
  both["neither replaces the other. A fully private ingestion path usually needs BOTH."]
  pending["CREATING THIS ONLY REQUESTS THE CONNECTION: it arrives on the target as Pending"]
  approve["someone with rights on the TARGET must approve it, quite possibly a different team"]
  clean["so a clean apply does NOT mean a working private path, and ingestion keeps failing until approval"]
  msg["request_message: the ONLY updatable field, and the only thing that lets an approver act confidently"]
  gid["group_id: target-type specific, e.g. blob or dfs or namespace or sqlServer. Deliberately NOT enum-validated because the legal set grows."]
  byname["takes cluster_name BY NAME, unlike the customer-managed-key sibling which takes cluster_id: the inconsistency is the provider's"]
  region["private_link_resource_region needed only for a CROSS-REGION target, and its absence fails at apply not plan"]
  this["terraform-azurerm-kusto-cluster-managed-private-endpoint"]
  res["azurerm_kusto_cluster_managed_private_endpoint.this"]
  out["outputs: id, name, target, group_id, has_request_message. Approval STATE is not exposed by the provider."]

  dir -->|"read this first"| mpe
  mpe -->|"contrast"| pe
  pe -->|"conclusion"| both
  both -->|"choose knowingly"| this
  pending -->|"read this too"| approve
  approve -->|"consequence"| clean
  clean -->|"disclosed"| this
  msg -->|"smooths approval"| approve
  gid -->|"the target sub-resource"| this
  byname -->|"identity"| this
  region -->|"cross-region only"| this
  this -->|"creates"| res
  res -->|"emits"| out

  classDef me fill:#0078D4,stroke:#004578,color:#fff;
  classDef keystone fill:#004578,stroke:#001f3f,color:#fff;
  classDef sib fill:#eef2f7,stroke:#b8c4d0,color:#1b1b1b;
  class this me;
  class res keystone;
  class dir,mpe,pe,both,pending,approve,clean,msg,gid,byname,region,out sib;

Resource inventory

Schema notes that bite — confirmed against the live provider schema:

• Everything except request_message is force-new. Replacing the endpoint tears down the cluster's private route to the target — and the new request needs approving again.
• The cluster is taken by cluster_name, not cluster_id — unlike azurerm_kusto_cluster_customer_managed_key in the same family, which takes cluster_id. The inconsistency is the provider's.
• group_id has no enum in the schema. The legal set depends on the target's resource type, so a wrong value fails at apply.
• Creation does not wait for approval. The apply returns once the request is registered.
• private_link_resource_region is required for a cross-region target, and its absence fails at apply rather than at plan.
• The approval state is not exposed on this resource. There is no attribute to read it from.
• No tags. The universal tail is timeouts only.

⚠️ Note the asymmetry, because it explains the whole operational shape of this resource: requesting needs nothing on the target. So a caller with rights only on the cluster can create a request that no one is empowered to approve — and nothing in the apply signals that. The approval permission usually belongs to a different team, which is why request_message matters.

• The Microsoft.Kusto resource provider registered on the subscription.
• An existing Kusto cluster.
• An existing target resource that supports Azure Private Link, and the correct group_id for its resource type.
• Someone with approval rights on the target, and an agreement that they will act on the request. Treat this as a real prerequisite rather than a formality — an unapproved request is the most common reason a private ingestion path never completes.
• ℹ️ If the target is in a different region from the cluster, private_link_resource_region must be supplied.

terraform-azurerm-kusto-cluster-managed-private-endpoint/
├── providers.tf   # required_version + the pinned azurerm provider. No provider block.
├── variables.tf   # name, cluster_name, resource_group_name, the target, group_id, request_message, timeouts
├── main.tf        # the keystone azurerm_kusto_cluster_managed_private_endpoint.this
├── outputs.tf     # id, name, cluster_name, target, group_id, has_request_message
├── README.md      # this document
├── SCOPE.md       # the cross-module contract
├── LICENSE        # MIT
└── .gitignore

provider "azurerm" {
  features {}
}

module "adx_mpe_storage" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"

  name                = "mpe-landing-blob"
  cluster_name        = module.adx.name
  resource_group_name = module.rg.name

  private_link_resource_id = module.st_landing.id
  group_id                 = "blob"

  request_message = "ADX prod cluster needs private read access to the landing container. Contact: data-platform@example.com"
}

ℹ️ The caller configures the provider, its authentication, and the mandatory features {} block. This module declares none of them.

⚠️ This apply succeeds and the connection is Pending. Nothing flows until someone with rights on the storage account approves it — see example 4.

Consumes

Emits

The examples below reference existing resources by ID or name rather than creating them; this module owns only its own resource. Those references are declared inputs:

variable "ehns_id" {
  description = "id of an existing ehns that these examples reference but do not create."
  type        = string
}

variable "st_archive_id" {
  description = "id of an existing st archive that these examples reference but do not create."
  type        = string
}

variable "st_lake_id" {
  description = "id of an existing st lake that these examples reference but do not create."
  type        = string
}

variable "vnet_subnet_ids" {
  description = "subnet ids of an existing vnet that these examples reference but do not create."
  type        = map(string)
}

MANAGED private endpoint (this module)
    lives in : the Kusto service's own managed network
    direction: CLUSTER  ---->  a data source
    use when : ingestion must read from a storage account / event hub / SQL server
               whose firewall denies public access

PRIVATE endpoint (terraform-azurerm-private-endpoint)
    lives in : YOUR virtual network
    direction: your clients  ---->  THE CLUSTER
    use when : analysts and dashboards must reach the cluster privately

module "adx_mpe_storage" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"

  name                = "mpe-landing-blob"
  cluster_name        = module.adx.name
  resource_group_name = module.rg.name

  private_link_resource_id = module.st_landing.id
  group_id                 = "blob"

  request_message = "ADX prod needs private read access to the landing container."
}

module "adx_mpe_eventhub" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"

  name                = "mpe-telemetry-ns"
  cluster_name        = module.adx.name
  resource_group_name = module.rg.name

  private_link_resource_id = var.ehns_id
  group_id                 = "namespace"

  request_message = "ADX prod ingests telemetry from this namespace."
}

# On the TARGET resource, by someone with rights there:
az network private-endpoint-connection approve \
  --resource-group rg-data-landing \
  --name <connection-name> \
  --resource-name stlandingprod \
  --type Microsoft.Storage/storageAccounts \
  --description "Approved for ADX prod ingestion"

# ❌ Nothing to act on.
# request_message omitted

# ⚠️ Technically present, useless in practice.
request_message = "please approve"

# ✅ An approver can act on this without asking anyone.
request_message = "ADX cluster 'adxprod' (rg-analytics-prod) requires private read access to the landing container for scheduled ingestion. Owner: data-platform@example.com. Change ref: CHG-4821."

module "adx_mpe_remote" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"

  name                = "mpe-westus-archive"
  cluster_name        = module.adx.name        # eastus
  resource_group_name = module.rg.name

  private_link_resource_id     = var.st_archive_id # westus2
  group_id                     = "blob"
  private_link_resource_region = "westus2"

  request_message = "ADX prod (eastus) reads archive blobs from westus2."
}

locals {
  sources = {
    landing = { id = module.st_landing.id, group = "blob" }
    lake    = { id = var.st_lake_id, group = "dfs" }
    telemetry = { id = var.ehns_id, group = "namespace" }
  }
}

module "adx_mpe" {
  source   = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"
  for_each = local.sources

  name                = "mpe-${each.key}"
  cluster_name        = module.adx.name
  resource_group_name = module.rg.name

  private_link_resource_id = each.value.id
  group_id                 = each.value.group

  request_message = "ADX prod private access to ${each.key}."
}

group_id = "blob"      # storage: blob access
group_id = "dfs"       # storage: Data Lake Gen2 hierarchical namespace
group_id = "table"     # storage: table service
group_id = "queue"     # storage: queue service
group_id = "namespace" # Event Hubs / Service Bus
group_id = "sqlServer" # SQL
group_id = "vault"     # Key Vault

Error: Invalid value for variable

  private_link_resource_id must be a full Azure Resource ID beginning
  "/subscriptions/..." — not a name, and not an endpoint URL.

# Changing only request_message:
  ~ resource "azurerm_kusto_cluster_managed_private_endpoint" "this" {
      ~ request_message = "please approve" -> "ADX prod ingestion; owner data-platform@example.com"
    }
Plan: 0 to add, 1 to change, 0 to destroy.

# THIS module — by name:
cluster_name = module.adx.name

# Its customer-managed-key sibling — by ID:
cluster_id = module.adx.id

# module.adx_mpe_storage.azurerm_kusto_cluster_managed_private_endpoint.this will be destroyed

output "adx_private_sources" {
  value = {
    for k, m in module.adx_mpe : k => {
      target      = m.private_link_resource_id
      group       = m.group_id
      explained   = m.has_request_message # expect true
    }
  }
}

provider "azurerm" {
  features {}
}

module "rg" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-analytics-prod"
  location = "eastus"
}

# 1 · A locked-down source: no public network access.
module "st_landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-storage-account.git?ref=v1.0.0"

  name                = "stlandingprod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  containers = {
    landing = {}
  }
}

# 2 · The cluster.
module "adx" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster.git?ref=v1.0.0"

  name                = "adxprod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  sku = {
    name     = "Standard_E8ads_v5"
    capacity = 2
  }
}

module "adx_db" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-database.git?ref=v1.0.0"

  name                = "telemetry"
  resource_group_name = module.rg.name
  location            = module.rg.location
  cluster_name        = module.adx.name
}

# 3 · Private EGRESS: the cluster reaches the locked-down storage account. This is the module.
module "adx_mpe_landing" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster-managed-private-endpoint.git?ref=v1.0.0"

  name                = "mpe-landing-blob"
  cluster_name        = module.adx.name
  resource_group_name = module.rg.name

  private_link_resource_id = module.st_landing.id
  group_id                 = "blob"

  request_message = "ADX 'adxprod' ingests from the landing container. Owner: data-platform@example.com."
}

# 4 · Private INGRESS: analysts reach the cluster. A DIFFERENT resource — see example 1.
module "adx_private_endpoint" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-private-endpoint.git?ref=v1.0.0"

  name                = "pe-adxprod"
  resource_group_name = module.rg.name
  location            = module.rg.location
  subnet_id           = var.vnet_subnet_ids["private-endpoints"]

  private_service_connection = {
    name                           = "adxprod"
    private_connection_resource_id = module.adx.id
    subresource_names              = ["cluster"]
    is_manual_connection           = false
  }
}

# 5 · A least-privilege data-plane grant for the ingestion identity: write, not read.
module "adx_ingest_role" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-database-principal-assignment.git?ref=v1.0.0"

  name                = "ingest-pipeline-ingestor"
  resource_group_name = module.rg.name
  cluster_name        = module.adx.name
  database_name       = module.adx_db.name

  principal_id   = var.ingest_group_object_id
  principal_type = "Group"
  tenant_id      = var.tenant_id
  role           = "Ingestor"
}

output "analytics_private_paths" {
  value = {
    egress_target    = module.adx_mpe_landing.private_link_resource_id
    egress_explained = module.adx_mpe_landing.has_request_message
    ingress_endpoint = module.adx_private_endpoint.id
  }
}

variable "name" {
  type = string
  # ⚠️ Force-new. Replacing tears down the private route and needs fresh approval.
}

variable "cluster_name" {
  type = string
  # ⚠️ By NAME. The customer-managed-key sibling takes cluster_id — the provider is inconsistent.
}

variable "resource_group_name" {
  type = string
  # The CLUSTER's resource group. The target may live elsewhere.
}

variable "private_link_resource_id" {
  type = string
  # Validated: must begin "/subscriptions/". Force-new.
}

variable "group_id" {
  type = string
  # ⚠️ Target-type specific (blob, dfs, table, queue, namespace, sqlServer, vault, …) and deliberately
  #    NOT enum-validated: the legal set grows, so a hard-coded list would reject valid future values.
}

variable "private_link_resource_region" {
  type    = string
  default = null
  # Required only for a cross-region target; absence fails at apply, not plan.
}

variable "request_message" {
  type    = string
  default = null
  # 💡 The only updatable field. Name the cluster, the purpose and an owner — the approver is
  #    usually on another team and sees no other context.
}

variable "timeouts" {
  type = object({
    create = optional(string)
    read   = optional(string)
    update = optional(string)
    delete = optional(string)
  })
  default = null
  # ℹ️ Creation returns when the request is REGISTERED, not when it is approved.
}

ℹ️ The connection's approval state is not emitted, because the provider exposes no attribute for it. Confirm approval on the target resource's private-endpoint connections — this module says so rather than emitting a guess.

• The name misleads, and the cost of the misunderstanding is silence. A managed private endpoint gives the cluster egress; a private endpoint gives clients ingress. Someone wanting private access to the cluster who reaches for this module gets a resource that applies cleanly, reports healthy, and does nothing they intended. That is why the direction leads both the design intent and the diagram.

• A successful apply means "requested", not "connected". The connection arrives on the target as Pending and needs approval from someone who may have no involvement in the Terraform run. Nothing in state distinguishes pending from approved, so the natural assumption — clean apply means working path — is wrong here.

• The permission asymmetry explains that. Requesting needs nothing on the target; only approving does. So a caller with cluster-only rights can create requests indefinitely without being able to complete any of them.

• request_message is the only lever on the human part of the process, and the only updatable field. It can be corrected without recreating the endpoint or re-triggering approval, which makes it cheap to improve.

• Two of the four timeouts knobs do nothing, and nothing reports that. The provider's read and delete functions both acquire the combined create/update timeout rather than their own, and since neither runs against a new resource, both resolve to the update timeout. So timeouts.read is never consulted — a refresh is governed by timeouts.update, defaulting to 60 minutes rather than the 5 the schema advertises — and timeouts.delete is never consulted either. Setting either one is accepted, changes nothing, and produces no warning.

• group_id is unvalidated on purpose. Its legal values depend entirely on the target's resource type and grow as Azure extends private-link support. An enum would be wrong within a release or two. private_link_resource_id is validated instead, because "must be a Resource ID" does not go stale.

• Everything else is force-new, and replacement is expensive twice over: it drops the private route, and the replacement request must be approved again.

• The cluster is referenced by name here and by ID in the sibling CMK module. Copying between the two produces a type error rather than a subtle bug, which is the better failure — but it is worth knowing before it happens.

• No tags. Tag the cluster and the target instead.

• Before applying: confirm the direction you actually need — egress (this) or ingress (terraform-azurerm-private-endpoint).
• Before applying: confirm the correct group_id for the target's resource type.
• Before applying: identify who will approve, and write a request_message they can act on.
• After applying: verify approval on the target. A clean apply is not enough.

terraform init -backend=false
terraform validate
terraform fmt -check

• Pin the source to a tag — ?ref=v1.0.0 — never a branch.
• Plan-only from here. A human applies from CI.
• ⚠️ After apply, check the target's private-endpoint connections and get the request approved. This step is not Terraform's and is not visible in state.
• ⚠️ Create and approve this before the data connection that depends on it.
• ⚠️ Any change other than request_message is a replacement, which drops the route and needs fresh approval.

terraform validate and terraform fmt -check are the offline gate. They confirm:

• the five required inputs are present and typed string;
• private_link_resource_id begins /subscriptions/;
• the timeouts value is an object of Go duration strings - but note that a key the type does not declare is SILENTLY DISCARDED rather than refused, so a misspelling here produces no error anywhere and the provider default quietly applies;
• the module declares no provider block.

What only plan and apply exercise:

• whether group_id is legal for the target's resource type;
• whether private_link_resource_region is needed and correct;
• whether the cluster exists and the caller can write managed private endpoints to it.

What no Terraform command checks at any stage:

• whether the connection was ever approved — the provider exposes no attribute for it, and the apply succeeds either way;
• whether anyone with approval rights on the target is aware the request exists.

Outputs:

cluster_name             = "adxprod"
group_id                 = "blob"
has_request_message      = true
id                       = "/subscriptions/00000000-.../clusters/adxprod/managedPrivateEndpoints/mpe-landing-blob"
name                     = "mpe-landing-blob"
private_link_resource_id = "/subscriptions/00000000-.../storageAccounts/stlandingprod"

⚠️ Nothing here tells you the connection is approved. That is the point of the note in the Outputs table — check the target.

• azurerm_kusto_cluster_managed_private_endpoint — provider documentation for this resource.
• azurerm_private_endpoint — the other direction, for private access into the cluster.
• Managed private endpoints in Azure Data Explorer — the service's own account, including approval.
• Azure Private Link sub-resource names — the authoritative source for group_id values per resource type.
• Sibling modules in this family: terraform-azurerm-kusto-cluster, terraform-azurerm-kusto-database, terraform-azurerm-kusto-eventhub-data-connection, terraform-azurerm-kusto-cluster-customer-managed-key, terraform-azurerm-kusto-cluster-principal-assignment, terraform-azurerm-kusto-database-principal-assignment.
• This module's SCOPE.md.

💙 "Infrastructure as Code should be standardized, consistent, and secure."