LangSmith on Azure

Self-hosted deployment on AKS, managed with Terraform.

What Pass 1 provisions

Pass 1 creates all Azure infrastructure. No Kubernetes workloads are deployed yet — that happens in Pass 2.

Step 1 — Configure terraform.tfvars

Run the interactive wizard from terraform/azure/:

cd terraform/azure
make quickstart

The wizard generates infra/terraform.tfvars covering: subscription, region, naming, AKS sizing, ingress controller, DNS/TLS, backend services, Key Vault, and security add-ons. Each section includes explanatory context and cost estimates.

Prefer to edit manually? Copy the example instead:

cp infra/terraform.tfvars.example infra/terraform.tfvars
vi infra/terraform.tfvars

Minimum required values:

# ── Identity ─────────────────────────────────────────────────────────────
subscription_id = "YOUR_AZURE_SUBSCRIPTION_ID"

# ── Location ─────────────────────────────────────────────────────────────
location = "eastus"

# ── Naming / tagging ─────────────────────────────────────────────────────
identifier  = "-prod"      # suffix appended to all resource names
environment = "prod"       # dev | staging | prod

# ── Deployment tier ───────────────────────────────────────────────────────
# Production (recommended):
postgres_source   = "external"    # Azure DB for PostgreSQL
redis_source      = "external"    # Azure Cache for Redis Premium
clickhouse_source = "in-cluster"  # in-cluster (dev/POC) or external (prod)

# ── DNS + TLS ────────────────────────────────────────────────────────────
dns_label              = "langsmith-prod"   # → langsmith-prod.eastus.cloudapp.azure.com
tls_certificate_source = "letsencrypt"
letsencrypt_email      = "you@example.com"

# ── Sizing ────────────────────────────────────────────────────────────────
sizing_profile = "production"   # minimum | dev | production | production-large

Step 2 — Bootstrap secrets

make setup-env

setup-env.sh writes infra/secrets.auto.tfvars (gitignored, chmod 600) — Terraform picks this file up automatically, no shell exports needed.

• First run: prompts for PostgreSQL password, LangSmith license key, admin password, and admin email. Generates api_key_salt, jwt_secret, and four Fernet encryption keys locally.
• Subsequent runs: reads all values silently from Azure Key Vault — no prompts.

Step 3 — Preflight check

make preflight

Validates before you spend 20 minutes on a failing apply:

• az CLI version and active login
• Active subscription — prints name so you can confirm it is correct
• 11 required Azure resource providers registered (Microsoft.ContainerService, Microsoft.DBforPostgreSQL, Microsoft.Cache, Microsoft.KeyVault, Microsoft.Storage, and others)
• RBAC: requires Contributor + User Access Administrator (or Owner) at subscription scope
• terraform.tfvars exists with location and subscription_id set
• secrets.auto.tfvars exists and has a non-empty langsmith_license_key
• terraform, kubectl, and helm binaries on PATH

Step 4 — Initialize Terraform

make init

Downloads the AzureRM provider, initializes the backend, and updates module sources. Required once per fresh clone and after any provider version change.

Step 5 — Apply infrastructure

make apply   # ~15–20 min on first run

make apply creates all Azure resources in the correct order. On first run this takes approximately 15–20 minutes. Subsequent applies (e.g. enabling Pass 3) are much faster.

Step 6 — Pass 1.5: Cluster access and K8s secrets

After make apply completes, get cluster credentials and push secrets into the cluster:

make kubeconfig    # fetches AKS credentials, merges into ~/.kube/config
make k8s-secrets   # Key Vault → langsmith-config-secret in the langsmith namespace

make k8s-secrets reads 8 secrets from Key Vault and creates or updates langsmith-config-secret. Safe to re-run — uses --dry-run=client | kubectl apply so it updates in place without recreating the secret.

Verify Pass 1

# All nodes should be Ready
kubectl get nodes

# Bootstrap components — all Running
kubectl get pods -n cert-manager     # 3 pods
kubectl get pods -n keda             # 3 pods
kubectl get pods -n ingress-nginx    # 1 pod (if using nginx)

# NGINX LoadBalancer — save the EXTERNAL-IP
kubectl get svc ingress-nginx-controller -n ingress-nginx

# Workload Identity service account — should have client-id annotation
kubectl get sa langsmith-ksa -n langsmith \
  -o jsonpath='{.metadata.annotations}'

# View all Terraform outputs
terraform -chdir=infra output

# Key outputs consumed by Helm scripts
terraform -chdir=infra output -raw keyvault_name
terraform -chdir=infra output -raw storage_account_name
terraform -chdir=infra output -raw storage_container_name
terraform -chdir=infra output -raw storage_account_k8s_managed_identity_client_id

Or run everything — cluster credentials, K8s secrets, Helm values generation, and deploy — in one shot after make apply:

make deploy-all   # kubeconfig → k8s-secrets → init-values → deploy

Teardown

Always uninstall Helm before destroying infrastructure:

make uninstall   # removes Helm releases, LGP CRD, langsmith namespace (removes Azure Load Balancer)
make destroy     # terraform destroy — safe now that LB is gone
make clean       # removes local secrets, generated values, local tfstate (LAST)

What Pass 2 deploys

Pass 2 installs the LangSmith Helm chart. It reads all secrets from langsmith-config-secret (created in Pass 1.5) and all infrastructure configuration from Terraform outputs.

Prerequisites: Pass 1 complete (make apply) and Pass 1.5 complete (make kubeconfig && make k8s-secrets).

Two deployment paths

Helm path (recommended)

Step 1 — Generate Helm values

cd terraform/azure
make init-values

make init-values reads terraform output and terraform.tfvars and generates helm/values/values-overrides.yaml with all fields populated:

• config.hostname — your FQDN (from dns_label or langsmith_domain)
• config.initialOrgAdminEmail — the first org admin account
• config.existingSecretName: langsmith-config-secret — secrets reference
• config.blobStorage — storage account name + container + Workload Identity client ID
• Workload Identity annotations for 5 service accounts (backend, platform-backend, queue, ingest-queue, host-backend)
• Ingress + TLS block (cert-manager annotation, TLS secret name)
• Postgres and Redis external secret references (if postgres_source = "external" / redis_source = "external")

Also copies the sizing overlay and any enabled addon overlays from helm/values/examples/ into helm/values/.

Step 2 — Deploy LangSmith

make deploy   # ~10 min

make deploy handles:

1. Validates values-overrides.yaml exists (fails fast with make init-values hint if missing)
2. Refreshes kubeconfig via az aks get-credentials
3. Annotates the LoadBalancer service with service.beta.kubernetes.io/azure-dns-label-name — required for Azure to assign the DNS label to the public IP
4. Creates the letsencrypt-prod cert-manager ClusterIssuer if tls_certificate_source = "letsencrypt" (idempotent)
5. Runs preflight checks: kubectl, helm, az, terraform on PATH; cluster connectivity; Helm repo updated
6. Verifies langsmith-config-secret exists — auto-creates from Key Vault if missing
7. Builds and logs the values chain: values.yaml → values-overrides.yaml → sizing overlay → addon overlays
8. Guards against stuck Helm releases: auto-rolls back pending-upgrade state before proceeding
9. Runs helm upgrade --install langsmith langchain/langsmith --timeout 20m
10. Waits for core deployments to roll out
11. Annotates the langsmith-ksa service account with the Workload Identity client ID
12. Prints the access URL and login credentials location

Or run Pass 1.5 + Pass 2 in one shot after make apply:

make deploy-all   # kubeconfig → k8s-secrets → init-values → deploy

# macOS
brew install watch
watch kubectl get pods -n langsmith

# Without watch
while true; do clear; kubectl get pods -n langsmith; sleep 3; done

Terraform path (alternative)

Use the Terraform path when you want the Helm release, K8s secrets, and Workload Identity service account managed in Terraform state.

# Copy and configure app vars
cp app/terraform.tfvars.example app/terraform.tfvars
vi app/terraform.tfvars   # set admin_email at minimum

# Pull infra outputs into app/infra.auto.tfvars.json + terraform init
make init-app

# Deploy Helm release + K8s secrets + WI service account via Terraform
make apply-app

Feature flags in app/terraform.tfvars:

sizing              = "production"   # minimum | dev | production | production-large
enable_agent_deploys  = true         # Pass 3 — LangSmith Deployments
enable_agent_builder  = true         # Pass 4 — Agent Builder (requires agent_deploys)
enable_insights       = true         # Pass 5 — Insights / ClickHouse
enable_polly          = true         # Pass 5 — Polly (requires agent_deploys)

End-to-end via Terraform (Pass 1 + Pass 2 in one shot):

make deploy-all-tf   # apply → init-values → init-app → apply-app

Verify Pass 2

# All pods Running or Completed (~17 pods)
kubectl get pods -n langsmith

# Ingress host + TLS assigned
kubectl get ingress -n langsmith

# TLS certificate issued
kubectl get certificate -n langsmith
# Expected: READY: True

# Helm release status
helm list -n langsmith

Expected pod state (all Running after ~5 minutes):

langsmith-ace-backend-xxxxx              1/1   Running     0   5m
langsmith-backend-xxxxx                  1/1   Running     0   5m
langsmith-backend-auth-bootstrap-xxxxx   0/1   Completed   0   5m
langsmith-backend-ch-migrations-xxxxx    0/1   Completed   0   5m
langsmith-backend-migrations-xxxxx       0/1   Completed   0   5m
langsmith-clickhouse-0                   1/1   Running     0   5m
langsmith-frontend-xxxxx                 1/1   Running     0   5m
langsmith-ingest-queue-xxxxx             1/1   Running     0   5m
langsmith-platform-backend-xxxxx         1/1   Running     0   5m
langsmith-playground-xxxxx               1/1   Running     0   5m
langsmith-queue-xxxxx                    1/1   Running     0   5m

Open https://<HOSTNAME> and log in with initialOrgAdminEmail and the admin password from Key Vault:

# Retrieve admin password
az keyvault secret show \
  --vault-name $(terraform -chdir=infra output -raw keyvault_name) \
  --name langsmith-admin-password \
  --query value -o tsv

Values chain

make deploy applies Helm values files in this order (last file wins on conflicts):

1. helm/values/values.yaml                             — Azure base (NGINX, Blob WI, no Istio)
2. helm/values/values-overrides.yaml                   — hostname, WI client-id, auth, postgres/redis
3. helm/values/langsmith-values-sizing-<profile>.yaml  — resource requests + HPA settings
4. (addon files — only when enable_* flags are set)

All files in helm/values/ are gitignored (generated or contain live secrets). Source templates live in helm/values/examples/ and are copied by make init-values.

Day-2 operations

make status         # 10-section health check
make status-quick   # skip Key Vault + K8s secret queries (faster)
make deploy         # re-deploy after any Helm value changes
make init-values    # re-generate values after Terraform changes
make kubeconfig     # refresh cluster credentials
make k8s-secrets    # re-create langsmith-config-secret from Key Vault

What Pass 3 adds

Pass 3 enables LangSmith Deployments — deploy and manage LangGraph graphs as API servers directly from the LangSmith UI. Three new pods are added to the cluster:

Prerequisite: Pass 2 running.

Step 1 — Scale node pool

Before enabling, bump default_node_pool_min_count to at least 5. The operator spawns agent deployment pods on demand and needs node headroom:

# infra/terraform.tfvars
default_node_pool_min_count = 5      # operator pods need headroom
enable_deployments          = true

Step 2 — Apply, regenerate values, deploy

cd terraform/azure
make apply          # scale up node pool (~5 min)
make init-values    # picks up enable_deployments = true → generates addon overlay
make deploy         # rolls out host-backend + listener + operator

make init-values appends the LangSmith Deployments addon overlay (langsmith-values-agent-deploys.yaml) to the values chain. It automatically injects:

config:
  deployment:
    enabled: true                          # REQUIRED — without this, listener and operator are skipped silently
    url: "https://<your-hostname>"         # must match config.hostname (with protocol)
    tlsEnabled: true                       # set based on tls_certificate_source

Step 3 — Verify

# All three pods Running
kubectl get pods -n langsmith | grep -E "host-backend|listener|operator"

# LangSmith Deployments CRDs registered
kubectl get crd | grep langchain

# List LangSmith Deployments (empty on first deploy — populated when you create a deployment)
kubectl get lgp -n langsmith

Expected: langsmith-host-backend, langsmith-listener, and langsmith-operator all Running. Total pod count: ~20 Running + 3 Completed jobs.

KEDA scaling for Deployments workers

KEDA is already installed in Pass 1. With enable_deployments = true, the operator creates KEDA ScaledObject resources for each agent deployment's worker queue. Worker pods scale down to zero when idle and scale up based on Redis queue depth.

No KEDA configuration is needed in terraform.tfvars — the operator manages it automatically when creating agent deployments.

Terraform path

If you are using the Terraform Helm path (Pass 2 via make apply-app), enable LangSmith Deployments in app/terraform.tfvars:

enable_agent_deploys = true

Then:

make init-app     # refresh infra outputs
make apply-app    # update Helm release

What Pass 4 adds

Pass 4 enables the Agent Builder — visual AI-assisted creation and management of LangGraph agents from the LangSmith UI. No terraform apply is needed for this pass — it only requires make init-values && make deploy.

Prerequisite: Pass 3 must be enabled (enable_deployments = true). Pass 4 requires enable_deployments = true — enabling Agent Builder without Deployments causes a preflight error.

Step 1 — Enable in terraform.tfvars

# infra/terraform.tfvars
enable_deployments   = true    # Pass 3 — required prerequisite
enable_agent_builder = true    # Pass 4

Step 2 — Regenerate values and deploy

cd terraform/azure
make init-values    # appends langsmith-values-agent-builder.yaml to values chain
make deploy         # rolling update — ~10 min for bootstrap Job to complete

make init-values appends the Agent Builder addon overlay (langsmith-values-agent-builder.yaml) to the values chain. This overlay:

• Enables the Agent Builder UI and its two supporting services
• Sets backend.agentBootstrap: true — a post-install job that registers Agent Builder as a LangSmith Deployment and creates the required ConfigMap
• Sets conservative agent worker pod resources (1 CPU / 1 Gi) instead of the chart's default 4 CPU / 8 Gi

Step 3 — Verify

# Static pods Running, bootstrap Job Completed
kubectl get pods -n langsmith | grep -E "tool-server|trigger-server|Bootstrap"

# Operator-managed dynamic pods (4 pods — api-server, queue, redis, postgres StatefulSet)
kubectl get pods -n langsmith | grep agent-builder

# Operator-managed LangSmith Deployment for Agent Builder
kubectl get lgp -n langsmith

Expected: 3 static pods (tool-server, trigger-server, bootstrap Job) + 4 dynamic pods. Total: ~26 pods. After make deploy, an Agent Builder section appears in the LangSmith UI navigation.

kubectl rollout restart deployment langsmith-frontend -n langsmith

Workload Identity for Agent Builder

Both langsmith-agent-builder-tool-server and langsmith-agent-builder-trigger-server need Workload Identity to access Azure Blob Storage. Their federated credentials are pre-registered in modules/k8s-cluster/main.tf — no additional setup is needed.

If you add a new pod that needs Blob access, update service_accounts_for_workload_identity in modules/k8s-cluster/variables.tf and run terraform apply -target=module.aks.

Terraform path

If using the Terraform Helm path, enable in app/terraform.tfvars:

enable_agent_deploys  = true   # required prerequisite
enable_agent_builder  = true

Then:

make init-app
make apply-app

What Pass 5 adds

Pass 5 enables two features — Insights and Polly — both of which require Pass 3 (LangSmith Deployments). They are independent of each other: you can enable either one without the other.

Insights — AI-powered trace analytics (Clio). Surfaces patterns and anomalies in LangSmith traces. Clio deploys as a dynamic LangGraph deployment via the operator when first invoked from the UI. No new static pods are added.

Polly — AI-powered evaluation and monitoring agent. Runs as a dynamic LangGraph deployment. Sets resource limits for the Polly worker (2 CPU / 4 Gi request, 4 CPU / 8 Gi limit, scales 1–5 replicas).

No terraform apply is needed for Pass 5 — only make init-values && make deploy.

Prerequisite: Pass 3 must be enabled (enable_deployments = true). Pass 4 and Pass 5 are independent — both require Pass 3 but not each other.

Step 1 — Enable in terraform.tfvars

# infra/terraform.tfvars
enable_deployments = true    # Pass 3 — required prerequisite
enable_insights    = true    # Pass 5 — Insights / Clio analytics
enable_polly       = true    # Pass 5 — Polly AI evaluation agent

You can enable just one:

enable_insights = true    # Insights only (Polly not needed)
# or
enable_polly    = true    # Polly only (Insights not needed)

Step 2 — Regenerate values and deploy

cd terraform/azure
make init-values    # appends insights and polly addon overlays to values chain
make deploy         # rolling update — ~5 min

make init-values appends the addon overlays based on clickhouse_source in terraform.tfvars:

• clickhouse_source = "in-cluster" → generates a minimal overlay (config.insights.enabled: true only). The Helm chart manages ClickHouse internally.
• clickhouse_source = "external" → generates a full overlay with clickhouse.external.enabled: true and a langsmith-clickhouse secret reference. You must create this secret with the ClickHouse host and credentials before deploying.

Step 3 — Verify

# ClickHouse already running from Pass 2
# Insights and Polly deploy as dynamic pods when first invoked from the UI
kubectl get pods -n langsmith | grep -E "clickhouse|polly|clio"

# Watch for dynamic pods when you first use Insights in the UI
kubectl get pods -n langsmith -w

# Confirm Insights is enabled in Helm values
helm get values langsmith -n langsmith | grep -A3 insights
# Expected: enabled: true

Pod count after Pass 5 is identical to after Pass 4 at rest (~22 running). Clio and Polly appear as dynamic pods when invoked from the UI.

kubectl rollout restart deployment langsmith-frontend -n langsmith

Terraform path

If using the Terraform Helm path, enable in app/terraform.tfvars:

enable_agent_deploys = true   # required prerequisite
enable_insights      = true
enable_polly         = true

Then:

make init-app
make apply-app

All 5 passes summary

After completing all passes, your deployment runs:

Core Variables

Deployment Tier

PostgreSQL

Redis

AKS Node Pools

Ingress Controller

DNS and TLS

LangSmith Application

Blob Storage

Key Vault

Network (BYO VNet)

High Availability

Optional Modules

Addon Pass Flags

These flags are read by init-values.sh and deploy.sh. Terraform ignores them — they only affect which Helm addon overlay files are generated.

Sensitive Variables (set via setup-env.sh)

These are written to secrets.auto.tfvars by make setup-env and stored in Azure Key Vault by Terraform. Never set these inline in terraform.tfvars.