> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-trino-dialect.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Infrastructure Requirements Reference
This document consolidates infrastructure prerequisites, node specifications, networking, storage, and IAM requirements across all supported ClickHouse Private deployment platforms.
**Units**
* `Gi` and `Mi` are Kubernetes [resource quantities](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/) (binary: 1 `Gi` = 1 GiB = 2^30 bytes). They appear wherever a value is written exactly as it goes into Helm values or manifests: pod resources (`Memory: 16Gi`), volume sizes (`keeper.storage.resources.requests`), cache sizing (`bytesPerGiRAM`).
* **GiB** (binary) is used in prose for hardware capacities such as instance RAM, local SSD, and temp storage.
* **GB** and **GiB** in node boot/OS disk rows follow how the provider's console or CLI labels the field: GB for GCP (`--disk-size`) and Azure (`--node-osdisk-size`), GiB for AWS EKS node group disk sizes.
***
## Prerequisites by Platform
### AWS (EKS)
| Requirement | Details |
| -------------------- | -------------------------------------------------------------------------- |
| AWS CLI | Installed and configured |
| `kubectl` | Installed |
| `helm` | v3.x installed |
| `skopeo` | Recommended for copying container images (retains all architectures) |
| EKS cluster IAM role | Created via AWS console "Create recommended role" with default permissions |
| Node IAM roles | Created via AWS console "Create recommended role" with default permissions |
| OIDC provider | Created for the EKS cluster |
### GCP (GKE)
| Requirement | Details |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| GCP Project | Billing enabled |
| `gcloud` CLI | Installed and authenticated |
| `kubectl` | Installed |
| `helm` | v3.x installed |
| `skopeo` | Installed (for copying container images) |
| AWS credentials | Read access to ClickHouse ECR (for image copying) |
| Bastion host | In the VPC for accessing the private cluster |
| Deployer service account | With appropriate permissions (see [IAM / Service Account Requirements](#iam-service-account-requirements)) |
| **(FIPS only)** Red Hat UBI8 image | Pulled via GKE for certificate generation |
### Azure (AKS)
| Requirement | Details |
| ------------------ | --------------------------------------------------------------------------------------------- |
| Azure subscription | With permissions to create AKS clusters, ACR, Storage Accounts, VNets, and Managed Identities |
| `az` CLI | Installed and authenticated |
| `kubectl` | Installed |
| `helm` | v3.x installed |
| `skopeo` | Installed (for copying container images) |
| AWS credentials | Read access to ClickHouse ECR (for image copying) |
| OIDC issuer | Enabled on the AKS cluster (`--enable-oidc-issuer`) |
| Workload Identity | Enabled on the AKS cluster (`--enable-workload-identity`) |
### Bare Metal
| Requirement | Details |
| -------------------------- | ------------------------------------------------------------------------------------- |
| Kubernetes | Any conformant distribution, v1.33+. RKE2, kubeadm, and k3s are all suitable. |
| `kubectl` | Installed |
| `helm` | v3.x installed |
| `skopeo` | For copying container images between registries |
| Private container registry | OCI-compatible and reachable from the cluster, to hold the mirrored images and charts |
| Object storage | MinIO AIStor recommended on-prem |
| Object storage credentials | Static access key / secret key |
***
## Node Group Specifications
### Keeper sizing: node vs pod
The Keeper tables below describe two different things: the recommended instance type sizes the **node**, while the recommended resources size the Keeper **pod**. Each node runs a single Keeper pod, and nothing resizes the pod automatically, so size it for its expected peak rather than its starting point, with requests equal to limits as described in [Pod QoS: Guaranteed](#pod-qos-guaranteed-recommended). The resource rows list two load profiles: **moderate** (2-4 vCPU / 8Gi) and **high** (4 vCPU / 16Gi); high-rate insert workloads are high-load for Keeper. The recommended instance types are sized for the high profile; a moderate-load pod can run on a smaller node, subject to the allocatable-memory check below.
The pod cannot consume the node's full memory capacity. Kubernetes schedules the Guaranteed Keeper pod by its memory request (equal to its limit) against the node's **allocatable** memory, after subtracting the requests of DaemonSets and any other pods on that node. Use this check rather than the node's advertised capacity:
```bash theme={null}
kubectl get node \
-o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.memory,ALLOCATABLE:.status.allocatable.memory
kubectl describe node # see "Allocated resources" for DaemonSet and other pod requests
```
As an initial hardware-planning allowance on common 16-64 GiB nodes, leave roughly 4-8 GiB for kubelet/system reservations, the eviction threshold, and node-level workloads, then verify the actual `allocatable` value and allocated requests for the chosen Kubernetes distribution and node shape. Reservations scale with node size and configuration, so 4-8 GiB is not a universal formula. The recommended 16Gi Keeper pod therefore needs a node with at least 24 GiB **only if** the checks above still leave 16 GiB allocatable to it; a 32 GiB node is the safer baseline and leaves room to raise the limit later. Sizing the node to exactly match the pod leaves no way to grow without replacing it.
When picking the pod's memory figure, memory is the binding constraint: Keeper holds its entire dataset in memory, and resident usage grows with the logical metadata size. Track the `ClickHouseAsyncMetrics_KeeperApproximateDataSize` metric to watch that trend, and size memory from the expected metadata volume rather than a fixed guess.
A fixed allocation can be outgrown silently, so alert on Keeper memory headroom ([Keeper sizing alerts](/cloud/clickhouse-private/reference/metrics-and-alerts#keeper-sizing-alerts) has an example rule) and watch the growth of the Keeper metadata size. High-rate small inserts are the usual driver of metadata growth; batching inserts is the primary mitigation.
### AWS (EKS)
The ARM64 options in the AWS tables below apply to standard deployments only. For a FIPS deployment, run Server, Keeper, and Operator on x86\_64 nodes with FIPS-enabled node images. See [Deploy ClickHouse with FIPS compliance](/cloud/clickhouse-private/tutorials/deploy-government) for the FIPS-specific topology.
#### Keeper Node Group
| Property | Value |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AMI (x86) | `AL2023_x86_64` |
| AMI (arm64) | `AL2023_ARM_64` |
| Disk size | 20 GiB (node boot disk) |
| Keeper data volume | 50Gi recommended; a separate disk attached to the node in addition to the boot disk, provisioned dynamically by the Helm chart via the `keeper.storage` values (chart default 10Gi); see the [chart values reference](/cloud/clickhouse-private/reference/chart-values) |
| Recommended instance type | `m7g.2xlarge` |
| Min nodes | 3 per ClickHouse cluster |
| Kubernetes label (x86) | `clickhouseGroup: keeper` |
| Kubernetes label (arm64) | `clickhouseGroup: keeper-arm64` |
| Taint | `clickhouse.com/do-not-schedule: true, NoSchedule` |
| Taint (arm64 only) | `clickhouse.com/arch: arm64, NoSchedule` |
| Recommended resources | CPU: 2-4, Memory: 8Gi (moderate load); CPU: 4, Memory: 16Gi (high load) |
#### Server Node Group
| Property | Value |
| ------------------------- | ----------------------------------------------------------------- |
| AMI (x86) | `AL2023_x86_64` |
| AMI (arm64) | `AL2023_ARM_64` |
| Disk size | 20 GiB |
| Recommended instance type | `m7gd.16xlarge` (strongly recommended "d" series with NVMe SSD) |
| Min nodes | Equal to desired number of ClickHouse replicas |
| Kubernetes label (x86) | `clickhouseGroup: server` |
| Kubernetes label (arm64) | `clickhouseGroup: server-arm64` |
| Taint | `clickhouse.com/do-not-schedule: true, NoSchedule` |
| Taint (arm64 only) | `clickhouse.com/arch: arm64, NoSchedule` |
| Recommended resources | CPU: 64, Memory: 256Gi |
| Launch template | Custom template required for NVMe SSD RAID setup (see source doc) |
#### Operator Node Group (x86)
| Property | Value |
| --------------------- | --------------------------------------------------------------------------- |
| AMI | `AL2023_x86_64` |
| Disk size | 20 GiB |
| Minimum instance size | xlarge |
| Instance type | Any x86-compatible type |
| Notes | Can be an existing node group if already x86-compatible with minimum xlarge |
**All AWS nodes require [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) for authentication.**
For keeper and server node groups, **create one node group per AZ** if you wish to support cluster autoscaler across AZs.
Autoscaler tags (if using cluster-autoscaler):
* `k8s.io/cluster-autoscaler/enabled: true`
* `k8s.io/cluster-autoscaler/$CLUSTER_NAME: owned`
### GCP (GKE)
#### Keeper Node Pool
| Property | Value |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Machine type | `n2-standard-8` |
| Disk type | `pd-ssd` |
| Disk size | 20 GB (node boot disk) |
| Keeper data volume | 50Gi recommended; a separate disk attached to the node in addition to the boot disk, provisioned dynamically by the Helm chart via the `keeper.storage` values (chart default 10Gi); see the [chart values reference](/cloud/clickhouse-private/reference/chart-values) |
| Node label | `clickhouseGroup=keeper` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Min nodes | 3 per ClickHouse cluster |
| Workload metadata | `GKE_METADATA` (for Workload Identity) |
| Recommended resources | CPU: 2-4, Memory: 8Gi (moderate load); CPU: 4, Memory: 16Gi (high load) |
#### Server Node Pool
| Property | Value |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Machine type (dev) | `n2-standard-8` |
| Machine type (prod) | `n2-standard-64` |
| Disk type | `pd-standard` + ephemeral local SSD |
| Disk size | 20 GB |
| Local SSD | Attached as ephemeral storage (`--ephemeral-storage-local-ssd count=N`); 375 GiB each, striped into RAID-0 when `count > 1`. Scale `count` so the array holds the cache (`bytesPerGiRAM * server_memory`) plus headroom for images/logs |
| Node label | `clickhouseGroup=server` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Autoscaling | min 1, max 10 |
| Workload metadata | `GKE_METADATA` (for Workload Identity) |
| Recommended resources | CPU: 4, Memory: 16Gi (dev); CPU: 64, Memory: 256Gi (prod) |
| SSD cache | `emptyDir` backed by ephemeral local SSD; enabled by `isOnEmptyDir=true` (GCP base config default) |
#### Default Node Pool
| Property | Value |
| ------------ | --------------- |
| Machine type | `n2-standard-4` |
| Disk type | `pd-standard` |
| Disk size | 50 GB |
| Nodes | 1 |
### Azure (AKS)
#### Keeper Node Pool
| Property | Value |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| VM size | `Standard_D8ds_v5` |
| OS | AzureLinux |
| OS disk | 32 GB Premium SSD (Managed) |
| Keeper data volume | 50Gi recommended; a separate disk attached to the node in addition to the boot disk, provisioned dynamically by the Helm chart via the `keeper.storage` values (chart default 10Gi); see the [chart values reference](/cloud/clickhouse-private/reference/chart-values) |
| Min nodes | 3 per ClickHouse cluster |
| Kubernetes label | `clickhouseGroup=keeper` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Recommended resources | CPU: 2-4, Memory: 8Gi (moderate load); CPU: 4, Memory: 16Gi (high load) |
#### Server Node Pool
| Property | Value |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| VM size (dev) | `Standard_D16ds_v5` |
| VM size (prod) | `Standard_D48ds_v5` |
| OS | AzureLinux |
| OS disk type | `Ephemeral` (no managed disk cost; lands on VM cache disk) |
| Min nodes | Equal to desired number of ClickHouse replicas |
| Kubernetes label | `clickhouseGroup=server` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Recommended resources | CPU: 16, Memory: 64Gi (dev); CPU: 48, Memory: 192Gi (prod) |
| SSD cache | `hostPath` pointing at `/mnt` (Azure mounts local SSD there automatically); `isOnEmptyDir=false`, `hostPath=/mnt`; no DaemonSet or RAID required |
**Temp storage by instance size (`Standard_D*ds_v5`, Intel Icelake):**
| VM Size | vCPUs | RAM | Temp Storage | Cache headroom (\~75%) |
| ------------------- | ----- | ------- | ------------ | ---------------------- |
| `Standard_D16ds_v5` | 16 | 64 GiB | 600 GiB | \~450 GiB |
| `Standard_D32ds_v5` | 32 | 128 GiB | 1200 GiB | \~900 GiB |
| `Standard_D48ds_v5` | 48 | 192 GiB | 1800 GiB | \~1350 GiB |
#### System / Operator Node Pool
| Property | Value |
| -------- | ------------------------------------------------------------------------------------- |
| VM size | `Standard_D4s_v3` or larger |
| Notes | Default system pool created with the cluster; no ClickHouse labels or taints required |
For keeper and server node pools, **create one pool per availability zone** if you wish to support cluster autoscaler across zones.
### Bare Metal
Assign nodes to distinct roles and label and taint them so the operator schedules ClickHouse pods correctly and nothing else lands on them. At least **three failure domains** are recommended so the operator can spread Keeper across them (topology spread with max skew 1 and a minimum of 3 domains on `topology.kubernetes.io/zone`). Map each domain to a real fault-isolation boundary such as a rack, chassis, or power feed where possible.
#### Keeper Nodes
| Property | Value |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Count | 3 (one per failure domain) |
| Node label | `clickhouseGroup=keeper` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Zone label | a distinct `topology.kubernetes.io/zone` per node, see the [Kubernetes documentation on topology constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) for more context |
| Storage | local NVMe persistent volume via a StorageClass, as for the Server nodes; Keeper fsyncs every committed Raft write, so fsync latency matters more than capacity. A network-attached volume with provisioned (not burstable) performance is an acceptable fallback |
| Keeper data volume | 100Gi recommended; provisioned by the Helm chart via the StorageClass above and sized by the `keeper.storage` values (chart default 10Gi); see the [chart values reference](/cloud/clickhouse-private/reference/chart-values) |
| Recommended resources | CPU: 2-4, Memory: 8Gi (moderate load); CPU: 4, Memory: 16Gi (high load) |
#### Server Nodes
| Property | Value |
| --------------------- | --------------------------------------------------------------------------------------- |
| Count | 3+ (spread across the failure domains) |
| Node label | `clickhouseGroup=server` |
| Taint | `clickhouse.com/do-not-schedule=true:NoSchedule` |
| Zone label | spread across the failure domains |
| Local NVMe SSD | Required for the filesystem cache; mounted at `/nvme/disk` (RAID-0 if multiple devices) |
| Recommended resources | Sized to the workload |
#### Operator Nodes
At least one additional node for the operator and cluster add-ons. No ClickHouse labels or taints are required.
***
## Pod QoS: Guaranteed (recommended)
ClickHouse server, keeper, and child compute group pods should run with the Kubernetes
[**Guaranteed**](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed)
QoS class. In practice, this means that for every container in the pod,
`resources.requests` equals `resources.limits` for **both** CPU and memory, and
both fields are set. Every deployment example in this documentation follows
this pattern.
### Why
ClickHouse's memory tracker relies on a stable memory ceiling. When
`requests != limits`, the effective ceiling and the OOM-kill boundary diverge
and behavior becomes hard to reason about:
* If the memory tracker uses the request value, queries are killed while the
node still has headroom (over-conservative).
* If it uses the cgroup limit, the pod can exceed its request and be
OOM-killed by the kernel with no ClickHouse-side warning.
Guaranteed QoS also gives the pod the highest priority under node pressure --
see the [Kubernetes Pod QoS docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
for the eviction and scheduling implications.
### Choosing a different QoS class
The operator does not enforce Guaranteed QoS -- you can deliberately run
ClickHouse under `Burstable` (or another class) if, for example, you are
trading predictability for pack density. Before you do, make sure you
understand the OOM trade-offs above and the wider Kubernetes resource model:
* [Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
* [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
For the applied examples, see any of the [deployment tutorials](/cloud/clickhouse-private/tutorials/deploy-aws)
or the [compute-compute separation guide](/cloud/clickhouse-private/how-to/configure-compute-compute-separation).
***
## Networking Requirements
### AWS
| Component | Specification |
| --------------- | ---------------------------------------------- |
| VPC CIDR | `10.20.0.0/16` (example) |
| IPv6 | Not used |
| Tenancy | Default |
| Minimum AZs | 3 (for HA) |
| Public subnets | 3 (one per AZ, `/20` each) |
| Private subnets | 3 (one per AZ, `/18` each) |
| NAT gateways | 1 per AZ |
| VPC endpoints | S3 gateway |
| DNS hostnames | Enabled |
| DNS resolution | Enabled |
| CNI | IPv4 required (e.g., Amazon VPC CNI with IRSA) |
Example subnet layout (us-west-2):
| Type | AZ | CIDR |
| ------- | ---------- | ---------------- |
| Public | us-west-2a | `10.20.192.0/20` |
| Public | us-west-2b | `10.20.208.0/20` |
| Public | us-west-2c | `10.20.224.0/20` |
| Private | us-west-2a | `10.20.0.0/18` |
| Private | us-west-2b | `10.20.64.0/18` |
| Private | us-west-2c | `10.20.128.0/18` |
### GCP
| Component | Specification |
| -------------------------- | --------------------------------------------------------------------- |
| VPC mode | Custom subnet |
| GKE subnet range | `10.1.0.0/24` (example) |
| Pod secondary range | `10.244.0.0/14` (must be on `/14` boundary; 3rd octet divisible by 4) |
| Service secondary range | `10.252.0.0/20` (must be on `/20` boundary) |
| Master IPv4 CIDR | `172.16.0.0/28` |
| Private nodes | Enabled (internal IPs only) |
| Private endpoint | Enabled (no public API; FedRAMP) |
| Master authorized networks | GKE subnet + bastion subnet |
| Cloud NAT | Required for outbound internet access from private nodes |
| Private Google Access | Enabled on subnet |
### Azure
| Component | Specification |
| ------------------ | --------------------------- |
| VNet address space | `10.20.0.0/16` (example) |
| Node subnet | `10.20.0.0/22` |
| Network plugin | Azure CNI with overlay mode |
| Pod CIDR | `192.168.0.0/16` |
| Service CIDR | `10.0.0.0/16` |
| DNS service IP | `10.0.0.10` |
| Availability zones | 3 (for HA) |
### Bare Metal
| Component | Specification |
| --------------------------- | ---------------------------------------------------------------------------- |
| CNI | Any CNI providing standard IPv4 pod networking (Cilium, Calico, Canal, etc.) |
| Failure domains | At least 3, expressed via `topology.kubernetes.io/zone` node labels |
| Object storage reachability | S3 endpoint reachable from the ClickHouse pods |
| Client access | Via port-forward or a load balancer / ingress |
***
## Storage Requirements
### AWS
| Resource | Details |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| S3 bucket | Standard class, encryption enabled, same region as EKS cluster. **Do not enable S3 Lifecycle policies** -- see [Manage data lifecycle](/cloud/clickhouse-private/how-to/manage-data-lifecycle) |
| S3 bucket scope | One per ClickHouse cluster, or shared with unique `keyPrefix` per cluster |
| EBS CSI driver | Required; install via Helm with IRSA |
| StorageClass | `gp3-encrypted` (provided by Helm chart) or custom |
| NVMe cache | "d" series instances (e.g., `m7gd.*`); mounted via EC2 launch template RAID script at `/nvme/disk` |
| Cache sizing | `bytesPerGiRAM * pod_memory_limit` (e.g., 11.3Gi \* 250Gi = 2825Gi) |
| VolumeSnapshot CRDs | Required by operator |
### GCP
| Resource | Details |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCS bucket | Standard class, uniform bucket-level access, same region as GKE cluster. **Do not enable Object Lifecycle Management** -- see [Manage data lifecycle](/cloud/clickhouse-private/how-to/manage-data-lifecycle) |
| GCS S3-compatible endpoint | `https://storage.googleapis.com` |
| GCS region setting | `auto` |
| StorageClass | `premium-rwo` (GKE built-in) or custom via Helm chart with `pd.csi.storage.gke.io` provisioner |
| Topology key | `topology.gke.io/zone` (not `topology.kubernetes.io/zone`) |
| Local SSD cache | `emptyDir` backed by ephemeral local SSD (`--ephemeral-storage-local-ssd`); GCP base config sets `isOnEmptyDir=true` |
| VolumeSnapshot CRDs | Required by operator |
### Azure
| Resource | Details |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Storage Account | Standard LRS, StorageV2, public blob access disabled, same region as AKS cluster. **Do not enable Blob Lifecycle Management rules** -- see [Manage data lifecycle](/cloud/clickhouse-private/how-to/manage-data-lifecycle) |
| Blob container scope | One per ClickHouse cluster, or shared with unique `keyPrefix` per cluster |
| Azure Disk CSI driver | Pre-installed in AKS; provisioner: `disk.csi.azure.com` |
| StorageClass | `premium-ssd-v2` (PremiumV2\_LRS); created by the Helm chart when `storageClass.create=true` |
| SSD cache | `emptyDir` backed by temp disk; AKS bind-mounts `/mnt/aks/kubelet/` to `/var/lib/kubelet` when `kubelet_disk_type=Temporary`; `isOnEmptyDir=true`; no DaemonSet or RAID required |
| Cache sizing | `bytesPerGiRAM * pod_memory_limit` targeting \~75% of temp storage (e.g., 7Gi \* 64Gi = 448Gi for `Standard_D16ds_v5`) |
| VolumeSnapshot CRDs | Required by operator |
### Bare Metal
| Resource | Details |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Object storage | MinIO AIStor recommended on-prem. **Do not enable object lifecycle / expiration rules** -- see [Manage data lifecycle](/cloud/clickhouse-private/how-to/manage-data-lifecycle) |
| Object storage credentials | Static access key / secret key, supplied to server and keeper via `additionalEnvVars` from a Kubernetes Secret |
| Server filesystem cache | Local NVMe SSD mounted at `/nvme/disk` (RAID-0 if multiple devices); mounted into Server pods by the operator as a HostPath (default) or EmptyDir volume, not through a StorageClass |
| Keeper storage | Local NVMe persistent volume via a StorageClass. A network-attached volume is an acceptable fallback with provisioned (not burstable) performance; avoid sharing it with other I/O-heavy workloads |
| Cache sizing | `cacheDiskSize` set to 70-80% of usable NVMe, or `bytesPerGiRAM * pod_memory_limit` |
| S3 region | Set `isOnPremiseInstance=true`; a region is not required for on-prem object stores |
| VolumeSnapshot CRDs | Required by operator |
***
## IAM / Service Account Requirements
### AWS
#### ECR Pull Role
* **Role name:** `ClickHouseAirgapECRPullRole`
* **Purpose:** Pull images from ClickHouse private ECR
* **Permissions:** `ecr:GetDownloadUrlForLayer`, `ecr:BatchGetImage`, `ecr:BatchCheckLayerAvailability`, `ecr:DescribeImages`, `ecr:ListImages`, `ecr:GetAuthorizationToken`
* **Action:** Provide the role ARN to the ClickHouse team
#### ClickHouse Server/Keeper S3 Role
* **One role per provisioned ClickHouse cluster**
* **Naming convention:** `CH-S3-$NAME-$REGION-$ORDINAL-Role`
* `$NAME`: cluster name (e.g., `default-xx-01`)
* `$REGION`: shortened region (e.g., `uw2` for `us-west-2`)
* `$ORDINAL`: reserved, set to `00`
* Example: `CH-S3-default-xx-01-uw2-00-Role`
* **Permissions:** `s3:*`, `s3:ListBucket` on the bucket resource
* **Trust policy:** IRSA via `sts:AssumeRoleWithWebIdentity` with EKS cluster's OIDC provider
#### EBS CSI Driver Role
* **Example name:** `ClickHouse_EksEbsCsiDriverRole`
* **Managed policy:** `arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy`
* **Trust policy:** IRSA with EKS cluster's OIDC provider
### GCP
#### Deployer Service Account
| Role | Purpose |
| --------------------------------------- | ------------------------------------------ |
| `roles/compute.networkAdmin` | Create/manage VPC, subnets, NAT, routers |
| `roles/container.admin` | Create/manage GKE clusters and node pools |
| `roles/storage.admin` | Create/manage GCS buckets |
| `roles/iam.serviceAccountAdmin` | Create/manage service accounts |
| `roles/iam.serviceAccountUser` | Use service accounts for workload identity |
| `roles/iam.workloadIdentityUser` | Configure workload identity bindings |
| `roles/artifactregistry.admin` | Create/manage Artifact Registry repos |
| `roles/resourcemanager.projectIamAdmin` | Manage IAM policies on the project |
#### ClickHouse Workload Identity Service Account
* **GCP SA:** `clickhouse@$PROJECT.iam.gserviceaccount.com`
* **K8s SA (manual):** `clickhouse` in namespace `ns-$CLUSTER_NAME`
* **K8s SA (helm-created):** `ch-$CLUSTER_NAME-sa` in namespace `ns-$CLUSTER_NAME`
* **K8s SA annotation:** `iam.gke.io/gcp-service-account=`
* **IAM binding member:** `serviceAccount:$PROJECT.svc.id.goog[/]`
* **Bucket permissions:** `roles/storage.objectAdmin`, `roles/storage.legacyBucketReader`
* **Artifact Registry permissions:** `roles/artifactregistry.reader`
### Azure
#### Deployer Permissions
| Role | Purpose |
| --------------------------------------------- | ---------------------------------------------- |
| `Contributor` on resource group | Create/manage VNet, AKS, ACR, Storage Accounts |
| `User Access Administrator` on resource group | Assign roles to the managed identity |
| `AcrPush` on ACR | Push images during setup |
#### ClickHouse Managed Identity
* **Type:** User-Assigned Managed Identity (UAMI)
* **K8s SA (helm-created):** `ch-$CLUSTER_NAME-sa` in namespace `ns-$CLUSTER_NAME`
* **K8s SA annotation:** `azure.workload.identity/client-id=`
* **Federated credential subject:** `system:serviceaccount:ns-$CLUSTER_NAME:ch-$CLUSTER_NAME-sa`
* **Role assignment:** `Storage Blob Data Contributor` on the Storage Account
### Bare Metal
* **Object storage credentials:** Static access key / secret key for MinIO AIStor, held in a Kubernetes Secret (`object-storage-credentials`) and injected into server and keeper pods via `additionalEnvVars`.
* **Registry pull secret:** `registry-credentials` docker-registry secret in the `clickhouse-operator-system` and cluster namespaces, if your registry requires authentication.
* **Service account:** No cloud identity binding required.
***
## Cluster Naming Conventions
The naming convention for ClickHouse clusters is consistent across all platforms:
```
$DESCRIPTOR-$LETTERS-$ORDINAL
```
| Component | Description | Example |
| ------------- | ------------------------------------- | --------- |
| `$DESCRIPTOR` | Descriptive name (letters only) | `default` |
| `$LETTERS` | Reserved; any two letters | `xx` |
| `$ORDINAL` | Incrementing ordinal starting at `01` | `01` |
**Example name:** `default-xx-01`
The cluster name must be **unique within the target Kubernetes cluster** and is used in:
* Kubernetes namespace: `ns-$CLUSTER_NAME`
* Kubernetes resources: prefixed with `c-$CLUSTER_NAME`
* IAM role names (AWS): `CH-S3-$CLUSTER_NAME-$REGION-$ORDINAL-Role`
* S3/GCS key prefixes: `ch-s3-` (unique per cluster)
***
## Required Kubernetes Components
| Component | AWS | GCP | Azure | Bare Metal |
| ------------------- | -------------------------------- | ------------------------------------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| CNI (IPv4) | Amazon VPC CNI (recommended) | GKE built-in | Azure CNI overlay (built-in) | Any IPv4 CNI |
| CSI driver | aws-ebs-csi-driver | GKE built-in (pd.csi.storage.gke.io) | AKS built-in (disk.csi.azure.com) | StorageClass for Keeper (local PVs with hostname-only affinity are sufficient); Server cache uses HostPath/EmptyDir |
| DNS | CoreDNS | GKE built-in | AKS built-in (CoreDNS) | Any in-cluster DNS (e.g. CoreDNS) |
| VolumeSnapshot CRDs | Required | Required | Required | Required |
| Autoscaler | Cluster Autoscaler (recommended) | GKE node pool autoscaling | AKS cluster autoscaler | N/A |
| Cert manager | Not required (unless FIPS) | Required for FIPS | Not required (unless FIPS) | Not required (unless FIPS) |