> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-trino-dialect.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Deploy ClickHouse with FIPS Compliance (Government)
This tutorial covers deploying ClickHouse Private with FIPS 140-3 compliance on AWS. It builds on the standard AWS deployment, with stricter node architecture and AMI requirements, then adds the FIPS-specific Kubernetes configuration.
**Deploying on GKE?**
The FIPS-specific Kubernetes steps below, including certificate generation, certificate secrets, and the `openSSL` Helm flags, are cloud-agnostic. For GCP, provision x86\_64 infrastructure with [Deploy on GCP GKE](/cloud/clickhouse-private/tutorials/deploy-gcp), then apply those steps here, substituting the GCS storage settings and Workload Identity from the GCP guide.
***
## Prerequisites
Before starting this tutorial, you must have completed the following from [tutorials/deploy-aws.md](/cloud/clickhouse-private/tutorials/deploy-aws):
* **Step 1:** IAM role for ECR access
* **Step 2:** Container images copied to your ECR using the FIPS version tags below
* **Step 3:** VPC created
* **Step 4:** EKS cluster created
* **Step 5:** FIPS node groups created using the x86\_64 requirements below instead of the standard ARM64 recommendations
* **Step 6:** S3 bucket and IAM roles created
### FIPS node group requirements
Use x86\_64 nodes for ClickHouse Server, ClickHouse Keeper, and the ClickHouse Operator. The cryptographic provider validation for the current Server and Keeper FIPS images does not include Linux on ARM64. A FIPS-enabled ARM node image does not extend that validation to the cryptographic module inside the ClickHouse containers.
AWS supports two x86\_64 node image paths for this deployment:
* Use `BOTTLEROCKET_x86_64_FIPS`, which is available directly as an EKS managed node group AMI type.
* Build a custom EKS-optimized Amazon Linux 2023 x86\_64 AMI with [FIPS mode enabled](https://docs.aws.amazon.com/linux/al2023/ug/fips-mode.html), then select it through an EC2 launch template. AWS publishes `AL2023_x86_64_STANDARD`, but does not expose a separate `AL2023_x86_64_FIPS` managed node group AMI type. See the [EKS AMI build scripts](https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-build-scripts.html).
The Bottlerocket option avoids maintaining a custom AMI. Use the AL2023 option when your organization standardizes on AL2023 and already has a custom AMI build and patching process.
| Node group | Architecture | FIPS node AMI options | Instance guidance | Kubernetes label |
| ---------- | ------------ | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- |
| Server | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | x86 "d" series with local NVMe and a 1:4 vCPU-to-memory ratio; [`m5d.16xlarge`](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html) is a compatible GovCloud example | `clickhouseGroup: server` |
| Keeper | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | `m8i.2xlarge` when available; `m7i.2xlarge`, `m6i.2xlarge`, or `m5.2xlarge` are compatible alternatives | `clickhouseGroup: keeper` |
| Operator | x86\_64 | `BOTTLEROCKET_x86_64_FIPS` or custom FIPS-enabled AL2023 x86\_64 | Minimum xlarge, using an x86-compatible instance type | No dedicated label required |
Configure instance-store NVMe using the matching AL2023 or Bottlerocket tab in [Step 5 of the standard AWS deployment](/cloud/clickhouse-private/tutorials/deploy-aws#server-node-group). In the Helm command below, set `SSD_CACHE_IS_ON_EMPTY_DIR=true` for Bottlerocket, which binds instance-store NVMe to `/var/lib/kubelet`. Set it to `false` for AL2023, which mounts instance-store NVMe at `/nvme/disk`.
### FIPS Version Tags
Use the following FIPS-specific image tags when copying artifacts:
| Artifact | Tag |
| ------------------------------ | --------------------- |
| clickhouse-server | `26.2.1.525-fips` |
| clickhouse-keeper | `26.2.1.258-fips` |
| clickhouse-operator | `main-1.20441.1-fips` |
| helm/clickhouse-operator-helm | `1.20441.1` |
| helm/onprem-clickhouse-cluster | `1.8.7` |
### S3 Bucket Naming Restriction
**The S3 bucket name cannot contain any periods (`.`)** when using FIPS. The S3 FIPS endpoints require virtual-host-style addressing, which does not work when the bucket name contains periods.
***
## Step 1: Create FIPS Certificates
FIPS compliance requires TLS certificates generated using FIPS-approved algorithms in a FIPS-enabled environment. You need three sets of certificates:
| Certificate | Private Key | Public Key | SAN Required |
| ----------- | ------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Server | `server.key` | `server.crt` | `*.c-${CLUSTER_NAME}-server-headless.ns-${CLUSTER_NAME}.svc.cluster.local`, `c-${CLUSTER_NAME}-server-any.ns-${CLUSTER_NAME}.svc.cluster.local` |
| Client | `client.key` | `client.crt` | (none) |
| Keeper | `keeper.key` | `keeper.crt` | `*.c-${CLUSTER_NAME}-keeper-headless.ns-${CLUSTER_NAME}.svc.cluster.local` |
You also need a CA certificate (`ca.crt` and `ca.key`), either self-signed or from your organization's certificate authority.
**Requirements:**
* Minimum RSA 3072-bit key size
* SHA-256 or higher signature algorithm
* Generated in a FIPS-enabled environment
* A unique set of certificates (excluding CA) must be created for each ClickHouse cluster
For the full certificate generation procedure using a Red Hat UBI8 container with FIPS crypto policy, see [how-to/configure-fips-certificates.md](/cloud/clickhouse-private/how-to/configure-fips-certificates).
***
## Step 2: Install Kubernetes Prerequisites
### Install VolumeSnapshot CRDs
```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
```
### Install StorageClass (Optional)
Only if you do not have a custom or existing StorageClass:
```bash theme={null}
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
CHART_VERSION=1.8.7
helm install clickhouse-prerequisites \
oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
--version=$CHART_VERSION \
-n default \
--set-json="storageClass.create=true" \
--set-json="createCluster=false" \
--set-json="serviceAccount.create=false"
```
***
## Step 3: Install Operator
Note the FIPS ECR endpoint format: `0000000000.dkr-ecr-fips.us-west-2.on.aws` (not the standard `.amazonaws.com` endpoint).
```bash theme={null}
# update ECR_HOST as needed
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
# should use the version of the operator's helm chart, not of the operator itself (eg 1.20441.1, not main-1.20441.1)
OPERATOR_VERSION=1.20441.1
# image tag: the FIPS operator build (note the main- prefix and the -fips suffix).
# Always set this explicitly: the FIPS variant is selected by the tag, so an
# unset tag will not give you a FIPS build.
OPERATOR_IMAGE_TAG=main-1.20441.1-fips
# set AZs as determined by VPC subnets
AZ_LIST='["us-west-2a","us-west-2b","us-west-2c"]'
helm install clickhouse-operator \
oci://$ECR_HOST/helm/clickhouse-operator-helm \
--version=$OPERATOR_VERSION \
--create-namespace \
-n clickhouse-operator-system \
--set-json="image.repository=\"$ECR_HOST/clickhouse-operator\"" \
--set-json="image.tag=\"$OPERATOR_IMAGE_TAG\"" \
--set-json='cilium.enabled=false' \
--set-json='idleScalerEnabled=false' \
--set-json='webhooks.enabled=false' \
--set-json='operator.debug=true' \
--set-json='operator.metricsScraper.enabled=false' \
--set-json="operator.availabilityZones=$AZ_LIST" \
--set-json='operator.featureFlags.backupOnASeparatePod=true' \
--set-json='operator.featureFlags.serverCMEKEnabled=true'
```
***
## Step 4: Create Certificate Secrets
Secrets must be installed in the same Kubernetes namespace where the ClickHouse cluster will run. If your cluster name is `default-xx-01`, the namespace will be `ns-default-xx-01`.
Ensure the namespace exists, then create the secrets. **Do not change the secret names or key names.**
```bash theme={null}
CERT_DIR=.
CLUSTER_NAME=default-xx-01
# server
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-server-cert-secret \
--from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
--from-file=server.crt="$CERT_DIR/server/server.crt" \
--from-file=server.key="$CERT_DIR/server/server.key" \
--from-file=client.crt="$CERT_DIR/client/client.crt" \
--from-file=client.key="$CERT_DIR/client/client.key"
# keeper
kubectl create secret generic -n ns-${CLUSTER_NAME} ${CLUSTER_NAME}-keeper-cert-secret \
--from-file=ca.crt="$CERT_DIR/ca/ca.crt" \
--from-file=keeper.crt="$CERT_DIR/keeper/keeper.crt" \
--from-file=keeper.key="$CERT_DIR/keeper/keeper.key"
```
***
## Step 5: Deploy ClickHouse Cluster
Use the correct FIPS version of the endpoints for ECR and S3. FIPS endpoints can be found at [https://aws.amazon.com/compliance/fips/](https://aws.amazon.com/compliance/fips/).
**Guaranteed QoS (recommended)**
ClickHouse workloads should run with matching `requests` and `limits` for
both CPU and memory. The single `SERVER_CPU`/`SERVER_MEMORY`/`KEEPER_CPU`/
`KEEPER_MEMORY` values below are applied to **both** `resources.requests`
and `resources.limits` in the helm invocation that follows, which places
the pods in the [Guaranteed](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
QoS class. If you need to run with a different QoS class, review
[Pod QoS: Guaranteed (recommended)](/cloud/clickhouse-private/reference/infrastructure-requirements#pod-qos-guaranteed-recommended)
first for the trade-offs.
**FIPS deployments set the image tags explicitly**
The standard tutorials leave `server.image.tag` / `keeper.image.tag` unset and take
the versions pinned in the cluster Helm chart. FIPS is the exception: the FIPS
variant is selected by the image tag rather than by a separate chart, so the `-fips`
tags below must be set explicitly. Keep them aligned with the versions in the
[Component Versions reference](/cloud/clickhouse-private/reference/versions).
```bash theme={null}
# this will be the `default` user's password
PASSWORD='My super secret p@$$w0rd'
HASHED_PASSWORD=`echo -n "$PASSWORD" | shasum -a 256 | awk '{printf $1}' | base64`
# update values below as needed,
# the ecr host should be the one in your account
# the s3 bucket should be the one created earlier
CLUSTER_NAME=default-xx-01
ECR_HOST=0000000000.dkr-ecr-fips.us-west-2.on.aws
SA_IAM_ROLE_ARN=arn:aws:iam::0000000000:role/CH-S3-${CLUSTER_NAME}-uw2-00-Role
S3_BUCKET_REGION=us-gov-west-1
S3_BUCKET_URL=https://my-clickhouse-data.us-gov-west-1.amazonaws.com
S3_BUCKET_NAME=my-clickhouse-data
# s3 key prefix can use any UUID value, but must be unique for all clusters storing data in the bucket specified above
S3_KEY_PREFIX=ch-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')
# these must match the x86_64 clickhouseGroup labels added to the node groups
SERVER_NG_GROUP=server
KEEPER_NG_GROUP=keeper
# these values should change depending on selected instance sizes
# be sure to take daemon set requirements into account when setting CPU and MEMORY values
SERVER_CPU=64
SERVER_MEMORY=256Gi
# Keeper: high-load profile; for moderate load 2-4 CPU / 8Gi also works (see the infrastructure reference)
KEEPER_CPU=4
KEEPER_MEMORY=16Gi
# Update the `verificationMode` field to control how TLS certificates are verified.
# Defaults to `none`. See Context.h in the ClickHouse source code for more details.
SERVER_VERIFICATION_MODE=none
KEEPER_VERIFICATION_MODE=none
# Currently, usage of `strict` is not recommended and you might encounter issues with the Operator.
# If you require mTLS cert auth, use `relaxed` (Teleport integration requires it)
# SERVER_VERIFICATION_MODE=relaxed
# KEEPER_VERIFICATION_MODE=relaxed
CHART_VERSION=1.8.7
# As a general rule, set cache size to 70-80% of the allocatable instance-store SSD capacity,
# accounting for DaemonSets and cloud-provider reserved disk space.
# The example below sizes the cache for the four 600 GB NVMe devices on m5d.16xlarge.
CACHE_DISK_SIZE_GI=1700Gi
# Select according to the Server node AMI:
# - Bottlerocket with instance-store NVMe bound to /var/lib/kubelet: true
# - AL2023 with instance-store NVMe mounted at /nvme/disk: false
SSD_CACHE_IS_ON_EMPTY_DIR=true
helm install $CLUSTER_NAME \
oci://$ECR_HOST/helm/onprem-clickhouse-cluster \
--version=$CHART_VERSION \
-n ns-$CLUSTER_NAME \
--create-namespace \
--set-json="account.hashedPassword=\"$HASHED_PASSWORD\"" \
--set-json="server.image.repository=\"$ECR_HOST/clickhouse-server\"" \
--set-json="server.storage.s3.endpoint=\"$S3_BUCKET_URL\"" \
--set-json="server.storage.s3.region=\"$S3_BUCKET_REGION\"" \
--set-json="server.storage.s3.bucketName=\"$S3_BUCKET_NAME\"" \
--set-json="server.storage.s3.keyPrefix=\"$S3_KEY_PREFIX\"" \
--set-json='server.arm64=false' \
--set-json="server.image.tag=\"26.2.1.525-fips\"" \
--set-json="server.podPolicy.nodeSelector.clickhouseGroup=\"$SERVER_NG_GROUP\"" \
--set-json="server.openSSL.enabled=true" \
--set-json="server.openSSL.required=true" \
--set-json="server.openSSL.verificationMode=$SERVER_VERIFICATION_MODE" \
--set-json="keeper.image.repository=\"$ECR_HOST/clickhouse-keeper\"" \
--set-json='keeper.arm64=false' \
--set-json="keeper.image.tag=\"26.2.1.258-fips\"" \
--set-json="keeper.podPolicy.nodeSelector.clickhouseGroup=\"$KEEPER_NG_GROUP\"" \
--set-json="keeper.openSSL.enabled=true" \
--set-json="keeper.openSSL.required=true" \
--set-json="keeper.openSSL.verificationMode=$KEEPER_VERIFICATION_MODE" \
--set-json="serviceAccount.annotations={\"eks.amazonaws.com/role-arn\":\"$SA_IAM_ROLE_ARN\"}" \
--set-json='server.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
--set-json='keeper.tolerations=[{"effect":"NoSchedule","key":"clickhouse.com/do-not-schedule","operator":"Exists"}]' \
--set-json="server.ssdCacheConfiguration.cacheDiskSize=\"$CACHE_DISK_SIZE_GI\"" \
--set-json="server.ssdCacheConfiguration.isOnEmptyDir=$SSD_CACHE_IS_ON_EMPTY_DIR" \
--set-json="server.podPolicy.resources.limits.cpu=\"$SERVER_CPU\"" \
--set-json="server.podPolicy.resources.limits.memory=\"$SERVER_MEMORY\"" \
--set-json="server.podPolicy.resources.requests.cpu=\"$SERVER_CPU\"" \
--set-json="server.podPolicy.resources.requests.memory=\"$SERVER_MEMORY\"" \
--set-json="keeper.podPolicy.resources.limits.cpu=\"$KEEPER_CPU\"" \
--set-json="keeper.podPolicy.resources.limits.memory=\"$KEEPER_MEMORY\"" \
--set-json="keeper.podPolicy.resources.requests.cpu=\"$KEEPER_CPU\"" \
--set-json="keeper.podPolicy.resources.requests.memory=\"$KEEPER_MEMORY\""
```
**Key differences from the standard AWS deployment:**
| Setting | Standard | FIPS |
| ------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Node architecture | ARM64 recommended on AWS | x86\_64 required |
| Node AMI | Standard EKS-optimized AMI | `BOTTLEROCKET_x86_64_FIPS` or a custom FIPS-enabled x86\_64 AMI |
| ECR endpoint | `0000000000.dkr.ecr.us-west-2.amazonaws.com` | `0000000000.dkr-ecr-fips.us-west-2.on.aws` |
| Server image tag | `26.2.1.525` | `26.2.1.525-fips` |
| Keeper image tag | `26.2.1.258` | `26.2.1.258-fips` |
| Operator image tag | `main-1.20441.1` | `main-1.20441.1-fips` |
| S3 endpoint | (default) | Explicit FIPS endpoint URL |
| `server.openSSL.enabled` | not set | `true` |
| `server.openSSL.required` | not set | `true` |
| `keeper.openSSL.enabled` | not set | `true` |
| `keeper.openSSL.required` | not set | `true` |
| Architecture flags | `server.arm64=true`, `keeper.arm64=true` | `server.arm64=false`, `keeper.arm64=false` |
| Cache configuration | `bytesPerGiRAM` | `cacheDiskSize` (explicit size); `isOnEmptyDir=true` for Bottlerocket or `false` for AL2023 |
***
## Step 6: Verify Installation
### Port-forward on TLS Port
With FIPS mode, ClickHouse uses TLS on port **9440** instead of the standard port 9000.
```bash theme={null}
kubectl port-forward svc/c-default-xx-01-server-any 9440:9440 -n ns-default-xx-01
```
### Create Client Configuration
Create a `client-config.xml` file to tell the ClickHouse client where to find the CA certificate. Update the path to your `ca.crt` file:
```xml theme={null}
/path/to/certs/ca.crt
```
### Connect and Run a Query
```bash theme={null}
clickhouse client --host localhost --port 9440 --password $PASSWORD --secure --config=client-config.xml
```
Run a test query:
```sql theme={null}
clickhouse-cloud :) select 1;
```
Expected output:
```
SELECT 1
Query id: 825591bf-a8e5-4995-ac9f-afb864854ba2
┌─1─┐
1. │ 1 │
└───┘
1 row in set. Elapsed: 0.001 sec.
```
**Important:** Port 9000 (plaintext native) and port 8123 (plaintext HTTP) will refuse connections when `openSSL.required=true`. Use port 9440 (native TLS) or port 8443 (HTTPS) instead.
***
## Next Steps
* **Teleport integration** (optional): For secure database access via Teleport Enterprise, see [Integrate Teleport with ClickHouse Private](/cloud/clickhouse-private/how-to/integrate-teleport). This requires `SERVER_VERIFICATION_MODE=relaxed` and `KEEPER_VERIFICATION_MODE=relaxed`.