The following scenarios for deploying DPOD Cloud Agent on k8s cloud providers are provided as an examples/samples and may change between different cloud providers, k8s versions and k8s cluster custom configuration.
Set variables according to your environment:
DPOD_CLOUD_AGENT_VERSION="1.0.21.0" DPOD_CLOUD_AGENT_OPERATOR_VERSION="1.1.0"
Configure Amazon EKS for Cloud Agent Deployment
Configure cluster nodes for registry mirror (containerd)
Create EC2 Launch Template for the cluster nodes.
In the Launch Template
Advanced details
->User data
add the following configuration:
Change the configuration according to your environment (at least thehost
attribute).MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash mkdir -p /etc/containerd/certs.d/icr.io printf 'server = "https://icr.io"\n[host."https://my-container-registry.example.com/v2/dpod-cloud-agent"]\n capabilities = ["pull", "resolve"]\n skip_verify = false\n override_path = true\n' | tee /etc/containerd/certs.d/icr.io/hosts.toml mkdir -p /etc/containerd/certs.d/cp.icr.io printf 'server = "https://cp.icr.io"\n[host."https://my-container-registry.example.com/v2/dpod-cloud-agent"]\n capabilities = ["pull", "resolve"]\n skip_verify = false\n override_path = true\n' | tee /etc/containerd/certs.d/cp.icr.io/hosts.toml systemctl restart containerd --//--
Make sure to create the cluster with
Node Group
that uses the newLaunch Template
Configure Google GKE for Cloud Agent Deployment
Configure cluster nodes for registry mirror (containerd)
A DaemonSet can be used to configure the GKE cluster nodes containerd runtime.
GKE nodes are automatically configured using the deprecated registry mirror format so the example DaemonSet is continuing this method.
The user may change the DamonSet logic to convert the configuration to the new format
Update the following Environment variables in the DaemonSet yaml:
ICR_IO_MIRROR
- the url for the private registry containsicr.io
imagesCP_ICR_IO_MIRROR
- the url for the private registry containscp.icr.io
images
apiVersion: apps/v1 kind: DaemonSet metadata: name: node-startup namespace: kube-system labels: k8s-app: node-startup spec: selector: matchLabels: name: node-startup updateStrategy: type: RollingUpdate template: metadata: labels: name: node-startup spec: nodeSelector: cloud.google.com/gke-container-runtime: "containerd" hostPID: true containers: - name: startup-script image: gcr.io/google-containers/startup-script:v2 imagePullPolicy: Always securityContext: privileged: true env: - name: ICR_IO_MIRROR value: "https://my-container-registry.example.com/v2/dpod-cloud-agent" - name: CP_ICR_IO_MIRROR value: "https://my-container-registry.example.com/v2/dpod-cloud-agent" - name: STARTUP_SCRIPT value: | #!/usr/bin/sh set -o errexit set -o pipefail set -o nounset CONTAINERD_CONFIG_FILE_PATH="/etc/containerd/config.toml" CONTAINERD_CONFIG_PATH_EXIST=$(grep 'config_path' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true) if [[ -z "$ICR_IO_MIRROR" ]]; then echo "ERROR: Environment variable ICR_IO_MIRROR is not set in containers.spec.env" exit 1 fi if [[ -z "$CP_ICR_IO_MIRROR" ]]; then echo "ERROR: Environment variable CP_ICR_IO_MIRROR is not set in containers.spec.env" exit 1 fi echo "INFO: Handling mirror for icr.io" # check if the mirror exist ICR_IO_MIRROR_EXIST=$(grep -xF '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."icr.io"]' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true) if [[ ${ICR_IO_MIRROR_EXIST} -eq 0 ]]; then echo -e '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."icr.io"]\n endpoint = ["'$ICR_IO_MIRROR'"]' >> ${CONTAINERD_CONFIG_FILE_PATH} else echo "INFO: icr.io mirror already exist" fi echo "INFO: Handling mirror for cp.icr.io" # check if the mirror exist ICR_IO_MIRROR_EXIST=$(grep -xF '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."cp.icr.io"]' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true) if [[ ${ICR_IO_MIRROR_EXIST} -eq 0 ]]; then echo -e '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."cp.icr.io"]\n endpoint = ["'$CP_ICR_IO_MIRROR'"]' >> ${CONTAINERD_CONFIG_FILE_PATH} else echo "INFO: cp.icr.io mirror already exist" fi # handling registry certificates (usually needed for private registry) echo "INFO: Adding certificates for registries to node trusted ca" ICR_IO_MIRROR_SERVER=$(echo ${ICR_IO_MIRROR} | awk -F/ '{print $3}') CP_ICR_IO_MIRROR_SERVER=$(echo ${CP_ICR_IO_MIRROR} | awk -F/ '{print $3}') openssl s_client -showcerts -connect ${ICR_IO_MIRROR_SERVER} </dev/null 2>/dev/null | openssl x509 -outform PEM | sudo tee /etc/ssl/certs/${ICR_IO_MIRROR_SERVER}.crt >/dev/null openssl s_client -showcerts -connect ${CP_ICR_IO_MIRROR_SERVER} </dev/null 2>/dev/null | openssl x509 -outform PEM | sudo tee /etc/ssl/certs/${CP_ICR_IO_MIRROR_SERVER}.crt >/dev/null /usr/sbin/update-ca-certificates echo "INFO: Reloading systemd management configuration" systemctl daemon-reload echo "INFO: Restarting containerd..." systemctl restart containerd
Deploy DPOD Cloud Agent on k8s cluster using OLM
Starting k8s 1.25 Pod Security Admission
became stable
feature and adopted by most of the cloud providers which requires some configurations in order to deploy OLM and use to deploy operators and applications.https://kubernetes.io/docs/concepts/security/pod-security-admission/
Change the DPOD Cloud Agent NS labels to conform with
Pod Security Admission
apiVersion: v1 kind: Namespace metadata: labels: kubernetes.io/metadata.name: example-dpod-cloudagent-ns pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latest pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/audit-version: latest pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/warn-version: latest name: example-dpod-cloudagent-ns
Create the DPOD Cloud Agent
CatalogSource
in OLM NSBased on OLM documentation in order to create subscription the
CatalogSource
should be created in the target NS (where the subscription for DPOD Cloud Agent is created) or inOLM
NS (ClusterScope)
https://olm.operatorframework.io/docs/troubleshooting/subscription/#a-subscription-in-namespace-x-cant-install-operators-from-a-catalogsource-in-namespace-yUse the following definition to create
CatalogSource
which conform withPod Security Admission
configured for theOLM
NS:apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ibm-dpod-cloud-agent-catalog namespace: olm spec: displayName: IBM DataPower Operations Dashboard Cloud Agent image: icr.io/cpopen/dpod-cloud-agent-operator-catalog:${DPOD_CLOUD_AGENT_OPERATOR_VERSION}-amd64 publisher: IBM sourceType: grpc grpcPodConfig: securityContextConfig: restricted
Create OperatorGroup for the new namespace for ClusterScope
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: dpod-cloudagent-group namespace: example-dpod-cloudagent-ns
Install the DPOD Cloud Agent operator
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ibm-dpod-cloud-agent-operator namespace: example-dpod-cloudagent-ns spec: channel: stable-v1.0 installPlanApproval: Automatic name: dpod-cloud-agent-operator source: ibm-dpod-cloud-agent-catalog sourceNamespace: olm startingCSV: dpod-cloud-agent-operator.v${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
Create CR to deploy the DPOD Cloud Agent
Create LoadBalancer services for DPOD Cloud Agent on Amazon EKS
Deploy manager LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-mng-lb-svc namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance spec: type: LoadBalancer ports: - port: 443 targetPort: 8443 name: https-manager selector: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent sessionAffinity: None
Deploy LoadBalancer service based on the number of messaging brokers replicas.
Deploy messaging bootstrap LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-bse-lb-svc namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance spec: type: LoadBalancer ports: - port: 30100 targetPort: 29092 name: kafka-bootstrap selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent sessionAffinity: None
Deploy broker-0 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-0 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance spec: type: LoadBalancer ports: - port: 30101 targetPort: 29092 name: kafka-broker-0 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-0 sessionAffinity: None
Deploy broker-1 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-1 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance spec: type: LoadBalancer ports: - port: 30102 targetPort: 29092 name: kafka-broker-1 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-1 sessionAffinity: None
Deploy broker-2 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-2 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance spec: type: LoadBalancer ports: - port: 30103 targetPort: 29092 name: kafka-broker-2 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-2 sessionAffinity: None
Create LoadBalancer services for DPOD Cloud Agent on Google GKE
Deploy manager LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-mng-lb-svc namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer ports: - port: 443 targetPort: 8443 name: https-manager selector: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent sessionAffinity: None
Deploy LoadBalancer service based on the number of messaging brokers replicas.
Deploy messaging bootstrap LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-bse-lb-svc namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer ports: - port: 30100 targetPort: 29092 name: kafka-bootstrap selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent sessionAffinity: None
Deploy broker-0 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-0 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer ports: - port: 30101 targetPort: 29092 name: kafka-broker-0 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-0 sessionAffinity: None
Deploy broker-1 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-1 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer ports: - port: 30102 targetPort: 29092 name: kafka-broker-1 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-1 sessionAffinity: None
Deploy broker-2 LB service
apiVersion: v1 kind: Service metadata: name: dpod-cloud-agent-prod-msg-dir-lb-svc-2 namespace: example-dpod-cloudagent-ns labels: app.kubernetes.io/component: dpod-cloud-agent-manager app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/managed-by: dpod-cloud-agent-operator app.kubernetes.io/name: dpod-cloud-agent-manager app.kubernetes.io/part-of: dpod-cloud-agent app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION} annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer ports: - port: 30103 targetPort: 29092 name: kafka-broker-2 selector: app.kubernetes.io/component: dpod-cloud-agent-messaging app.kubernetes.io/instance: dpod-cloud-agent-prod app.kubernetes.io/name: dpod-cloud-agent-messaging app.kubernetes.io/part-of: dpod-cloud-agent statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-2 sessionAffinity: None
Update the DNS records
Get the LB external address
kubectl get svc -n example-dpod-cloudagent-ns |grep lb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) dpod-cloud-agent-prod-mng-lb-svc LoadBalancer 10.100.12.184 34.41.36.111 443:30997/TCP dpod-cloud-agent-prod-msg-bse-lb-svc LoadBalancer 10.100.13.1 34.66.80.161 30100:31232/TCP dpod-cloud-agent-prod-msg-dir-lb-svc-0 LoadBalancer 10.100.12.9 35.223.187.112 30101:31830/TCP dpod-cloud-agent-prod-msg-dir-lb-svc-1 LoadBalancer 10.100.12.10 35.223.182.103 30101:31830/TCP dpod-cloud-agent-prod-msg-dir-lb-svc-2 LoadBalancer 10.100.12.11 35.223.116.42 30101:31830/TCP