IBM DataPower Operations Dashboard v1.0.21.x

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

The following scenarios for deploying DPOD Cloud Agent on k8s cloud providers are provided as an examples/samples and may change between different cloud providers, k8s versions and k8s cluster custom configuration.

Set variables according to your environment:

DPOD_CLOUD_AGENT_VERSION="1.0.21.0"
DPOD_CLOUD_AGENT_OPERATOR_VERSION="1.1.0"

Configure Amazon EKS for Cloud Agent Deployment

Configure cluster nodes for registry mirror (containerd)

  • Create EC2 Launch Template for the cluster nodes.

  • In the Launch Template Advanced details -> User data add the following configuration:
    Change the configuration according to your environment (at least the host attribute). 

    MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash

mkdir -p /etc/containerd/certs.d/icr.io
printf 'server = "https://icr.io"\n[host."https://my-container-registry.example.com/v2/dpod-cloud-agent"]\n    capabilities = ["pull", "resolve"]\n    skip_verify = false\n    override_path = true\n' | tee /etc/containerd/certs.d/icr.io/hosts.toml
mkdir -p /etc/containerd/certs.d/cp.icr.io
printf 'server = "https://cp.icr.io"\n[host."https://my-container-registry.example.com/v2/dpod-cloud-agent"]\n    capabilities = ["pull", "resolve"]\n    skip_verify = false\n    override_path = true\n' | tee /etc/containerd/certs.d/cp.icr.io/hosts.toml
systemctl restart containerd

--//--

  • Make sure to create the cluster with Node Group that uses the new Launch Template

Configure Google GKE for Cloud Agent Deployment

Configure cluster nodes for registry mirror (containerd)

A DaemonSet can be used to configure the GKE cluster nodes containerd runtime.

GKE nodes are automatically configured using the deprecated registry mirror format so the example DaemonSet is continuing this method.
The user may change the DamonSet logic to convert the configuration to the new format

Update the following Environment variables in the DaemonSet yaml:

  • ICR_IO_MIRROR - the url for the private registry contains icr.io images

  • CP_ICR_IO_MIRROR- the url for the private registry contains cp.icr.io images

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-startup
  namespace: kube-system
  labels:
    k8s-app: node-startup
spec:
  selector:
    matchLabels:
      name: node-startup
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: node-startup
    spec:
      nodeSelector:
        cloud.google.com/gke-container-runtime: "containerd"
      hostPID: true
      containers:
        - name: startup-script
          image: gcr.io/google-containers/startup-script:v2
          imagePullPolicy: Always
          securityContext:
            privileged: true
          env:
            - name: ICR_IO_MIRROR
              value: "https://my-container-registry.example.com/v2/dpod-cloud-agent"
            - name: CP_ICR_IO_MIRROR
              value: "https://my-container-registry.example.com/v2/dpod-cloud-agent"
            - name: STARTUP_SCRIPT
              value: |
                #!/usr/bin/sh

                set -o errexit
                set -o pipefail
                set -o nounset

                CONTAINERD_CONFIG_FILE_PATH="/etc/containerd/config.toml"
                CONTAINERD_CONFIG_PATH_EXIST=$(grep 'config_path' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true)

                if [[ -z "$ICR_IO_MIRROR" ]]; then
                    echo "ERROR: Environment variable ICR_IO_MIRROR is not set in containers.spec.env"
                    exit 1
                fi

                if [[ -z "$CP_ICR_IO_MIRROR" ]]; then
                    echo "ERROR: Environment variable CP_ICR_IO_MIRROR is not set in containers.spec.env"
                    exit 1
                fi

                echo "INFO: Handling mirror for icr.io"
                # check if the mirror exist
                ICR_IO_MIRROR_EXIST=$(grep -xF '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."icr.io"]' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true)
                if [[ ${ICR_IO_MIRROR_EXIST} -eq 0 ]]; then
                    echo -e '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."icr.io"]\n  endpoint = ["'$ICR_IO_MIRROR'"]' >> ${CONTAINERD_CONFIG_FILE_PATH}

                else
                    echo "INFO: icr.io mirror already exist"
                fi

                echo "INFO: Handling mirror for cp.icr.io"
                # check if the mirror exist
                ICR_IO_MIRROR_EXIST=$(grep -xF '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."cp.icr.io"]' ${CONTAINERD_CONFIG_FILE_PATH} | wc -l || true)
                if [[ ${ICR_IO_MIRROR_EXIST} -eq 0 ]]; then
                    echo -e '[plugins."io.containerd.grpc.v1.cri".registry.mirrors."cp.icr.io"]\n  endpoint = ["'$CP_ICR_IO_MIRROR'"]' >> ${CONTAINERD_CONFIG_FILE_PATH}

                else
                    echo "INFO: cp.icr.io mirror already exist"
                fi

                # handling registry certificates (usually needed for private registry)
                echo "INFO: Adding certificates for registries to node trusted ca"
                ICR_IO_MIRROR_SERVER=$(echo ${ICR_IO_MIRROR} | awk -F/ '{print $3}')
                CP_ICR_IO_MIRROR_SERVER=$(echo ${CP_ICR_IO_MIRROR} | awk -F/ '{print $3}')

                openssl s_client -showcerts -connect ${ICR_IO_MIRROR_SERVER} </dev/null 2>/dev/null | openssl x509 -outform PEM | sudo tee /etc/ssl/certs/${ICR_IO_MIRROR_SERVER}.crt >/dev/null
                openssl s_client -showcerts -connect ${CP_ICR_IO_MIRROR_SERVER} </dev/null 2>/dev/null | openssl x509 -outform PEM | sudo tee /etc/ssl/certs/${CP_ICR_IO_MIRROR_SERVER}.crt >/dev/null
                /usr/sbin/update-ca-certificates

                echo "INFO: Reloading systemd management configuration"
                systemctl daemon-reload
                echo "INFO: Restarting containerd..."
                systemctl restart containerd

Deploy DPOD Cloud Agent on k8s cluster using OLM

Starting k8s 1.25 Pod Security Admission became stable feature and adopted by most of the cloud providers which requires some configurations in order to deploy OLM and use to deploy operators and applications.https://kubernetes.io/docs/concepts/security/pod-security-admission/

  • Change the DPOD Cloud Agent NS labels to conform with Pod Security Admission 

    apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: example-dpod-cloudagent-ns
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
  name: example-dpod-cloudagent-ns

  • Create the DPOD Cloud Agent CatalogSource in OLM NS

    • Based on OLM documentation in order to create subscription the CatalogSource should be created in the target NS (where the subscription for DPOD Cloud Agent is created) or in OLM NS (ClusterScope)
      https://olm.operatorframework.io/docs/troubleshooting/subscription/#a-subscription-in-namespace-x-cant-install-operators-from-a-catalogsource-in-namespace-y

    • Use the following definition to create CatalogSource which conform with Pod Security Admission configured for the OLM NS:

      apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-dpod-cloud-agent-catalog
  namespace: olm
spec:
  displayName: IBM DataPower Operations Dashboard Cloud Agent
  image: icr.io/cpopen/dpod-cloud-agent-operator-catalog:${DPOD_CLOUD_AGENT_OPERATOR_VERSION}-amd64
  publisher: IBM
  sourceType: grpc
  grpcPodConfig:
    securityContextConfig: restricted

  • Create OperatorGroup for the new namespace for ClusterScope

    apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: dpod-cloudagent-group
  namespace: example-dpod-cloudagent-ns

  • Install the DPOD Cloud Agent operator

    apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-dpod-cloud-agent-operator
  namespace: example-dpod-cloudagent-ns
spec:
  channel: stable-v1.0
  installPlanApproval: Automatic
  name: dpod-cloud-agent-operator
  source: ibm-dpod-cloud-agent-catalog
  sourceNamespace: olm
  startingCSV: dpod-cloud-agent-operator.v${DPOD_CLOUD_AGENT_OPERATOR_VERSION}

  • Create CR to deploy the DPOD Cloud Agent

Create LoadBalancer services for DPOD Cloud Agent on Amazon EKS

Deploy manager LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-mng-lb-svc
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external 
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
spec:
  type: LoadBalancer
  ports:
    - port: 443
      targetPort: 8443
      name: https-manager
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
  sessionAffinity: None

Deploy LoadBalancer service based on the number of messaging brokers replicas.

  • Deploy messaging bootstrap LB service

    apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-bse-lb-svc
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external 
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
spec:
  type: LoadBalancer
  ports:
    - port: 30100
      targetPort: 29092
      name: kafka-bootstrap
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
  sessionAffinity: None

  • Deploy broker-0 LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-0
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external 
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
spec:
  type: LoadBalancer
  ports:
    - port: 30101
      targetPort: 29092
      name: kafka-broker-0
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-0
  sessionAffinity: None

  • Deploy broker-1 LB service

    apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-1
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external 
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
spec:
  type: LoadBalancer
  ports:
    - port: 30102
      targetPort: 29092
      name: kafka-broker-1
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-1
  sessionAffinity: None

  • Deploy broker-2 LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-2
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external 
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
spec:
  type: LoadBalancer
  ports:
    - port: 30103
      targetPort: 29092
      name: kafka-broker-2
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-2
  sessionAffinity: None

Create LoadBalancer services for DPOD Cloud Agent on Google GKE

Deploy manager LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-mng-lb-svc
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    cloud.google.com/l4-rbs: "enabled"
spec:
  type: LoadBalancer
  ports:
    - port: 443
      targetPort: 8443
      name: https-manager
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
  sessionAffinity: None

Deploy LoadBalancer service based on the number of messaging brokers replicas.

  • Deploy messaging bootstrap LB service

    apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-bse-lb-svc
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    cloud.google.com/l4-rbs: "enabled"
spec:
  type: LoadBalancer
  ports:
    - port: 30100
      targetPort: 29092
      name: kafka-bootstrap
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
  sessionAffinity: None

  • Deploy broker-0 LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-0
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    cloud.google.com/l4-rbs: "enabled"
spec:
  type: LoadBalancer
  ports:
    - port: 30101
      targetPort: 29092
      name: kafka-broker-0
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-0
  sessionAffinity: None

  • Deploy broker-1 LB service

    apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-1
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    cloud.google.com/l4-rbs: "enabled"
spec:
  type: LoadBalancer
  ports:
    - port: 30102
      targetPort: 29092
      name: kafka-broker-1
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-1
  sessionAffinity: None

  • Deploy broker-2 LB service

apiVersion: v1
kind: Service
metadata:
  name: dpod-cloud-agent-prod-msg-dir-lb-svc-2
  namespace: example-dpod-cloudagent-ns
  labels:
    app.kubernetes.io/component: dpod-cloud-agent-manager
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/managed-by: dpod-cloud-agent-operator
    app.kubernetes.io/name: dpod-cloud-agent-manager
    app.kubernetes.io/part-of: dpod-cloud-agent
    app.kubernetes.io/version: ${DPOD_CLOUD_AGENT_OPERATOR_VERSION}
  annotations:
    cloud.google.com/l4-rbs: "enabled"
spec:
  type: LoadBalancer
  ports:
    - port: 30103
      targetPort: 29092
      name: kafka-broker-2
  selector:
    app.kubernetes.io/component: dpod-cloud-agent-messaging
    app.kubernetes.io/instance: dpod-cloud-agent-prod
    app.kubernetes.io/name: dpod-cloud-agent-messaging
    app.kubernetes.io/part-of: dpod-cloud-agent
    statefulset.kubernetes.io/pod-name: dpod-cloud-agent-prod-msg-2
  sessionAffinity: None

Update the DNS records

  • Get the LB external address

    kubectl get svc -n example-dpod-cloudagent-ns |grep lb
NAME                                     TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)           
dpod-cloud-agent-prod-mng-lb-svc         LoadBalancer   10.100.12.184   34.41.36.111     443:30997/TCP     
dpod-cloud-agent-prod-msg-bse-lb-svc     LoadBalancer   10.100.13.1     34.66.80.161     30100:31232/TCP   
dpod-cloud-agent-prod-msg-dir-lb-svc-0   LoadBalancer   10.100.12.9     35.223.187.112   30101:31830/TCP 
dpod-cloud-agent-prod-msg-dir-lb-svc-1   LoadBalancer   10.100.12.10    35.223.182.103   30101:31830/TCP
dpod-cloud-agent-prod-msg-dir-lb-svc-2   LoadBalancer   10.100.12.11    35.223.116.42    30101:31830/TCP

  • No labels

Copyright © 2015 MonTier Software (2015) Ltd.