...

Federated architecture best fits customers that execute high load (thousands of transactions per second or more) in their gateways.

The cell environment implements the federated architecture by distributing DPOD's Store and DPOD's agents across different federated servers.

The cell environment has two main components:

• Cell Manager - a DPOD server (usually virtual) that manages all Federated Cell Members (FCMs), as well as providing central DPOD services such as the Web Console, reports, alerts, resource monitoring, etc.
• Federated Cell Members (FCMs) - DPOD servers (usually physical with very fast local storage) that include Store data nodes and agents (Syslog and WS-M) for collecting, parsing and storing data.

The cell environment does not replicate any data between the members, so adding more members will not provide any HA / DR capabilities.

The following diagram describes the cell environment:

Image RemovedImage Added

Prerequisites

1. Before installing a cell environment, make sure to complete the sizing process with IBM Support Team to get recommendations for the hardware and architecture suitable for your requirements.
2. DPOD cell manager and federated cell members must be of the same version (minimum version is 1.0.8.6).
3. DPOD cell manager is usually virtual and can be installed in both Appliance Mode or Non-Appliance Mode with Medium Load architecture type, as detailed in the Hardware and Software Requirements.
4. DPOD federated cell members (FCMs) can be one of the following:
   1. Physical servers installed in Non-appliance Mode with HighMode (based on RHEL) with High_20dv architecture type, as detailed in the Hardware and Software Requirements.
      Physical servers are used when the cell is required to process high levels of transactions per second (TPS) load.
   2. Virtual servers installed in Non-appliance Mode with Medium architecture type or higher, as detailed in the Hardware and Software Requirements.
      Virtual servers are used when the cell is required to process moderate levels of moderate transactions per second (TPS) load, or when the cell is part of a non-production environment where the production cell uses physical servers (to keep environments architecture similar).
5. A All DPOD cell environment members must have be identical - only physical or only virtual cell members (cannot mix physical and virtual cell members in the same cell).All DPOD federated cell members must have , and with the same resources , such as (CPUs, RAM, disk type and storage capacity).
6. Physical federated cell members with NVMe drives require 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Cell Member Configuration for Configuring Cell Members with 4 CPU Sockets and NVMe Disks.
7. Each cell component (manager / FCM) should have two network interfaces:
   
   1. Internal network interface - dedicated for DPOD inter-communication between the cell components.
   2. External network interface - for DPOD users to access the communicating with the rest of the network. This includes users accessing the DPOD Web Console (on the cell manager) and for , communication between DPOD and the Monitored Gateways (on both the cell manager and the members).
   3. Internal interface - for internal DPOD components inter-communication (should be a 10Gb Ethernet interface).
8. Network ports should be opened in the network firewall as detailed below:

...

...

...

...

...

...

DPOD Cell Manager

...

Each Monitored Device

...

5550 (TCP)

...

HTTP/S

...

Monitored device administration management interface.
If the SOMA port is different than 5550 - the port should be changed accordingly.

...

DPOD Cell Manager

...

TCP and UDP 53

...

DNS services. Static IP address may be used.

...

DPOD Cell Manager

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

DPOD Cell Manager

...

Organizational mail server

...

25 (TCP)

...

SMTP

...

Send reports by email

...

DPOD Cell Manager

...

LDAP

...

TCP 389 / 636 (SSL).

TCP 3268 / 3269 (SSL)

...

LDAP

...

Authentication & authorization. Can be over SSL.

...

NTP Server

...

DPOD Cell Manager

...

123 (UDP)

...

NTP

...

Time synchronization

...

Users IPs         

...

DPOD Cell Manager

...

443 (TCP)

...

HTTP/S

...

DPOD's Web Console

...

Admins IPs         

...

DPOD Cell Manager

...

22 (TCP)

...

TCP

...

SSH

...

Each DPOD Federated Cell Member

...

TCP and UDP 53

...

DNS services

...

Each DPOD Federated Cell Member

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

Each Monitored Device

...

Each DPOD Federated Cell Member

...

60000-60003 (TCP)

...

TCP

...

SYSLOG Data

...

Each Monitored Device

...

Each DPOD Federated Cell Member

...

60020-60023 (TCP)

...

HTTP/S

...

WS-M Payloads

...

NTP Server

...

Each DPOD Federated Cell Member

...

123 (UDP)

...

NTP

...

Time synchronization

...

Admins IPs         

...

Each DPOD Federated Cell Member

...

22 (TCP)

...

TCP

...

SSH

Cell Manager Installation

Prerequisites

• Make sure to meet the prerequisites listed at the top of this page.
• Install the following software package (RPM) on the cell manager: bc

DPOD Installation

Install DPOD:

• For Appliance Mode, follow the procedure: Appliance Installation.
  For Non-appliance Mode, follow the procedure: Non-Appliance Installation.
• During installation, when prompted to choose the data disk type (SSD / non SSD), choose the cell members disk type (should be SSD) instead of the cell manager disk type.
• During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.

After DPOD installation is complete, execute the following operating system performance optimization commands and reboot the server:

sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="2G"/g' /etc/init.d/MonTier-es-raw-trans-Node-1
/app/scripts/tune-os-parameters.sh
reboot

Federated Cell Member Installation

The following section describes the installation process of a single Federated Cell Member (FCM). Please repeat the procedure for every FCM installation.

Prerequisites

• Make sure to meet the prerequisites listed at the top of this page.
• Install the following software package (RPM) on the cell member: bc
• The following software packages (RPMs) are recommended for system maintenance and troubleshooting, but are not required: telnet client, net-tools, iftop, tcpdump, pciutils, nvme-cli
  

Configuring Mount Points of Cell Member

The cell member is usually a bare metal server with NVMe disks for maximizing server I/O throughput.
Each of the Store's logical nodes (service) will be bound to specific physical processor, disks and memory using NUMA (Non-Uniform Memory Access) technology.

Required information

The following table contains the list of OS mount points that should be configured along with additional information that must be gathered before federating the DPOD cell member to the cell environment.
Please copy this table, use it during the procedure, and complete the information in the empty cells as you follow the procedure:

...

* Lines marked with asterisk (*) are relevant only in case DPOD sizing team recommends 9 disks instead of 6 disks per cell member. You may remove these lines in case you have only 6 disks per cell member.

Identifying disk bays and disk serial numbers

To identify which of the server's NVMe disk bays is bound to which of the CPUs, use the hardware manufacture documentation.
Write down the disk bay as well as the disk's serial number by visually observing the disk.

Identifying disk OS paths

To list the OS path of each disk, execute the following command and write down the disk OS path (e.g.: /dev/nvme0n1) according to the disk's serial number (e.g.: PHLE8XXXXXXC3P2EGN):

nvme -list

Expected output:
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev 
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     PHLE8XXXXXXC3P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46
/dev/nvme1n1     PHLE8XXXXXXM3P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46
/dev/nvme2n1     PHLE8XXXXXX83P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46
/dev/nvme3n1     PHLE8XXXXXXN3P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46
/dev/nvme4n1     PHLE8XXXXXX63P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46
/dev/nvme5n1     PHLE8XXXXXXJ3P2EGN   SSDPE2KE032T7L                           1         3.20 TB / 3.20 TB          512 B + 0 B      QDV1LV46

Identifying PCI slot numbers

To list the the PCI slot for each disk OS path, execute the following command and write down the PCI slot (e.g.: 0c:00.0) according to the last part of the disk OS path (e.g.: nvme0n1):

lspci -nn | grep NVM | awk '{print $1}' | xargs -Innn bash -c "printf 'PCI Slot: nnn     '; ls -la /sys/dev/block | grep nnn"

Expected output:
PCI Slot: 0c:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:2 -> ../../devices/pci0000:07/0000:07:00.0/0000:08:00.0/0000:09:02.0/0000:0c:00.0/nvme/nvme0/nvme0n1
PCI Slot: 0d:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:5 -> ../../devices/pci0000:07/0000:07:00.0/0000:08:00.0/0000:09:03.0/0000:0d:00.0/nvme/nvme1/nvme1n1
PCI Slot: ad:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:1 -> ../../devices/pci0000:ac/0000:ac:02.0/0000:ad:00.0/nvme/nvme2/nvme2n1
PCI Slot: ae:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:0 -> ../../devices/pci0000:ac/0000:ac:03.0/0000:ae:00.0/nvme/nvme3/nvme3n1
PCI Slot: c5:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:3 -> ../../devices/pci0000:c4/0000:c4:02.0/0000:c5:00.0/nvme/nvme4/nvme4n1
PCI Slot: c6:00.0     lrwxrwxrwx. 1 root root 0 May 16 10:26 259:4 -> ../../devices/pci0000:c4/0000:c4:03.0/0000:c6:00.0/nvme/nvme5/nvme5n1

Tip: you may execute the following command to list the details of all PCI slots with NVMe disks installed in the server:
lspci -nn | grep -i nvme | awk '{print $1}' | xargs -Innn lspci -v -s nnn

Tip: you may execute the following command to list all disk OS paths in the server:
ls -la /sys/dev/block

Identifying NUMA nodes

To list the NUMA node of each PCI slot, execute the following command and write down the NUMA node (e.g.: 1) according to the PCI slot (e.g.: 0c:00.0):

lspci -nn | grep -i nvme | awk '{print $1}' | xargs -Innn bash -c "printf 'PCI Slot: nnn'; lspci -v -s nnn | grep NUMA"

Expected output:
PCI Slot: 0c:00.0	Flags: bus master, fast devsel, latency 0, IRQ 45, NUMA node 1
PCI Slot: 0d:00.0	Flags: bus master, fast devsel, latency 0, IRQ 52, NUMA node 1
PCI Slot: ad:00.0	Flags: bus master, fast devsel, latency 0, IRQ 47, NUMA node 2
PCI Slot: ae:00.0	Flags: bus master, fast devsel, latency 0, IRQ 49, NUMA node 2
PCI Slot: c5:00.0	Flags: bus master, fast devsel, latency 0, IRQ 51, NUMA node 3
PCI Slot: c6:00.0	Flags: bus master, fast devsel, latency 0, IRQ 55, NUMA node 3

Example of required information

This is an example of how a row of the table should look like:

...

Verifying NVMe disks speed

Execute the following command and verify all NVMe disks have the same speed (e.g.: 8GT/s):

lspci -nn | grep -i nvme | awk '{print $1}' | xargs -Innn bash -c "printf 'PCI Slot: nnn'; lspci -vvv -s nnn | grep LnkSta:"

Expected output:
PCI Slot: 0c:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
PCI Slot: 0d:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
PCI Slot: ad:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
PCI Slot: ae:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
PCI Slot: c5:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
PCI Slot: c6:00.0		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

Configuring mount points

Configure the mount points according to the table with all gathered information.
It is highly recommended to use LVM (Logical Volume Manager) to allow flexibility for future storage needs.

The following example uses LVM. You may use it for each mount point (replace vg_data2 with vg_data22/vg_data222/vg_data3 etc.):

pvcreate -ff /dev/nvme0n1
vgcreate vg_data2 /dev/nvme0n1
lvcreate -l 100%FREE -n lv_data vg_data2
mkfs.xfs -f /dev/vg_data2/lv_data

The following example is the line that should be added to /etc/fstab for each mount point (replace vg_data2 and /data2 with the appropriate values from the table):

/dev/vg_data2/lv_data    /data2                   xfs     defaults        0 0

Create a directory for each mount point (replace /data2 with the appropriate values from the table):

mkdir -p /data2

Inspecting final configuration

Execute the following command and verify mount points:

...

...

1. 1. , communication with DNS, NTP, SMTP, LDAP, and anything else on the network.
   2. This design was driven by customer requirements and allows separation between the two types of communications, which may be used to enhance the security (e.g.: deny end-users from being able to access the inter-cell communication).
   3. We recommend that all the internal network interfaces have IP addresses which belong to a single subnet (the internal subnet), and also all the external network interfaces have IP addresses which belong to a single subnet (the external subnet). Having an internal subnet that is different from the external subnet makes it easier to configure the servers without using static routing and easier to configure the network firewall rules.
   4. A diagram demonstrating this is available in Firewall Rules for DPOD Cell Environment.
2. Network rules should be defined as detailed in Firewall Rules for DPOD Cell Environment.

Cell Manager Installation

Prerequisites

• Make sure to meet the prerequisites listed at the top of this page.
• For Non-appliance Mode, follow the procedure: Prepare Pre-Installed Operating System.

DPOD Installation

• For Appliance Mode, follow the procedure: Appliance Installation.
  For Non-appliance Mode, follow the procedure: Non-Appliance Installation.
  During installation, when prompted to choose the data disk type (SSD / non SSD), choose the cell members disk type (should be SSD) instead of the cell manager disk type.
  During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.
• Once installation is done, execute the following operating system performance optimization commands and reboot the server:

sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="2G"/g' /etc/init.d/MonTier-es-raw-trans-Node-1
/app/scripts/tune-os-parameters.sh
reboot

Federated Cell Member Installation

The following section describes the installation process of a single Federated Cell Member (FCM). Please repeat the procedure for every FCM installation.

Prerequisites

• Make sure to meet the prerequisites listed at the top of this page.
• Follow the procedure: Prepare Pre-Installed Operating System.
• The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...).
• Physical federated cell members with 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Configuring Cell Members with 4 CPU Sockets and NVMe Disks.

DPOD Installation

• Physical servers should use RHEL as the operating system (and not CentOS).
• Use Non-appliance Mode and follow the procedure: Non-Appliance Installation
  During installation, the four-letter Installation Environment Name should be identical to the one that was chosen during the Cell Manager installation.
  During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.
• Once installation is done, execute the following operating system performance optimization command and reboot the server:

/app/scripts/tune-os-parameters.sh
reboot

After the server reboots, make sure httpd service is running and can be restarted successfully. If an error is displayed during the service restart, please see if the following information helps in resolving it: https://access.redhat.com/solutions/1180103

systemctl restart httpd

Configuring Mount Points of Cell Member

List of Mount Points

The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...). The data disks should be mounted to different mount points. The required mount points are:

• In case the server has 6 disks: /data2, /data22, /data3, /data33, /data4, /data44
• In case the server has 9 disks: /data2, /data22, /data222, /data3, /data33, /data333, /data4, /data44, /data444

Mapping Mount Points to Disks

Map the mount points to disks:

• In case of physical federated cell members with 4 CPU sockets and NVMe disks - use the information gathered at Configuring Cell Members with 4 CPU Sockets and NVMe Disks to map the mount point with the proper disk:

• For all other types of federated cell members servers - you may map the mount points to any disk.

Creating Mount Points

Use LVM (Logical Volume Manager) to create the mount points. You may use the following commands as an example of how to configure a single mount point (/data2 on disk nvme0n1 in this case):

pvcreate -ff /dev/nvme0n1
vgcreate vg_data2 /dev/nvme0n1
lvcreate -l 100%FREE -n lv_data vg_data2
mkfs.xfs -f /dev/vg_data2/lv_data
echo "/dev/vg_data2/lv_data    /data2                   xfs     defaults        0 0" >> /etc/fstab
mkdir -p /data2
mount /data2

Inspecting final configuration

Execute the following command and verify mount points (this example is for 6 disks per cell member and does not include other mount points that should exist):

lsblk

Expected output:
NAME                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1               259:42    0   2.9T  0 disk 
└─vg_data4data2-lv_data    253:70    0   2.9T  0 lvm  /data4

OS Configuration of Cell Member before Federation

Installing NUMA software

Execute the following command:

yum install numactl

Configuring local OS based firewall

Most Linux-based OS use a local firewall service (e.g.: iptables / firewalld). Since the OS of the Non-Appliance Mode DPOD installation is provided by the user, it is under the user's responsibility to allow needed connectivity to and from the server.

Configure the local firewall service to allow connectivity as described in the prerequisites section at the top of this page.

DPOD Installation

Install DPOD:

• Use Non-appliance Mode and follow the procedure: Non-Appliance Installation
  This installation (before the federation process is executed later) is similar to a standard All-In-One standalone DPOD installation.
  In order for this installation to complete successfully, all prerequisites for DPOD installation should be met as described in the installation procedure, including the 3 disk drives requirement.
• The four-letter Installation Environment Name should be identical to the one that was chosen during the Cell Manager installation.
• During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.

After DPOD installation is complete, execute the following operating system performance optimization commands and reboot the server:

/app/scripts/tune-os-parameters.sh
reboot

Cell Member Federation

In order to federate and configure the cell member, run the following script in the cell manager once per cell member.

...

data2
nvme1n1               259:5    0   2.9T  0 disk 
└─vg_data22-lv_data   253:11   0   2.9T  0 lvm  /data22
nvme2n1               259:1    0   2.9T  0 disk 
└─vg_data3-lv_data    253:9    0   2.9T  0 lvm  /data3
nvme3n1               259:0    0   2.9T  0 disk 
└─vg_data33-lv_data   253:10   0   2.9T  0 lvm  /data33
nvme4n1               259:3    0   2.9T  0 disk 
└─vg_data44-lv_data   253:8    0   2.9T  0 lvm  /data44
nvme5n1               259:4    0   2.9T  0 disk 
└─vg_data4-lv_data    253:7    0   2.9T  0 lvm  /data4

Cell Member Federation

In order to federate and configure the cell member, run the following script in the cell manager once per cell member.

Important: The script should be executed using the OS root user.OS root user, and also requires remote root access over SSH from the cell manager to the cell member.

Execute the script suitable for your environment:

• In case of a physical federated cell members with 4 CPU sockets and NVMe disks:
  

  Code Block
  language bash theme RDark

...

• /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member>

...

• -i physical

  
  

• In case of a physical federated cell member with 2 CPU sockets or SSD disks:
  

  Code Block
  language bash theme RDark
  /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -

...

• i physical -n true

  
  

• In case of a virtual federated cell member:
  

  Code Block
  language bash theme RDark
  /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i virtual

...

The script writes two log files , - one in the cell manager and one in the cell member. The log file names are mentioned in the script's output.
In case of a failure, the script will try to rollback the configuration changes it made, so the problem can be fixed before rerunning it again.

Cell Member Federation Post Steps

Updating service files for 4-CPU cell members

DPOD cell member is using NUMA (Non-Uniform Memory Access) technology.
The default cell member configuration binds DPOD's agent to NUMA node 0 and the Store's nodes to NUMA node 1.
If the server has 4 CPUs, the user should update the service files of nodes 2 and 3 and change the bound NUMA nodes to 2 and 3 respectively.

Note: In case the cell member has only 2 CPUs - skip this step.

To verify the amount of CPUs installed on the server, use the NUMA utility:

...

...

the configuration changes it made, so the problem can be fixed before rerunning it again.
If the rollback fails, and the cell member services do not start successfully, it might be required to uninstall DPOD from the cell member, reinstall and federate it again.

Updating Configuration for Physical Federated Cell Members with 4 CPU Sockets and NVMe Disks

Note: If the cell member server does not have 4 CPU sockets or does not have NVMe disks - skip this step.

To update the service files, execute the following commands:

sed -i 's#/usr/bin/numactl --membind=1 --cpunodebind=1#/usr/bin/numactl --membind=2 --cpunodebind=2#g' /etc/init.d/MonTier-es-raw-trans-Node-23
sed -i 's#/usr/bin/numactl --membind=1 --cpunodebind=1#/usr/bin/numactl --membind=3 --cpunodebind=3#g' /etc/init.d/MonTier-es-raw-trans-Node-34

To verify the NUMA configuration for all services, execute the following command:

grep numactl /etc/init.d/*

Updating

...

Configuration for Federated Cell Members with at least 384GB

...

RAM

...

Note: In case If the cell member server has less than 384GB RAM - skip this step.

...

Code Block
language bash theme RDark
sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-2 sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-3 sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-4

Restarting Services of the Federated Cell Member

Stop all the application services using app-util.sh → option 2 (stop all)

Start all the application services using app-util.sh → option 1 (start all)

Cell Member Federation Verification

After a successful federation, you will be able to see the new federated cell member in the Manage → System → Nodes page.
For  For example:

Also, the new agents will be shown in the agents list in the Manage → Internal Health → Agents page:

...

Configure the monitored gateways to use the federated cells agents. Please follow instructions on Adding Monitored DevicesGateways.