Versions Compared

Version Old Version 71 New Version Current
Changes made by

Shachar Greenberg

Amit Munwes

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Federated architecture best fits customers that execute high load (thousands of transactions per second or more) in their gateways, where the vast majority of the transactions is executed on-premise.

The The cell environment implements the federated architecture by distributing DPOD's Store and DPOD's processing (using DPOD's agents ) across different federated servers.

The cell environment has two main components:

  • Cell Manager - a DPOD server (usually virtual or physical) that manages all Federated Cell Members (FCMs), as well as providing central DPOD services such as the Web Console, reports, alerts, resource monitoring, etc.
  • Federated Cell Member Members (FCMFCMs) - a DPOD server servers (usually physical with very fast local high speed storage) that includes include Store data nodes and agents (Syslog and WS-M) for collecting, parsing and storing data. There could be one or more federated cell members per cell.

The following diagram describes the Cell Environment:

Image Removed

The following procedure describes the process of establishing a DPOD cell environment.

Prerequisites

The cell environment does not replicate any data between the members, so adding more members will not provide any HA / DR capabilities.

The following diagram describes the cell environment:

Image Added

Prerequisites

  1. Before installing a cell environment, make sure to complete the sizing process with IBM Support Team to get recommendations for the hardware and architecture suitable for your requirements.
  2. DPOD cell manager and federated cell members must be with members must be of the same version (minimum version is v1.0.8.5)..
  3. DPOD cell manager is usually virtual and can be installed in both Appliance Mode or Non-Appliance Mode with Medium Load architecture both Appliance Mode or Non-Appliance Mode with Medium Load architecture type, as detailed in the the Hardware and Software Requirements. The manager server can be both virtual or physical.Physical
  4. DPOD federated cell member members (FCMFCMs) must be installed in Non-appliance Mode with High_20dv architecture can be one of the following:
    1. Physical servers installed in Non-appliance Mode (based on RHEL) with High_20dv architecture type, as detailed in the Hardware and Software Requirements.
    Virtual DPOD federated cell member (FCM) must be installed in Non-appliance Mode with Medium architecture
    1. Physical servers are used when the cell is required to process high transactions per second (TPS) load.
    2. Virtual servers installed in Non-appliance Mode with Medium architecture type or higher, as detailed in the Hardware and Software Requirements
     and in Virtual Cell Environment Installation
    1. .
  5. All DPOD federated cell member (FCM) must have the exactly the same resources such CPUs , Memory , disk type and storage capacity.
  6. Each cell component (manager / FCM) should have two network interfaces:
    1. External interface - for DPOD users to access the Web Console (on the cell manager) and for communication between DPOD and Monitored Gateways (on both the cell manager and the members).
    2. Internal interface - for internal DPOD components inter-communication (should be a 10Gb Ethernet interface).
  7. Network ports should be opened in the network firewall as detailed below:

...

From

...

To

...

Ports (Defaults)

...

Protocol

...

Usage

...

DPOD Cell Manager

...

Each Monitored Device

...

5550 (TCP)

...

HTTP/S

...

Monitored device administration management interface

...

DPOD Cell Manager

...

TCP and UDP 53

...

DNS services. Static IP address may be used.

...

DPOD Cell Manager

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

DPOD Cell Manager

...

Organizational mail server

...

25 (TCP)

...

SMTP

...

Send reports by email

...

DPOD Cell Manager

...

LDAP

...

TCP 389 / 636 (SSL).

TCP 3268 / 3269 (SSL)

...

LDAP

...

Authentication & authorization. Can be over SSL.

...

NTP Server

...

DPOD Cell Manager

...

123 (UDP)

...

NTP

...

Time synchronization

...

Each Monitored Device

...

DPOD Cell Manager

...

60000-60003 (TCP)

...

TCP

...

SYSLOG Data

...

Each Monitored Device

...

DPOD Cell Manager

...

60020-60023 (TCP)

...

HTTP/S

...

WS-M Payloads

...

Users IPs         

...

DPOD Cell Manager

...

443 (TCP)

...

HTTP/S

...

DPOD's Web Console

...

Admins IPs         

...

DPOD Cell Manager

...

22 (TCP)

...

TCP

...

SSH

...

Each DPOD Federated Cell Member

...

TCP and UDP 53

...

DNS services

...

Each DPOD Federated Cell Member

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

NTP Server

...

Each DPOD Federated Cell Member

...

123 (UDP)

...

NTP

...

Time synchronization

...

Each Monitored Device

...

Each DPOD Federated Cell Member

...

60000-60003 (TCP)

...

TCP

...

SYSLOG Data

...

Each Monitored Device

...

Each DPOD Federated Cell Member

...

60020-60023 (TCP)

...

HTTP/S

...

WS-M Payloads

...

Admins IPs         

...

Each DPOD Federated Cell Member

...

22 (TCP)

...

TCP

...

SSH

...

Cell Manager Installation

Prerequisites

  • DPOD cell manager  should be installed in Non-Appliance Mode with Medium Load architecture type, as detailed in the Hardware and Software Requirements. The manager server can be both virtual or physical.
  • Install the following software package (RPM): bc

Installation

...

Note
Important !!  During the Cell Manager installation, the user will be asked to choose the data disk type (SSD / non SSD) - choose the cell members disk type (should be SSD)
Note

As described in the prerequisites section, the cell manager should have two network interfaces.

When installing DPOD, the user is prompted to choose the IP address for the Web Console - this should be the IP address of the external network interface.

  • After DPOD installation is complete, the user should execute the following operating system performance optimization script:
Code Block
themeRDark
/app/scripts/tune-os-parameters.sh

Federated Cell Member Installation

The following section describes the installation process of a single Federated Cell Member (FCM). User should repeat the procedure for every FCM installation.

Note
Important !!  The initial installation (until the federation process is executed) of both Cell Manager and Cell Members is a standard All-In-One standalone DPOD installation .
In order for the initial installation to complete successful  all per-requisites for DPOD installation should be met as described on Hardware and Software Requirements  and Prepare Pre-Installed Operating System (including the 3 disk drives)

Prerequisites

  • DPOD federated cell member (FCM) should be installed in Non-appliance Mode with High_20dv architecture type, as detailed in the Hardware and Software Requirements.
  • Install the following software package (RPM): bc
  • The following software packages (RPMs) are recommended for system maintenance and troubleshooting, but are not required: telnet client, net-tools, iftop, tcpdump, pciutils, nvme-cli

Installation

DPOD Installation

  • Install DPOD in Non-Appliance Mode: Installation Procedure
  • The four-letter Installation Environment Name should be identical to the one that was chosen for the Cell Manager.
Note

As described in the prerequisites section, the federated cell member should have two network interfaces.

When installing DPOD, the user is prompted to choose the IP address for the Web Console - this should be the IP address of the external network interface (although the FCM does not run the Web Console service).

  • After DPOD installation is complete, the user should execute the following operating system performance optimization script:
Code Block
themeRDark
/app/scripts/tune-os-parameters.sh
Note

User should reboot the server for the new performance optimization to take effect.

Preparing Cell Member for Federation

Preparing Mount Points

The cell member is usually a "bare metal" server with NVMe disks for maximizing server throughput.

Each of the Store's logical node (service) will be bound to a specific physical processor, disks and memory using NUMA (Non-Uniform Memory Access) technology.

The default cell member configuration assumes 6 NVMe disks which will serve 3 Store logical nodes (2 disks per node).

The following OS mount points should be configured by the user before federating the DPOD cell member to the cell environment.

Note

We highly recommend the use of LVM (Logical Volume Manager) to allow flexible storage for future storage needs.

Empty cells in the following table should be completed by the user, based on their specific hardware:

...

How to Identify Disk OS Path and Disk Serial

...

Identify all NVMe Disks installed on the server

Code Block
themeRDark
lspci -nn | grep NVM

expected output :

5d:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]
5e:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]
ad:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]
ae:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]
c5:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]
c6:00.0 Non-Volatile memory controller [0108]: Intel Corporation Express Flash NVMe P4500 [8086:0a54]

Locate disk's NUMA node
Use the disk PCI slot  listed in previous command  to identify the NUMA node (the first disk PCI slot is :  5d:00.0 )

Code Block
themeRDark
linenumberstrue
lspci  -s 5d:00.0 -v

expected output :

5d:00.0 Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500 (prog-if 02 [NVM Express])
        Subsystem: Lenovo Device 4712
        Physical Slot: 70
        Flags: bus master, fast devsel, latency 0, IRQ 93, NUMA node 1
        Memory at e1310000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at e1300000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI-X: Enable+ Count=129 Masked-
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Virtual Channel
        Capabilities: [180] Power Budgeting <?>
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [270] Device Serial Number 55-cd-2e-41-4f-89-0f-43
        Capabilities: [2a0] #19
        Capabilities: [2d0] Latency Tolerance Reporting
        Capabilities: [310] L1 PM Substates
        Kernel driver in use: nvme
        Kernel modules: nvme

...

Identify NVMe Disks path
Use the disk PCI slot  listed in previous command  to identify the disk's block device path

Code Block
themeRDark
ls -la /sys/dev/block |grep  5d:00.0

expected output :
lrwxrwxrwx. 1 root root 0 Nov  5 08:06 259:4 -> ../../devices/pci0000:58/0000:58:00.0/0000:59:00.0/0000:5a:02.0/0000:5d:00.0/nvme/nvme0/nvme0n1

...

Code Block
themeRDark
nvme -list |grep nvme0n1

expected output :

/dev/nvme0n1     PHLE822101AN3P2EGN   SSDPE2KE032T7L                           1           3.20  TB /   3.20  TB    512   B +  0 B   QDV1LV45

...

  1. Use the disk bay number and the disk serial number (visually identified) and correlate them with the output of the disk tool to identify the disk OS path.
Example for Mount Points and Disk Configurations

...

Example for LVM Configuration
Code Block
themeRDark
pvcreate -ff /dev/nvme0n1
vgcreate vg_data2 /dev/nvme0n1
lvcreate -l 100%FREE -n lv_data vg_data2
mkfs.xfs -f /dev/vg_data2/lv_data

pvcreate -ff /dev/nvme1n1
vgcreate vg_data22 /dev/nvme1n1
lvcreate -l 100%FREE -n lv_data vg_data22
mkfs.xfs /dev/vg_data22/lv_data

/etc/fstab file:

Code Block
themeRDark
/dev/vg_data2/lv_data    /data2                   xfs     defaults        0 0
/dev/vg_data22/lv_data   /data22                   xfs     defaults        0 0
/dev/vg_data3/lv_data    /data3                   xfs     defaults        0 0
/dev/vg_data33/lv_data   /data33                   xfs     defaults        0 0
/dev/vg_data4/lv_data    /data4                   xfs     defaults        0 0
/dev/vg_data44/lv_data   /data44                   xfs     defaults        0 0

Create directories for the new data mount points

Code Block
themeRDark
mkdir -p /data2 /data22 /data3 /data33 /data4 /data44
Example for the Final Configuration for 3 Store's nodes
Note

This example does not include other mount points needed, as describe in Hardware and Software Requirements.

Code Block
themeRDark
# lsblk

NAME                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1             259:0    0   2.9T  0 disk
└─vg_data2-lv_data  253:6    0   2.9T  0 lvm  /data2
nvme1n1             259:5    0   2.9T  0 disk
└─vg_data22-lv_data 253:3    0   2.9T  0 lvm  /data22
nvme2n1             259:1    0   2.9T  0 disk
└─vg_data3-lv_data  253:2    0   2.9T  0 lvm  /data3
nvme3n1             259:2    0   2.9T  0 disk
└─vg_data33-lv_data 253:5    0   2.9T  0 lvm  /data33
nvme4n1             259:4    0   2.9T  0 disk
└─vg_data44-lv_data 253:7    0   2.9T  0 lvm  /data44
nvme5n1             259:3    0   2.9T  0 disk
└─vg_data4-lv_data  253:8    0   2.9T  0 lvm  /data4

Install NUMA Software

Code Block
themeRDark
yum install numactl

Preparing Local OS Based Firewall

Most Linux-based OS uses a local firewall service (e.g.: iptables / firewalld).

Since the OS of the Non-Appliance Mode DPOD installation is provided by the user, it is under the user's responsibility to allow needed connectivity to and from the server.

User should make sure needed connectivity detailed in Network Ports Table is allowed on the OS local firewall service.

Note

When using DPOD Appliance mode installation for the cell manager, local OS based firewall service is handled by the cell member federation script.

Cell Member Federation

In order to federate and configure the cell member, run the following script in the cell manager once per cell member.

For instance, to federate two cell members, the script should be run twice (in the cell manager) - first time with the IP address of the first cell member, and second time with the IP address of the second cell member.

Important: The script should be executed using the OS root user.

Code Block
themeRDark
/app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member>
For example: /app/scripts/configure_cell_manager.sh -a 172.18.100.34 -g 172.17.100.33
Example for a Successful Execution
Code Block
themeRDark
 /app/scripts/configure_cell_manager.sh -a 172.18.100.36 -g 172.17.100.35

2018-10-22_16-13-16 INFO Cell Configuration
2018-10-22_16-13-16 INFO ===============================
2018-10-22_16-13-18 INFO 
2018-10-22_16-13-18 INFO Log file is : /installs/logs/cell_manager_configuration-2018-10-22_16-13-16.log
2018-10-22_16-13-18 INFO 
2018-10-22_16-13-18 INFO Adding new cell member with the following configuration :
2018-10-22_16-13-18 INFO Cell member internal address 172.18.100.36
2018-10-22_16-13-18 INFO Cell member external address 172.17.100.35
2018-10-22_16-13-18 INFO Syslog agents using TCP ports starting with 60000
2018-10-22_16-13-18 INFO Wsm agents using TCP ports starting with 60020 
2018-10-22_16-13-18 INFO 
2018-10-22_16-13-18 INFO During the configuration process the system will be shut down, which means that new data will not be collected and the Web Console will be unavailable for users.
2018-10-22_16-13-18 INFO Please make sure the required network connectivity (e.g. firewall rules) is available between all cell components (manager and members) according to the documentation.
2018-10-22_16-13-18 INFO 
2018-10-22_16-13-20 INFO Please choose the IP address for the cell manager server internal address followed by [ENTER]:
2018-10-22_16-13-20 INFO 1.) 172.18.100.32
2018-10-22_16-13-20 INFO 2.) 172.17.100.31
1
2018-10-22_16-14-30 INFO Stopping application ...
2018-10-22_16-15-16 INFO Application stopped successfully. 
root@172.18.100.36's password: 
2018-10-22_16-21-41 INFO Cell member configuration ended successfully.
2018-10-22_16-21-45 INFO Stopping application ...
2018-10-22_16-22-31 INFO Application stopped successfully. 
2018-10-22_16-22-31 INFO Starting application ...

Note that the script writes two log file, one in the cell manager and one in the cell member. The log file names are mentioned in the script's output.

Example for a Failed Execution

...

themeRDark

...

    1. Virtual servers are used when the cell is required to process moderate transactions per second (TPS) load, or when the cell is part of a non-production environment where the production cell uses physical servers (to keep environments architecture similar).
  2. All DPOD cell members must be identical - only physical or only virtual (cannot mix physical and virtual cell members in the same cell), and with the same resources (CPUs, RAM, disk type and storage capacity).
  3. Physical federated cell members with 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Configuring Cell Members with 4 CPU Sockets and NVMe Disks.
  4. Each cell component (manager / FCM) should have two network interfaces:
    1. Internal network interface - dedicated for DPOD inter-communication between the cell components.
    2. External network interface - for communicating with the rest of the network. This includes users accessing the DPOD Web Console (on the cell manager), communication between DPOD and the Monitored Gateways, communication with DNS, NTP, SMTP, LDAP, and anything else on the network.
    3. This design was driven by customer requirements and allows separation between the two types of communications, which may be used to enhance the security (e.g.: deny end-users from being able to access the inter-cell communication).
    4. We recommend that all the internal network interfaces have IP addresses which belong to a single subnet (the internal subnet), and also all the external network interfaces have IP addresses which belong to a single subnet (the external subnet). Having an internal subnet that is different from the external subnet makes it easier to configure the servers without using static routing and easier to configure the network firewall rules.
    5. A diagram demonstrating this is available in Firewall Rules for DPOD Cell Environment.
  5. Network rules should be defined as detailed in Firewall Rules for DPOD Cell Environment.

Cell Manager Installation

Prerequisites

DPOD Installation

  • For Appliance Mode, follow the procedure: Appliance Installation.
    For Non-appliance Mode, follow the procedure: Non-Appliance Installation.
    During installation, when prompted to choose the data disk type (SSD / non SSD), choose the cell members disk type (should be SSD) instead of the cell manager disk type.
    During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.
  • Once installation is done, execute the following operating system performance optimization commands and reboot the server:
Code Block
languagebash
themeRDark
sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="2G"/g' /etc/init.d/MonTier-es-raw-trans-Node-1
/app/scripts/tune-os-parameters.sh
reboot


Federated Cell Member Installation

The following section describes the installation process of a single Federated Cell Member (FCM). Please repeat the procedure for every FCM installation.

Prerequisites

  • Make sure to meet the prerequisites listed at the top of this page.
  • Follow the procedure: Prepare Pre-Installed Operating System.
  • The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...).
  • Physical federated cell members with 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Configuring Cell Members with 4 CPU Sockets and NVMe Disks.

DPOD Installation

  • Physical servers should use RHEL as the operating system (and not CentOS).
  • Use Non-appliance Mode and follow the procedure: Non-Appliance Installation
    During installation, the four-letter Installation Environment Name should be identical to the one that was chosen during the Cell Manager installation.
    During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.
  • Once installation is done, execute the following operating system performance optimization command and reboot the server:
Code Block
languagebash
themeRDark
/app/scripts/tune-os-parameters.sh
reboot

After the server reboots, make sure httpd service is running and can be restarted successfully. If an error is displayed during the service restart, please see if the following information helps in resolving it: https://access.redhat.com/solutions/1180103

Code Block
languagebash
themeRDark
systemctl restart httpd

Configuring Mount Points of Cell Member

List of Mount Points

The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...). The data disks should be mounted to different mount points. The required mount points are:

  • In case the server has 6 disks: /data2, /data22, /data3, /data33, /data4, /data44
  • In case the server has 9 disks: /data2, /data22, /data222, /data3, /data33, /data333, /data4, /data44, /data444

Mapping Mount Points to Disks

Map the mount points to disks:

Mount PointsDisks
/data2, /data22 and /data222 (if exists)Disks connected to NUMA node 1
/data3, /data33 and /data333 (if exists)Disks connected to NUMA node 2
/data4, /data44 and /data444 (if exists)Disks connected to NUMA node 3
  • For all other types of federated cell members servers - you may map the mount points to any disk.

Creating Mount Points

Use LVM (Logical Volume Manager) to create the mount points. You may use the following commands as an example of how to configure a single mount point (/data2 on disk nvme0n1 in this case):

Code Block
languagebash
themeRDark
pvcreate -ff /dev/nvme0n1
vgcreate vg_data2 /dev/nvme0n1
lvcreate -l 100%FREE -n lv_data vg_data2
mkfs.xfs -f /dev/vg_data2/lv_data
echo "/dev/vg_data2/lv_data    /data2                   xfs     defaults        0 0" >> /etc/fstab
mkdir -p /data2
mount /data2

Inspecting final configuration

Execute the following command and verify mount points (this example is for 6 disks per cell member and does not include other mount points that should exist):

Code Block
languagebash
themeRDark
lsblk

Expected output:
NAME                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1               259:2    0   2.9T  0 disk 
└─vg_data2-lv_data    253:0    0   2.9T  0 lvm  /data2
nvme1n1               259:5    0   2.9T  0 disk 
└─vg_data22-lv_data   253:11   0   2.9T  0 lvm  /data22
nvme2n1               259:1    0   2.9T  0 disk 
└─vg_data3-lv_data    253:9    0   2.9T  0 lvm  /data3
nvme3n1               259:0    0   2.9T  0 disk 
└─vg_data33-lv_data   253:10   0   2.9T  0 lvm  /data33
nvme4n1               259:3    0   2.9T  0 disk 
└─vg_data44-lv_data   253:8    0   2.9T  0 lvm  /data44
nvme5n1               259:4    0   2.9T  0 disk 
└─vg_data4-lv_data    253:7    0   2.9T  0 lvm  /data4

Cell Member Federation

In order to federate and configure the cell member, run the following script in the cell manager once per cell member.

Important: The script should be executed using the OS root user, and also requires remote root access over SSH from the cell manager to the cell member.

Execute the script suitable for your environment:

  • In case of a physical federated cell members with 4 CPU sockets and NVMe disks:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i physical


  • In case of a physical federated cell member with 2 CPU sockets or SSD disks:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i physical -n true


  • In case of a virtual federated cell member:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i virtual


The script writes two log files - one in the cell manager and one in the cell member. The log file names are mentioned in the script's output.
In case of a failure, the script will try to rollback the configuration changes it made, so the problem can be fixed before rerunning it again.

Cell Member Federation Post Steps

NUMA configuration

DPOD cell member is using NUMA (Non-Uniform Memory Access) technology. The default cell member configuration binds DPOD's agent to CPU 0 and the Store's nodes to CPU 1rerunning it again.
If the server has 4 CPUs, the user should edit the service files of nodes 2 and 3 and change the bind CPU to 2 and 3 respectively.

Identifying NUMA Configuration

To identify the amount of CPUs installed on the server, use the NUMA utility:

Code Block
themeRDark
numactl -s

Example output for 4 CPU server :

policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3
Alter Store's Node 2 and 3 (OPTIONAL - only if the server has 4 CPUs)

The services files are located on the directory /etc/init.d/ with the name MonTier-es-raw-trans-Node-2 and MonTier-es-raw-trans-Node-3.

...

themeRDark

...

rollback fails, and the cell member services do not start successfully, it might be required to uninstall DPOD from the cell member, reinstall and federate it again.

Updating Configuration for Physical Federated Cell Members with 4 CPU Sockets and NVMe Disks

Note: If the cell member server does not have 4 CPU sockets or does not have NVMe disks - skip this step.

To update the service files, execute the following commands:

Code Block
languagebash
themeRDark
sed -i 's#/usr/bin/numactl --membind=1 --cpunodebind=1#/usr/bin/numactl --membind=2 --cpunodebind=2#g' /etc/init.d/MonTier-es-raw-trans-Node-3
sed -i 's#/usr/bin/numactl --membind=1 --cpunodebind=1#/usr/bin/numactl --membind=3 --cpunodebind=3#g' /etc/init.d/MonTier-es-raw-trans-Node-4

To verify the NUMA configuration for all services, execute the following command:

Code Block
languagebash
themeRDark
grep numactl /etc/init.d/*

Updating Configuration for Federated Cell Members with at least 384GB RAM

Note: If the cell member server has less than 384GB RAM - skip this step.

To update the service files, execute the following command:

Code Block
languagebash
themeRDark
sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-2
sed -i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-3
OLDsed VALUE : numa="/usr/bin/numactl --membind=1 --cpunodebind=1"
NEW VALUE : numa="/usr/bin/numactl --membind=3 --cpunodebind=3"-i 's/^NODE_HEAP_SIZE=.*/NODE_HEAP_SIZE="64G"/g' /etc/init.d/MonTier-es-raw-trans-Node-4

Restarting Services of the Federated Cell Member

Stop all the application services using app-util.sh → option 2 (stop all)

Start all the application services using app-util.sh → option 1 (start all)

Cell Member Federation Verification

After a successful executionfederation, you will be able to see the new federated cell member in the Manage → System → Nodes page.
For example, after federating cell member the page should look as follows For example:

Also, the new agents will be shown in the agents list in the Manage → Internal Health → Agents page.
For example, if the cell manager has two agents and there is a federated cell member with additional four agents, the page will show six agents:

Configure the Monitored

...

Gateways to Use the Federated Cell Member Agents

It is possible to configure entire monitored device or just a specific domain to the federated cell member's agents.

To configure monitored device / specific domain please Configure the monitored gateways to use the federated cells agents. Please follow instructions on Adding Monitored DevicesGateways.