Versions Compared

Version Old Version 1 New Version Current
Changes made by

Shachar Greenberg

Amit Munwes

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Image Removed

Overview

The remote collector deployment should assist in 2 scenarios: 

  • Data should be collected across several deployments but a consolidate single view is required (only one Local nodes is required).
  • When a Local Node is reaching a CPU limit and an offload of work is required (can offload up to 20% CPU in high load).

In order to setup a new Remote Collector server you will need to install another new DPOD server based on the prerequisites below. The Node that will contain the Data and the console will be called "Local Node" and the second installation (contains only the Syslog and WS-M agent) will be called "remote collector".

Prerequisites

  1. Two DPOD installations must be with the same version (minimum version is v1.0.7 )
  2. The remote collector DPOD installations should be configured with the "medium" architecture type as detailed in the Hardware and Software Requirements
  3. Each installation will requires some different ports to be opened in the firewall - see table 1
  4. There are no requirements regarding the Environment name of each DPOD installation
  5. The two DPODs need to be able to communicate with each other and with the monitored DataPower devices

Setup steps

In order to configure the local node and remote collector(s), run the following script in the local node once per remote collector .

Code Block
configure_local_node.sh -a <IP address of the remote collector>
For example: configure_local_node.sh -a 192.168.0.5

The script will configure both the local node and remote collector.
Run this script once for each remote collector that you want to add - e.g. if you want to add two remote collectors, run the script twice (in the local node), first time with the IP address of the first remote collector, and second time with the IP of the second remote collector.

Optional parameters:

Code Block
configure_local_node.sh -a <IP address of the remote collector> -s <initial syslog agents port> -w <initial WSM agents port>
For example: configure_local_node.sh -a 192.168.0.5 -s 70000 -w 70050

The defaults are port 60000 for the initial syslog agents port and 60020 for the initial WSM agents port

Output

Example for a successful execution - note that the script writes two log file, one in the local node and one in the remote collector, the log file names are mentioned in the script's output.

Image Removed

Example for a failed execution, you will need to check the log file for further information.
in case of a failure, the script will try to rollback the configuration changes it made, so you can try to fix the problem and run it again.

Image Removed

...

Image Removed

Also, the new agents will be shown in the agents in the Manage → Internal Health → Agents page.
For example, we have one local node with two agents and two remote collectors with two agents each, the page will show six agents:

Image Removed

Configure The Monitored Device to Remote Collector's Agents

It is possible to configure entire monitored device to remote collector's agent or just a specific domain.

To configure monitored device / specific domain please follow instructions on Adding Monitored Devices

...

Warning

We recommend using the script described in the previous section.
There is no need to take any manual steps if you already run the script.

The following communication and ports are used in a remote collector deployment scenario (table 1). Perform the following commands to accomplish this task on each DPOD local firewall:

Run in Local Node -
Change the XXXX to the IP of the Remote Collector

Code Block
iptables -I INPUT -p tcp -s XXXX/24 --dport 9300:9309 -j ACCEPT
service iptables save
service iptables restart

After running the commands, run the following command and search the output for two entries showing port 9300 (shown in red in the below screenshot)

Code Block
iptables -L -n

...

From

...

To

...

Ports (Defaults)

...

Protocol

...

Usage

...

Local Node DPOD Appliance  

...

Each Monitored Device

...

5550 (TCP)

...

HTTP/S

...

Monitored Device administration management interface

...

Local Node DPOD Appliance 

...

TCP and UDP 53

...

DNS services

...

Local Node DPOD Appliance  

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

Local Node DPOD Appliance  

...

Organizational mail server

...

25 (TCP)

...

SMTP

...

Send reports by email

...

Local Node DPOD Appliance  

...

LDAP

...

TCP 389 / 636 (SSL).

TCP 3268 / 3269 (SSL)

...

LDAP

...

Authentication & authorization. Can be over SSL

...

NTP Server

...

Local Node DPOD Appliance  

...

123 (UDP)

...

NTP

...

Time synchronization

...

Each Monitored Device

...

Local Node DPOD Appliance  

...

60000-60009 (TCP)

...

TCP

...

SYSLOG Data

...

Each Monitored Device

...

Local Node DPOD Appliance  

...

60020-60029 (TCP)

...

HTTP/S

...

WS-M Payloads

...

FROM Users IPs         

...

Local Node DPOD Appliance  

...

443 (TCP)

...

HTTP/S

...

Access to with IBM DataPower Operations Dashboard Console

...

FROM Admins IPs         

...

Local Node DPOD Appliance  

...

22 (TCP)

...

TCP

...

SSH

...

Remote Collector DPOD Appliance  

...

Each Monitored Device

...

5550 (TCP)

...

HTTP/S

...

Monitored Device administration management interface

...

Remote Collector DPOD Appliance 

...

TCP and UDP 53

...

DNS services

...

Remote Collector DPOD Appliance  

...

NTP Server

...

123 (UDP)

...

NTP

...

Time synchronization

...

Remote Collector DPOD Appliance  

...

Organizational mail server

...

25 (TCP)

...

SMTP

...

Send reports by email

...

Remote Collector DPOD Appliance  

...

LDAP

...

TCP 389 / 636 (SSL).

TCP 3268 / 3269 (SSL)

...

LDAP

...

Authentication & authorization. Can be over SSL

...

NTP Server

...

Remote Collector DPOD Appliance  

...

123 (UDP)

...

NTP

...

Time synchronization

...

Each Monitored Device

...

Remote Collector DPOD Appliance  

...

60000-60009 (TCP)

...

TCP

...

SYSLOG Data

...

Each Monitored Device

...

Remote Collector DPOD Appliance  

...

60020-60029 (TCP)

...

HTTP/S

...

WS-M Payloads

...

FROM Users IPs         

...

Remote Collector DPOD Appliance  

...

443 (TCP)

...

HTTP/S

...

Access to with IBM DataPower Operations Dashboard Console

...

FROM Admins IPs         

...

Remote Collector DPOD Appliance  

...

22 (TCP)

...

TCP

...

SSH

...

In the Local Node
Using putty or any other ssh client, issue the following command: 

Code Block
sed -i -e "s/^SERVICES_SIXTH_GROUP=\".*MonTier-SyslogAgent-1 MonTier-HK-WdpServiceResources MonTier-HK-WdpDeviceResources/SERVICES_SIXTH_GROUP=\"MonTier-HK-WdpServiceResources MonTier-HK-WdpDeviceResources/g" /etc/sysconfig/MonTier

In the Local Node
Using putty or any other ssh client, issue the following command: 

Code Block
mv /etc/init.d/MonTier-SyslogAgent-1 /etc/init.d/Disabled-MonTier-SyslogAgent-1
mv /etc/init.d/MonTier-SyslogAgent-2 /etc/init.d/Disabled-MonTier-SyslogAgent-2
mv /etc/init.d/MonTier-SyslogAgent-3 /etc/init.d/Disabled-MonTier-SyslogAgent-3
mv /etc/init.d/MonTier-SyslogAgent-4 /etc/init.d/Disabled-MonTier-SyslogAgent-4
mv /etc/init.d/MonTier-SyslogAgent-5 /etc/init.d/Disabled-MonTier-SyslogAgent-5
mv /etc/init.d/MonTier-SyslogAgent-6 /etc/init.d/Disabled-MonTier-SyslogAgent-6
mv /etc/init.d/MonTier-SyslogAgent-7 /etc/init.d/Disabled-MonTier-SyslogAgent-7
mv /etc/init.d/MonTier-SyslogAgent-8 /etc/init.d/Disabled-MonTier-SyslogAgent-8
mv /etc/init.d/MonTier-SyslogAgent-9 /etc/init.d/Disabled-MonTier-SyslogAgent-9
mv /etc/init.d/MonTier-SyslogAgent-10 /etc/init.d/Disabled-MonTier-SyslogAgent-10


mv /etc/init.d/MonTier-WsmAgent-1 /etc/init.d/Disabled-MonTier-WsmAgent-1
mv /etc/init.d/MonTier-WsmAgent-2 /etc/init.d/Disabled-MonTier-WsmAgent-2
mv /etc/init.d/MonTier-WsmAgent-3 /etc/init.d/Disabled-MonTier-WsmAgent-3
mv /etc/init.d/MonTier-WsmAgent-4 /etc/init.d/Disabled-MonTier-WsmAgent-4
mv /etc/init.d/MonTier-WsmAgent-5 /etc/init.d/Disabled-MonTier-WsmAgent-5

...

In the Remote Collector
Using putty or any other ssh client, issue the following commands:

Code Block
mv /etc/init.d/MonTier-es-raw-trans-Node-1 /etc/init.d/Disabled-MonTier-es-raw-trans-Node-1
mv /etc/init.d/MonTier-es-raw-trans-Node-2 /etc/init.d/Disabled-MonTier-es-raw-trans-Node-2
mv /etc/init.d/MonTier-es-raw-trans-Node-3 /etc/init.d/Disabled-MonTier-es-raw-trans-Node-3
mv /etc/init.d/MonTier-es-raw-trans-Node-4 /etc/init.d/Disabled-MonTier-es-raw-trans-Node-4

mv /etc/init.d/MonTier-Derby /etc/init.d/Disabled-MonTier-Derby

mv /etc/init.d/MonTier-HK-ESRetention /etc/init.d/Disabled-MonTier-HK-ESRetention

mv /etc/init.d/MonTier-HK-SyslogKeepalive /etc/init.d/Disabled-MonTier-HK-SyslogKeepalive
mv /etc/init.d/MonTier-HK-WsmKeepalive /etc/init.d/Disabled-MonTier-HK-WsmKeepalive

mv /etc/init.d/MonTier-HK-WdpDeviceResources /etc/init.d/Disabled-MonTier-HK-WdpDeviceResources
mv /etc/init.d/MonTier-HK-WdpServiceResources /etc/init.d/Disabled-MonTier-HK-WdpServiceResources

mv /etc/init.d/MonTier-Reports /etc/init.d/Disabled-MonTier-Reports

mv /etc/init.d/MonTier-UI /etc/init.d/Disabled-MonTier-UI

sed -i -e "s/^SERVICES_FIRST_GROUP=\".*/SERVICES_FIRST_GROUP=\"\"/g" /etc/sysconfig/MonTier
sed -i -e "s/^SERVICES_SECOND_GROUP=\".*/SERVICES_SECOND_GROUP=\"\"/g" /etc/sysconfig/MonTier
sed -i -e "s/^SERVICES_THIRD_GROUP=\".*/SERVICES_THIRD_GROUP=\"\"/g" /etc/sysconfig/MonTier
sed -i -e "s/\MonTier-HK-WdpServiceResources MonTier-HK-WdpDeviceResources//g" /etc/sysconfig/MonTier
sed -i -e "s/^SERVICES_SEVENTH_GROUP=\".*/SERVICES_SEVENTH_GROUP=\"\"/g" /etc/sysconfig/MonTier

...

Overview

Federated architecture best fits customers that execute high load (thousands of transactions per second or more) in their gateways.

The cell environment implements the federated architecture by distributing DPOD's Store and DPOD's agents across different federated servers.

The cell environment has two main components:

  • Cell Manager - a DPOD server (usually virtual) that manages all Federated Cell Members (FCMs), as well as providing central DPOD services such as the Web Console, reports, alerts, resource monitoring, etc.
  • Federated Cell Members (FCMs) - DPOD servers (usually physical with very fast local storage) that include Store data nodes and agents (Syslog and WS-M) for collecting, parsing and storing data.

The cell environment does not replicate any data between the members, so adding more members will not provide any HA / DR capabilities.

The following diagram describes the cell environment:

Image Added

Prerequisites

  1. Before installing a cell environment, make sure to complete the sizing process with IBM Support Team to get recommendations for the hardware and architecture suitable for your requirements.
  2. DPOD cell manager and federated cell members must be of the same version.
  3. DPOD cell manager is usually virtual and can be installed in both Appliance Mode or Non-Appliance Mode with Medium Load architecture type, as detailed in the Hardware and Software Requirements.
  4. DPOD federated cell members (FCMs) can be one of the following:
    1. Physical servers installed in Non-appliance Mode (based on RHEL) with High_20dv architecture type, as detailed in the Hardware and Software Requirements.
      Physical servers are used when the cell is required to process high transactions per second (TPS) load.
    2. Virtual servers installed in Non-appliance Mode with Medium architecture type or higher, as detailed in the Hardware and Software Requirements.
      Virtual servers are used when the cell is required to process moderate transactions per second (TPS) load, or when the cell is part of a non-production environment where the production cell uses physical servers (to keep environments architecture similar).
  5. All DPOD cell members must be identical - only physical or only virtual (cannot mix physical and virtual cell members in the same cell), and with the same resources (CPUs, RAM, disk type and storage capacity).
  6. Physical federated cell members with 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Configuring Cell Members with 4 CPU Sockets and NVMe Disks.
  7. Each cell component (manager / FCM) should have two network interfaces:
    1. Internal network interface - dedicated for DPOD inter-communication between the cell components.
    2. External network interface - for communicating with the rest of the network. This includes users accessing the DPOD Web Console (on the cell manager), communication between DPOD and the Monitored Gateways, communication with DNS, NTP, SMTP, LDAP, and anything else on the network.
    3. This design was driven by customer requirements and allows separation between the two types of communications, which may be used to enhance the security (e.g.: deny end-users from being able to access the inter-cell communication).
    4. We recommend that all the internal network interfaces have IP addresses which belong to a single subnet (the internal subnet), and also all the external network interfaces have IP addresses which belong to a single subnet (the external subnet). Having an internal subnet that is different from the external subnet makes it easier to configure the servers without using static routing and easier to configure the network firewall rules.
    5. A diagram demonstrating this is available in Firewall Requirements for DPOD Cell Environment.
  8. Network rules should be defined as detailed in Firewall Requirements for DPOD Cell Environment.

Cell Manager Installation

  • Make sure to meet the prerequisites listed at the top of this page.
  • For Non-appliance Mode, follow the procedure: Prepare Pre-Installed Operating System.
  • For Non-appliance Mode, follow the procedure: Non-Appliance Installation.
    For Appliance Mode, follow the procedure: Appliance Installation.
    During installation, when prompted to choose the data disk type (SSD / non SSD), choose the cell members disk type (should be SSD) instead of the cell manager disk type.
    During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.

Federated Cell Member Installation

The following section describes the installation process of a single Federated Cell Member (FCM). Please repeat the procedure for every FCM installation.

  • Make sure to meet the prerequisites listed at the top of this page.
  • Follow the procedure: Prepare Pre-Installed Operating System.
  • Physical servers should use RHEL as the operating system (and not CentOS).
  • The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...).
  • Physical federated cell members with 4 CPU sockets and NVMe disks require special disks and mount points configuration to ensure performance. See Configuring Cell Members with 4 CPU Sockets and NVMe Disks.
  • Use Non-appliance Mode and follow the procedure: Non-Appliance Installation
    During installation, the four-letter Installation Environment Name should be identical to the one that was chosen during the Cell Manager installation.
    During installation, when prompted to choose the IP address for the Web Console, choose the IP address of the external network interface.
  • Make sure httpd service is running and can be restarted successfully. If an error is displayed during the service restart, please see if the following information helps in resolving it: https://access.redhat.com/solutions/1180103
Code Block
languagebash
themeRDark
systemctl restart httpd

Configuring Mount Points of Cell Member

List of Mount Points

The cell member server should contain disks according to the recommendations made in the sizing process with IBM Support Team, which includes disks for OS, install, and data (one for /data and 6 to 9 additional disks for /data2/3/4...). The data disks should be mounted to different mount points. The required mount points are:

  • In case the server has 6 disks: /data2, /data22, /data3, /data33, /data4, /data44
  • In case the server has 9 disks: /data2, /data22, /data222, /data3, /data33, /data333, /data4, /data44, /data444

Mapping Mount Points to Disks

Map the mount points to disks:

Mount PointsDisks
/data2, /data22 and /data222 (if exists)Disks connected to NUMA node 1
/data3, /data33 and /data333 (if exists)Disks connected to NUMA node 2
/data4, /data44 and /data444 (if exists)Disks connected to NUMA node 3
  • For all other types of federated cell members servers - you may map the mount points to any disk.

Creating Mount Points

Use LVM (Logical Volume Manager) to create the mount points. You may use the following commands as an example of how to configure a single mount point (/data2 on disk nvme0n1 in this case):

Code Block
languagebash
themeRDark
pvcreate -ff /dev/nvme0n1
vgcreate vg_data2 /dev/nvme0n1
lvcreate -l 100%FREE -n lv_data vg_data2
mkfs.xfs -f /dev/vg_data2/lv_data
echo "/dev/vg_data2/lv_data    /data2                   xfs     defaults        0 0" >> /etc/fstab
mkdir -p /data2
mount /data2

Inspecting final configuration

Execute the following command and verify mount points (this example is for 6 disks per cell member and does not include other mount points that should exist):

Code Block
languagebash
themeRDark
lsblk

Expected output:
NAME                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1               259:2    0   2.9T  0 disk 
└─vg_data2-lv_data    253:0    0   2.9T  0 lvm  /data2
nvme1n1               259:5    0   2.9T  0 disk 
└─vg_data22-lv_data   253:11   0   2.9T  0 lvm  /data22
nvme2n1               259:1    0   2.9T  0 disk 
└─vg_data3-lv_data    253:9    0   2.9T  0 lvm  /data3
nvme3n1               259:0    0   2.9T  0 disk 
└─vg_data33-lv_data   253:10   0   2.9T  0 lvm  /data33
nvme4n1               259:3    0   2.9T  0 disk 
└─vg_data44-lv_data   253:8    0   2.9T  0 lvm  /data44
nvme5n1               259:4    0   2.9T  0 disk 
└─vg_data4-lv_data    253:7    0   2.9T  0 lvm  /data4

Cell Member Federation

In order to federate and configure the cell member, run the following script in the cell manager, once per cell member.

Important: The script should be executed using the OS root user, and also requires remote root access over SSH from the cell manager to the cell member.

Execute the script suitable for your environment:

  • In case of a physical federated cell members with 4 CPU sockets and NVMe disks:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i physical


  • In case of a physical federated cell member with 2 CPU sockets or SSD disks:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i physical -n true


  • In case of a virtual federated cell member:

    Code Block
    languagebash
    themeRDark
    /app/scripts/configure_cell_manager.sh -a <internal IP address of the cell member> -g <external IP address of the cell member> -i virtual


The script writes two log files - one in the cell manager and one in the cell member. The log file names are mentioned in the script's output.
In case of a failure, the script will try to rollback the configuration changes it made, so the problem can be fixed before rerunning it again.
If the rollback fails, and the cell member services do not start successfully, it might be required to uninstall DPOD from the cell member, reinstall and federate it again.

Reboot the Federated Cell Member

Execute the following command to reboot the cell member:

Code Block
languagebash
themeRDark
reboot

Cell Member Federation Verification

After a successful federation, you will be able to see the new federated cell member in the Manage → System → Nodes page. For example:

Image Added

Also, the new agents will be shown in the agents list in the Manage → Internal Health → Agents page:

Image Added

Configure the Monitored Gateways to Use the Federated Cell Member Agents

Configure the monitored gateways to use the federated cells agents. Please follow instructions on Adding Monitored Gateways.