Component Diagram

The diagram below depicts DPOD's components and their interactions.

(Note: DPOD was designed for deployment using either a standalone (All in one) or distributed topology. When distributed topology is used, some of the components above will reside on different virtual/physical servers.)

Big Data Store

The data flowing into DPOD through the stream processing component is stored in a Big Data store.

A new DPOD installation sets up a Big Data store that is pre-configured and optimized for DPOD's requirements. As the storage is managed automatically by DPOD’s internal components, no regular maintenance is required by users.

Configuration Database

DPOD uses an internal relational database to store its system configuration. DPOD's components read configuration properties from this database.
Configuration data is updated via the console (web interface), and is accessible by web interface and several internal components.
DPOD also stores report templates and scheduling information in this database.

Log Collection agents (syslog agents)

The log collection agents receive Syslog log records from the monitored devices' Syslog log target, which is configured by DPOD during initial system configuration. The agents then parse the log entries, store them in the Big Data store and send them to the stream analyzing component.

DPOD automatically creates a log target on each domain (including the default domain).

Each monitored device's log target can send logs to a single log collection agent. However, a single log collection agent may subscribe to Syslog targets originating from multiple domains on a single device or several domains on different devices.

Using DPOD's user interface, a DPOD system administrator may tune the links between Syslog targets and specific log collection agent (based on the Syslog target's logging rate). Doing so helps balance the network transmissions across the system and enhances throughput and log processing on DPOD's side.

During DPOD's installation process, the installer gathers environment planning and sizing request data. Based on this information, the installer creates and configures a number of log collection agents.

Sampling Agents (Device Resources and Service Resources)

DPOD’s sampling agents use DataPower's SOMA (SOAP configuration Management) interface for retrieving operational data for both devices (e.g. CPU, memory, load, file system and sensors) and services (e.g. service memory, service configuration). Sampling records are stored in the Big Data store.

The sampling interval used may be configured. The default sampling interval is 30 seconds for device data and 300 seconds for service data and can be changed.

WS-Management Agents (WS-M)

The DPOD WS-M agents, using the WS-Management agent on monitored devices, are responsible for processing and storing the monitored devices' service payload recording from WS-Proxy services,

Payload recording is switched off by default, and has to be enabled manually through DPOD's user interface. For security reasons, enabling the WS-M Agents in IDG is a manual process as well. Once this is done, a subscription is registered in the WS-M agent on the monitored device.

Once a payload is recorded by a monitored device's WS-M agent from a WS-Proxy policy, the data is pushed to DPOD’s WS-M agent, which stores it in the Big Data store.

Payload recording puts the system under substantial load and therefore has to be manually enabled and its maximum duration is limited to 60 minutes.

Stream Processing

DPOD uses the Stream Processing component to stream, parse and analyze incoming data. It collects data from the log collection and WS-M agents, and operates automatically without requiring maintenance.

User Interface (Web Console) 

DPOD's user interface (Web Console) is a web-based user interface. The Console can be accessed via HTTPS and requires user/password authentication.

The console enables the user to troubleshoot, analyze and gain insights into transaction activity on monitored devices. Privileged users may also use it to update the system configuration.

Reports

DPOD allows users to generate various reports. These reports may be run on an ad-hoc or scheduled basis. The Reports component is responsible for processing and generating these user reports.

The system is installed with a number of built-in reports (e.g. Services Elapsed Time, System Errors and Service Memory), and privileged users are able to configure new custom reports via DPOD’s Web Console.

Reports can be saved as CSV files on the DPOD appliance file system, or sent as mail attachment via SMTP and a custom web service.

Gateway Maintenance Activities

A maintenance activity defines the set of maintenance actions required for a specific goal. An example of such an activity is "Perform Secure Backup on device X".
Additionally, the maintenance activity contains other specific definitions for the action. This may include for instance which certificate should be used for the Secure Backup or which deployment policy should be used for a configuration sync operation.

Current provided activities are: backup and configuration sync.

DPOD allows users to define a plan that includes a set of target gateways on which activity will be performed and a receipt on how to perform the activities.

These Activities can be integrated into an organisation wide DevOps process by REST invocation.

Alerts

DPOD can publish alerts when certain predefined events occur, for example, when device CPU is over 80%. Alerts can be viewed and managed from the Alerts Setup page.

An alert consists of: 

Alert Query - the metadata that defines the alert parameters (for example, count all the system errors occured in the last 10 minutes in domain DMZ). There are several builtin queries. Users customized threshold is a pramater in the query.

Alert Execution - one execution of the alert query

Alert Publishing  - when an alert returns (positive) results, it will be published to interested parties via email or syslog

Internal Alerts

DPOD can send alerts on its internal services and component health  status.

Transactions Event Feeder

This component handle the creation and publishing of a single aggrgated logical transaction record.

One of the common usage is to push flat transactional data to external system or centralized data to have a flexiable and easy acces with analytics tools.

DevOps Services Portal

DPOD now provides a new self service DevOps portal for traditional services. This is a new portal dashboard where end-users can:

• Execute two new actions on SOAP Web services with local WSDL: validate and promote
  • The Validate action uploads your new WSDL and schema files to a temporary location, validates its compilation and creates a temporary WS-Gateway to ensure the object is up.
  • Promote action allows uploading a new WSDL and schema files to the target location (device and domain) and creating a new version of the service.
• Execute two new actions on SOAP Web services with remote WSDL: validate and promote
  • The Validate action updates a new or existing URL of a remote WSDL, validates its compilation, and creates a temporary WS-Gateway to ensure the object is up.
  • The Promote action updates a new or existing URL of a remote WSDL in a target location (Device and Domain), and creates a newer version of the service.

All actions require permissions set by a security policy (custom roles).

Each action may be extended or customized using Python scripts. Example scripts are open source and may be obtained from a git repository - see docs