IBM DataPower Operations Dashboard v1.0.14.0

Note: A more recent version of DPOD is available. See DPOD Documentation for the latest documentation.

What's New

Owned by Amit Munwes
Last updated: Aug 11, 2021
11 min read

The features listed below were added to DPOD. The document's icon conventions are listed at the bottom.

New Features 1.0.14.0

Improved User Experience

  • Many filters (including device, domain, service, API name, product, etc.) support wildcards (*) for matching.

  • Filters across the system support operators (including reports and alerts).

  • A new context menu in main tables (transactions, early failing, raw messages, etc.) allows easier filtering and copy.

  • Custom dashboards may be used as the home page.

  • Some widgets that could not be added to custom dashboards, such as Transactions List, are now supported.

  • CSV export size has been increased to 2,500 records instead of 1,000.

  • Any alert and report can now be duplicated, no matter if they arrive out-of-the-box or created from scratch.

API Connect Integration

  • Provider organization and consumer organization can be used in custom roles to limit user access.

  • Many filters have been added to API Connect dashboards, including provider organization, HTTP response code, HTTP method, elapsed time, request/response size, etc.

  • Added Top Consumers table in Recent Activity dashboard.

  • Status column in API Connect Transactions page was replaced by HTTP response code column.

  • Transaction error analysis now detects some more common error scenarios with APIC v10.

Deprecated Extended Transactions

  • The deprecated Extended Transaction functionality, which required instrumentation of the processing policy, and was replaced in 1.0.8.0 by a non-intrusive alternative, has been removed.

 

New Features 1.0.13.0

Improved User Experience

  • Quickly move to next and previous transactions from the transaction page.

  • Home page under User Preferences has many more options.

API Connect Integration

  •  Added support for Web Socket Upgrade policy (requires gateway firmware 10.0.2+).

Supported Platforms

  • Added support for RHEL 8.3 (non-appliance installation only).

 

New Features 1.0.12.0

Improved User Experience

  • Added filter operators, such as "Equal", "Not Equal", "Greater Than" and "Lower Than" to all dashboards.

  • Improved Time filter - add previous and next time period buttons, add new options and allow to manually edit custom time.

  • New Report: Service Request/Response Size Stats.

  • Gateways table is now sorted by name.

New Task Scheduling Mechanism

  • New task scheduling mechanism enhances the system stability and improves the response time of tasks including reports and alerts.

  • New Scheduler Info page under the Manage menu displays information about the scheduled tasks.

 

New Features 1.0.11.0

Correlated Transactions Tab

  • New tab in the transaction drill-down page, showing a list of all other transactions with the same global transaction ID.

  • The list shows both APIC and Gateway transactions.

Raw Messages over Time Dashboard

  • Show amount of raw messages over time.

  • Show total amount of raw messages grouped by message codes.

API Connect Integration

  • APIC transactions status is solely determined by their front-end response code.

Improved User Experience

  • New user preference to enable dependency between device, domain and service filters.

  • New Alert: API Error Message Count Alert.

  • New Alert: Trans. Over 30 sec by Service Name Alert.

  • New Alert: Unavailable Devices Alert.

  • New Report: Top Transactions Count per Interval.

  • New Report: Top Documents Count per Interval.

  • New Report: Top Documents Count per Agent per Interval.

  • New Report: Service Latency Report - Extended.

  • New Report: Service and URI Total Transactions Report.

Supported Platforms

  • Added support for Gateway/APIC v10

 

New Features 1.0.10.1

Important security fixes.

 

New Features 1.0.10.0

API Connect Integration

  • Support for DataPower API Gateway.

  • Extended transactions page now shows API Connect transactions (when extended transactions are enabled).

  • Reworked API Connect transaction drill-down page to show more accurate transaction flow and latency information.

LDAP Integration

  • Improved integration with LDAP to allow more flexibility and simplicity in the LDAP registry configuration.

Improved User Experience

  • New report - latency statistics grouped by gateway/domain/service.

  • Reports and Alerts - Support TLSv1.2 for SMTP with STARTTLS.

  • New gateway information panel.

  • New internal alert on misconfigured DPOD log targets.

  • Better support for IDG 2018 SOMA error messages.

  • Speed up Failed Objects sampling to make alerts more reliable.

  • API Connect API Policies dashboard is now sorted by API Name and Policy Name.

  • API Connect URI and URL calls dashboards can now be filtered by API attributes.

 

New Features 1.0.9.0

Appliance Migration Assistant

  • A step by step wizard to backup and restore gateway configuration in order to migrate from one gateway to another.

  • The migration assistant backs up a source gateway, performs validation on both source and target, and restores the backup to the target gateway.

Services Configuration Changes Analysis

  • A new dashboard showing all recent service configuration changes and their effect on latency and on the number of errors.

  • The transaction drill-down page indicates whether a recent configuration change was made to the underlying service.

Reports and Alerts Improvements and New History Dashboards

  • Reports and alerts executions are stored and can be downloaded from new history dashboards.

  • More filter types are available for API Connect reports and alerts, as reports and alerts are now bound to a product (Gateway or API Connect).

API Connect Integration Improvements 

  • Added context variable capturing to the policy variables capture.

  • Handle more cases of early failed transactions (OAuth, 403, etc).

  • Improved error analysis and payload recording.

New Admin Interface

  • Most of the system's admin functionality, such as stopping or starting services, is available through a new web console.

Improved User Experience

  • New dashboard showing stats of gateway's MQ objects, such as Active and Idle connections.

  • Added a new alert and a system health metric for MQ Connections Utilization.

  • Added a new alert for monitoring services that did not run in the last X minutes.

  • Added a new alert for API Connect transactions that ended with a specific error code or with an error.

  • Added a new Transactions Latency Statistics report.

  • Added support for Sync and Firmware upgrade for IDG 2018.x firmwares.

 

New Features 1.0.8.6

Custom Dashboards 

  • Admins can design custom dashboards containing DPOD widgets.

  • Custom dashboards are available to all users from the Dashboards menu.

API Connect Integration Improvements 

  • New option to trace and view policy variables for specific API Connect transactions (by Catalog and API Name).

  • Improved API Connect Side Calls analysis.

Support for Monitoring Transactions in the Default Domain 

  • DPOD now supports monitoring transactions in the default domain.

Improved User Experience

  • When removing a monitored device from DPOD - all DPOD's log targets can be removed from the monitored device.

 

New Features 1.0.8.5

API Connect Integration Improvements 

  • Many new dashboards in the API-C product view, including APIs activity, latency, HTTP response codes and consumers.

  • New dashboard for comparing latency between API versions.

  • New dashboard for comparing API assembly policies between APIs or versions.

  • Several filters have been added to API-C pages with auto-complete, such as catalog, space, product, plan, API name, API version and consumer application name.

  • Detection of API Connect early failed transactions improved significantly.

Automation Scripts for Creating Federated Architecture

  •  DPOD's processing can now be distributed across different federated installations in order to handle very high transaction loads (thousands of transactions per second).

DevOps Services Portal Improvements

  • New functionality to import a service into multiple devices.

  • New functionality to view and filter services by user comments.

Request and Response Size (requires firmware 7.6.0.8+ or 7.7.1.2+)

  • In the Transactions list page, results can be filtered using request and response size of the transactions.

  • Request and response sizes are displayed in the transaction details page.

  • Response size has been added to Activity Distribution dashboard.

Improved User Experience

  • Yesterday, Last Week and Last Month added to the time filter.

  • Raw messages can now be filtered by From/To B2B Partner Profiles.

  • Reports and alerts queries can have named parameters.

  • New reports: average service latency, average request size and services above specific elapsed time.

  • New alert: message code threshold.

  • Backup, Sync and Firmware upgrade plans can now be canceled before they start to execute.

Supported Platforms

  • Added support for RHEL 7.5.

  • Upgraded Java run-time and Store versions.

 

New Features 1.0.8.0

System Health

  • An overall health overview for all gateways according to user defined metrics.

  • Drill-down to view the device's health in the last 60 minutes.

Non-Intrusive Extended Transactions ( requires firmware 7.7.1.1+)

  • Extended Transactions, which are transactions that span more than one monitored devices/services, are now being tracked in a non-intrusive way over HTTP protocol, without the need for processing policy instrumentation.

  • The pre v1.0.8, intrusive Extended Transaction functionality will be deprecated in the near future.

API Connect Integration Improvements

  • Support for API Connect version 2018.2.5 compatibility mode.

  • New DPOD Roles to allow and revoke access to API Connect field such as API Name, Space and Plan names.

  • API Connect views can now be set as the default system view.

  • Service URL/URI Calls dashboards are now available in API Connect view.

Performance and History

  • User can choose between supporting more TPS but gathering less data per transaction, or collecting more data and supporting less TPS. This option is available during log target setup.

APM Integration

  • Providing a publisher to Splunk with a Splunk application as a reference implementation for the integration (requires firmware 7.7.1+).

  • Publishing more analytical data fields.

  • Simplify setup of implementing APM integration.

DevOps Services Portal Improvements

  • New functionality to stop or start a service based on custom role permissions.

Docker Support

  • Publish DPOD developer edition image in Docker Hub.

Improved User Experience

  • All dashboards can be exported to a CSV file.

  • Domain filter now support asterisk wildcard.

  • New REST API to create WS-M subscriptions.

  • Device Resources page now shows Ethernet Interfaces statistics.

  • New alert on more than 5 API Connect APIs with latency above X ms.

  • Provide retry functionality for failed publishing via SMTP or syslog of alerts, internal health and maintenance jobs.

  • Admins can use a new page to delete transactions logs that are stored in DPOD.

 

New Features 1.0.7.0

Tenant Support 

  • Added the ability to monitor transactions information on appliance with Tenants.

  • Resource monitoring and appliance maintenance are not available for tenants.

API Connect Integration

Maintenance Plans Improvements

Improved User Experience

 

New Features 1.0.6.0

DevOps Services Portal 

  • DPOD now provides a new self service DevOps portal for traditional services. This is a new portal dashboard where end-users can:

    • Execute two new actions on SOAP Web services with local WSDL: validate and promote

      • The Validate action uploads your new WSDL and schema files to a temporary location, validates its compilation and creates a temporary WS-Gateway to ensure the object is up.

      • Promote action allows uploading a new WSDL and schema files to the target location (device and domain) and creating a new version of the service.

    • Execute two new actions on SOAP Web services with remote WSDL: validate and promote

      • The Validate action updates a new or existing URL of a remote WSDL, validates its compilation, and creates a temporary WS-Gateway to ensure the object is up.

      • The Promote action updates a new or existing URL of a remote WSDL in a target location (Device and Domain), and creates a newer version of the service.

  • All actions require permissions set by a security policy (custom roles).

  • Each action may be extended or customized using Python scripts. Example scripts are open source and may be obtained from a git repository - see docs 

Early Failing Requests dashboard

  • Added a new dashboard for early failing requests (such as TLS/SSL failures) - where a request is received but is unable to reach the execution phase of a Processing Rule of a Service - (requires firmware 7.5.2.4+)

Service API Dashboard

  • DPOD now provides a new dashboard that provides statistics pertaining to service URIs. These statistics include the number of invocations and min/max/avg latency in addition to the Service API (URL) Dashboard

Improved User Experience

  • Users can now save favorite filter combinations and use them in subsequent sessions.

  • A new free text search field was added to the Transactions List page and Raw Messages Page where the user may enter a single value that will be searched across various data fields.

  • Service filter now supports auto completion.

  • All applicable filters support multi-select of values.

  • Users can now change the size of the series (number of values) that is displayed in charts of type "top N".

  • A recent notifications panel was added to The Navigation Bar. It displays the recent UI notifications for the user (not persisted).

  • A new "Share" button added. It allows sharing the current page (including filters, transaction ID etc.) by copying/pasting the current URL or by sending it via email

  • Time range filters now provides more options: last 10 minutes, start of day, start of week, this month.

  • Consistent behavior across pages of zoom into a transaction or use its value as a filter criteria.

Data Migration Procedure

  • From time to time, DPOD customers may be required to install a new DPOD installation while preserving the data already collected. for example:

    • Customer deployed DPOD Appliance mode at version v1.0.0 with CentOS 6.7 and now requires to migrate to CentOS 7.2 introduced at version v1.0.2+.

    • Customer deployed DPOD on a virtual server but load increased and now requires a physical server.

    • Customer deployed DPOD Appliance mode but wants to migrate to Non-Appliance mode (RHEL) to better comply with their organization security requirements and standards.

  • New procedure and tools were introduced to support customers with migration of an existing DPOD Store data to a new DPOD installation in each of the scenarios above.

 

New Features 1.0.5.0

API Connect Integration

DPOD now has new capabilities when coupled with API Connect v5.0.7.2+ (API-C) and DPOD

  • Display a dedicated transaction list with only API-C transactions

  • Filter by API-C fields (e.g. API name, API version, catalog, HTTP method, API urls and a dozen more fields)

  • Calculate and display latency graphs for each policy execution inside an API (including custom policies)

  • Provide both API-C and DataPower views of the same transaction, and the ability to switch between them

  • A new dashboard of APIs and the date they were last executed in the system.

  • API payloads may now be captured

  • Automatically run setup DPOD log targets for each new API-C domain created. (This requires enablement in the monitored device configuration). Further configuration based on domain name prefix is available.

 To activate API-C integration, perform (from the Web Console) a re-configuration of all your log targets in the DataPower that runs the API-C domain. You can also use the auto-setup feature to handle API-C domains created in the future.

Maintenance Plan for Syncing DataPower Appliance Configuration () - see limitations

DPOD now provides a multitude of syncing capabilities:

  • Sync a single source configuration (domains/devices from a single appliance) to multiple targets (domains/devices) accordingly. Wildcard patterns supported.

  • A script may be executed as part of the sync lifecycle - before running, on completion and before/after each single sync.

  • Periodic sync task scheduling 

  • The domain or device may be quiesced before execution

  • Configuration sync with password map alias, provided that phrase and all objects are already at target and completely identical (firmware 7.5.2.4 and above)

 To use this feature, a remote file system for temporary storage is required. Ensure to read and understand this feature's limitations, as it disables runtime while syncing

Maintenance Plan for DataPower Backup and Appliance Configuration - see limitations

DPOD now provides a multitude of backup capabilities:

  • Backup a group of devices/domains. Wildcard patterns supported.

  • Domain or device export or secure backup

  • Ability to quiesce the domain or device before execution

  • A script may be executed as part of the backup lifecycle - before running, on completion and before/after each single backup

 To use this feature, a remote file system for temporary storage is required. Ensure to read and understand this feature's limitations, as it disables runtime while performing the backup

Side Call Analysis - firmware 7.6+

  • DPOD now provides latency analysis for calls performed within a transaction using protocols such as HTTP, MQ, LDAP, ODBC.

  • DPOD now provides details of the order of execution for these side calls.

 To activate Side Call Analysis on DPOD run re-configuration of all your log targets in the DataPower that runs the API-C domain.

External APM Integration () - firmware 7.6+

  • DPOD now provides the ability to push transaction aggregated data as one JSON record to external APMs (such as Tivoli APM) or log aggregators (such as Splunk).

 To activate this feature, a series of manual configuration steps will need to be performed, and a system shutdown will be required.

Service API Dashboard

  • DPOD now provides a new dashboard that shows statistics pertaining to a service URL, such number of invocations and min/max/avg latency.

Introducing DPOD environment

  • A monitored device may now be attached to several DPOD installations.

  • Naming each DPOD installation and reflection of this.

  • Node naming and configuration editing from the UI.
     During installation / upgrade you will need to set the system variables.

Feature Improvements

  • New alert for transaction errors

  • New report: export basic monitored device service configuration

  • Improved High Availability Scenarios 

    • A limited Active / Active scenario is now supported if the same monitored device is connected to two separate and independent DPOD installations

    • New REST APIs introduced, to support Active / Standby scenario

  • New filter to search logs by B2B message id or B2B partner id

  • Improved error Analysis by providing simple human readable errors that are most common in:

    • SSL scenarios within a transaction

    • Side calls scenarios within a transaction (when the Side Call feature is enabled)

    • API Connect scenarios within a transaction (when the API-C feature is enabled)

Deprecated features in future versions

  • The Extended Transaction feature is expected to be deprecated for customers using firmware above 7.6. This feature will be replaced using a Non-Intrusive technique in upcoming releases.





The following conventions are used in this document:

 - Some actions are required to enable the feature.
 - This is a Tech Preview feature. Consult the release notes for more information.



IBM DataPower Operations Dashboard (DPOD) v1.0.14.0