Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...


Warning

This is a tech preview feature that currently has performance issues.
We are working to make that a fully supported feature.


When APM Integration is enabled, a single Syslog record (in JSON format) is sent for each gateway transaction to an external APM, log aggregator or any other Syslog server (e.g.: IBM APM, Splunk, ELK, etc.).

The Syslog record contains information about the transaction which is aggregated from several sources of information DPOD has.

The feature requires DataPower FW 7.6+.

Value to Customers

  • This feature allows customers to easily display gateway information on their APMs or log aggregators such as IBM APM, Splunk or ELK. This isolates the customer from changes to DataPower's log structure and saves the need to parse Syslog records.
  • Customers may link from the displayed transaction in their APM to DPOD's transaction details, in order to enhance troubleshooting efforts.
  • DPOD customers can use this feature to externalize DPOD information for data warehouse purposes.
  • DPOD customers can retain only the summarized transaction details instead of all log records. This will increase history retention time period and minimize storage requirementsGateway transactions can be displayed in the broader context of the entire transaction flow across the different components the APM is monitoring.
  • The APM gains access to knowledge only DPOD has about the gateway transactions (imagine that DPOD can tell the APM why the transaction failed in case of an error because DPOD analyzes the gateway logs).
  • Parsing and storing the JSON records which DPOD produces for each transaction takes much less storage space and parsing efforts than collecting the raw log records that the gateway is producing using log targets.
  • APM users can quickly drill down to any gateway transaction in DPOD for further analysis using deep links.
  • This feature may be used to export DPOD information for data warehousing purposes, allowing reports and dashboards in the organization's standard tools.

Transaction Record Structure

The following table describes the fields that are logged with this feature.

Field NameDescriptionPossible Values
deviceNameDataPower gateway nameString
domainNameDataPower domain name where the transaction was executedString
latencyElapsedThe elapsed time of the transaction in millisecondslong
microSecTimestampTimestamp format of the time the transaction startedString
microSecTimestampStartFor internal useString
microSecTimestampFinishFor internal useString
serviceTypeService type as defined in the gatewayString - mpgw,wsp,xml-firewall,b2bgw
serviceUriRequest URIString
serviceUrlRequest URLString
srcNodeNameThe name of the DPOD node that captured the transactionString
isErrorIndication whether the transaction completed with errorsBoolean true/false
isTechnicalErrorIndication whether the transaction completed with errorsBoolean  true/false
clientIpThe client IP of the machine (or load balancer) where the transaction started.String
serviceNameThe service the transaction ran on.String
transactionIdDataPower transaction ID (TID)String
transactionGlobalIdDataPower global transaction ID (GTID)26 chars long
timeZoneThe time zone used to log transaction startString format +ZZ:ZZ
docAddedTimeInMilFor internal uselong
timeInMilTransaction start time since Epoch in millisecondslong number
timeHHMMSSFull time of transaction start

String format HHMMSS where:

HH: 00-23
MM: 00-59
SS: 00-59

requestSizeThe request sizelong
aggRecordVersionEstimated FW version of the gateway that executed the transaction. (For internal use)String

...

Code Block
languagejava
titleJSON Example
linenumberstrue
{
  "_index": "[logical-tran-compact_i3][0]",
  "_type": "wdpLogicalTransChild",
  "_id": "ea5ae3c55b45be5500056a13_348659",
  "_timestamp": "2018-07-11T08:22:45.457Z",
  "_version": 5,
  "_operation": "INDEX",
  "_source": {
    "deviceName": "1cb3a54303a9",
    "domainName": "Infra_Domain",
    "latencyElapsed": 2,
    "microSecTimestamp": "2018-07-11T11:22:45.313729+03:00",
    "microSecTimestampStart": "2018-07-11T11:22:45.313729+03:00",
    "microSecTimestampFinish": "2018-07-11T11:22:45.315558+03:00",
    "serviceType": "xmlfirewall",
    "serviceUri": "/UpdateWantedMenProfiles_WHSW/Service.asmx",
    "serviceUrl": "http://Infra.HA:2555/UpdateWantedMenProfiles_WHSW/Service.asmx",
    "srcNodeName": "NODE0",
    "isError": false,
    "isTechnicalError": false,
    "clientIp": "172.77.77.5",
    "serviceName": "WSS_Loopback.XMLFW",
    "transactionId": "348659",
    "transactionGlobalId": "ea5ae3c55b45be5500056a13",
    "timeZone": "+03:00",
    "docAddedTimeInMil": 1531297365329,
    "timeHHMMSS": "11:22:45",
    "timeInMil": 1531297365313,
    "aggRecordVersion": "7.6.0.0+"
  }
}

Feature enablement

For each syslog agent in the system perform the following:

  1. Edit the file /app/flume/syslog_agents/conf/MonTier-SyslogAgent-nn/flume_syslog.conf
  2. Change the following property to true instead of false:
    MonTier-SyslogAgent-nn.sinks.syslogElasticSinknnn.serializer.enableLogicalTx = true

Stop and start Syslog agentsTo enable/disable this feature, please execute the following script: app_logical_tran.sh