Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Warning

This is a tech preview feature

...

that currently has performance issues.
We are working to make that a fully supported feature.


When APM Integration is enabled, a single Syslog record (in JSON format) is sent for each gateway transaction to an external APM, log aggregator or any other Syslog server (e.g.: IBM APM, Splunk, ELK, etc.).

The Syslog record contains information about the transaction which is aggregated from several sources of information DPOD has.

The feature requires DataPower FW 7.6+.

Value to Customers

...

  • Gateway transactions can be displayed in the broader context of the entire transaction flow across the different components the APM is monitoring.
  • The APM gains access to knowledge only DPOD has about the gateway transactions (imagine that DPOD can tell the APM why the transaction failed in case of an error because DPOD analyzes the gateway logs).
  • Parsing and storing the JSON records which DPOD produces for each transaction takes much less storage space and parsing efforts than collecting the raw log records that the gateway is producing using log targets.
  • APM users can quickly drill down to any gateway transaction in DPOD for further analysis using deep links.
  • This feature may be used to export DPOD information for data warehousing purposes, allowing reports and dashboards in the organization's standard tools.

Transaction Record Structure

The following table describes the fields that are logged with this feature.

serviceTypeService type as defined in
Field NameDescriptionPossible Values
deviceNameDataPower mpgw,wsp,xml-firewall,b2bgw
transactionGlobalIdDataPower global transaction ID (GTID)26 chars long
transactionIdDataPower transaction ID (TID)long number
srcNodeNameThe name of the DPOD node that captured the transactiongateway nameString
domainNameDataPower domain name where the transaction was executedString
deviceNameDataPower device nameString
timeYearOnlyYear of transaction startformat YYYY
timeMonthNumMonth number of transaction start1-12
timeDayDay of month of transaction start1-31
timeHHMMSSFull time of transaction start

format HHMMSS where:

HH: 00-23
MM: 00-59
SS: 00-59

timeHourHour of transaction start00-23
timeMinuteMinute of transaction start0-59
timeSecondSecond of transaction start00-59
timeMicroSec

Microsecond of transaction start

timeZoneThe time zone used to log transaction startformat +ZZZZ
timeInMilTransaction start time since Epoch in millisecondslong number
timeInMicroSec

Transaction start time since Epoch in microseconds

timeDayInYearDay of year of transaction start1-365
timeSecondInDaySecond in the Day of transaction start0-86399
timeMinuteInDayMinute in the Day of transaction start0-3599
timeDayInWeekDay in a week of transaction start1-7. 1- Sunday, 7-Saturday.
microSecTimestampTimestamp format of the time the transaction startedYYYY-MM-DDTHH:mm:ss.SSSSSS+ZZZZ
aggRecordVersionEstimated FW version of the DataPower that executed the transaction. (For internal use)String
technicalServiceNameService Name.
Note: in WSP we are not currently providing an operation name
String
technicalErrorMessageError message relating to the transaction.
This field will only be populated when the transaction completed with error.
StringlatencyElapsedThe elapsed time of the transaction in millisecondslong
microSecTimestampTimestamp format of the time the transaction startedString
microSecTimestampStartFor internal useString
microSecTimestampFinishFor internal useString
serviceTypeService type as defined in the gatewayString - mpgw,wsp,xml-firewall,b2bgw
serviceUriRequest URIString
serviceUrlRequest URLString
srcNodeNameThe name of the DPOD node that captured the transactionString
isErrorIndication whether the transaction completed with errorsBoolean true/false
isTechnicalErrorIndication whether the transaction completed with errorsBoolean  true/false
aggErrorCodeError Code in DataPowerclientIpThe client IP of the machine (or load balancer) where the transaction started.String
messageserviceNameThe Syslog line that DPOD discerned is most likely to reflect the error causeString
aggIndTXErrorIndication that information on error transaction was receivedtrue/false
aggIndTXFinishedIndication that information on transaction completion was receivedtrue/false
aggIndTXStartedIndication that information on starting transaction arrivedtrue/false
aggFirstTxOfGtxIndication on whether this is the first transaction in a group. In this case -there might be a following transaction logged with the same GTIDtrue/false
microSecTimestampFinishFor internal uselong
aggUuidGtidEpochSecondsservice the transaction ran on.String
transactionIdDataPower transaction ID (TID)String
transactionGlobalIdDataPower global transaction ID (GTID)26 chars long
timeZoneThe time zone used to log transaction startString format +ZZ:ZZ
docAddedTimeInMilFor internal uselongdocAddedTimeInMil
For internal usetimeInMilTransaction start time since Epoch in millisecondslong number
WDPTutXUuidGtidTidFor internal uselong
WDPTutXUuidGtidDeviceIdFor internal useString
aggUuidGtidTimstFor internal useString
microSecTimestampStart For internal usetimeHHMMSSFull time of transaction start

String format HHMMSS where:

HH: 00-23
MM: 00-59
SS: 00-59

requestSizeThe request sizelong
aggRecordVersionEstimated FW version of the gateway that executed the transaction. (For internal use)String


Code Block
languagejava
titleJSON Example
linenumberstrue
{

  "_index" : "[logical-tran-compact_i1i3][0]",
   "_type" : "wdpLogicalTranswdpLogicalTransChild",

  "_id" : "802d48ad5976a98f00080cc4ea5ae3c55b45be5500056a13_527556348659",

  "_timestamp" : "20172018-07-25T0211T08:1522:2345.279Z457Z",

  "_version" : 45,

  "_operation" : "INDEX",

  "_source" : {
      "serviceTypedeviceName" : "mpgw1cb3a54303a9",
 
    "timeDayInYeardomainName" : "206Infra_Domain",
 
    "transactionGlobalIdlatencyElapsed" : "802d48ad5976a98f00080cc4",
      "timeDayInWeek" : "2",
 
    "microSecTimestampStartmicroSecTimestamp" : "20172018-07-25T0511T11:1422:3945.570000+0000",
      "timeHHMMSS" : "05:14:39",
 313729+03:00",
    "aggUuidGtidTimstmicroSecTimestampStart" : "20172018-07-25 0211T11:14:39",
      "aggFirstTxOfGtx" : "true22:45.313729+03:00",
      "aggIndTXStartedmicroSecTimestampFinish" : "true",
      "deviceName" : "idg7600",
      "timeSecond" : "39",
      "aggUuidGtidEpochSeconds" : "1500948879",
      "timeSecondInDay" : "18879",
      "timeMinuteInDay" : "314",
      "timeYearOnly" : "2017",
      "timeInMicroSec" : "1500948879569000",
      "srcNodeName" : "NODE0",
      "timeDay" : "25",
      "microSecTimestamp" : "2017-07-25T05:14:39.569000+0000",
      "WDPTutXUuidGtidDeviceId" : "802d48ad",
 : "2018-07-11T11:22:45.315558+03:00",
    "timeInMilserviceType" : "1500948879569xmlfirewall",
 
    "timeZoneserviceUri" : "+0000/UpdateWantedMenProfiles_WHSW/Service.asmx",
 
    "transactionIdserviceUrl" : "527556",
      "timeMinute" : "14",
      "timeMonthNum" : "07",
      "domainName" : "APIMgmt_ACB198F9A6",
 http://Infra.HA:2555/UpdateWantedMenProfiles_WHSW/Service.asmx",
    "timeMicroSecsrcNodeName" : "569000NODE0",

     "timeHourisError" : "05"false,
 
    "WDPTutXUuidGtidTidisTechnicalError" : "00080cc4"false,
 
    "aggRecordVersionclientIp" : "7172.677.077.0+5",
 
    "technicalServiceNameserviceName" : "GetDeliveryStatusWSS_MHJVLoopback.MPGWXMLFW",
 
    "docAddedTimeInMiltransactionId" : "1500948882966348659",
 
    "technicalErrorMessagetransactionGlobalId" : "Invalid JSON formatea5ae3c55b45be5500056a13",
 
    "isTechnicalErrortimeZone" : "1+03:00",
 
    "aggErrorCodedocAddedTimeInMil" : "0x02130008"1531297365329,
 
    "messagetimeHHMMSS" : "<11>2017-07-25T05:14:39.570000+0000 MonTierLocalId-8 [0x02130008][mpgw][error] mpgw(GetDeliveryStatus_MHJV.MPGW): trans(527556)[error][192.168.0.112] gtid(802d48ad5976a98f00080cc4): Invalid JSON format\r",
 11:22:45",
    "aggIndTXErrortimeInMil" : "true"1531297365313,
 
    "microSecTimestampFinishaggRecordVersion" : "2017-07-25T05:14:39.573000+0000",
      "aggIndTXFinished" : "true"
   7.6.0.0+"
  }
}

Feature enablement

Perform the following steps to enable this feature

  1. Install and configure Store plugin.
  2. Configure each syslog agent.
  3. Deploy Event Publisher
  4. Stop and start all system services.

Plugin Configuration

...

Configure each Store data node (for example: number 2 and 4) as follow:

  1. cd /app/elasticsearch_nodes/config/MonTier-es-raw-trans-Node-2 
  2. Edit the elasticsearch.yml file

Uncomment the following parameter:

...

Configure Syslog agent

For each syslog agent in the system perform the following:

...

To enable/disable this feature, please execute the following script: app_logical_tran.sh