| Warning |
|---|
This is a tech preview feature |
...
that currently has performance issues. |
When APM Integration is enabled, a single Syslog record (in JSON format) is sent for each gateway transaction to an external APM, log aggregator or any other Syslog server (e.g.: IBM APM, Splunk, ELK, etc.).
The Syslog record contains information about the transaction which is aggregated from several sources of information DPOD has.
The feature requires DataPower FW 7.6+.
Value to Customers
...
- Gateway transactions can be displayed in the broader context of the entire transaction flow across the different components the APM is monitoring.
- The APM gains access to knowledge only DPOD has about the gateway transactions (imagine that DPOD can tell the APM why the transaction failed in case of an error because DPOD analyzes the gateway logs).
- Parsing and storing the JSON records which DPOD produces for each transaction takes much less storage space and parsing efforts than collecting the raw log records that the gateway is producing using log targets.
- APM users can quickly drill down to any gateway transaction in DPOD for further analysis using deep links.
- This feature may be used to export DPOD information for data warehousing purposes, allowing reports and dashboards in the organization's standard tools.
Transaction Record Structure
The following table describes the fields that are logged with this feature.
| Field Name | Description | Possible Values | ||
|---|---|---|---|---|
| deviceName | Service type as defined inDataPower | mpgw,wsp,xml-firewall,b2bgw | ||
| transactionGlobalId | DataPower global transaction ID (GTID) | 26 chars long | ||
| transactionId | DataPower transaction ID (TID) | long number | ||
| srcNodeName | The name of the DPOD node that captured the transaction | gateway name | String | |
| domainName | DataPower domain name where the transaction was executed | String | ||
| deviceName | DataPower device name | String | ||
| timeYearOnly | Year of transaction start | format YYYY | ||
| timeMonthNum | Month number of transaction start | 1-12 | ||
| timeDay | Day of month of transaction start | 1-31 | ||
| timeHHMMSS | Full time of transaction start | format HHMMSS where: HH: 00-23 | ||
| timeHour | Hour of transaction start | 00-23 | ||
| timeMinute | Minute of transaction start | 0-59 | ||
| timeSecond | Second of transaction start | 00-59 | timeMicroSec | Microsecond of transaction start |
| timeZone | The time zone used to log transaction start | format +ZZZZ | ||
| timeInMil | Transaction start time since Epoch in milliseconds | long number | timeInMicroSec | Transaction start time since Epoch in microseconds |
| timeDayInYear | Day of year of transaction start | 1-365 | ||
| timeSecondInDay | Second in the Day of transaction start | 0-86399 | ||
| timeMinuteInDay | Minute in the Day of transaction start | 0-3599 | ||
| timeDayInWeek | Day in a week of transaction start | 1-7. 1- Sunday, 7-Saturday. | ||
| microSecTimestamp | Timestamp format of the time the transaction started | YYYY-MM-DDTHH:mm:ss.SSSSSS+ZZZZ | ||
| aggRecordVersion | Estimated FW version of the DataPower that executed the transaction. (For internal use) | String | ||
| technicalServiceName | Service Name. Note: in WSP we are not currently providing an operation name | String | ||
| technicalErrorMessage | Error message relating to the transaction. This field will only be populated when the transaction completed with error. | StringlatencyElapsed | The elapsed time of the transaction in milliseconds | long |
| microSecTimestamp | Timestamp format of the time the transaction started | String | ||
| microSecTimestampStart | For internal use | String | ||
| microSecTimestampFinish | For internal use | String | ||
| serviceType | Service type as defined in the gateway | String - mpgw,wsp,xml-firewall,b2bgw | ||
| serviceUri | Request URI | String | ||
| serviceUrl | Request URL | String | ||
| srcNodeName | The name of the DPOD node that captured the transaction | String | ||
| isError | Indication whether the transaction completed with errors | Boolean true/false | ||
| isTechnicalError | Indication whether the transaction completed with errors | Boolean true/false | ||
| aggErrorCode | Error Code in DataPowerclientIp | The client IP of the machine (or load balancer) where the transaction started. | String | |
| messageserviceName | The Syslog line that DPOD discerned is most likely to reflect the error cause | String | ||
| aggIndTXError | Indication that information on error transaction was received | true/false | ||
| aggIndTXFinished | Indication that information on transaction completion was received | true/false | ||
| aggIndTXStarted | Indication that information on starting transaction arrived | true/false | ||
| aggFirstTxOfGtx | Indication on whether this is the first transaction in a group. In this case -there might be a following transaction logged with the same GTID | true/false | ||
| microSecTimestampFinish | For internal use | long | ||
| aggUuidGtidEpochSecondsservice the transaction ran on. | String | |||
| transactionId | DataPower transaction ID (TID) | String | ||
| transactionGlobalId | DataPower global transaction ID (GTID) | 26 chars long | ||
| timeZone | The time zone used to log transaction start | String format +ZZ:ZZ | ||
| docAddedTimeInMil | For internal use | long | docAddedTimeInMil | |
| For internal usetimeInMil | Transaction start time since Epoch in milliseconds | long number | ||
| WDPTutXUuidGtidTid | For internal use | long | ||
| WDPTutXUuidGtidDeviceId | For internal use | String | ||
| aggUuidGtidTimst | For internal use | String | ||
| microSecTimestampStart | For internal usetimeHHMMSS | Full time of transaction start | String format HHMMSS where: HH: 00-23 | |
| requestSize | The request size | long | ||
| aggRecordVersion | Estimated FW version of the gateway that executed the transaction. (For internal use) | String |
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
"_index" : "[logical-tran-compact_i1i3][0]",
"_type" : "wdpLogicalTranswdpLogicalTransChild",
"_id" : "802d48ad5976a98f00080cc4ea5ae3c55b45be5500056a13_527556348659",
"_timestamp" : "20172018-07-25T0211T08:1522:2345.279Z457Z",
"_version" : 45,
"_operation" : "INDEX",
"_source" : {
"serviceTypedeviceName" : "mpgw1cb3a54303a9",
"timeDayInYeardomainName" : "206Infra_Domain",
"transactionGlobalId" : "802d48ad5976a98f00080cc4",
"timeDayInWeek" latencyElapsed": "2",
"microSecTimestampStartmicroSecTimestamp" : "20172018-07-25T0511T11:1422:3945.570000+0000",
"timeHHMMSS" : "05:14:39",
313729+03:00",
"aggUuidGtidTimstmicroSecTimestampStart" : "20172018-07-25 0211T11:14:39",
"aggFirstTxOfGtx" : "true22:45.313729+03:00",
"aggIndTXStartedmicroSecTimestampFinish" : "true",
"deviceName" : "idg7600",
"timeSecond" : "39",
"aggUuidGtidEpochSeconds" : "1500948879",
"timeSecondInDay" : "18879",
"timeMinuteInDay" : "314",
"timeYearOnly" : "2017",
"timeInMicroSec" : "1500948879569000",
"srcNodeName" : "NODE0",
"timeDay" : "25",
"microSecTimestamp" : "2017-07-25T05:14:39.569000+0000",
"WDPTutXUuidGtidDeviceId" : "802d48ad",
: "2018-07-11T11:22:45.315558+03:00",
"timeInMilserviceType" : "1500948879569xmlfirewall",
"timeZoneserviceUri" : "+0000/UpdateWantedMenProfiles_WHSW/Service.asmx",
"transactionIdserviceUrl" : "527556",
"timeMinute" : "14",
"timeMonthNum" : "07",
"domainName" : "APIMgmt_ACB198F9A6",
http://Infra.HA:2555/UpdateWantedMenProfiles_WHSW/Service.asmx",
"timeMicroSecsrcNodeName" : "569000NODE0",
"timeHourisError" : "05"false,
"WDPTutXUuidGtidTidisTechnicalError" : "00080cc4"false,
"aggRecordVersionclientIp" : "7172.677.077.0+5",
"technicalServiceNameserviceName" : "GetDeliveryStatusWSS_MHJVLoopback.MPGWXMLFW",
"docAddedTimeInMiltransactionId" : "1500948882966348659",
"technicalErrorMessagetransactionGlobalId" : "Invalid JSON formatea5ae3c55b45be5500056a13",
"isTechnicalErrortimeZone" : "1+03:00",
"aggErrorCodedocAddedTimeInMil" : "0x02130008"1531297365329,
"messagetimeHHMMSS" : "<11>2017-07-25T05:14:39.570000+0000 MonTierLocalId-8 [0x02130008][mpgw][error] mpgw(GetDeliveryStatus_MHJV.MPGW): trans(527556)[error][192.168.0.112] gtid(802d48ad5976a98f00080cc4): Invalid JSON format\r",
11:22:45",
"aggIndTXErrortimeInMil" : "true"1531297365313,
"microSecTimestampFinishaggRecordVersion" : "2017-07-25T05:14:39.573000+0000",
"aggIndTXFinished" : "true"
7.6.0.0+"
}
} |
Feature enablement
Perform the following steps to enable this feature
- Install and configure Store plugin.
- Configure each syslog agent.
- Deploy Event Publisher
- Stop and start all system services.
Plugin Configuration
...
Configure each Store data node (for example: number 2 and 4) as follow:
- cd /app/elasticsearch_nodes/config/MonTier-es-raw-trans-Node-2
- Edit the elasticsearch.yml file
uncomment the following parameters:
...
Configure Syslog agent
For each syslog agent in the system perform the following:
...
To enable/disable this feature, please execute the following script: app_logical_tran.sh