This is a tech preview feature introduced in v1.0.5. When used, a syslog record will be sent for each DataPower transaction.
The syslog record used is a json-formatted data object, containing information aggregated from several sources relating to the transaction.
The feature requires DataPower FW 7.6+
Value to Customers
- This feature allows customers to easily display DataPower information on their APMs or log aggregator such as IBM APMs , Splunk or ELK. This removes the dependency on, and isolates the customer from changes to DataPower's log structure.
- Customers may add a link to the syslog record, which redirects from each transaction into DPOD' transaction details, in order to enhance troubleshooting efforts.
- DPOD customers can use this feature to externalize DPOD information for Data Warehouse purposes.
- DPOD customers can retain summarized transaction details. This will increase history information retention time period and minimize storage requirements.
Transaction Record Structure
The following table describes the fields that are logged with this feature.
| Field Name | Description | Possible Values |
|---|---|---|
| serviceType | Service type as defined in DataPower | mpgw,wsp,xml-firewall,b2bgw |
| transactionGlobalId | DataPower global transaction ID (GTID) | 26 chars long |
| transactionId | DataPower transaction ID (TID) | long number |
| srcNodeName | The name of the DPOD node that captured the transaction | |
| domainName | DataPower domain name where the transaction was executed | String |
| deviceName | DataPower device name | String |
| timeYearOnly | Year of transaction start | format YYYY |
| timeMonthNum | Month number of transaction start | 1-12 |
| timeDay | Day of month of transaction start | 1-31 |
| timeHHMMSS | Full time of transaction start | format HHMMSS where: HH: 00-23 |
| timeHour | Hour of transaction start | 00-23 |
| timeMinute | Minute of transaction start | 0-59 |
| timeSecond | Second of transaction start | 00-59 |
| timeMicroSec | Microsecond of transaction start | |
| timeZone | The time zone used to log transaction start | format +ZZZZ |
| timeInMil | Transaction start time since Epoch in milliseconds | long number |
| timeInMicroSec | Transaction start time since Epoch in microseconds | |
| timeDayInYear | Day of year of transaction start | 1-365 |
| timeSecondInDay | Second in the Day of transaction start | 0-86399 |
| timeMinuteInDay | Minute in the Day of transaction start | 0-3599 |
| timeDayInWeek | Day in a week of transaction start | 1-7. 1- Sunday, 7-Saturday. |
| microSecTimestamp | Timestamp format of the time the transaction started | YYYY-MM-DDTHH:mm:ss.SSSSSS+ZZZZ |
| aggRecordVersion | Estimated FW version of the DataPower that executed the transaction. (For internal use) | String |
| technicalServiceName | Service Name. Note: in WSP we are not currently providing an operation name | String |
| technicalErrorMessage | Error message relating to the transaction. This field will only be populated when the transaction completed with error. | String |
| isTechnicalError | Indication whether the transaction completed with errors | true/false |
| aggErrorCode | Error Code in DataPower | String |
| message | The Syslog line that DPOD discerned is most likely to reflect the error cause | String |
| aggIndTXError | Indication that information on error transaction was received | true/false |
| aggIndTXFinished | Indication that information on transaction completion was received | true/false |
| aggIndTXStarted | Indication that information on starting transaction arrived | true/false |
| aggFirstTxOfGtx | Indication on whether this is the first transaction in a group. In this case -there might be a following transaction logged with the same GTID | true/false |
| microSecTimestampFinish | For internal use | long |
| aggUuidGtidEpochSeconds | For internal use | long |
| docAddedTimeInMil | For internal use | long |
| WDPTutXUuidGtidTid | For internal use | long |
| WDPTutXUuidGtidDeviceId | For internal use | String |
| aggUuidGtidTimst | For internal use | String |
| microSecTimestampStart | For internal use | String |
JSON Example
{
"_index" : "logical-tran-compact_i1",
"_type" : "wdpLogicalTrans",
"_id" : "802d48ad5976a98f00080cc4_527556",
"_timestamp" : "2017-07-25T02:15:23.279Z",
"_version" : 4,
"_operation" : "INDEX",
"_source" : {
"serviceType" : "mpgw",
"timeDayInYear" : "206",
"transactionGlobalId" : "802d48ad5976a98f00080cc4",
"timeDayInWeek" : "2",
"microSecTimestampStart" : "2017-07-25T05:14:39.570000+0000",
"timeHHMMSS" : "05:14:39",
"aggUuidGtidTimst" : "2017-07-25 02:14:39",
"aggFirstTxOfGtx" : "true",
"aggIndTXStarted" : "true",
"deviceName" : "idg7600",
"timeSecond" : "39",
"aggUuidGtidEpochSeconds" : "1500948879",
"timeSecondInDay" : "18879",
"timeMinuteInDay" : "314",
"timeYearOnly" : "2017",
"timeInMicroSec" : "1500948879569000",
"srcNodeName" : "NODE0",
"timeDay" : "25",
"microSecTimestamp" : "2017-07-25T05:14:39.569000+0000",
"WDPTutXUuidGtidDeviceId" : "802d48ad",
"timeInMil" : "1500948879569",
"timeZone" : "+0000",
"transactionId" : "527556",
"timeMinute" : "14",
"timeMonthNum" : "07",
"domainName" : "APIMgmt_ACB198F9A6",
"timeMicroSec" : "569000",
"timeHour" : "05",
"WDPTutXUuidGtidTid" : "00080cc4",
"aggRecordVersion" : "7.6.0.0+",
"technicalServiceName" : "GetDeliveryStatus_MHJV.MPGW",
"docAddedTimeInMil" : "1500948882966",
"technicalErrorMessage" : "Invalid JSON format",
"isTechnicalError" : "1",
"aggErrorCode" : "0x02130008",
"message" : "<11>2017-07-25T05:14:39.570000+0000 MonTierLocalId-8 [0x02130008][mpgw][error] mpgw(GetDeliveryStatus_MHJV.MPGW): trans(527556)[error][192.168.0.112] gtid(802d48ad5976a98f00080cc4): Invalid JSON format\r",
"aggIndTXError" : "true",
"microSecTimestampFinish" : "2017-07-25T05:14:39.573000+0000",
"aggIndTXFinished" : "true"
}
}
Feature enablement
Perform the following steps to enable this feature
- Install and configure Store plugin.
- Configure each syslog agent.
- Deploy Event Publisher
- Stop and start all system services.
Plugin Configuration
Configure each Store data node (for example: number 2 and 4) as follow:
- cd /app/elasticsearch_nodes/config/MonTier-es-raw-trans-Node-2
- Edit the elasticsearch.yml file
Uncomment the following parameter:
Parameter Values Description events.publisher.plugin.disable false or true enable or disable the plugin
Configure Syslog agent
For each syslog agent in the system perform the following:
- cd /app/flume/syslog_agents/conf/MonTier-SyslogAgent-nn/ (replace nn with the agent number)
- edit the flume_syslog.conf file
- Locate in each agent rows with pattern: MonTier-SyslogAgent-nn.sinks.syslogElasticSinknnn.batchSize = (replace nnn with sink name 1 to 10. Also - replace nn with the agent number)
- Add below each of these rows the line below (substituting nn and nnn as described in step 3 above)
MonTier-SyslogAgent-nn.sinks.syslogElasticSinknnn.serializer.enableLogicalTx = false