Versions Compared

Old Version 7

changes.mady.by.user Amit Munwes

Saved on

compared with

New Version Current

changes.mady.by.user Amit Munwes

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Warning
  • This feature is considered a tech preview feature.
  • The nature of this solution is intrusive, as it requires manual modification of the customer services. The customer is responsible of modifying their services and testing them thoroughly before using the modified services.
  • Only Transactions page and Service Activity dashboard are tuned to work with the custom log records. It is not planned to add support in other pages.

...

1. Overview

To enable the customer to send custom information to the product, the customer may modify their services and add a special transform action that will send additional information to the product that will be displayed within the Web Console.

...

2. Implementing the Solution

2.1. The Need to Modify the Services

The product heavily relies on Syslog records sent by the Gateway to detect transactions and their attributes. To enable the customer to add custom information, the information must be provided as Syslog messages that are sent as part of the transaction. This can only be achieved by modifying the services, and manually adding a special transform action that will generate the required messages the product expects.

2.2. Syslog Record Format

The product is expecting a Syslog message in the following format and with specific log category and severity. See the XSLT template file for an example how to create such a Syslog message. Limitations of the Syslog message is documented under “Syslog Records Limitations”.

#^R2^A1:192.168.110.22^A2:Website User 123^A3:Loans^A4:Ping^A5:Approve^A6:0^A7:1^A8:1^A9:Backend returned HTTP 500^AA:Credit system request failed^AB:BCF345^AC:OK^AD:OK^AE:Ignored error^AF:1477802055111^AG:1477802066123^AH:100^AI:200^AJ:300^AK:400^AL:1000^AM:123^AN:456^AO:http://MNG:92/api/Approve.asmx^AP:192.168.1.85^AQ:chrome^AR:chrome-mobile^AS:111^AT:222^AU:333^AV:444^AW:555^AX:1.1^AY:2.2^AZ:3.3^B1:4.4^B2:5.5^B3:1^B4:0^B5:1^B6:0^B7:1^B8:Str1^B9:Str2^BA:Str3^BB:Str4^BC:Str5#^

2.3. XSLT File

2.3.1. Customizing the XSLT File

The customer will be provided with an XSLT template file that records all default information the product expects and creates the appropriate Syslog message. The customer may use it as an implementation reference. An additional XSLT file will be provided as a customization example. These XSLT files should be examined by the customer, modified as required and thoroughly tested before used.

...

Custom fields in the XSLT have the notation: <!-- User code goes here -->
All user code must be added only within these notations. For example, for the custom field "customServiceName":


2.3.2. List of Custom Fields in XSLT


Field

Type

Description

customConsumer

String

A custom consumer name provided by the customer, based on the payload, headers, certificate etc.

customServiceName

String

A custom service name provided by the customer, based on the payload, headers or just to have meaningful service names.

customOperationName

String

A custom operation name provided by the customer, based on the payload, headers or just to have meaningful operation names.

isCustomError

0/1

Indicates whether the transaction was considered erroneous by the customer.

customErrorMessage

String

A custom error message provided by the customer. Trim this message after trimming technicalErrorMessage if total length of Syslog is still more than 1024 3072 bytes. For future use.

customErrorCode

String

A custom error code provided by the customer. For future use.

customCompletionStatus

String

A custom completion status provided by the customer from the following list: STARTED, IN_PROGRESS, COMPLETED, ERROR, UNKNOWN, MISSING. For future use.

customCompletionReason

String

A custom completion reason in case completion status has been provided and more explanation is required. For future use.

customClientIp

String

A custom client IP provided by the customer (e.g. using X-Forwarded-For header).

customUserAgent

String

A custom user agent provided by the customer. For future use.

customNumber1-5

Number

For future use

customFloat1-5

Number

For future use

customBoolean1-5

0/1

For future use

customString1-5

String

For future use




2.3.3. Using Customized XSLT File

The customer may choose how to use the XSLT file after customizing it:

  1. Upload the customized XSLT file to all Gateways monitored by the product. The customer may define the exact location and name of the XSLT file.
  2. Copy the entire XSLT or parts of it, modify it, and use it within another XSLT they already have, as long as the generated Syslog message is in the same format the product is expecting. See an example of a Syslog message above and see the XSLT file for more information on how to create it.
  3. There are 2 main implementation strategies.
    1. If the customer has main routing services, they can modify the routing services only. This will provide them information about all the transactions of the routing services. However, latency analysis and error analysis will be limited, since the product has no information about what happened in the services that were executed by the routing services.
    2. The customer may modify all their services. This will provide much more information to the product, but requires manual modification of all services.
  4. A new transform action must be placed at the end of each relevant response rule, and each relevant error rule of a monitored service (may be implemented first on main routing services and if required on additional services later).
  5. In case of a "One Way" service with no response rule, a new transform action must be placed at the end of the request rule.
  6. If the new transform action is not located at the end of the processing rule, time measuring will be affected (see Transaction Times limitation below).

2.4. Modifying the Services

2.4.1. Summary

 

  1. There are 2 main implementation strategies.
    1. If the customer has main routing services, they can modify the routing services only. This will provide them information about all the transactions of the routing services. However, latency analysis and error analysis will be limited, since the product has no information about what happened in the services that were executed by the routing services.
    2. The customer may modify all their services. This will provide much more information to the product, but requires manual modification of all services.
  2. A new transform action must be placed at the end of each relevant response rule, and each relevant error rule of a monitored service (may be implemented first on main routing services and if required on additional services later).
  3. In case of a "One Way" service with no response rule, a new transform action must be placed at the end of the request rule.
  4. If the new transform action is not located at the end of the processing rule, time measuring will be affected (see Transaction Times limitation below).

2.4.2. Detailed Explanation


  1. Place a new transform action in the processing policy of the services you want to monitor. The transform action must be placed at the end of each relevant response rule, and each relevant error rule. In both cases, it should be placed after the result action, as the last action of the rule. The service must have both response and error rules.

  2. The transform action should use the XSLT file that was previously uploaded to the device or the customer’s own XSLT that generates the Syslog message. The Input and Output of the transform action should both be NULL. The rest of the rule needs not be changed. The output of the result action will stay the same as it is.


  3. In case of a "One Way" service, meaning that it has no response rule processing, the action must be placed at the end of the request rule.

...

In case of a Multi-Protocol Gateway or a Web-Service Proxy with the "var://service/mpgw/skip-backside" set to 1 (similar to a loopback behavior):


 

2.5. Testing the Modified Services

Once the transform action is in place, the special Syslog messages can be viewed in the "view logs" window in the Gateway while transactions are processed. The messages can be recognized by the prefix "#^R2". For example:

3. Features

The custom information that is provided by the customer is mostly used to be displayed when searching and investigating transactions. However, sometimes it is inevitable to display the technical information of the services. For example, in the Explore (service configuration) section, it only makes sense to display the name of the objects exactly as they are defined in the Gateway.

3.1. Existing Features

3.1.1. Dashboards

All existing pages will be available. However, a few charts will not be displayed and some dashboards can only display technical information about services, as described in the Solution Limitations section below.

3.1.2. Investigate

All existing pages will be available. However, there are situations where technical information about services is displayed, as described in the Solution Limitations section below.

3.1.3. Explore

All existing pages will be available. However, only technical information about the services is displayed because of the nature of this feature.

3.1.4. Reports

All out-of-the-box reports will be available. However, some reports will display technical information about the services, as described in the Solution Limitations section below.

4. Solution Limitations

4.1. Supported Services

The supported service types are:

...

Services that are not “One-Way” must have both response and error rules so the product can display both successful and failed transactions.

4.2. Features of Existing Version

4.2.1. Dashboards

  1. Service Latency: network latency charts (x4) will not be displayed.
  2. Probes in Use: technical names of the services will be displayed.
  3. Service Memory: technical names of the services will be displayed.
  4. In case the customer uses the strategy of modifying only the routing services, the back-end analysis across the product will provide information about the services invoked by the routing services, and not the real back-end systems.
  5. When searching within Payload, the search cannot be combined together with Status or Client IP filters. This is in addition to the Service filter, which is also not available in the current version of the product.
  6. Since payload recording is enabled per domain, there could be a situation where the product receives payload information about transactions that belong to services that were not modified by the customer, and thus has no custom information about these transactions. In such a case, the product will display the technical information about the transaction and will mark the transaction ID with a special asterisk. Also, when displaying such a transaction in the Transaction page, only technical information will be displayed and a proper disclaimer will be used in the title next to the transaction number.
  7. Extended Transaction feature is designed to work with the technical information of the transactions. If the customer is planning on using this feature, they would also need to customize the Extended Transaction XSLT files.

4.2.2. Investigate

  1. In case the customer uses the strategy of modifying only the routing services, the back-end analysis across the product will provide information about the services invoked by the routing services, and not the real back-end systems.
  2. When searching within Payload, the search cannot be combined together with Status or Client IP filters. This is in addition to the Service filter, which is also not available in the current version of the product.
  3. Since payload recording is enabled per domain, there could be a situation where the product receives payload information about transactions that belong to services that were not modified by the customer, and thus has no custom information about these transactions. In such a case, the product will display the technical information about the transaction and will mark the transaction ID with a special asterisk. Also, when displaying such a transaction in the Transaction page, only technical information will be displayed and a proper disclaimer will be used in the title next to the transaction number.
     
  4. Extended Transaction feature is designed to work with the technical information of the transactions. If the customer is planning on using this feature, they would also need to customize the Extended Transaction XSLT files.

4.2.3. Reports

  • Service Memory: technical names of the services will be displayed.

4.3. Syslog Records

4.3.1. Syslog Record Format

The product is expecting a Syslog message in a specific format and with specific log category and severity.

4.3.2. Syslog Record Length

Syslog records are limited to 1024 to 3072 bytes. Therefore, the total size of the custom record must not exceed this limit. Custom values and messages that are overridden by the customer must be short and descriptive.

4.3.3. Number of Syslog Records

The product expects one and only one custom Syslog record per transaction.

4.4. Transaction Times

  1. The XSLT measures time and latencies of the transaction, one of which is the total latency of the transaction. This value will be slightly less than the real value presented in the latency records, since the response is sent to the client after the XSLT is executed (although the transform action is located after the response action). This means that the latency of transmission to the client will not be included in the measured time of the XSLT. The difference should in most cases be just a few milliseconds.
  2. In case the transform action is placed before the result action (not recommended), the measured time will have another slight difference compared to the latency records generated by the Gateway. This difference is the latency of the execution of the result action, which follows the transform action. The difference should in most cases be just a few milliseconds.
  3. The execution of the XSLT template (without special customizations that might be modified by the customer) has been tested in lab conditions and added a very small overhead of a few milliseconds to the total latency of the transactions.
  4. Time analysis of a single transaction that is displayed in the console in transaction level will still be accurate and detailed with the limitation of the existing version, meaning whenever there is a latency syslog record as part of the transaction.

4.5. Request Size in XML Firewall Services

The request size of XML Firewall services cannot be obtained by the XSLT.

4.6. Future Features of the Product

Due to the nature of this solution, it totally depends on the information provided by the XSLT. As the product evolves, it might have in the future features that rely on other sources of information, which cannot be provided by an XSLT. In such a case, those features will not be enabled to customers that rely on the XSLT solution.

 

5. Customization and deployment

5.1.1. Turn On Custom Transaction Log Records

In System Parameters page, set Transactions Source to “logical”.


5.1.2. Testing Custom Transaction Log Records

In Transactions page, you should see the custom values that were provided (such as custom service name & IP address).

In Service Activity dashboard, you should see the custom values that were provided (such as custom service name) as well as a new filter - Consumer.


5.1.3. Turn Off Custom Transaction Log Records

To turn off this feature and display technical details (the default mode of the product), in System Parameters page, set Transactions Source back to “syslog”.

...