Symptoms

• Some transactions are not showing payloads, even though payload capture via WS-M was enabled for them.

• In /logs/wsmAgents/agent-flume_MonTier-WsmAgent-X.log (where X is the number of the DPOD WSM agent you use), you will see the following message(s):

  12/07/2016 02:12:29,852- WARN o.m.a.f.h.NHttpFileServer [I/O-dispatch-3] wsmHttpSource1 -98b515b9-ed96-45dd-a999-248f6f19f1a3- Request ignored - With size 18615431 that  limit 10485760

Cause

Each WS-M event contains 4 payloads: Front-End request, Back-End request, Back-End response and Front-End response. The size of a single event equals to the size of all 4 payloads combined.
The DataPower batches several transactions (WS-M events) together before sending them to DPOD as a single request. The default setting is grouping 20 events together.

DPOD's WS-M Agent sets a size limit for incoming request. The default limit is 10MB, meaning DPOD will drop any requests that are larger than 10MB.

Using those defaults may cause DPOD to drop payload events. Consider the case where transaction payload size is 1mb (300kb back-end and front-end requests, and 200kb back-end and front-end responses). With the defaults in place, DataPower groups 20 transactions before sending them to DPOD. This will generate a total request size will of 20MB. DPOD will then drop this event (as it exceeds the 10MB limit) and will not show the payloads.

Resolution

Changing how many payloads the DataPower will batch together

Both the number of transactions DataPower groups and sends to DPOD with each batch and the maximum batch size are configurable in DPOD (and not in the DataPower).

Change the following system parameters:

• WS-M Subscription Push Max Elements  - Number of transactions DataPower groups together (the default is 20)

• WS-M Subscription Max Envelope Size - Maximum batch size that the DataPower will send to DPOD (the default is 10485760 = 10MB)

Changing the max batch size DPOD can handle 

Edit /app/flume/wsm_agents/conf/MonTier-WsmAgent-X/flume_wsm.conf (where X is the number of the WSM Agent you want to condigure).

The relevant configuration key is MonTier-WsmAgent-X.sources.wsmHttpSource1.requestSizeLimitInBytes - max batch size that DPOD will process (the default is 10485760 = 10MB)

Best Practices

It is better to lower the number of the batched payloads (wdp.wsm.push.max_elements) than to increase the batch size DPOD can accept (wdp.wsm.push.max_envelope_size and wsmHttpSource1.requestSizeLimitInBytes). Making more smaller requests allows both the DataPower and DPOD to work in parallel and requires less memory on both appliances. Specifically, setting wsmHttpSource1.requestSizeLimitInBytes to a high value may put some stress on DataPower and DPOD machines with a low amount of RAM.

In cases where specific transaction/batch of transaction is larger than 10MB, try and raise the limit only for one specific WS-M agent and select this agent when subscribing to WS-M.