Symptoms
- In Devices page, under "Log Targets Stats", the number of "Events Dropped" is increasing over time.
- For example, "Events Dropped" shows 42009, "Events Dropped - 10m" shows 32384 and "Events Dropped - 1h" shows 2361.
- Some transactions may not be captured by DPOD, or some transactions are captured with partial information (e.g. they stay in "INPROCESS" status).
Cause
The "Log Targets Stats" table provides the status of the Gateway's Log Targets, as reported by DataPower. This means that the Gateway's Log Targets are unable to send Syslog records to DPOD.
This might be caused by:
- There might be a network issue preventing the Gateway from connecting to DPOD or sending Syslog records to DPOD.
This includes firewalls, routing configuration on the Gateway, network outage, bandwidth utilization etc. - DPOD Syslog agents might be down (constantly or temporarily, for example if DPOD agents were restarted or DPOD version has been upgraded).
- DPOD Syslog agents are malfunctioning due to lack of resources (especially relevant for virtual environments which are overloaded with CPU or memory).
Resolution
- Make sure DPOD Syslog agents are accessible from the Gateway that is experiencing the dropped events.
You can use the troubleshooting tools on the Gateway Web UI to try to connect to DPOD Syslog agents. If the tools fail to connect, that needs to be fixed by the network administrator. - Restart the Syslog agents using the Admin Console, and see if the problem still exists (you need to wait at least 2 hours to check the Devices page again).
- Inspect the logs of DPOD Syslog agents and search for any errors that might explain the issue. You might see CPU starvation errors or long GC error messages which indicate there is a lack of resources for DPOD machine.