Overview
When an Exinda device is under an extreme amount of load, the system processes rise in their RAM use and can start using the 'swap' space - hard drive space that the RAM has access to, in order to put data that will not fit in RAM space. Taking and putting information into swap from / to the RAM takes a lot longer than it does to access information directly from RAM, but if the RAM capacity is full, the information requires a place to stay. When the Exinda is using a significant amount of swap, it is possible that network slowness will occur at the same time.
Cause
This can be caused by a Denial of Service attack that affects the network and that causes the number of new connections per second on the Exinda to go higher than the specifications. In order to verify if this is correct, please have the software specs of your Exinda available (Please ask support for the specs in case you do not have them). Go to Monitor-->System-->Connections. Pay close attention to both graphs and try to look for sudden spikes in the number of connections, specially for the new ones (bottom graph). See if these numbers have exceeded or are too close to the hardware specs.
In order to investigate what could be the cause, go to Monitor-->Service Levels-->TCP Health. In this report you will see three counters:
Aborted - Connections were established, but were closed by a RST (reset) issued by either the client or server rather than a clean close. High numbers of aborted connections can point to network or server problems.
Refused - A SYN packet was observed and a RST or ICMP "connection refused" message was received in response. This usually means the server is up, but the application is unavailable or not working correctly. It can also indicate a TCP port scan is occurring.
Ignored - A SYN packet was observed, but no SYN-ACK response was received. This usually means the server is not responding, does not exist, is not accessible, or is ignoring the connection request. It can also indicate a TCP port scan is occurring.
When a DoS attack occurs, usually the number of Refused and Ignored connections suddenly spike up. If this is the case (a sudden spike for the day of the occurrence) browse through the drop down menu options to check what Internal Hosts, External Hosts and Applications are involved. See if one application or Internal Host in specific has substantially more ignored/refused connections than the rest, this usually tells you what host/application are the target. The External Hosts can also show the spike, but maybe the numbers are spread across equally through different hosts, meaning that the attack is coming from different sources?
Resolution
Given that the Exinda is not a security device, the only way of avoiding this is through the use of a firewall connected after the Exinda. The purpose of the Exinda is to monitor all traffic going through it, and as such, allocates resources to each connection. This is mitigated in v7.4.2 firmware, where resources will not be allocated until a full three way handshake between the client and server has been made.
Additional Information
After the attach or flood of such connections, the memory utilization might still remain high. Process collectord does not release the memory it acquires during the flood. To release memory from collectord run the following commands
en
conf t
service collector restart
Priyanka Bhotika
Comments