Dell EMC Series Server Sizing Manual page 23

NIC Failure test
We performed the hard NIC failure test by removing one NIC cable from the
active node that was involved in active recording. After the NIC failure, writing to
the same node failed. When the network fails, the server must recognize the
failure, then it must establish a new connection. Also, when the network fails TCP
socket connections are left open and remain open on the cluster until Isilon's
OneFS forces them closed, which allows the server to continue writing.
We can force the open TCP sockets to close for a duration of less than 2 minutes
by reducing the TCP keep idle and TCP keep interval timeout to the
optimum values recommended by Isilon Engineering.
To reduce the video loss duration due to the TCP Socket Open condition, set
the persistent values in the sysctl.config file as follows to reduce the impact
duration time significantly:
isi_sysctl_cluster
net.inet.tcp.keepidle=61000
isi_sysctl_cluster
net.inet.tcp.keepintvl=5000
Refer to the KB article: 000089232 for further information about how to
configure these parameters.
Note
NIC failure impact can be overcome by using NIC aggregation in Active/Passive
Failure aggregation mode, which is explained in the next test case. Connectivity
to the nodes that are not affected by the network outage continues to be
available throughout the test scenario and no impact was observed.
NIC Failure test with NIC aggregation in Active/Passive
We did a hard NIC failure test with Active/Passive aggregation by removing the
active NIC port cable. After the network failure, writing to the same node
continued and the NIC that was passive was immediately changed to the active
NIC. The NIC failure caused no apparent loss.
Note
NIC aggregation in Active/Passive mode remedies only a network
disconnection/NIC failure that happens on the Isilon node or the corresponding
switch port where it is connected.
Node Poweroff Test
To simulate an unexpected single node hard failure, we held down the power
button until the node powered off. This causes the servers that were writing to
that node to reconnect to a new node. In our tests, the servers on the failed node
reconnected to a new node, but did not start writing again for an aggregate
(reconnect and start writing) duration of up to 52 seconds while waiting for
writing to the SMB share to be re-started.
The second issue is that the removal or addition of a node causes an interrupt to
the cluster. Therefore, video servers writing to the other nodes might experience
a short interruption. The duration of the interruption can be reduced by modifying
the OneFS environment variables.
Testing and validation
Tests conducted
23