Infiniband Fabric; Infiniband Switches; Default Bonded Interfaces - Oracle Exalogic Elastic Cloud Owner's Manual

6.6.1 InfiniBand Fabric

Exalogic machines use a unified 40 Gb per second InfiniBand quad data rate (QDR)
fabric for internal communication.
Applications running on compute nodes communicate with applications on other
compute nodes using this InfiniBand network. Exalogic machines communicate with
Oracle Exadata Database Machines for database connectivity via IPoIB. Exalogic
machines can be connected to an external network, including a standard database
hosted on a machine outside of the Exalogic machine, via the InfiniBand-to-10 Gb
Ethernet gateways using Ethernet over InfiniBand (EoIB). Each Exalogic machine
configuration includes at least 2 such gateways, which also act as InfiniBand switches
connecting all compute nodes and the storage appliance within the Exalogic machine.

6.6.2 InfiniBand Switches

Sun Network QDR InfiniBand Gateway Switches (part number NM2-GW) are used as
the leaf switches in the Exalogic machine. They connect to the Host Channel Adapters
(HCAs) of Exalogic compute nodes.
These switches (NM2-GW) also act as Ethernet gateways to connect your Exalogic
machine to the external LAN over Ethernet. For more information, see
Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand
Gateway
Sun Datacenter InfiniBand Switch 36 (part number NM2-36P) is used in multirack
configuration scenarios (an Exalogic machine to another Exalogic machine, and an
Exalogic machine to an Oracle Exadata Database Machine) only. This switch is not
connected or used in an Exalogic machine single rack.

6.6.3 Default Bonded Interfaces

After the Sun Network QDR InfiniBand Gateway Switches are connected to Exalogic
compute nodes, the following bonded interfaces are configured:
• IP over InfiniBand (IPoIB) - bond0 link (ib0/ib1 for Oracle Linux, and ibp0/
ibp1 for Oracle Solaris)
ib0 or ibp0 represents the HCA port 0 of compute nodes, and ib1 or ibp1
represents the HCA port 1 of compute nodes.
Switch.
Note:
In the Exalogic machine, InfiniBand switches (both leaf and spine switches)
are automatically configured to separate the IP over InfiniBand (IPoIB) traffic
and the Ethernet over InfiniBand (EoIB) traffic.
Note:
Depending on your application deployment and isolation requirements, you
can create additional bonded IP subnet interfaces over this default IPoIB link.
For more information, see the "Application Isolation by Subnetting over IPoIB"
topic in the Oracle Exalogic Enterprise Deployment Guide.
Introduction to Oracle Exalogic Network Configuration
Understand Network Requirements and Configuration 6-11
Connectivity