Summary of Contents for Mellanox Technologies MCX4421A-XCQN
Page 1
ConnectX®-4 Lx Single and Dual 10/25 Gb/s Ethernet SFP28 Port Adapter Card User Manual for Open Compute Project P/N: MCX4421A-XCQN, MCX4411A-ACAN, MCX4412-ACAN, MCX4411A-ACQN, MCX4421A- ACQN Rev 2.5 Mellanox Technologies www.mellanox.com...
Page 2
THIS HARDWARE , SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS ﺴWITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.
Adapter LEDs Operations on page 64 • Added Mechanical Drawing of Single-port Cards on page July 2016 • Added MCX4421A-XCQN and updated necessary sections accordingly. • Updated Features and Benefits on page 16. • Updated PCI Express Interface on page 18.
About this Manual This User Manual describes Mellanox Technologies ConnectX®-4 Lx 10/25 Gb/s Ethernet Sin- gle and Dual SFP28 port PCI Express x8 adapter cards for Open Compute Project, spec 2.0. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express Rev 2.5 Mellanox Technologies...
Introduction Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con- nectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest perform- ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.
RoHS MT27711A0-FDCF-AE - single-port cards Adapter IC Part Number MT27712A0-FDCF-AE - dual-port cards 4117 for Physical Function (PF) Device ID (decimal) 4118 for Virtual Function (VF) a. See OCP 2.0 Spec Stacking Height on page 14 Rev 2.5 Mellanox Technologies...
Figure 1: Type 1 Vertical Stack Front View - Single Port Cards Figure 2: Type 1 Vertical Stack Front View - Dual Port Cards Operating Systems/Distributions • RHEL/CentOS • FreeBSD • VMware • OpenFabrics Enterprise Distribution (OFED) • OpenFabrics Windows Distribution (WinOF-2) Rev 2.5 Mellanox Technologies...
Introduction Connectivity • Interoperable with 1/10/25 Gb/s Ethernet switches • Passive copper cable with ESD protection • Powered connectors for optical and active cable support Rev 2.5 Mellanox Technologies...
Virtualization A consolidated compute and storage network achieves significant cost-performance advan- Storage Acceleration tages over multi-fabric networks. Standard block and file access protocols can leverage RDMA for high-performance storage access. Rev 2.5 Mellanox Technologies...
Page 17
(VM) within the server. The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be NC-SI connected to a BMC. Enabled for MCX4421A-XCQN, MCX4411A-ACQN and MCX- 4421A-ACQN. Supported in MCX4421A-XCQN, MCX4411A-ACQN and MCX4421A-ACQN.
• Auto-negotiates to x8, x4, x2, or x1 • Support for MSI/MSI-X mechanisms LEDs Interface Section 7.6, There are two I/O LEDs per port. For LEDs specifications, please refer to “Adapter LEDs Operations,” on page 64 Rev 2.5 Mellanox Technologies...
4. Remove the card from its package. 5. Please note that the card must be placed on an antistatic surface. 6. Check the card for visible signs of damage. Do not attempt to install the card if damaged. Rev 2.5 Mellanox Technologies...
(x16 in a x8). Align the adapter connector edge with the PCI Express connector slot. screws are open. 4. Place the adapter card on the clips without applying any pressure. Open Clips Open Clips Rev 2.5 Mellanox Technologies...
Determine the correct orientation of the connector to the card before inserting the connector. Do not try and insert the connector upside down. This may damage the adapter card. Rev 2.5 Mellanox Technologies...
3. Turn off the system and disconnect the power cord from the server. 3.6.2 Card Un-installation Please note that the following images are for illustration purposes only. 1. Verify that the system is powered off and unplugged. 2. Wait 30 seconds. Rev 2.5 Mellanox Technologies...
Hardware Installation 1. To remove the card, disengage clip 1 and 2 on connector. Clip 2 Connector Side Clip 1 2. To disconnect connector A, gently pull the adapter card upwards Connector Side Rev 2.5 Mellanox Technologies...
Identify the Card in Your System 3.7.1 On Windows 1. Open Device Manager on the server. Click start => Run, and then enter “devmgmt.msc”. 2. Expand System Devices and locate your Mellanox ConnectX-4 Lx adapter card. Rev 2.5 Mellanox Technologies...
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID.
Step 1. The following example shows a system with an installed Mellanox HCA: # lspci -v | grep Mellanox 86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family Subsystem: Mellanox Technologies Device 0014 Download the ISO image to your host.
The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. 1. The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option. Rev 2.5 Mellanox Technologies...
At this point, Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine. 4.1.3.1 Installation Procedure Login to the installation machine as root. Step 1. Mount the ISO image on your machine. Step 2. host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt Rev 2.5 Mellanox Technologies...
Page 29
PSID: MT_0DB0110010 Versions: Current Available 2.9.1000 Status: No matching image found Reboot the machine if the installation script performed firmware updates to your network Step 4. adapter hardware. Otherwise, restart the driver by running: "/etc/init.d/openibd restart" Rev 2.5 Mellanox Technologies...
Page 30
Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files. See the relevant chapters in this manual for details. The list of the modules that will be loaded automatically upon boot can be found in the /etc/ file. infiniband/openib.conf Rev 2.5 Mellanox Technologies...
Logs dir: /tmp/MLNX_OFED_LINUX-x.x-x.logs 4.1.3.4 openibd Script As of MLNX_OFED v2.2-1.0.0 the script supports pre/post start/stop scripts: openibd This can be controlled by setting the variables below in the /etc/infiniband/openibd.conf file. OPENIBD_PRE_START OPENIBD_POST_START OPENIBD_PRE_STOP OPENIBD_POST_STOP Example: OPENIBD_POST_START=/sbin/openibd_post_start.sh Rev 2.5 Mellanox Technologies...
Mount the ISO image on your machine and copy its content to a shared location in your net- Step 2. work. # mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>-<CPU arch>.iso /mnt You can download it from http://www.mellanox.com => Products => Software => Ether- net Drivers. Rev 2.5 Mellanox Technologies...
Page 33
# sudo rpm --import RPM-GPG-KEY-Mellanox warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) <support@mel- lanox.com>" From : /repos/MLNX_OFED/<MLNX_OFED file>/RPM-GPG-KEY-Mellanox Is this ok [y/N]: Check that the key was successfully imported.
To update the firmware to the version included in MLNX_OFED package, you can either: • Run # yum install mlnx-fw-updater • Update the firmware to the latest version available on Mellanox Technologies’ Web site Section 4.1.7, “Updating Firmware After Installation,” on page 37 as described in 4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool...
Step 3. "/etc/apt/sources.list.d/mlnx- with the following content: _ofed.list" deb file:/<path to extracted MLNX_OFED package> ./ Download and install Mellanox Technologies GPG-KEY. Step 4. # wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - Check that the key was successfully imported.
OEM card and now you wish to (manually) update firmware on your adapter card(s), you need to perform the steps below. The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies’ Web site: (http://www.mellanox.com =>...
Follow these steps below to add the Mellanox's x.509 public key to your system: Prior to adding the Mellanox's x.509 public key to your system, please make sure: • the 'mokutil' package is installed on your system • the system is booted in UEFI mode Rev 2.5 Mellanox Technologies...
After the signature has been removed, a massage as the below will no longer be presented upon module loading: "Request for unknown module key 'Mellanox Technologies signing key: 61feb074fc7292f958419386ffdd9d5ca999e403' err -11" However, please note that a similar message as the following will still be presented: "my_module: module verification failed: signature and/or required key missing - taint-...
Depending on the application of the user's system, it may be necessary to modify the default con- figuration of network adapters based on the ConnectX® adapters. In case tuning is required, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 2.5 Mellanox Technologies...
Installing the incorrect .exe file is prohibited. If you do so, an error message will be displayed. For example, if you try to install a 64-bit .exe on a 32-bit machine, the wizard will display the follow- ing (or a similar) error message: Rev 2.5 Mellanox Technologies...
Step 2. # MLNX_WinOF2-1_35_All_x64.exe /v"/l*vx [LogFile]" [Optional] If you do not want to upgrade your firmware version Step 3. # MLNX_WinOF2-1_35_All_x64.exe /v" MT_SKIPFWUPGRD=1" Click Next in the Welcome screen. Step 4. 1. MT_SKIPFWUPGRD default value is False Rev 2.5 Mellanox Technologies...
Page 43
Driver Installation Read then accept the license agreement and click Next. Step 5. Select the target folder for the installation. Step 6. Rev 2.5 Mellanox Technologies...
Page 44
If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both an OEM card and a Mellanox card, only the Mellanox card will be updated. Select a Complete or Custom installation, follow Step a and on. Step 8. Rev 2.5 Mellanox Technologies...
Page 45
• Performance tools - install the performance tools that are used to measure performance in user environment. • Documentation - contains the User Manual and Release Notes. • Management tools - installation tools used for management, such as mlxstat. b. Click Install to start the installation. Rev 2.5 Mellanox Technologies...
Page 46
Driver Installation In case firmware upgrade option was checked in Step 7.you will be notified if a firmware Step 9. upgrade is required. Click Finish to complete the installation. Step 10. Rev 2.5 Mellanox Technologies...
Upon installation completion, the inf files can be located at: • %ProgramFiles%\Mellanox\MLNX_WinOF2\Drivers\<OS> To see the Mellanox network adapters, display the Device Manager and pull down the “Net- work adapters” menu. 1. MT_NDPROPERTY default value is True 2. MT_SKIPFWUPGRD default value is False Rev 2.5 Mellanox Technologies...
Open a CMD console-> Click Start-> Task Manager-> File-> Run new task-> and enter Step 1. CMD. Extract the driver and the tools: Step 2. # MLNX_WinOF2-1_35_All_x64 /a • To extract only the driver files. # MLNX_WinOF2-1_35_All_x64 /a /vMT_DRIVERS_ONLY=1 Rev 2.5 Mellanox Technologies...
Page 49
Driver Installation Click Next to create a server image. Step 3. Click Change and specify the location in which the files are extracted to. Step 4. Rev 2.5 Mellanox Technologies...
Page 50
Driver Installation Click Install to extract this folder, or click Change to install to a different folder. Step 5. To complete the extraction, click Finish. Step 6. Rev 2.5 Mellanox Technologies...
If the machine has a standard Mellanox card with an older firmware version, the firmware will be automatically updated as part of the WinOF-2 package installation. 4.2.8 Performance Tuning For further information on WinOF-2 performance, please refer to the Performance Tuning Guide for Mellanox Network Adapters. Rev 2.5 Mellanox Technologies...
4.3.1 Hardware and Software Requirements Table 9 - Hardware and Software Requirements Requirements Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies’ HCA devices: • MT27630 ConnectX®-4 LX (EN) (firmware: fw-ConnectX4-Lx) Device ID For the latest list of device IDs, please visit Mellanox website.
• ESX 6.0 File: mft-4.3.0.25-10EM-600.0.0.2768847.x86_64.vib MD5SUM: 6a3bd63882f2ca3f696c1a96d62112df 2. Install the image according to the steps described in the MFT User Manual. The following procedure requires custom boot image downloading, mounting and boot- ing from a USB device. Rev 2.5 Mellanox Technologies...
Base MAC: 0000e41d2da206d4 Versions: Current Available 12.12.0780 12.12.1100 Status: Update required Perform FW update? [y/N]: y Device #1: Up to date Device #2: Updating FW ... Done Restart needed for updates to take effect. Log File: /var/log/mlxup/mlxup-yyyymmdd.log Rev 2.5 Mellanox Technologies...
• Verify that your switch and adapter port are compatible • Check that the latest driver is loaded Link light is on, but with no communication • Check that both the adapter and its link are set to the same speed established and duplex settings Rev 2.5 Mellanox Technologies...
(MFT) Once installed, run: mst start mst status flint –d <mst_device> q ibstat Ports Information lbv_devinfo To download the latest firmware version refer to Firmware Version http://www.mellanox.com/supportdownloader Upgrade cat /var/log/messages Collect Log File dmesg > system.log Rev 2.5 Mellanox Technologies...
EMC: CE / FCC/ VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. c. Simulated inside air tunnel 15.2 mm high and 73 mm wide. Airflow direction - SFP28 to ConnectX-4 Rev 2.5 Mellanox Technologies...
EMC: CE / FCC/ VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. c. Simulated inside air tunnel 15.2 mm high and 73 mm wide. Airflow direction - SFP28 to ConnectX-4 Lx. Rev 2.5 Mellanox Technologies...
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. c. Simulated inside air tunnel 15.2 mm high and 73 mm wide. Airflow direction - SFP28 to ConnectX-4 Lx. Rev 2.5 Mellanox Technologies...
EMC: CE / FCC / VCCI / ICES / RCM RoHS: RoHS-R6 a. Typical power for ATIS traffic load. b. For both operational and non-operational states. c. Simulated inside air tunnel 15.2 mm high and 73 mm wide. Airflow direction - SFP28 to ConnectX-4 Lx. Rev 2.5 Mellanox Technologies...
• A constant Green indicates a valid logical (data activity) link/activity link without data transfer. • A blinking Green indicates a valid logical link with data transfer. • If LED1 is off, then the logical link has not been estab- lished. Rev 2.5 Mellanox Technologies...
Specifications Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm. Figure 5: Mechanical Drawing of Single-port Cards Rev 2.5 Mellanox Technologies...
Figure 7: Rear View of Module With Pin Placement Table 16 - SFP28 Connector Pinout Symbol Name Description VeeT Transmitter Ground (Common with Receiver Ground) TX_Fault Transmitter Fault. TX_Disable Transmitter Disable. Laser output disabled on high or open. 2-wire Serial Interface Data Line Rev 2.5 Mellanox Technologies...
Page 68
MOD_ABS pulls line low to indicate module is plugged in. e. LOS is open collector output. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal. Rev 2.5 Mellanox Technologies...
PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 FCI B2B connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification. Figure 8: PCIe x8 Connector Pinout Rev 2.5 Mellanox Technologies...
MAC for the Ethernet protocol. The revision indicated on the labels in the following figures do not necessarily repre- sent the latest revision of the card. Figure 9: MCX4421A-XCQN Board Label (Example) Figure 10: MCX4411A-ACAN Board Label (Example) Rev 2.5...
6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula- tions. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. Rev 2.5 Mellanox Technologies...
Page 73
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure. CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20. Rev 2.5 Mellanox Technologies...
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 2.5 Mellanox Technologies...
Page 75
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave. PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus récentes CEI 60 825-1 Rev 2.5 Mellanox Technologies...
6. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim- mungen erfolgen. 7. Regionale und nationale elektrische Bestimmungen t Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bes- timmungen installiert werden. Rev 2.5 Mellanox Technologies...
Page 77
8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon- takt führen. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1 Rev 2.5 Mellanox Technologies...
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. Rev 2.5 Mellanox Technologies...
Page 79
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos. PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes: IEC 60825-1 Rev 2.5 Mellanox Technologies...
Need help?
Do you have a question about the MCX4421A-XCQN and is the answer not in the manual?
Questions and answers