Fujitsu PRIMERGY BX900 InfiniBand User Manual

Connection blade and mezzanine card v4.0 (linux, windows)

Hide thumbs Also See for PRIMERGY BX900 InfiniBand:

User manual (811 pages)

System message manual (430 pages)

Hardware reference manual (69 pages)

page of 94

/ 94
Bookmarks

Quick Links

Download this manual

User Manual - English

PRIMERGY BX900 / BX400 InfiniBand Modules

(56Gb)

Connection Blade and Mezzanine Card V4.0

(Linux®, Windows®)

Edition April 2013

Need help?

Do you have a question about the PRIMERGY BX900 InfiniBand and is the answer not in the manual?

Questions and answers

Summary of Contents for Fujitsu PRIMERGY BX900 InfiniBand

Page 1 User Manual - English PRIMERGY BX900 / BX400 InfiniBand Modules (56Gb) Connection Blade and Mezzanine Card V4.0 (Linux®, Windows®) Edition April 2013...
Page 2 Gesellschaft für Technik-Dokumentation mbH www.cognitas.de Copyright and Trademarks Copyright © Fujitsu Technology Solutions GmbH 2013. All rights reserved. Delivery subject to availability; right of technical modifications reserved. All hardware and software names used are trademarks of their respective manufacturers.
Page 3 Contents Introduction ......5 Notational Conventions ..... 5 Target Group .
Page 4 Contents Network Configuration Information ....24 2.6.1 Network configuration diagram ....24 2.6.2 Configuration sheet .
Page 5 Introduction This manual describes the configuration, installation and basic use of the InfiniBand connection blade and Mezzanine card software. The package includes tools for general and advanced management functionality. It also describes the installation of OFED (OpenFabrics Enterprise Distribution) for Linux, the identification of the Mezzanine card and Win OpenFabrics instal- lation for Windows.
Page 6 Introduction Fujitsu Support Please contact your Fujitsu Technical Support if you require assistance: http://support.ts.fujitsu.com/com/support/index.html If you purchased the products from Fujitsu Japan, please consult your system engineer. Target Group This manual is intended for users, developers, and system administrators responsible for setting up and maintaining switch system platforms using InfiniBand fabrics and networks.
Page 7 Introduction Overview Overview This manual explains the use of the software management. Managed devices must have a CPU and a management board. This board has an operating system and a toolset to operate, manage, and upgrade the device. Further information is provided on the PRIMERGY ServerBooks DVD: –...
Page 8 PRIMERGY BX900 Introduction PRIMERGY BX900 The PRIMERGY BX900 Blade Server system is a modular server system that integrates up to 18 server modules, eight connection blade modules and two Management Modules (MMB). The InfiniBand connection blade module provides networking and Switch functions to PRIMERGY BX900 Blade Server system.
Page 9 Introduction Inserting Connection Blade into PRIMERGY BX900 Inserting Connection Blade into PRIMERGY BX900 In order to support 4x FDR InfiniBand, 4 lanes per one port are required in backplane. Due to this reason, the form factor of connection blade is double wide - single height.
Page 10 Connecting Mezzanine Cards on Server Blade Introduction Connecting Mezzanine Cards on Server Blade The figure below indicates the connection relation between connection blade (CB) and Mezzanine Card on server blade. InfiniBand (IB) Mezzanine Cards can be set to both Mezzanine I slot and Mezzanine II slot. When Mezzanine Card is installed in Mezzanine II slot, the dual port of the card will be available and one of two ports will be connected to CB in CB5/6, the other one will be CB7/8.
Page 11 Description of InfiniBand Connection Blade Description of InfiniBand Connection Blade The PRIMERGY BX900 InfiniBand connection blade module features 18 internal and 18 external 4x14Gbps (FDR) ports. The below figure shows the front view with the 18 external ports, whereas the internal ones are connected to the server blades over Midplane.
Page 12 Description of InfiniBand Connection Blade Introduction 1.8.2 LEDs Connection Blade LEDs To indicate overall status of the connection blade, two LEDs are located at the left side of the front panel. Figure 6: LED-ID/Status Position Color Function Left side: top Blue Identify active (Identify LED)
Page 13 Introduction Description of InfiniBand Connection Blade Port LEDs For every external port a pair of LEDs exists. Figure 7: Port-Leds Position Color Function Above port: right Green Link up (Port Physical Link) No Link Above port: left Yellow Link up (Port Logical Link) Yellow blinking Link active Link down...
Page 14 Technical Data Introduction Technical Data 1.9.1 IB Connection Blade Internal Ports: External Ports: Lanes per Port: 4 (4x) Max Speed per Lane: 14Gbps (FDR) Product Weight: 9.4 Lbs / 4.3 kg Form Factor: Double wide Height: 27.9 mm / 1.1" Width: 388.2 mm / 15.3"...
Page 15 InfiniBand hardware installed. A blade can be properly installed with all required InfiniBand drivers and software during Red Hat Enterprise Linux installation. If you purchased the products from Fujitsu Japan, you can download it from http://jp.fujitsu.com/platform/server/primergy/downloads/. Please note, that only Red Hat Enterprise Linux Version 5.7 or higher, and 6.1 or higher are supported!
Page 16 Ê Download OFED from Fujitsu Technology Solutions Web site (http://support.ts.fujitsu.com/com/support/downloads.html > Driver & Downloads) Ê If you purchased the products from Fujitsu Japan, you download it from http://jp.fujitsu.com/platform/server/primergy/downloads/. Warning This software is the driver package of ConnectX-3 IB Mezzanine card (56Gbps).
Page 17 Linux Installation OFED Installation Run: rpm -e --allmatches openmpi libcxgb4 ofed-docs-1.4.1-2.el5 openib-1.4.1-6.el5 libibverbs libibverbs-utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm-utils libsdp opensm-libs compat-dapl compat-dapl-devel dapl dapl-devel dapl-utils perftest rds-tools ibutils infiniband-diags qperf libibverbs libibverbs-utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm-utils libsdp opensm-libs compat-dapl compat-dapl- devel compat-dapl-utils dapl dapl-devel dapl-utils perftest...
Page 18 OFED Installation Linux Installation 4. Execute Installer again and enter y. [root@localhost mnt]# ./mlnxofedinstall --without-32bit --without-fw-update This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed. Do you want to continue?[y/N]:y Starting MLNX_OFED_LINUX-1.5.3-3.0.0 installation ...
Page 19 Linux Installation OFED Installation 2.3.2 OFED installation on RHEL6.2 1. Mount iso file and move to /mnt. [root@localhost /]# mount -ro loop MLNX_OFED_LINUX-1.5.3- 3.0.0-rhel6.2-x86_64.iso /mnt/ [root@localhost /]# [root@localhost /]#cd /mnt/ 2. Execute Installer and enter y [root@localhost mnt]# ./mlnxofedinstall --without-32bit --without-fw-update This program will install the MLNX_OFED_LINUX package on your machine.
Page 20 OFED Installation Linux Installation Uninstall the packages. [root@localhost mnt]# [root@localhost mnt]# rpm -e --nodeps --allmatches rdma rdma-1.0-14.el6.noarch libibverbs libibverbs-utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm-utils opensm-libs dapl rds-tools ibutils infinipath-psm libibverbs libibverbs- utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm-utils opensm-libs compat-opensm-libs dapl rds-tools infinipath- psm opensm-libs libipathverbs dapl libibcm libibmad...
Page 21 Only with RHEL5.7 or RHEL6.1, OFED may be updated by the following steps: Ê Download OFED from Fujitsu Technology Solutions Web site (http://support.ts.fujitsu.com/com/support/downloads.html > Driver & Downloads) Ê If you purchased the products from Fujitsu Japan, you download it from http://jp.fujitsu.com/platform/server/primergy/downloads/. 2.3.3.1 Updating of OFED on RHEL5.7 1.
Page 22 OFED Installation Linux Installation [root@ localhost mnt]# rpm -e --nodeps --allmatches libibverbs- devel [root@ localhost mnt]# rpm -e --nodeps --allmatches libibverbs [root@ localhost mnt]# If these packages are not installed, the following error message is returned: [root@ localhost mnt]# rpm -e --nodeps --allmatches libibverbs-devel error: package libibverbs-devel is not installed [root@localhost mnt]# rpm -e --nodeps --allmatches...
Page 23 Linux Installation OFED Installation 4. Unmount /mnt directory. [root@localhost mnt]# cd / [root@ localhost ~]# umount /mnt [root@ localhost ~]# 5. Restart the OS. [root@ localhost ~]# reboot 6. Check the modification of limits.conf file. [root@ localhost ~]# tail -2 /etc/security/limits.conf * soft memlock unlimited * hard memlock unlimited [root@ localhost ~]#...
Page 24 OFED Installation Linux Installation 3. Move to /root and force the following packages to be uninstalled: – libibverbs – libibcm – librdmacm – infinipath-psm – fcoe-utils [root@ localhost Packages]# cd / [root@ localhost ~]# rpm -e --nodeps libibverbs libibcm librdmacm infinipath-psm fcoe-utils [root@ localhost ~]# If these packages are not installed, the following error message is returned:...
Page 25 Linux Installation OFED Installation Preparing... ####################################### kernel-ib ####################################### Installing kernel-ib-devel RPM Preparing... ####################################### kernel-ib-devel ####################################### Installation finished successfully. [root@ localhost mnt]# 5. Unmount /mnt directory. [root@localhost mnt]# cd / [root@ localhost ~]# umount /mnt [root@ localhost ~]# 6. Restart the OS. [root@ localhost ~]# reboot 7.
Page 26 OFED Installation Linux Installation mlx4_core Parameters Ê log_num_mtt Log maximum number of memory translation table segments per HCA (default is 20; max is 20). Ê log_mtts_per_seg Log number of MTT entries per segment (1-7) (int) (default is 3; max is 7). Example of /etc/modpobe.d/mlx4.conf Add the following parameters: options mlx4_core log_num_mtt=20 log_mtts_per_seg=4...
Page 27 Linux Installation Updating Firmware after Installation Updating Firmware after Installation If you wish to burn newer firmware, you have to download it from Fujitsu Technology Solutions Web site (http://support.ts.fujitsu.com/com/support/downloads.html > Driver & Downloads). If you purchased the products from Fujitsu Japan, you download it from http://jp.fujitsu.com/platform/server/primergy/downloads/.
Page 28 Subnet Manager Linux Installation 2.5.2 Confirmation of OpenSM running Enter the following command to confirm that OpenSM is running: # /etc/init.d/opensmd status OpenSM is running properly, if the result of this command displays "running". enter the following command to run OpenSM, if the result displays "stopped": # /etc/init.d/opensmd start 2.5.3 Confirmation of the master SM...
Page 29 Linux Installation Network Configuration Information Network Configuration Information The following information (1)-(6) is needed to maintain and troubleshooting an InfiniBand network. The information should be saved when configuring the InfiniBand network. 1. Network configuration diagram 2. Configuration sheet 3. The server on which the MASTER is found 4.
Page 30 Network Configuration Information Linux Installation 2.6.2 Configuration sheet Server mezz# mezz# OS host- Static iRMC eth0 eth1 eth2 eth3 ib0 ib1 blade 1 GUID 2 GUID name /DHCP Slot1 Slot2 Slot3 Slot4 Slot5 Slot6 Slot7 Slot8 Slot9 Slot10 Slot11 Slot12 Slot13 Slot14 Slot15...
Page 31 Linux Installation Network Configuration Information – Please fill in the GUID of the Mezzanine Card installed in Mezzanine I slot in the column of "mezz#1 GUID". Similarly, please fill in the GUID of the Mezzanine Card installed in Mezzanine II slot in the column of "mezz#2 GUID".
Page 32 Network Configuration Information Linux Installation Figure 10: Label on the back of the IB Mezzanine Card How to identify the node GUID: Ê Log in to the server blade as root. Ê Run the open terminal or telnet/ssh. Ê Execute the ibstat command, and identify the node GUID ("Node GUID: 0x****************").
Page 33 Linux Installation Network Configuration Information In the following example: the node GUID of Mezz slot1 is "0002c903002ff3e0", and Mezz slot2 is "0002c903002ff2a0". [root@localhost ~]# ibstat CA 'mlx4_0' CA type: MT4099 Number of ports: 2 Firmware version: 2.9.8220 Hardware version: 0 Node GUID: 0x0002c903002ff3e0 System image GUID: 0x0002c903002ff3e3 Port 1:...
Page 34 Network Configuration Information Linux Installation CA 'mlx4_1' CA type: MT4099 Number of ports: 2 Firmware version: 2.9.8220 Hardware version: 0 Node GUID: 0x0002c903002ff2a0 System image GUID: 0x0002c903002ff2a3 Port 1: State: Down Physical state: Polling Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02514868 Port GUID: 0x0002c903002ff2a1...
Page 35 Linux Installation Network Configuration Information 2.6.4 The result of the command "ibnetdiscover" GUID list It is useful to provide the text file (:.txt) which associates a GUID with the name of the device so that the ibnetdiscover command displays the result more meaningfully.
Page 36 Network Configuration Information Linux Installation "ibnetdiscover" command execution procedure 1. Connect to the CLI of IB connection blade via MMB (MMB Console Redirection). 2. Execute the command swinfo <BX900S2-CB3> swinfo <BX900S2-CB3 Information: GUID: 0x0002c9020040c7a8 MAC: 00:02:c9:11:20:5a IS4 Firmware revision: 7.2.326 CPU Firmware revision: Mellanox release mlnx405ex-1.0.2 build 2009-07-19 3.
Page 37 Linux Installation Network Configuration Information Figure 11: ibnetdiscover command: GUID list and arrangement drawing of devices BX900/BX400 InfiniBand Modules User Manual...
Page 38 Network Configuration Information Linux Installation The result format of ibnetdiscover 1. Provide the GUID list file (mmap.txt) in advance. 2. Search for the GUID obtained by the command swinfo in the result of ibnetdiscover ("S-" indicates IB connection blade, see RED underline) 3.
Page 39 Windows Installation Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed. A server can be properly installed with all required InfiniBand drivers and software during Windows (HPC) Server 2008 R2 SP1 installation or afterwards by using the Win OpenFabrics installation or OpenSM setup.
Page 40 Installation of WinOF Windows Installation 1. After inserting InfiniBand cards, make sure that “Other devices -> Unknown device” is shown in the Device Manager. Figure 12: Display of InfiniBand devices before installation of WinOF 2. Double click the Unknown device. Click Details tab and select Hardware Ids in Property.
Page 41 Windows Installation Installation of WinOF 4. Click [Next>] after the following window appears. Figure 13: InstallShield Wizard BX900/BX400 InfiniBand Modules User Manual...
Page 42 Installation of WinOF Windows Installation 5. After the following window appears, read License Agreement carefully. If you accept, tick "I accept the terms in the license agreement" and then click [Next>]. Figure 14: License Agreement User Manual BX900/BX400 InfiniBand Modules...
Page 43 Windows Installation Installation of WinOF 6. Click [Next>] after the following window appears (if you want to change the installation folder, specify the folder you want to install on by clicking [Change…]). Figure 15: Destination Folder BX900/BX400 InfiniBand Modules User Manual...
Page 44 Installation of WinOF Windows Installation 7. After the following window appears, untick "Check this box to configure your system for maximum 10GigE performance (Recommended)", and then click [Next>]. Figure 16: Maximum performance User Manual BX900/BX400 InfiniBand Modules...
Page 45 Windows Installation Installation of WinOF 8. Click [Install] after the following window appears. Figure 17: Installation BX900/BX400 InfiniBand Modules User Manual...
Page 46 Installation of WinOF Windows Installation 9. The following window appears after the installation process is finished. Click [Finish] without checking any boxes. Figure 18: Installation completed User Manual BX900/BX400 InfiniBand Modules...
Page 47 Windows Installation Installation of WinOF 10. Make sure that InfiniBand devices are recognized properly in Device Manager. Figure 19: InfiniBand devices BX900/BX400 InfiniBand Modules User Manual...
Page 48 Installation of WinOF Windows Installation 11. After WinOF installation, OpenSM should be set up. The 'OpenSM' service is registered to the [Services] Window after WinOF installation. Double-click 'OpenSM' to open properties. Figure 20: Configuration of OpenSM 12. Change 'Startup type' from Manual to Automatic (Delay Start). 13.
Page 49 Windows Installation Installation of WinOF 14. Open the 'Network Connections' Window to check if IPoIB networks are linked up. Disable the unused IPoIB adapters. Figure 21: Network status 15. Since no IP address is assigned to this new IPoIB adapter, assign an IP address to it.
Page 50 Installation of WinOF Windows Installation 17. In order to check if 'Network Direct' is enabled properly, execute the same command again. The message 'Installing IBAL provider: already installed' is shown if it is enabled properly. Figure 23: Installation of Network Direct 2 User Manual BX900/BX400 InfiniBand Modules...
Page 51 IB Connection Blade Management Connection In this chapter the connection of IB connection blade via Management Blade is explained. Follow the procedures below. Configurations for connecting to IB connection blade are explained in section “Configuration” on page 1. Login to Management Blade CLI via ssh. Please refer to "Server View Management Blade"...
Page 52 Connection IB Connection Blade Management Figure 25: Console redirect connection blade 4.1.1 Configuring the Switch for the First Time 1. Login (from MMB page_3_1) as admin and use admin as password. This starts the configuration wizard. However, the wizard may not start depending on a platform.
Page 53 IB Connection Blade Management Connection Wizard Session Display (Example) Comments Step 2: Use DHCP on mgmt0 Perform this step to obtain an IP interface? [yes] address for the switch. (mgmt0 is the management port of the switch.) If you wish the DHCP server to assign the IP address, type 'yes' and press <Enter>.
Page 54 Connection IB Connection Blade Management Wizard Session Display (Example) Comments You have entered the following infor- The wizard displays a summary of mation: your choices and then asks you to confirm the choices or to re-edit them. 1.Hostname: <switch name> Either press <Enter>...
Page 55 IB Connection Blade Management Connection Step 8: Domain name? Step 9: Enable IPv6? [yes] Step 10: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] Step 11: Admin password (Enter to leave unchanged)? You have entered the following information: 1. Hostname: BX900S2-CB3 2.
Page 56 Connection IB Connection Blade Management 4.1.2 Ethernet Connection via Management Blade IB connection blade provides the function of Ethernet connection via the Management LAN port on a Management Blade. Ethernet connection enables the user to connect to IB connection blade via telnet/ssh with CLI.
Page 57 IB Connection Blade Management Connection [ibuser@localhost ~]$ ssh 192.168.1.10 Mellanox MLNX-OS Switch Management Password: Mellanox Switch BX900S2-CB3 [standalone: master] > 4.1.4 WebUI Connection Ê Start a Web browser Google Chrome, Microsoft Internet Explorer 7.0 or Mozilla Firefox 3.0 or above. Ê...
Page 58 Configuration IB Connection Blade Management Configuration In order to configure the function of IB connection blade using CLI (Command Line Interface), the user needs to either connect the management console to Management Blade and run a terminal software or connect to Management Blade via ssh.
Page 59 IB Connection Blade Management Configuration 5. Enter "show" command to check the IP address. BX900S2-CB3 [standalone: master] (config) # show interfaces mgmt0 Interface mgmt0 state Admin up: Link up: IP address: 192.168.1.10 Netmask: 255.255.255.0 IPv6 enabled: Autoconf enabled: Autoconf route: Autoconf privacy: IPv6 addresses: IPv6 address:...
Page 60 Configuration IB Connection Blade Management 4.2.2 Save and restore the configuration The configuration data can be saved and restored by configuration command. Caution! Look out for the following during the operation of save and restore: – Do NOT turn off the power –...
Page 61 IB Connection Blade Management Configuration 2. To display the available configuration files and the active file, enter: BX900S2-CB3 [standalone: master] (config) # show configuration files initial (active) initial.bak myconf 3. Copy configuration file (myconf) to server by configuration upload command. BX900S2-CB3 [standalone: master] (config) # configuration upload myconf scp://user:password@192.168.1.10/tmp/conf1 How to restore the configuration...
Page 62 Basic Operation IB Connection Blade Management Basic Operation This chapter describes the basic operation by the commands and how to get the information for the maintenance. 4.3.1 How to check System Information This section describes how to check system information for this product. The firmware version can be checked for this product.
Page 63 IB Connection Blade Management Users and Privileges Target arch: Target hw: m460ex Built by: alia@fit15 Uptime: 22m 20.100s Product model: Host ID: 0002C95EB028 System memory: 75 MB used / 1952 MB free / 2027 MB total Swap: 0 MB used / 0 MB free / 0 MB total Number of CPUs: CPU load averages: 0.00 / 0.00 / 0.00 Users and Privileges...
Page 64 CLI modes IB Connection Blade Management CLI modes The CLI can be in one of three modes, and each mode makes available a certain group (or level) of commands for execution. The three command modes are: 1. Standard mode When the CLI is launched, it begins in Standard mode. This is the most restrictive mode and only has commands to query a restricted set of state information.
Page 65 IB Connection Blade Management Starting MLNX-OS Web User Interface Step1 Set up an Ethernet connection between the switch and a local network machine Step2 Open a Web browse connection between the switch and a local network Make sure the screen resolution is set to 1024*768 or higher. Step3 Type in the IP address of the switch or its DNS name in the following format: http://<switch_IP_address>.
Page 66 Starting MLNX-OS Web User Interface IB Connection Blade Management 4.6.2 After Log-in Display After you log in to MLNX-OS, a (default) status summary window will be displayed containing the following information: Figure 30: MLNX-OS status summary Figure 31: Active alerts User Manual BX900/BX400 InfiniBand Modules...
Page 67 IB Connection Blade Management Starting MLNX-OS Web User Interface Icon Description System Information Banner: Contains information on the switch system type. System Information Banner: Contains information on the: - OpenSM state - Chassis and SM HA role User credentials and Host name Action Pane: A display of the MLNX- OS menu icons Left Side Pane - Displays a list of the...
Page 68 Starting MLNX-OS Web User Interface IB Connection Blade Management 4.6.3 MLNX-OS WebUI The WebUI of MLNX-OS makes available the following menus (listed in order of appearance from left to right): – SETUP – SYSTEM – SECURITY – PORTS – STATUS User Manual BX900/BX400 InfiniBand Modules...
Page 69 IB Connection Blade Management Starting MLNX-OS Web User Interface 4.6.4 SETUP MENU The Setup menu makes available the following sub-menus (listed in order of appearance from top to bottom). Interfaces is used to obtain the status of, configure, or disable interfaces to the InfiniBand fabric. Thus, you can: set or clear the IP address and netmask of an interface;...
Page 70 Starting MLNX-OS Web User Interface IB Connection Blade Management NTP is used to set NTP (Network Time Protocol) and NTP servers. Licensing is used to manage MLNX-OS SwitchX licenses. User Manual BX900/BX400 InfiniBand Modules...
Page 71 IB Connection Blade Management Starting MLNX-OS Web User Interface 4.6.5 SYSTEM MENU The System menu makes available the following sub-menus (listed in order of appearance from top to bottom) Modules displays a (photo) graphic illustration of the system modules (front and rear views). By moving the mouse over the ports in the front view, a pop-up caption is displayed to indicate the status of the port.
Page 72 Starting MLNX-OS Web User Interface IB Connection Blade Management 4.6.6 SECURITY MENU The Security menu makes available the following sub-menus (listed in order of appearance from top to bottom). Users is used to manage (setting up, removing, modifying) user accounts. Admin Password is used to modify the system administrator password.
Page 73 IB Connection Blade Management Starting MLNX-OS Web User Interface 4.6.8 STATUS MENU The Status menu makes available the following sub-menus (listed in order of appearance from top to bottom). Summary displays general information about the switch system and the MLNX-OS SwitchX image, including: current date and time, hostname, uptime of system, installed MLNX- OS SwitchX image version, system memory,...
Page 74 Starting MLNX-OS Web User Interface IB Connection Blade Management Network provides a graphical display of network usage (transmitted and received packets) over time (1 day). It also provides per interface statistics. Logs displays the system log messages. It is possible to display either the currently saved system log or a continuous system log.
Page 75 If you cannot solve the problem, please contact the repair center: http://support.ts.fujitsu.com/com/support/index.html If you purchased the products from Fujitsu Japan, please consult your system engineer. Check the Log File of OpenSM (Linux only) Check the log file in the server (/var/log/opensm.log) that started up the Subnet Manager(OpenSM) as Master.
Page 76 Check the Log File of OpenSM (Linux only) Troubleshooting 3. IB connection blade If you have changed the above server blade and still get the same messages, you need to change IB connection blade. For details of how to specify the exact IB connection blade, please refer to section “The result of the command "ibnetdiscover"”...
Page 77 Troubleshooting Check the Log File of OpenSM (Linux only) 3. IB cable If you have changed the above IB connection blade and you still receive the messages, all IB cables connected to the IB connection blade need to be changed. Reporting Generic Notice type:1 num:128 (Link state change) from LID:LID number GUID:GUID number: the port of IB connection blade was disconnected in the subnet.
Page 78 IB Mezzanine Card Troubleshooting IB Mezzanine Card You can confirm place and time the error occured by referring to the following items on a server blade. Confirmation item Active link LED for IB Mezzanine card on the front side of the blade IB Mezzanine card driver message Active link LED for IB Mezzanine card When an active link LED for IB Mezzanine card on the front side of the server...
Page 79 4. If the LED is still turned off, please consult technical.support@ts.fujitsu.com If you purchased the products from Fujitsu Japan, please consult the system engineer. Mount the IB Mezzanine card more rigidly. Does IB connection blade Go to No.4...
Page 80 %d to %d. technical.support@ts.fujitsu.com This driver version supports only revisions If you purchased the products %d to %d. from Fujitsu Japan, please Internal error detected: buf[%02x]: %08x consult the system engineer. Failed to initialize user access region table, aborting. Couldn't map EQ doorbell for EQN 0x%06x Couldn't map interrupt clear register, aborting.
Page 81 NOP command failed to generate interrupt technical.support@ts.fujitsu.com (IRQ %d), aborting. If you purchased the products Failed to initialize completion queue table, from Fujitsu Japan, please aborting. consult the system engineer. Failed to initialize shared receive queue table, aborting. Failed to initialize queue pair table, aborting.
Page 82 Couldn't restore Mezzanine card PCI If you purchased the products Express Device Control register, aborting. from Fujitsu Japan, please consult the system engineer. Couldn't restore Mezzanine card PCI Express Link control register, aborting. Couldn't restore Mezzanine card reg %x, aborting.
Page 83 Troubleshooting IB Mezzanine Card Work after IB Mezzanine card has been replaced (Linux) 1. The server blade is mounted on the main body of BX900 and the power supply is turned on. 2. Linux is started, /sbin/lspci command is executed by the root account, and it is confirmed that the IB Mezzanine card is recognized.
Page 84 Error Memory Region. 0x0048 xxx: Failed to create If you purchased the products from Error receive descriptor pool. Fujitsu Japan, please consult the system engineer. 0x0049 xxx: Failed to create Error NDIS_PACKET pool to receive indications. 0x004A xxx: Failed to create...
Page 85 Subnet Administrator technical.support@ts.fujitsu.com Error query for port information failed. If you purchased the products from 0x0055 xxx: Subnet Administrator Fujitsu Japan, please consult the Error failed query for broadcast system engineer. group information. 0x0056 xxx: Subnet Administrator Error failed request to joining broadcast group.
Page 86 0x005E xxx: SetDeviceRegistra- Please consult Error tionAttributes failed. technical.support@ts.fujitsu.com 0x005F xxx: SetAdapterRegistra- If you purchased the products from Error tionAttributes failed. Fujitsu Japan, please consult the 0x0060 xxx: SetOffloadAttributes system engineer. Error failed. 0x0061 xxx: ipoib_create_adapter Error failed. 0x0062 xxx : ipoib_start_adapter Error failed.
Page 87 Please burn the last FW and technical.support@ts.fujitsu.com restart the mlx4_bus driver. If you purchased the products 0x0008 RUN_FW command failed from Fujitsu Japan, please Error with error %2.%n consult the system engineer. The adapter card is non- functional.%n Most likely a FW problem.%n Please burn the last FW and restart the mlx4_bus driver.
Page 88 Please burn the last FW and technical.support@ts.fujitsu.com restart the mlx4_bus driver. If you purchased the products 0x000D Too few QPs were requested from Fujitsu Japan, please Error (requested %2, reserved for consult the system engineer. FW %3).%n The adapter card is non- functional.%n...
Page 89 Error message technical.support@ts.fujitsu.com changes in each event. If you purchased the products from Fujitsu Japan, please consult the system engineer. Table 10: [source:ibbus] Work after IB Mezzanine card has been exchanged (Windows Server 2008 R2 SP1) 1. Turn on the server.
Page 90 When ambient temperature does not have abnormality, the IB connection blade must be replaced. Please consult technical.support@ts.fujitsu.com If you purchased the products from Fujitsu Japan, please consult the system engineer. Table 12: snmp trap User Manual BX900/BX400 InfiniBand Modules...
Page 91 Confirm that PSU or Power Supply source of BX900 or BX400 system does not have abnormality. Please consult technical.support@ts.fujitsu.com If you purchased the products from Fujitsu Japan, please consult the system engineer. lowPower 1.3.6.1.4.1.33049.2.1.2.8 [Content] Recover The MLNX-OS management appliance has been restored to its normal power.
Page 92 IB Connection Blade Troubleshooting 6. Restore composition definition information. Please refer to section “Save and restore the configuration” on page 7. Connect IB-Cable to the same port as before. 8. Confirm that LED of P and L/A lights. It might take about one minute to lighting LED.
Page 93 Index BX400 S1 Management Blade BX900 S2 login master Mezzanine card exchange Command line interface GUID configuration sheet hardware installation technical data disk Space for installation network configuration diagram disk space notational conventions for Windows installation OFED ethernet connection BX installation OpenFabrics Enterprise Distribution GUID...
Page 94 Index show asic-version show system guid software requirements Windows ssh connection standby Subnet Manager subnet manager OpenSM support switch LEDs technical support Windows installation User Manual BX900/BX400 InfiniBand Modules...

This manual is also suitable for:

Primergy bx400 infiniband