Summary of Contents for Mellanox Technologies Mellanox SX1018
Page 1
Mellanox MLNX-OS® User Manual for SX1018HP Ethernet Managed Blade Switch Rev 1.6.9 Software Version 3.3.4100 www.mellanox.com Mellanox Technologies Confidential...
Page 2
Rev 1.6.9 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.
Rev 1.6.9 About this Manual This manual provides general information concerning the scope and organization of this User’s Manual. Intended Audience This manual is intended for network administrators who are responsible for configuring and managing Mellanox Technologies’ SwitchX based Switch Platforms. Related Documentation The following table lists the documents referenced in this User’s Manual.
Rev 1.6.9 Glossary Table 3 - Glossary Authentication, Authorization, and Accounting. Authentication - verifies user credentials (username and password). Authorization - grants or refuses privileges to a user/client for accessing spe- cific services. Accounting - tracks network resources consumption by users. Address Resolution Protocol.
Page 9
Rev 1.6.9 Table 3 - Glossary MTU (Maximum Transfer Unit) The maximum size of a packet payload (not including headers) that can be sent /received from a port Network Adapter A hardware device that allows for communication between computers in a network PFC/FC Priority Based Flow Control applies pause functionality to traffic classes OR...
Rev 1.6.9 Introduction Mellanox® Operating System (MLNX-OS®) enables the management and configuration of Mel- lanox Technologies’ SwitchX® silicon based switch platforms. MLNX-OS supports the Virtual Protocol Interconnect (VPI) technology which enables it to be used for both Ethernet and Infini- Band technology providing the user with greater flexibility.
Page 11
Rev 1.6.9 Table 5 - Ethernet Features Feature Description Ethernet support • 48K Unicast MAC addresses • VLAN (802.1Q) - 4K • LAG/LACP (802.3ad), 16 links per LAG (36 LAGs) • Rapid Spanning Tree (802.1w) • Flow control (802.3x) • IGMP snooping v1,2 •...
Rev 1.6.9 Getting Started The procedures described in this chapter assume that you have already installed and powered on your switch according to the instructions in the Hardware Installation Guide, which was shipped with the product. Configuring the Switch for the First Time Connect to the HP Chassis Manager and run “connect interconnect <slot number 1-7>”.
Page 13
Rev 1.6.9 Table 7 - Configuration Wizard Session - IP Configuration by DHCP (Sheet 2 of 3) Wizard Session Display (Example) Comments Step 2: Use DHCP on mgmt0 interface? [yes] Perform this step to obtain an IP address for the switch.
Page 14
Rev 1.6.9 Table 7 - Configuration Wizard Session - IP Configuration by DHCP (Sheet 3 of 3) Wizard Session Display (Example) Comments You have entered the following information: The wizard displays a summary of your choices and then asks you to confirm the choices or to re-edit them.
Page 15
Rev 1.6.9 Table 8 - Configuration Wizard Session - IP Zeroconf Configuration Wizard Session Display - IP Zeroconf Configuration (Example) Mellanox configuration wizard Do you want to use the wizard for initial configuration? y Step 1: Hostname? [switch-112126] Step 2: Use DHCP on mgmt0 interface? [no] Step 3: Use zeroconf on mgmt0 interface? [no] yes Step 4: Default gateway? [192.168.10.1] Step 5: Primary DNS server?
Page 16
Rev 1.6.9 Table 9 - Configuration Wizard Session - Static IP Configuration Wizard Session Display - Static IP Configuration (Example) Mellanox configuration wizard Do you want to use the wizard for initial configuration? y Step 1: Hostname? [switch-112126] Step 2: Use DHCP on mgmt0 interface? [yes] n Step 3: Use zeroconf on mgmt0 interface? [no] Step 4: Primary IP address? 192.168.10.4 Mask length may not be zero if address is not zero (interface mgmt0)
Rev 1.6.9 Check the mgmt0 interface configuration before attempting a remote (for example, SSH) con- Step 5. nection to the switch. Specifically, verify the existence of an IP address. switch # show interfaces mgmt0 Interface mgmt0 state Admin up: Link up: IP address: 169.254.15.134 Netmask:...
Rev 1.6.9 Mellanox Switch switch > Login to the switch (default username is admin, password admin) Step 3. Once you get the prompt, you are ready to use the system. Refer to MLNX-OS Command Refer- Step 4. ence Guide for additional information on the CLI commands. Starting the Web Interface ...
Rev 1.6.9 Figure 3: Display After Login Licenses MLNX-OS software package can be extended with premium features. Installing a license allows you to access the specified premium features. This section is relevant only to switch systems with an internal management capability. The following licenses are offered with MLNX-OS software: Table 10 - MLNX-OS Licenses Valid on product...
Rev 1.6.9 Table 10 - MLNX-OS Licenses Valid on product Description UPGR-6036F-56E SX6036 56GbE link speed UPGR-1036-GW SX1036 InfiniBand, Ethernet L3, Gateway UPGR-1036F-56E SX1036 56GbE link speed UPGR-1024-GW SX1024 InfiniBand, Ethernet L3, Gateway UPGR-1024-56E SX1024 56GbE link speed LIC-fabric-inspector SX6036F/T/ InfiniBand fabric inspector monitoring and health.
Page 21
Rev 1.6.9 Figure 4: No Licenses Installed Enter your license key(s) in the text box. If you have more than one license, please enter each Step 3. license in a separate line. Click “Add Licenses” after entering the last license key to install them.
Page 22
Rev 1.6.9 Figure 5: Enter Licence Key(s) in Text Box All installed licenses should now be displayed. Figure 6: Installed License Mellanox Technologies Mellanox Technologies Confidential...
Rev 1.6.9 Save the configuration to complete the license installation. Step 4. If you do not save the installation session, you will lose the installed licenses at the next system boot. 2.4.3 Retrieving a Lost License Key In case of a lost MLNX-OS® license key, contact your authorized Mellanox reseller and provide the switch’s chassis serial number.
Rev 1.6.9 User Interfaces Command Line Interface (CLI) MLNX-OS® is equipped with an industry-standard CLI. The CLI is accessed through SSH or Telnet sessions, or directly via the console port on the front panel (if it exists). Refer to the MLNX-OS Command Reference Guide for complete set of commands, syntax and examples.
Rev 1.6.9 3.1.2 Syntax Conventions To help you identify the parts of a CLI command, this section explains conventions of presenting the syntax of commands. Table 12 - Syntax Conventions Syntax Convention Description Example < > Angled brackets Indicate a value/variable that <1...65535>...
Rev 1.6.9 show Display system configuration or statistics slogin Log into another system securely using ssh switch Configure switch on system telnet Log into another system using telnet terminal Set terminal parameters traceroute Trace the route packets take to a destination switch-11a596 [standalone: master] >...
Rev 1.6.9 The following session shows how to move between command modes: switch > (You start in Standard mode) switch > enable (Move to Enable mode) switch # (You are in Enable mode) switch # configure terminal (Move to Config mode) switch (config) # (You are in Config mode) switch (config) # exit...
Rev 1.6.9 Table 13 - Angled Brackets Parameter Description Parameter Description <network prefix> An IPv4 network prefix specifying a network. Used in conjunction with a net- mask to determine which bits are significant. e.g. “192.168.0.0”. <regular expression> An extended regular expression as defined by the “grep” in the man page. (The value you provide here is passed on to “grep -E”.) <node id>...
Rev 1.6.9 • Ethernet Management Make sure to save your changes before switching between menus or sub-menus. Click the “Save” button to the right of “Save Changes?”. Figure 7: WebUI 3.2.1 Setup Menu The Setup menu makes available the following submenus (listed in order of appearance from top to bottom): Table 14 - Setup Submenus Submenu Title...
Rev 1.6.9 Table 14 - Setup Submenus Submenu Title Description Routing Used to set, remove or display the default gateway, and the static and dynamic routes. Hostname Used to set or modify the hostname. Used to set or delete static hosts. Used to set, remove, modify or display static and dynamic name servers.
Rev 1.6.9 Table 15 - System Submenus Submenu Title Description Inventory Displays a table with the following information about the system modules: mod- ule name, type, serial number, ordering part number and Asic firmware version. Power Management Displays a table with the following information about the system power supplies: power supply name, power, voltage level, current consumption, and status.
Rev 1.6.9 Table 17 - Ports Submenus Submenu Title Description Phy Profile Provides the ability to manage phy profiles. Protocol type Manages the link protocol type 3.2.5 Status Menu The Status menu makes available the following sub-menus (listed in order of appearance from top to bottom): Table 18 - Status Submenus Submenu Title...
Rev 1.6.9 The IB SM Mgmt menu makes available the following sub-menus (listed in order of appearance from top to bottom): Table 19 - IB SM Mgmt Submenus Submenu Title Description Summary Displays the local Subnet Manager (SM) status (running time, failures, etc). Base SM Used to manage basic SM configuration (enabling SM, priority level, and restor- ing initial configuration).
Rev 1.6.9 Table 20 - Fabric Inspctr Submenus Submenu Title Description IB Nodes Displays information about InfiniBand nodes in the fabric. It is possible to filter display by the type of InfiniBand node (HCA adapter, switch, etc). IB Ports Displays all active InfiniBand ports in the fabric. It is possible to filter display by the type of InfiniBand port (HCA port, switch port, switch management port, etc), by the port rate (speed or width), by the Subnet Manager status on the node, by node traffic, etc.
Rev 1.6.9 System Management Management Interface 4.1.1 Configuring Management Interfaces with Static IP Addresses If your switch system was set during initialization to obtain dynamic IP addresses through DHCP and you wish to switch to static assignments, perform the following steps: Change to Config mode.
Rev 1.6.9 For all other systems (and software versions) DHCP is disabled by default. If a user connects through SSH, runs the wizard and turns off DHCP, the connection is immediately terminated as the management interface loses its IP address. <localhost># ssh admin@192.168.10.101 Mellanox MLNX-OS Switch Management Password:...
Page 39
Rev 1.6.9 Partition 1: SX_PPC_M460EX SX_3.3.3130 2013-03-20 21:32:25 ppc Partition 2: SX_PPC_M460EX SX_3.3.3130 2013-03-20 21:32:25 ppc Images available to be installed: image-PPC_M460EX-SX_3.3.3256.img SX_PPC_M460EX SX_3.3.3256 2013-03-20 21:32:25 ppc Serve image files via HTTP/HTTPS: no No image install currently in progress. Boot manager password is set. No image install currently in progress.
Page 40
Rev 1.6.9 switch (config) # show images Installed images: Partition 1: SX <old ver> 2013-04-28 16:02:50 Partition 2: SX <new ver> 2013-04-28 16:52:50 Images available to be installed: new_image.img SX <new ver> 2013-04-28 16:52:50 Serve image files via HTTP/HTTPS: no No image install currently in progress.
Page 41
Rev 1.6.9 new_image.img SX <new ver> 2011-04-28 16:52:50 Installed images: Partition 1: SX <old ver> 2011-04-28 16:02:50 Partition 2: SX <new ver> 2011-04-28 16:52:50 Last boot partition: 1 Next boot partition: 2 No boot manager password is set. switch (config) # Save current configuration.
Rev 1.6.9 When performing upgrade from the WebUI, make sure that the image you are trying to upgrade to is not located already in the system (i.e. fetched from the CLI). 4.2.2 Deleting Unused Images To delete unused images: Enter Config mode.
Page 43
Rev 1.6.9 Disable paging of CLI output. Run: switch-112094 [standalone: master] (config) # no cli default paging enable Display commands to recreate current running configuration. Run: switch-112094 [standalone: master] (config) # show running-config Copy the output to a text file. 4.2.3.1 Downloading Image Log into the system to obtain the serial number.
Page 44
Rev 1.6.9 Partition 1: <current version> 2010-09-19 03:46:25 Partition 2: <current version> 2010-09-19 03:46:25 Last boot partition: 1 Next boot partition: 1 No boot manager password is set. switch (config) # Install the MLNX-OS image. Run: Step 4. switch (config) # image install <image_name> Step 1 of 4: Verify Image 100.0% [#################################################################] Step 2 of 4: Uncompress Image...
Rev 1.6.9 In case you are downloading to an older software version which has never been run yet on the switch, use the following command sequence as well: switch (config) # no boot next fallback-reboot enable switch (config) # configuration write Reload the switch.
Rev 1.6.9 If one or more of the switch modules is programmed with a firmware version other than the default version, then MLNX-OS will automatically attempt to burn the default firmware version instead. If a firmware update takes place, then the login process will be delayed for a few min- utes.
Rev 1.6.9 • To save the configuration to the active configuration file, run: switch (config) # configuration write • To save the configuration to a user-specified file without making the new file the active configuration file, run: switch (config) # configuration write to myconf no-switch •...
Rev 1.6.9 4.4.1 BIN Configuration Files BIN configuration files are not human readable and cannot be edited. To create a new BIN configuration file switch (config) # configuration new my-filename To upload a BIN configuration file from a switch to an external file server switch (config) # configuration upload my-filename scp://root@my-server/root/tmp/my- filename ...
Rev 1.6.9 When applying a text-based configuration file, the configuration is appended to the switch’s existing configuration. Reboot is not required. Logging 4.5.1 Monitor To print logging events to the terminal: Set the modules or events you wish to print to the terminal. For example, run: switch (config) # logging monitor events notice switch (config) # logging monitor sx-sdk warning These commands print system events in severity “notice”...
Rev 1.6.9 4.6.1 Supported Events The following table presents the supported events and maps them to their relevant MIB OID. Table 22 - Supported Event Notifications and MIB Mapping Event Name Event Description MIB OID Comments asic-chip-down ASIC (chip) down Mellanox-EFM-MIB: Not supported asicChipDown...
Rev 1.6.9 Table 22 - Supported Event Notifications and MIB Mapping Event Name Event Description MIB OID Comments power-redundancy-mismatch Power redundancy mis- Mellanox-EFM-MIB: Supported for SX65XX only match powerRedundancyMis- match systems process-crash A process in the system Mellanox-EFM-MIB: has crashed procCrash process-exit A process in the system...
Page 52
Rev 1.6.9 Enter to Config mode. Run: Step 1. switch > switch > enable switch # configure terminal Set your mailhub to the IP address to be your mail client’s server – for example, Microsoft Out- Step 2. look exchange server. switch (config) # email mailhub <IP address>...
Rev 1.6.9 This is a test email. ==== Done. For further information, please refer to Mellanox MLNX-OS Command Reference Guide. USB Support 4.7.1 Accessing a USB Device for Read/Write MLNX-OS can access USB devices attached to switch systems. USB devices are automatically recognized and mounted upon insertion.
Rev 1.6.9 In order to block sending mDNS traffic from the management interface use the following command: switch (config) # no ha dns enable switch (config) # 4.10 User Management and Security 4.10.1 Authentication, Authorization and Accounting (AAA) AAA is a term describing a framework for intelligently controlling access to computer resources, enforcing policies, auditing usage, and providing the information necessary to bill for services.
Page 55
Rev 1.6.9 It is used for several reasons: • RADIUS facilitates centralized user administration • RADIUS consistently provides some level of protection against an active attacker For information on the RADIUS commands, please refer to Mellanox MLNX-OS Command Ref- erence Guide. 4.10.1.2 TACACS+ TACACS (Terminal Access Controller Access Control System), widely used in network environ- ments, is a client/server protocol that enables remote access servers to communicate with a cen-...
Rev 1.6.9 4.10.2 Secure Shell (SSH) It is recommended not to use more than 100 concurrent SSH sessions to the switch. 4.10.2.1 Adding a Host and Providing an SSH Key To add entries to the global known-hosts configuration file and its SSH value, perform the following steps: Change to Config mode Run: Step 1.
Rev 1.6.9 4.11 Network Management Interfaces 4.11.1 SNMP Simple Network Management Protocol (SNMP), is a network protocol for the management of a network and the monitoring of network devices and their functions. SNMP supports asynchro- nous event (trap) notifications and queries. MLNX-OS supports: •...
Page 58
Rev 1.6.9 Table 25 - Standard MIBs – Structure, Management Interface and General SNMP Standard Comments TCP-MIB RFC 4022 Management interface UDP-MIB RFC 4113 Management interface IP-FORWARD-MIB RFC 4292 Management interface HOST-RESOURCES-MIB, RFC 2790 Management interface HOST-RESOURCES-TYPE Table 26 - Standard MIBs – Chassis and Switch Standard Comments RFC1213-MIB...
Page 59
Rev 1.6.9 4.11.1.2 Private MIB Table 27 - Private MIBs Supported Comments MELLANOX-SMI-MIB Mellanox Private MIB main structure (no objects) MELLANOX-PRODUCTS-MIB List of OID – per managed system (sysObjID) MELLANOX-IF-VPI-MIB IfTable extensions MELLANOX-EFM-MIB Deprecated MIB (based on Mellanox-MIB) Traps definitions are supported. MELLANOX-ENTITY-MIB Enhances the standard ENTITY-MIB (contains GUID and ASIC revision).
Page 60
Rev 1.6.9 For additional information refer to MELLANOX-EFM-MIB. For event-to-MIB mapping, please refer to Table 22, “Supported Event Notifications and MIB Mapping,” on page 50. 4.11.1.4 Configuring SNMP To set up the SNMP: Activate the SNMP server on the MLNX-OS switch (in configure mode) using the following Step 1.
Page 61
Rev 1.6.9 Enter privacy password and its confirmation. Step 3. switch (config) # snmp-server user admin v3 prompt auth md5 priv des Auth password: ******** Confirm: ******** Privacy password: ******** Confirm: ******** switch (config) # To retrieve the system table, run the following SNMP command: snmpwalk -v3 -l authPriv -a MD5 -u admin -A “<Authentication password>”...
Rev 1.6.9 This particular event is used as an example only. Verify the list of traps and informs being sent to out of the system. Run: Step 5. switch (config) # show snmp events Events for which traps will be sent: asic-chip-down: ASIC (Chip) Down cpu-util-high: CPU utilization has risen too high disk-space-low: Filesystem free space has fallen too low...
Rev 1.6.9 Ethernet Switching Interface Interface Ethernet have the following physical set of configurable parameters • Admin state – enabling or disabling the interface. • Flow control – admin state per direction (send or receive) • MTU (Maximum Transmission Unit) – (1518-9216 bytes) •...
Page 64
Rev 1.6.9 Specific ports can be split by using a QSFP 1X4 breakout cable to split one 40 Gb/s port into 4 lanes (4 SFP+ connectors). These 4 lanes then go, one lane to each of the 4 SFP+ connectors. Some ports can be split into 2 10 Gb/s ports, using lanes 1 and 2 only.
Page 65
Rev 1.6.9 5.1.1.1 Changing the Module Type to a Split Mode To split a port of an interface: Shut down all the ports related to the interface. Run: Step 1. • in case of split-2, shut down the current interface only •...
Rev 1.6.9 The module-type can be changed only from the first member of the split and not from the interface that was split. The following warning will be displayed: The following interfaces will be unmapped: 1/4/1 1/ 4/2 1/4/3 1/4/4 Type when prompted Type 'yes' to confirm unsplit.
Rev 1.6.9 If the physical port is operationally up, this port will be an active member of the aggre- gation. Consequently, it will be able to convey traffic. 5.2.2 Configuring Link Aggregation Control Protocol (LACP) To configure LACP: Log in as admin. Step 1.
Rev 1.6.9 • Trunk – Trunk port is a port connecting 2 switches. It accepts only tagged frames with VLANs of which the port is a member. On egress, traffic sent from the Trunk port is tagged. By default, a Trunk port is, automatically, a member on all current VLANs. 5.3.1 Configuring Access Mode and Assigning Port VLAN ID (PVID) ...
Rev 1.6.9 host, is to converge quickly, while the required behavior of a port connected to a switch entity is to converge based on the RSTP parameters. Additionally, it adds security issues on a port and switch basis, allowing the operator to deter- mine the state and role of a port or the entire switch should an abnormal event occur.
Rev 1.6.9 BPDU filtering is configured per interface. When configured, the port does not send any BPDUs and drops all BPDUs that it receives. To configure BPDU filter, use the following command: switch (config interface etherent <inf>)# spanning-tree bpdufilter {enable , disable} Configuring BPDU filtering on a port connected to a switch can cause bridging loops because the port filters any BPDU it receives and goes to forwarding state.
Rev 1.6.9 ticast-group by sending a join request message towards the network router, and responds to que- ries sent from the network router by dispatching a join report. A given port can be either manually configured to be a router-port or it can be dynamically man- ifested when having received a query, hence, the network router is connected to this port.
Rev 1.6.9 Define the MRouter port on the VLAN. Run: Step 6. switch (config) # vlan 2 switch (config vlan 2) # ip igmp mrouter interface ethernet 1/1 switch (config vlan 2) # To change the Interface Switchport to Hybrid: Log in as admin.
Rev 1.6.9 Enter config mode. Run: Step 2. switch > enable switch # configure terminal Enable LLDP globally on the switch. Run: Step 3. switch (config) # lldp switch (config) # Enable LLDP per interface. Run: Step 4. switch (config interface ethernet 1/1) # lldp receive switch (config interface ethernet 1/1) # lldp transmit Show LLDP local information.
Page 76
Rev 1.6.9 • Transmits PFC-PAUSE frames when the receive threshold for a particular traffic class is reached • Provides the management capability for an administrator to configure the flow control properties on each port of the switch • Keeps flow control disabled for all priorities on all ports by default •...
Rev 1.6.9 Choose the desirable priority you want to enable using the command Step 4. dcb priority-flow-control priority <pri[0..7]> enable switch (config) # dcb priority-flow-control priority 5 enable To enable PFC per interface: Log in as admin. Step 1. Change to config mode.
Page 78
Rev 1.6.9 Log in as admin. Step 1. Enter config mode. Run: Step 2. switch > enable switch # configure terminal Run the command Step 3. dcb ets disable switch (config) # no dcb ets enable To configure the WRR bandwidth percentage: Log in as admin.
Rev 1.6.9 Run the command to verify the configuration. Step 5. show dcb ets switch (config) # show dcb ets ETS enabled Bandwidth -------------------------- Number of Traffic Class: 4 switch (config) # Access Control List An Access Control List (ACL) is a list of permissions attached to an object, to filter or match switches packets.
Rev 1.6.9 5.9.2 ACL Actions An ACL action is a set of actions can be activated in case the packet hits the ACL rule. To modify the VLAN tag of the egress traffic as part of the ACL “permit” rule: Create access-list action profile: Step 1.
Rev 1.6.9 There is no limitation on the number of mirroring sources and more than a single source can be mapped to a single analyzer destination. 5.10.1 Mirroring Sessions Port mirroring is performed by configuring mirroring sessions. A session is an association of a mirror port (or more) and an analyzer port.
Page 82
Rev 1.6.9 There is no limitation on the number of the source interfaces mapped to a mirroring session. Ingress and egress traffic flows of a specific source interface can be mapped to two dif- ferent sessions. The source interface can be a physical interface or a LAG. Port mirroring can be configured on a LAG interface but not on a LAG member.
Page 83
Rev 1.6.9 The system on the receiving end of the analyzer port must be set to handle the egress traffic. If it is not, it might discard it and indicate this in its statistics (packet too long). 5.10.1.3 Header Format Ingress traffic from the source interface can be manipulated in several ways depending on the network layout using the command header-format.
Rev 1.6.9 The default behavior in congestion situations is to drop any excessive frames that may clog the system. ETS, PFC and FC configurations do not apply to the destination port. 5.10.1.5 Truncation When enabled, the system can truncate the mirrored packets into smaller 64-byte packets (default) which is enough to capture the packets’...
Rev 1.6.9 5.11.2 Statistical Samples The sFlow agent samples interface counters time based. Polling interval is configurable to any value between 5-3600 seconds with the default being 20 seconds. The following statistics are gathered by the CPU: Table 31 - List of Statistical Counters Counter Description Total packets...
Rev 1.6.9 (Optional) Set the sampling rate of the mechanism. Run: Step 6. switch (config sflow) # sampling-rate 16000 This means that one every 16000 packet gets collected for sampling. (Optional) Set the maximum size of the data path sample. Run: Step 7.