Nvidia MSTFLINT Documentation
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Computer Hardware
  5. MSTFLINT
  6. Documentation
Nvidia MSTFLINT Documentation

Nvidia MSTFLINT Documentation

Hide thumbs Also See for MSTFLINT:
Table of Contents

Advertisement

Quick Links

Download this manual
 
 
 
 
 
 
 
MSTFLINT Package -Firmware Burning and
Diagnostics Tools Documentation v4.20.0
 
 
Exported on Apr/28/2022 11:57 AM

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the MSTFLINT and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia MSTFLINT

Summary of Contents for Nvidia MSTFLINT

  • Page 1               MSTFLINT Package -Firmware Burning and Diagnostics Tools Documentation v4.20.0     Exported on Apr/28/2022 11:57 AM...

  • Page 2: Table Of Contents

    Release Notes Update History..............6 General Information .................6 Package Tools ..................6 Dependencies..................7 mstflint Supported Operating Systems and Platforms........8 Supported Flash Types................. 11 Supported Interface Cards (NICs)............12 Supported Switch Systems Software ............12 Changes and New Features ............... 13 mstflint Bug Fixes in this Version..............
  • Page 3 Switches Options ................31 Command Parameters ................. 34 mstflint: Burning a Firmware Image ............35 mstflint: Managing an Expansion ROM Image ..........41 mstflint: Performing Checksum Calculation on Image/Device......41 mstflint: Querying the Firmware Image ............ 42 mstflint: Setting GUIDs and MACs ............43 mstflint: Verifying the Firmware Image ............
  • Page 4 Firmware Burning Related Issues ..............87 Secure Firmware Related Issues ..............89 Appendixes..................91 mstflint - Updating Firmware Using ethtool/devlink and .mfa2 File ...... 91 mstflint User Manual Revision History .............93 Release Notes Revision History .............96 mstflint Release Notes Change Log History ............ 96 mstflint Bug Fixes History ...............114...

  • Page 5: Overview

    Overview ® The NVIDIA  Firmware Tools (mstflint) package is a set of firmware management and debug tools for NVIDIA devices. The document describes mstflint features, tools content and configuration. The documentation here relates to: • Release Notes • User Manual...

  • Page 6: Release Notes

    Release Notes These are the release notes for mstflint. mstflint supports Linux operating system. Please see the supported platform table for further details. The tools functionality is identical in all operating systems unless otherwise noted.  Release Notes Update History Revision...

  • Page 7: Dependencies

    Category Tool Description Operati System mstvp Reads PCI device VPD Linux FreeBSD mstfwr Load Firmware after firmware update on ISFU capable devices.(5th Linux eset generation devices) FreeBSD mstfwt Extracts and prints trace messages generated by the firmware of 5th Linux race generation devices.

  • Page 8: Mstflint Supported Operating Systems And Platforms

    Supported Operating Systems and Platforms mstflint is supported on the following platforms: Table Legend:  + (Green) Supported and tested ** (Orange) Supported but not tested in MFT v4.20.0 Supported Operating Systems and Platforms Platform Status AliOS 7.2 aarch64 Alma 8.5 x86_64 Anolis 8.4...
  • Page 9 Platform Status Fedora 35 X86_64 FreeBSD 12-STABLE (12.2) x86_64 FreeBSD 13.0-STABLE x86_64 FreeBSD 14-CURRENT x86_64 KylinOS v10 SP2 x86_64 Mariner x86_64 MLNX-OS 64 Bit OEL 7.8 x86_64 OEL 7.9 x86_64 OEL 8.2 x86_64 OEL 8.3 x86_64 OpenEuler 20.3 x86_64 OpenEuler 20.3 SP1 x86_64 OpenEuler 20.3 SP3 x86_64...
  • Page 10 Platform Status RHEL 7.9 PPC64LE RHEL 8 RHEL 8 PPC64LE RHEL 8 x86_64 RHEL 8.1 PPC64LE (Power 9) RHEL 8.1 x86_64 RHEL 8.1 RHEL 8.2 x86_64 RHEL 8.2 PPC64LE RHEL 8.3 PPC64LE RHEL 8.3 x86_64 RHEL 8.4 x86_64 RHEL 8.4 PPC64LE RHEL 8.5 PPC64LE...

  • Page 11: Supported Flash Types

    Ubuntu 21.10 x86_64 Ubuntu 22.04 x86_64 UOS v20 1021e x86_64 UOS v20 1021e UOS v20 1030 x86_64 Supported Flash Types mstflint supports the following Flash types. Vendor Flash Family Tested P/N Winbond W25QxxBV W25Q32FVSSIG W25Q32FVSSIGS W25Q32FVSSIGT W25Q128FVSIGS W25Qxxx W25Q256JVBIMT...

  • Page 12: Supported Interface Cards (Nics)

    GD25LB256EBFRY GD25B256DFIGR Supported Interface Cards (NICs) With respect to mstflint, NVIDIA IC devices are divided into two groups: Group I and Group II (4th generation and 5th generation, respectively). The ICs are listed in the following table: IC Group IC Device Group II / 5th Generation •...

  • Page 13: Changes And New Features

    Changes and New Features Component/ Tool Description Operating System Rev. 4.20.0 Added support for NVIDIA ConnectX-7 adapter cards. mstflint mstflint cable/transceivers burning commands provide now validation and extract version from the image files. mtsconfig mlxconfig enables the users to apply token via MTUSB connected device.

  • Page 14: Mstflint Bug Fixes In This Version

    Keywords: mstfwmanager, pci_cr0, pciconf Discovered in Version: 4.18.0 Fixed in Release: 4.20.0 mstflint Known Issues The following table provides a list of known issues and limitations of mstflint. For a list of old Known Issues, please see Archived Known Issues file. Internal Ref.
  • Page 15 Internal Ref. Issue Keywords: mstlink, LID, InBand, OpenSM Discovered in Version: 4.20.0 2823492 Description: mstfwreset is not supported on DPU with GPU boards. Workaround: N/A Keywords: mstfwreset Discovered in Version: 4.18.0 2715716 Description: mstfwreset is not supported on secure-boot host devices. Workaround: N/A Keywords: mstfwreset Discovered in Version: 4.18.0 2752916 Description: The information of the IB/ETH protocols should not be stored on the same CSV file.
  • Page 16 Discovered in Version: 4.13.3-2 1431471 Description: In ConnectX-5 adapter cards, the time-stamp capability using mstflint, is supported only on the device using the "-d" flag, and not on the binary using the "-i" flag. Workaround: Use the “-d” flag to set the time-stamp.

  • Page 17: Introduction

    Introduction ® The mstflint package is a set of firmware management and debug tools for NVIDIA devices. mstflint can be used for: • Generating a standard or customized NVIDIA firmware image • Querying for firmware information • Burning a firmware image to a single NVIDIA device The list of the available tools in the package can be found in the .Release Notes v4.18.1...

  • Page 18: Compilation And Installation

    In-Band device access requires the local IB port to be in the ACTIVE state and connected to an IB fabric. All mstflint tools address the target hardware device using a PCI device. To list all available PCI devices, use the “lspci” command.

  • Page 19: Firmware Generation, Configuration, And Update Tools

    – Firmware Update and Query Tool The mstfwmanager is a firmware update and query utility which scans the system for available NVIDIA devices (only mst PCI devices) and performs the necessary firmware updates. For further information on firmware update, please refer to Booting HCA Device in Livefish...

  • Page 20: Querying The Device

    Do not use firmware Ctrl update --no-progress Do not show progress -o|--outfile OutputFileName  Write to specified output file --online    Fetch required FW images online from NVIDIA server --online-query-psid PSIDs  Query FW info, PSID(s) are comma separated --key key Key for custom download/update --download DirectoryName  ...

  • Page 21: Archived Images Content

    PCI Device Name: 0000:09:00.0 Port1 GUID: 0002c9000100d051 Port2 MAC: 0002c9000002 Versions: Current Available 2.31.5050 2.32.5000 Status: Update required --------- Found device(s) requiring firmware update. Please use -u flag to perform the update. Query all the devices. Querying Mellanox devices firmware ...  ...

  • Page 22: Mstarchive - Binary Files Compression Tool

    The directory with the binaries files -m|--mfa2-file mfa2_file Mfa2 file to parse  The .mfa2 file can be used with ethtool to burn adapter cards firmware. The procedure is described in mstflint - Updating Firmware Using ethtool/devlink and .mfa2 File section.

  • Page 23: Mstconfig - Changing Device Configuration Tool

    mstconfig – Changing Device Configuration Tool The mstconfig tool allows the user to change some of the device configurations without reburning the firmware. The configuration is also kept after reset. By default, mstconfig shows the configurations that will be loaded in the next boot. For 5th generation devices, it is also possible to query the default configurations and the configurations that are used by the current running firmware.

  • Page 24: Examples Of Mstconfig Usage

    -u|--key_uuid Keypair uuid clear_semaphore Clear the tool semaphore. i[show_confs] Display information about all configurations. q[uery] Query supported configurations. r[eset] Reset all configurations to their default value. s[et] Set configurations to a specific device. set_raw Set raw configuration file.(only Connect-IB/ConnectX-4/Lx) get_raw Gets raw configuration file (5th generation/Group II devices only) backup Backup configurations to a file (only Connect-IB/ConnectX-4/Lx).
  • Page 25   Device type: ConnectX-3 PCI device: 41:00.0 41:00.0 Device 1: ----------- Configurations: Next Boot SRIOV_EN True(1) NUM_OF_VFS WOL_MAGIC_EN_P1 False(0) WOL_MAGIC_EN_P2 False(0)  N/A means that the device default configuration is set.  For Array type parameters, the query command will not show a value for it. It will only show you the word "Array"...

  • Page 26: Using Mstconfig

    # mstconfig -d 41:00.0 reset Reset configuration device 41:00.0? ? (y/n) [n] : y Applying... Done!     -I- Please power-cycle device to load configurations. >mstconfig 41:00.0 query     Device 1: ---------- Device type: ConnectX-3 PCI Device: 41:00.0 Configurations: Next Boot SRIOV_EN True(1)
  • Page 27 Device #1: ---------- Device type: ConnectX4 PCI device: 41:00.0 Configurations: Next Boot SRIOV_EN NUM_OF_VFS   Apply Configuration? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load configurations. Using mstconfig to Set Preboot Settings For a full description of the preboot configurable parameters refer to Supported Configurations and their Parameters under "Preboot Settings".

  • Page 28: Mstconfig Commands

    Data: 0x0000000c Operation intended advanced users.   Are you sure you want to apply raw TLV file? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load configurations.  Never apply files from an unreliable source. mstconfig Commands mstconfig Backup Command The backup command is used to save the current non-volatile configurations (TLV) in the device into a file in raw TLV syntax so it can be restored anytime using the set_raw command.
  • Page 29 nv_debug_mode nv_global_pci_conf In order to include the nv_kdnet_data configuration in the template, change the 0 to 1, as demonstrated in the following example. Example: nv_kdnet_data   #mstconfig gen_xml_template /tmp/confs.txt /tmp/template.xml Saving output... Done!   #cat /tmp/template.xml <?xml version="1.0" encoding="UTF-8"?> <config xmlns="http://www.mellanox.com/config"> <nv_kdnet_data>...

  • Page 30: Mstconfig Supported Configurations And Parameters

    mstconfig create_conf Command The create_conf command assists in creating an NV configuration file that can be used for LifeCycle and Secure Firmware Updates purposes. The flow for creating a configuration file is the same as the flow for xml2bin. The user must provide the command with an XML file containing the required configurations and their values.

  • Page 31: Mstflint - Firmware Burning Tool

    Note that the textual values are case insensitive (either “True” or “true” are accepted). mstflint – Firmware Burning Tool The mstflint (Flash interface) utility performs the following functions: • Burns a binary firmware image to the Flash device attached to an adapter or a switch device.
  • Page 32 -blank_guids Burn the image with blank GUIDs and MACs (where applicable). These values can be set later using the "sg" command (see details below). Commands affected: burn -clear_semaphore Force clear the flash semaphore on the device. No command is allowed when this flag is used.
  • Page 33 NOTE: This flag is intended for advanced users only. Running in this mode may cause the firmware to hang. Run a quick query. When specified, mstflint will not perform full image integrity checks  during the query operation. This may shorten execution time when running over slow ...

  • Page 34: Command Parameters

    Non interactive mode - assume answer "yes" to all questions. Commands affected: all Note 1. The -mac and -macs options are applicable only to NVIDIA Ethernet adapter and switch devices. Note 2. When accessing SwitchX via I2C or PCI, the -override_cache_replacement flag must be set.

  • Page 35: Mstflint: Burning A Firmware Image

    Burning a Firmware Image The mstflint utility enables you to burn the Flash from a binary image.To burn the entire Flash from a raw binary image, use the following command line: # mstflint -d <device> -i <fw-file> [-guid <GUID> | -guids <4...
  • Page 36 To burn a firmware image: Update the firmware on the device, keeping the current GUIDs and VSD. (Note: This is the common way to use the tool. # mstflint -d 41:00.0 -i fw-4099-2_42_5000-MCX354A-FCB_A2.bin burn Update the firmware on the device, specifying the GUIDs to burn.
  • Page 37 The device flash MUST have all relevant device information (signatures, PSID, VPD, DEV_INFO, MFG_INFO, etc.) valid since MFA2 format does not have that information and without the burn process will fail. #mstflint -d <device> -i <mfa2 file> --psid <PSID string> (optionally) -- latest_fw (optionally) –silent (optionally) b (or burn) •...
  • Page 38 In large clusters the script can be executed on multiple hosts, each handling a different part of the cluster. Cable Burn Command # mstflint -d <device> --linkx <flags> <commands> where: Flags: <device>...
  • Page 39 Query misc. flash/firmware characteristics. Updating the Firmware Burning a firmware cable transceiver connected to the host is done using the "mstflint" tool. To do so, the user should use the "–linkx" flag. Firmware can be burnt in follow one of the methods: •...
  • Page 40 Querying a firmware cable transceiver connected to the host is done using the "flint" tool. To do so, the user should use the flag. "–linkx" # mstflint -d <device> --linkx --downstream_device_ids <ids> [--output_file <file_name>] q Query ports 1,2,5 Example: # mstflint -d <device> --linkx --downstream_device_ids 1,2 The system responds with information about the firmware version loaded into the transceivers.

  • Page 41: Mstflint: Managing An Expansion Rom Image

    For example: the sections that are changed when performing a firmware upgrade. To perform a checksum on the flash, run the following command line:# mstflint -d <mst device> checksumTo perform a checksum on a firmware image, run the following command line: # mstflint -i <image file>...

  • Page 42: Mstflint: Querying The Firmware Image

    • Default Update Method" field in Query Full output:{This field reflect the method which mstflint will use in order to update the device. The user can enforce a different method using the –no_fw_ctrl or the –ocr flags.The default methods are: •...

  • Page 43: Mstflint: Setting Guids And Macs

    On 4th generation/Group I devices, the “sg” command can operate on both the image file and the image on the flash. When running the “sg” command on an image on the flash, if the GUIDs/ MACs/UIDs in the image are non-blank, the mstflint will re-burn the current image using the given GUIDs/MACs/UIDs.
  • Page 44 GUIDs: f45214030001b8a0 f45214030001b8a1 f45214030001b8a2 f45214030001b8a3 MACs: 00300abadaba 00300abadabb VSD: PSID: MT_1090120019 # mstflint -i /tmp/image.bin -guid 0002c9000abcdef0 -mac 02c90abcdef0 sg You are about to change the Guids/Macs/Uids on the device: New Values Current Values Node GUID: 0002c9000abcdef0 f45214030001b8a0 Port1 GUID:...
  • Page 45 Device VSD: PSID: MT_1240110019 # mstflint -d 41:00.0 -uid 0000000000000001 -ocr sg guids_num=2 step_size=1 -W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang. Updating GUID section - OK Updating ITOC section - OK...
  • Page 46 In this method, the image is generated with blank GUIDs and CRCs. The GUIDs are set after the device is assembled using the mstflint "sg" command. To set GUIDs take less than 1 second when running on an image with blank GUIDs (through a PCI device).

  • Page 47: Mstflint: Verifying The Firmware Image

    -i ./fw-ConnectX3-rel.bin -guid 0x0002c9030abcdef0 -mac 0x0002c9bcdef1 -striped_image sg (Optional) After assembly, query the image on flash to verify there are no GUIDs on the device. sg# mstflint -i ./fw-ConnectX3-rel.bin -striped_image q Image type: FW Version: 2.31.5050 FW Release Date: 4.5.2014 Device ID:...

  • Page 48: Mstflint: Setting The Vsd

    MFA2 archive does not have the dTOC information by its definition, the binary comparison will ignore the device specific sections on the device. #mstflint -d <device> -i <fw image> --silent (optional) bc (or binary_compare) Verifying MFA2 Archive Binary verifying of MFA2 archive enables the user to verify that a given MFA2 archive contains the image that matches the given device.

  • Page 49: Mstflint: Disabling/Enabling Access To The Hardware

    Disabling/Enabling Access to the Hardware The secure host feature enables ConnectX family devices to block access to its internal hardware registers. The hardware access in this mode is allowed only if a correct 64 bits key is provided. ...

  • Page 50: Mstflint: Flash Operations

    # mstflint -d 41:00.0 0x20 Writing a dword to Flash To write one dword to Flash memory, use the following command line: # mstflint -d <device> ww addr data where: device The device the dword is written to. addr The address of the word to write.
  • Page 51 0x10008 0x5a445a44 Writing a dword to Flash Without Sector Erase To write one dword to Flash memory without sector erase , use the following command line: # mstflint -d <device> wwne addr data where: device The device the dword is written to..

  • Page 52: Mstflint: Comparing The Binary Image

    Setting a Timestamp on Image In order to set a timestamp on an image, run: # mstflint -i ./fw-4115.bin timestamp set [UTC time]  The user can either specify a combined date and time timestamp in UTC which conforms to ISO 8601, or let the tool use the machine’s time for the timestamp.
  • Page 53 In case it is not possible to modify the firmware image, it is possible to set the timestamp directly on the device by specifying the timestamp and firmware version tied to it. # mstflint -d /41:00.0/mt4115_pciconf0 timestamp set <UTC time> <Firmware version> Querying a Timestamp on Device To view the timestamp that was set on the device, run: # mstflint -d /41:00.0/mt4115_pciconf0 timestamp query...

  • Page 54: Mstflint Limitations

    Using Secure Host Secure Host feature is supported for all NVIDIA network adapters (listed in Group 1 and group 2). For group 1 network adapters, the user is required to generate and burn a firmware image that supports the feature (see “Generating/Burning a Firmware Supporting Secure Host”...
  • Page 55 Access the hardware while hardware access is disabled: # mstflint -d 41:00.0 E- Cannot open 41:00.0: HW access is disabled on the device. E- Run "mstflint -d 41:00.0 hw_access enable" in order to enable HW access. Enable hardware access: # mstflint -d 41:00.0...

  • Page 56: Mstflint: Secure Firmware Update

    The firmware rejects binaries that do not match the verification criteria. Secure FW utilizes the same ‘fail safe’ upgrade procedures, so events like power failure during update should not leave the device in an unstable state. The table below lists the impact of secure FW update on mstflint tools. Tool Flow...
  • Page 57 The motivation for signing with two keys is to allow a firmware update from both firmwares, the one that supports only 2048bit keys and the one that supports 4096bit keys. Examples: # mstflint -i /tmp/image.bin sign --private_key privatekey.pem --key_uuid "e0129552-13ba-11e7-a990-0cc47a6d39d2" # mstflint -i /tmp/image.bin sign --private_key privatekey_2048.pem --key_uuid "e0129552-13ba-11e7-...
  • Page 58 Files.If the security type permits legacy flash access commands, the --no_fw_ctrl flag can be used to command the mstflint to work in the non firmware controlled mode. This means that all the non-secure functionality will be supported using this flag, and the burn flow will work without requiring a signed image.Example:...
  • Page 59 To generate 4096 bits public keys, please follow the same steps as above, but use "file_public_key_4096" instead of "file_public_key".  For further information, see mstconfig xml2bin Command. Set the key's binary file in the firmware image using the mstflint set_public_keys command.  For further information, see Setting a “Public Keys” Section in a Binary Image File.

  • Page 60: Mstfwreset - Loading Firmware On 5Th Generation Devices Tool

    To sign with a 2048 bits private key only, make sure that the firmware image does not contain a 4096 bits key signature. Run the mstflint set_public_keys command with a 4096 bits keys section filled with zeros and then sign with a 2048 bits private key.

  • Page 61: Reset Levels And Types

    -t|--type <0,1> Run reset with the specified reset-type -m|--mst_flags Provide mst flags to be used when invoking mst restart step. For example: -- MST_FLAGS mst_flags="--with_fpga" -y|--yes  Answer “yes” on prompt --skip_driver| -s Skip driver start/stop stage (driver must be stopped manually) -v|--version Print tool version -h|--help ...

  • Page 62: Mstfwreset For Socket-Direct Nics

    mstfwreset for Socket-Direct NICs To reset the firmware on a socket-direct NIC, run the tool on all PCI devices related to the same NIC with function 0 simultaneously. See the following example on a Linux OSs: $ lspci -d 15b3: 08:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] 08:00.1...

  • Page 63: Mstfwreset Limitations

     • When running the reset command without specifying a reset level the minimal reset level will be performed. • When running the reset command without specifying a reset type the default reset type would be 0 (Full chip reset). To reset a device with a specific reset level to load new firmware, use the following command line: # mstfwreset -d 41:00.0...

  • Page 64: Tool Requirements

    # mstcongestion [option] [-d|--device <PCI DEVICE>] [--mode <MODE>] [--action <ACTION>] [-q|--query] [-h|-- help] [-v|--version] where: -d|--device <PCI DEVICE> NVIDIA PCI device address --mode <MODE>   Set Mode, options are: [aggressive | dynamic] --action <ACTION>   Set Action, options are: [disabled | drop | mark] Note: The “mark”...

  • Page 65: Disabling/Enabling Access To The Hardware

    -v, --version     Shows program's version number and exit --device DEVICE, -d DEVICE  Device to work with. --disable_rshim  When TRUE, the host does not have an RSHIM function to access the embedded CPU registers --disable_tracer When TRUE, the host will not be allowed to own the Tracer --disable_counter_rd When TRUE, the host will not be allowed to read Physical port counters ...

  • Page 66: 5Th Generation Devices

    # mstflint -d 41:00.0 set_key 22062011 Setting the HW Key - OK Restoring signature - OK  A driver restart is required to activate the new key. Access the HW while HW access is disabled: # mstflint -d 41:00.00 E- Cannot open 41:00.0: HW access is disabled on the device.
  • Page 67 # mstflint -d 41:00.0 hw_access enable 18022018 -I- The Secure Host was disabled successfully on the device. And the same as previous, providing the key can be done in interactive shell: # mstflint -d 41:00.0 hw_access enable Enter Key : ********...

  • Page 68: Debug Utilities

    Debug Utilities This section contains: • mstfwtrace Utility • mstregdump Utility • mstreg Utility • mstlink Utility • mstresourcedump Utility • mstresourceparse Utility mstfwtrace Utility The mstfwtrace utility extracts and prints trace messages generated by the firmware running on 5th generation (Group II) devices iRISCs. ...

  • Page 69: Mstregdump Utility

    [0x28edf929623dd] [0xa2] access_reg: register_id=0x9043 mstregdump Utility The mstdump utility dumps device internal configuration registers. The dump file is used by NVIDIA Support for hardware troubleshooting purposes. It can be applied on all NVIDIA devices. mstdregump Usage To run mstregdump: # mstregdump [-full] <mst device> > <dump file>...

  • Page 70: Mstreg Utility

    Registers can be get/set in unknown (RAW) mode by providing register ID and length.  Unknown (RAW) mode is risky as no checks are performed, please consult with NVIDIA support before using it. mstreg Usage mst driver must be started prior to running mstreg tool.
  • Page 71 NCFG PAOS PDDR PMDR PMLP PPAOS PPCNT PPLM PPLR PPRT PPTT PTAS PTYS ROCE_ACCL SBCM SBDCR SBPM SBPR SBSR SLRG SLRP SLTP ..Query a single access register (PAOS): mstreg -d 41:00.0 --show_reg PAOS Field Name | Address (Bytes) | Offset (Bits) | Size (Bits) | Access ========================================================================= oper_status 0x00000000...

  • Page 72: Mstlink Utility

    Set access register data (PAOS in unknown mode (RAW) with indexes: local_port=0x1 swid=0x0 and data e 1): mstreg -d 41:00.0 --reg_id 0x5006 --reg_len 0x10 --indexes "0x0.16:8=0x1,0x0.24:8=0x0" --yes --set "0x4.0:2=0x1" You are about to send access register id: 0x5006 with the following data: Address | Data =======================...
  • Page 73 (e.g. KR-startup) (Optional) --tx_group_map <group_num> Map ports to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices) --ports <ports> Ports to be mapped [1,2,3,4..] --test_mode <prbs_mode>...
  • Page 74 (Optional - Default Nominal) --iteration <iteration> Iteration Number of BER Collection --pc Clear Counters --set_external_phy Set External PHY Note: The flag is supported in NVIDIA Spectrum switch systems only. --twisted_pair_force_mode Twisted Pair Force Mode [MA(Master)/SL(Slave)] <twisted_pair_force_mode> --cable Perform operations on the cables  ...
  • Page 75 --offset <offset> Specific page offset to read/write      --write  <bytes> Perform write operation with specific data (list of bytes, separated by ',')      --prbs_select  <side> Module PRBS test mode side selector [MEDIA, HOST]        --prbs_mode <cmd> Perform PRBS test mode on the Module [EN(Enable),DS(Disable)] Set PRBS generator pattern...
  • Page 76         --rx_amplitude <value> Set Module Rx Output Amplitude [0(100-400mV),1(300-600mV),2(400-800mV),3(600-1200 mV)] --margin Read the SerDes eye margins per lane --measure_time <time> Measure time in seconds for single eye [10, 30, 60, 90, 120, 240, 480, 600 and 900] (Optional - Default 60 for PCIe and 30 for Network ports) --eye_select <eye_sel> ...
  • Page 77 Configure FEC: mstlink -d <device> -p <port_number> --fec RS Configure Port for Physical Test Mode: mstlink -d <device> -p <port_number> --test_mode EN (--rx_prbs PRBS31 --rx_rate 25G --tx_prbs PRBS7 --tx_rate 10G --invert_rx_polarity --invert_tx_polarity) Perform PRBS Tuning: mstlink -d <device> -p <port_number> --test_mode TU ...

  • Page 78: Margin Scan Tool

    -d <device> --show_module --json To show ports group map (for NVIDIA Quantum and NVIDIA Spectrum-2): mstlink –d<device> --show_tx_group_map 0 To assign ports to a specific group on NVIDIA Quantum and NVIDIA Spectrum-2  mstlink –d <device> --tx_group_map 1 –ports 1,2,3,5,4,8,7,8,9,10,11 Margin Scan Tool The margin scan tool is used for scanning PCIe [Gen4 speed] or Network ports [EDR\25G or HDR\PAM4 speeds]. ...

  • Page 79: Rx-To-Tx Loopback Mode Activation

    Clear the loopback configuration using the "NO loopback" option.  mstlink -d /dev/mst/mt4123_pciconf0 –loopback NO Enable the port.  mstlink -d /dev/mst/mt4123_pciconf0 –port_state UP Tool Usage with NIC vs. Switch (-p Flag) When using mstlink on NVIDIA Quantum HDR switch split ports, make sure you use the split formula below to calculate the port number:...

  • Page 80: Mstresourcedump Utility

    Any mstlink command for switch should include the "-p" flag to address the specific port in the switch. • When working with the NIC, if an MTUSB is used for communication with the NVIDIA NIC, to address port 2, use mstlink -d /dev/mst/mt4115_pciconf0 --gvmi_address <0xAddress>.

  • Page 81: Mstresourcedump Usage

    mstresourcedump Usage mstresourcedump [-h] [-v] {dump,query} where dump Dump command query Query command -h, --help Show help message and exit -v, --version Shows tool version and exit mstresourcedump query Usage  mstresourcedump query [-h] [--virtual-hca-id VIRTUAL_HCA_ID] --device DEVICE where -h, --help Show help message and exit --virtual-hca-id ...
  • Page 82 mstresourcedump dump Usage  mstresourcedump dump [-h] --device DEVICE --segment SEGMENT [--virtual-hca-id VIRTUAL_HCA_ID] [--index1 INDEX1] [-- index2 INDEX2] [--num-of-obj1 NUM_OF_OBJ1] [--num-of-obj2 NUM_OF_OBJ2] [--depth DEPTH] [--bin BIN] where -h, --help Show help message and exit --virtual-hca-id  The virtual HCA (host channel adapter, NIC) ID --index1 The first context index to dump (if supported for this segment) --index2...

  • Page 83: Mstresourceparse Utility

    Utility The mstresourceparse tool parses and prints data segments content. The parser's output is used by NVIDIA representatives for debugging and troubleshooting.  The tool applicable inputs for parsing can be the mstresourcedump outputs (bin file or the “human readable”...
  • Page 84 hw_version =   fw_version = 0x1063232c   --------------------------------------------------------------------------------   Segment - segment_command (0xfffa)   segment_header.segment_type = 0xfffa   segment_header.length_dw =   vhca_id =   segment_called = 0x2000   index1 = 0x21   index2 =   num_of_obj1 =   num_of_obj2 =  ...
  • Page 85 index1 = 0x21   index2 =   num_of_obj1 =   num_of_obj2 =   RAW DATA:   DWORD [0-3] :0x0005FFFA 0x20000000 0x00000021 0x00000000   DWORD [4] :0x00000000   --------------------------------------------------------------------------------   Segment - segment_notice (0xfff9)   segment_header.segment_type = 0xfff9   segment_header.length_dw =  ...

  • Page 86: Troubleshooting

    Troubleshooting You may be able to easily resolve the issues described in this section. If a problem persists and you are unable to resolve it yourself please contact your NVIDIA representative or Support. General Related Issues Issue Cause Solution Adapter is no longer identified by Happens due to burning the wrong Power cycle the server.

  • Page 87: Installation Related Issues

    Issue Cause Solution After resetting configuration Firmware loads the default Reboot the server using the tool on 5th generation configuration only upon (Group II) devices, the reboot configuration's value does not change Installation Related Issues Issue Cause Solution Unable to install the tool package Insufficient privileges Copy the tool's package to / on ESXi platform and the following...
  • Page 88 Add “-ocr” option to the 'mstflint' command. -E- Cannot open Device: 41:00.0. For example: B14 Operation not permitted mstflint -d 41:00.0 -ocr q MFE_CMDIF_GO_BIT_BUSY The following message is reported Updating only the EXP_ROM Allow “-allow_rom_change” option on screen when trying to remove (FlexBoot) for recent firmware to the “mstflint”...

  • Page 89: Secure Firmware Related Issues

    Issue Cause Solution Changing device setting such as ROM/ GUIDS Secure Firmware does not allow using the relevant mstflint commands result in changes to the device data unless failure with the following error: burning new Secure Firmware image. -E- <Operation> failed: Unsupported operation...
  • Page 90 • The FW image is not secured • The key is not applicable Secure Firmware fails when using mstflint mstflint brom and drom commands brom and drom commands. are not supported. When the CR space is in read only mode, the...

  • Page 91: Appendixes

    Appendixes • mstflint - Updating Firmware Using ethtool/devlink and .mfa2 File mstflint - Updating Firmware Using ethtool/devlink and .mfa2 File In order to flash the firmware on the device using ethtool, you need to prepare a .mfa2 firmware file using the mstarchive tool - see mstarchive –...
  • Page 92 For the firmware update to take effect, you need to either reboot the server or run: # mstfwreset -d 04:00.0 -y r Validate the firmware update by a query. Using mst: # mstflint -d 04:00.0 q Image type: Image type: FW Version: 16.26.0292 FW Release Date: 15.5.2019...

  • Page 93: Mstflint User Manual Revision History

    User Manual Revision History Revision Date Description 4.20.0 April 30, 2022 • Updated mslink commands, added cable operations. • Updated mstprivhost, added Embedded Arm CPU options.  • Updated mstresourcedump, added the "–mem" key. 4.18.0 November 30, • Updated mstlink Utility 2021 •...
  • Page 94 Revision Date Description • mstfwtrace Utility: Added the following options: • "--gvmi" and "--ignore_old_events" • Using mstconfig to Split a Port in a Remotely Managed Switch Mar. 2019 Converted to online html format; some reorganization. Dec. 2, 2018 • Added the following sections: •...
  • Page 95 Revision Date Description EXP_ROM_UEFI_ARM_ENABLE HOST_CHAINING_MODE HOST_CHAINING_DESCRIPTORS HOST_CHAINING_TOTAL_BUFFER_SIZE • Using Secure Host...

  • Page 96: Release Notes Revision History

    Note: If the NIC driver is unloaded, burning via DMA is unsupported (due to BME is unset) and regular burn flow will be executed instead.  Note: This capability is supported in 5th Generation devices only. Note: This capability requires mstflint kernel module to work. Bug Fixes See Bug Fixes.
  • Page 97 mstlink Added supported for switching between NRZ/PAM4 speeds for new devices that support HDR/200G speeds (ConnectX-6, ConnectX-6 Dx, Quantum, Spectrum-2). Rev. 4.13.3-2 Burning MFA2 Enables the user to extract (i.e. unzip) 4MB images from MFA2 archive, Images that matches the device type and device PSIB. If there are more than one matching image, the user may use –latest flag and burn the latest firmware, or choose the required image from...
  • Page 98 Exposes supported access registers, and allows users to obtain information regarding the registers fields and attributes, and to set and get data with specific register. mstflint Added image-reactivation feature which re-assigns the image signature to the previous image (in case the new image is faulty) enabling "fwreset"...
  • Page 99 General Removed the COMFIG COMPACT definition.  Linux General Added support for libibmad 12. Linux mstconfig Renamed the BOOT_RETRY_CNT1 parameter to BOOT_RETRY_CNT. Bug Fixes mstflint Bug Fixes History Rev. 4.11.0-5 mstconfig Added the below new params: • pci_atomics_disabled_ext_atomics_enabled • pci_atomics_enabled_ext_atomics_enabled •...
  • Page 100 Added compatibility to 1.1.X version, in addition to previously supported Compatibility 1.0.2 version. Rev. 4.11.0 .deb Package As of mstflint v4.12.0, the name of *.deb files will be changed from Name “mstflint-<version>.amd64.deb” to “mstflint_<version>_amd64.deb” e.g., from mstflint-4.10.0-104.amd64.deb to mstflint_4.10.0-104_amd64.deb Supported Devices Added support for Mellanox Quantum switch systems and ConnectX-6 Ready adapter cards.
  • Page 101 Note: On Multi Host devices with firmware version lower than 1x.23.xxxx, the flag "--pci_link_downtime 2.5" must be added to mstfwreset Secure Firmware mstflint now handles all the burn parameters when MCC is enabled and displays the secure-FW CS tokens. Supported Devices [Beta] Added support for BlueField SmartNIC.
  • Page 102 Secure Firmware Added support for Secure Firmware Update to ConnectX-4 adapter cards. Update Enabled signing the package with an RSA 4096 bit keys. Added support for setting the GUIDs when Secure Firmware Update is enabled. mstconfig Added the following mstconfig configuration parameters: •...
  • Page 103 Linux Added a flag to enforce working in a non-secure mode, if available (according to security type). Added expansion ROM CPU architecture to the mstflint query when the expansion ROM is available. mstlink Added a new tool that displays and configures port related data at the physical layer.
  • Page 104 General Added support for Innova IPsec 4 Lx EN /Innova Flex 4 Lx EN Linux mstflint package size has been reduced in Linux by separating the Linux / architecture specific RPMs, and in ESXi, by moving relevant tools to the ESXi OEM package.
  • Page 105 Linux Windows and queries the cables info FreeBSD Build Created one mstflint package for all 64 bits FreeBSD OSs FreeBSD mstfwmanager_pci Removed support for mstfwmanager_pci tool (it is deprecated), since all Linux the Linux tools can work without a kernel now. When required, use mstfwmanager instead.
  • Page 106 Allow RD counters • IP protocol used by flexboot Added the option to display the configuration’s default values. mstflint Added support to calculate checksum on selected sections in the firmware image. Added the option to attach a timestamp to the firmware image.
  • Page 107 • Wake on LAN Added support for the following configurations in ConnectX-3 and Linux/ ConnectX-3 Pro: Windows/ FreeBSD • InfiniBand Boot Settings • Preboot Boot Settings msttrace Added support for MEM mode in ConnectX-4 Windows cpld_update Added the cpld_update tool to the OEM package Linux mstfwreset Added support for resetting the firmware...
  • Page 108 Rev. 3.6.1 mstconfig Added mstconfig tool for changing non volotaile configuration on device Windows Burning Tools Added support for micron flash in mstflint and updated production burn Windows flow on Connect-IB Rev. 3.6.0 mstconfig Added mstconfig tool for changing non volatile configuration on device...
  • Page 109 Redesigned the utility to make its look and feel more user friendly VMware ESXi Added support for ConnectX-3 Pro VMware ESXi Rev. 3.5.0 mstflint/wqdump Redesigned the mstflint and wqdump utility to make their look and feel Linux/Windows more user friendly mstflint Added support for brom in Connect-IB Linux/Windows...
  • Page 110 It is no longer required to run mst start/stop when using Winmstflint Windows tools. The service is automatically loaded/unloaded when an mstflint tool is running. The mst service installation was removed from the setup Added support for SwitchX silicon devices...
  • Page 111 Windows • The ROM section in the image now contains multiple boot Windows images. Therefore mstflint was modified to display information for all of the images in the ROM section. • Added support to display/burn UEFI ROM/ Added an option to set the VSD and GUIDs in a binary image file. This is Windows useful for production to prepare images for pre-assembly flash burning.
  • Page 112 The ROM section in the image now contains multiple boot images. Linux Therefore the mstflint was modified to display information for all of the images in the ROM section Added support to display/burn UEFI ROM...
  • Page 113 Linux installation without-kernel which allow for a partial installation in order to avoid change problems with SW dependencies Now allows a non-root user to prepare mstflint RPMs Linux Added Mellanox ConnectX-2 and BridgeX support Linux/Windows mstflint Added a CRC check for the full image...

  • Page 114: Mstflint Bug Fixes History

    Updated the msti2c utility Linux Added the mget_temp utility which reads the temperature of the Linux ConnectX/ConnectX-2, InfiniScale IV, and BridgeX devices mstflint Bug Fixes History The table below lists the history of bugs fixed.  Internal Ref. Issue 2578580 Description: Fixed an issue that resulted in getting MVPD read errors from the mstfwmanager during fast reboot.
  • Page 115 Description: mstflint is currently not in ConnectX-6 Lx adapter cards. Keywords: mstflint Discovered in Version: 4.15.0 Fixed in Release: 4.17.0 2183083 Description: mstflint does not support using combined short flags without a separation between them. For example:  • Not recommended: -emc •...
  • Page 116 Discovered in Version: 4.14.0-1 Fixed in Release: 4.16.0 2110890 Description: If there is a json-c library installed on the machine, some of mstflint tools like mstflint and mstfwmanager will not be compiled successfully unless the installed library is removed. Keywords: mstlink Discovered in Version: 4.14.0-1...
  • Page 117 Internal Ref. Issue 2084837 Description: Setting the speeds (50GbE and 100GbE) for the new devices (Connect-X 6 and above, Quantum switches and above) requires specifying the number of lanes for the speed: mstlink -d <dev> --speeds [50G_2X | 50G_1X | 100G_2X | 100G_4X] For PRBS mode, to work with PAM4 speeds, use the same speed naming for (50GbE, and 100GbE).
  • Page 118 Keywords: mstflint Discovered in Version: 4.14.0-105 Fixed in Release: 4.16.0 2151018 Description: Occasionally, when burning MFA2 using mstflint, it might get stuck if in the middle of the process mstfwreset is executed.  Keywords: MFA2, mstflint Discovered in Version: 4.15.0 Fixed in Release: 4.16.0 2193807 Description: Cable firmware burning capability is not supported.
  • Page 119 Keywords: mstlink Discovered in Version: 4.13.0 Fixed in Release: 4.14.0-1 1797470 Description: Fixed an issue that prevented the mstflint tools from recognizing the second port on dual port devices. Keywords: mstflint, dual port devices Discovered in Version: 4.12.0 Fixed in Release: 4.13.0...
  • Page 120 Issue Fixed in Release: 4.13.0 1655224 Description: Decreased mstflint query timeout from 80 seconds to 8 seconds. In case the tool does not get a response from the device after 8 seconds, the following error message is displayed: "Cannot open Device: /dev/mst/mt4117_pciconf0. Resource unavailable".
  • Page 121 Internal Ref. Issue Fixed in Release: 4.8.0 1097425 Description: mstfwmanager does not handle Socket Direct adapters correctly. Keywords: mstfwmanager Discovered in Release: 4.7.0 Fixed in Release: 4.8.0 907531 Description: mstfwreset is not functional on MultiHost and Socket Direct NICs. Keywords: mstfwreset Discovered in Release: 4.6.0 Fixed in Release: 4.7.0 969322/...
  • Page 122 Internal Ref. Issue Discovered in Release: 4.4.0 Fixed in Release: 4.5.0...
  • Page 123 NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.

Table of Contents