Xilinx Alveo X3522 User Manual
  1. Manuals
  2. Brands
  3. Xilinx Manuals
  4. Computer Hardware
  5. Alveo X3522
  6. User manual

Xilinx Alveo X3522 User Manual

Hide thumbs Also See for Alveo X3522:
Table of Contents

Advertisement

Quick Links

Alveo X3522
User Guide
UG1523 (v1.0) October 18, 2022
Xilinx is creating an environment where employees, customers, and
partners feel welcome and included. To that end, we're removing non-
inclusive language from our products and related collateral. We've
launched an internal initiative to remove language that could exclude
people or reinforce historical biases, including terms embedded in our
software and IPs. You may still find examples of non-inclusive
language in our older products as we work to make these changes and
align with evolving industry standards. Follow this
link
for more
information.

Advertisement

Table of Contents
loading

Related Manuals for Xilinx Alveo X3522

Summary of Contents for Xilinx Alveo X3522

  • Page 1 Alveo X3522 User Guide UG1523 (v1.0) October 18, 2022 Xilinx is creating an environment where employees, customers, and partners feel welcome and included. To that end, we’re removing non- inclusive language from our products and related collateral. We’ve launched an internal initiative to remove language that could exclude people or reinforce historical biases, including terms embedded in our software and IPs.

  • Page 2: Table Of Contents

    Basic Adapter Configuration....................22 UEFI and PXE..........................23 Precision Time Protocol......................24 Chapter 5: Tuning ......................25 Huge Pages..........................25 Interrupt Affinity........................28 Dropping Multicast Traffic......................34 Performance Tuning on Linux....................36 UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 3 Driver Logging Levels....................... 60 Appendix A: Additional Resources and Legal Notices ......61 Xilinx Resources.........................61 Documentation Navigator and Design Hubs.................61 References..........................61 Revision History......................... 62 Please Read: Important Legal Notices................... 63 UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 4: Chapter 1: Introduction

    IMPORTANT! Before reading this guide, follow the instructions in Alveo X3522 Installation Guide (UG1522) to install and configure the network adapter and its drivers. This guide assumes that you have already installed the X3522 and that it can pass network traffic.

  • Page 5: Block Diagram

    ✓ Double-bit error detection Diagnostics ✓ Initiate self-generated traffic and self-test ✓ Debug bus Block Diagram A block diagram of the X3522 is shown in the following figure. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 6: Terms And Acronyms

    Transfers data on rising and falling edges of the clock. DSFP Dual Small Form-factor Pluggable. A 2-channel (dual) network transceiver design that is compatible with SFP+/ SFP28 transceivers. See https://dsfpmsa.org/. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 7 NUMA Non-Uniform Memory Access. A multiprocessing design where memory is local to clusters of processors. Onload A high performance user-level network stack from Xilinx, which accelerates TCP and UDP network I/O. PCIe Peripheral Component Interconnect Express. A high-speed serial bus used to connect components.
  • Page 8 Chapter 1: Introduction Table 2: Terms and Definitions (cont'd) Term Definition XCUX35 The FPGA used on an X3522. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 9: Chapter 2: X3 Migration Quickstart

    Xilinx. • TCPDirect is now supplied in its own package, separately from Onload. See the Alveo X3522 Installation Guide (UG1522), the Onload User Guide (UG1586) and the TCPDirect User Guide (SF-116303-CD). UG1523 (v1.0) October 18, 2022 www.xilinx.com...
  • Page 10 • Power draw varies more with traffic load. • The X3522 uses more power than the X2-series. • Its airflow requirements are consequently higher. See the Alveo X3522 Data Sheet (DS1002). DSFP28 Cages • The X3522 has two DSFP28 cages, each of which supports two ports.

  • Page 11: Software Migration

    • You must filter in software for unwanted data, including non-UDP/TCP frames. • Some example applications do not support the X3522. See the Alveo X3 ef_vi Conversion Guide (XN-201257-CD) and the ef_vi User Guide (SF-114063- CD). UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 12 Migration • The X3522 needs an updated version of sfptpd that recognizes its new PCIe vendor ID. • It will then be treated in the same way as other Xilinx/Solarflare NICs. • No changes are needed to configuration. See the Enhanced PTP User Guide (UG1602).

  • Page 13: Chapter 3: What's New

    On predecessors to the X3522 such as the X2522, packets are received into a 4 KB standard page: • Applications own and manage the packet buffers, and post them to the NIC. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 14 • The driver handles all the receive events and buffer management leaving receivers free to detect presence of new packets by watching the packet buffer memory for changes instead of consuming events. See the following figure: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 15: Transmit

    The intention is that every sender can have a dedicated CTPIO aperture and so avoid clashes. If CTPIO fails, fallback automatically occurs to store-and-forward, which will always succeed. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 16 As it is fully cut-through, the X3522 does not support TCP/IP checksum offload on transmit. Like its predecessors, the X3522 sends a poisoned frame on failure, but its improved design means this is significantly less likely to occur. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 17: Interrupts

    Ports The X3522 adapter has a single port per cage when using SFP cables or a pair of ports per cage with DSFP cables: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 18: Drivers

    # modprobe -r xilinx_efct # modprobe -r auxiliary # modprobe auxiliary # modprobe xilinx_efct # onload_tool reload --onload-only Note: If you are also using the sfc driver, omit the --onload-only flag. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 19: Features

    FPGA, meaning it can be easily customized to add new features. Tuning Changes in the design and behavior of the X3522 mean that it might require different system tuning settings from its predecessors. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 20 The net driver must check every frame, even those destined for Onload. ○ Configuring all-but-one core isolated from kernel threads is no longer viable for higher ○ traffic rates. Chapter 5: Tuning. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 21: Chapter 4: Basic Configuration

    • The LEDs for cage 1 show the status of ports 0 and 2. • The LEDs for cage 2 show the status of ports 1 and 3. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 22: Basic Adapter Configuration

    Ethtool is a standard Linux tool to view, set, and change Ethernet adapter settings. ethtool <-option> <interface> You can view settings without root permission. For example: • To identify the current configuration settings for the interface: ethtool <interface> UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 23: Uefi And Pxe

    The X3522 supports UEFI. It is enabled in the delivered factory configuration. Note: To change whether UEFI is enabled, you must program new configuration settings into the FPGA flash. See Chapter 6: Programming the X3522. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 24: Precision Time Protocol

    PXE. The host can then use the UEFI functions of the X3522 NIC to run PXE over that NIC. Precision Time Protocol Customers requiring configuration instructions for these adapters and Xilinx PTP in a PTP deployment should refer to the Enhanced PTP User Guide (UG1602). UG1523 (v1.0) October 18, 2022 www.xilinx.com...

  • Page 25: Chapter 5: Tuning

    1961 PCI-MSI 524320- edge enp1s0f0np2-rx-0 2230 PCI-MSI 524289- edge enp1s0f0np0-rx-1 1822 PCI-MSI 524321- edge enp1s0f0np2-rx-1 2231 PCI-MSI 524290- edge enp1s0f0np0-rx-2 2183 PCI-MSI 524322- edge enp1s0f0np2-rx-2 2184 PCI-MSI 524291- UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 26 So for this example, you would require a minimum of: 16 × (ROUNDUP (2048 ÷ 1024) + 1) which is 48 huge pages. You might later increase this value when tuning your application for best performance. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 27 1. Check the current 2048 KB huge page allocation by inspection of the files in /sys/ kernel/mm/hugepages/hugepages-2048kB: # sh -c 'cd /sys/kernel/mm/hugepages/hugepages-2048kB; grep "" *' free_hugepages:0 nr_hugepages:0 nr_hugepages_mempolicy:0 nr_overcommit_hugepages:0 resv_hugepages:0 surplus_hugepages:0 In the output: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 28: Interrupt Affinity

    IMPORTANT! For affinity tuning to succeed the irqbalance service must be disabled. In all cases, including using Onload, you must follow the instructions in Disable the Irqbalance Service. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 29 Use the following command to view the topology of your processor: $ lstopo-no-graphics --no-legend --no-io --of txt You can then determine which CPU cores share an L3 cache. For example: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 30 • To view the interrupts assigned to a particular interface: $ ls /sys/class/net/<interface>/device/msi_irqs/ For example to view the interrupts assigned to the enp1s0f0np0 interface: $ ls /sys/class/net/enp1s0f0np0/device/msi_irqs/ UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 31 PCI-MSI 49283075-edge enp1s0f0np0-rx-3 122: PCI-MSI 49283076-edge enp1s0f0np0-rx-4 124: PCI-MSI 49283077-edge enp1s0f0np0-rx-5 126: PCI-MSI 49283078-edge enp1s0f0np0-rx-6 128: PCI-MSI 49283079-edge enp1s0f0np0-rx-7 130: PCI-MSI 49283080-edge enp1s0f0np0-tx-00 132: PCI-MSI 49283081-edge onld-0 133: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 32 UDP v4 traffic sent to port 1234 at address 192.168.10.200, then to use receive queue 3: # ethtool -N enp1s0f0np0 flow-type udp4 dst-ip 192.168.10.200 dst-port 1234 queue 3 Added rule with ID 0 UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 33 (see Linux man pages). Use of this call and how a particular application can be tuned is beyond the scope of this guide. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 34: Dropping Multicast Traffic

    -S <interface> will be incremented. The filter can be deleted using its ID to reference it. For example: $ sudo ethtool -N enp2s0f0np0 delete 255 UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 35 This will also be the case for any other layer 2 multicast protocols and IPv6 traffic. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 36: Performance Tuning On Linux

    ○ • Increasing the interrupt moderation interval will: generate less interrupts ○ reduce CPU utilization (because there are less interrupts to process) ○ increase latency ○ UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 37 To turn off interrupt moderation, set an interval of zero (0): ethtool –C <interface> rx-usecs 0 To set the TX interrupt moderation interval: ethtool –C <interface> tx-usecs <interval> UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 38 CPU cache size all influence the affect of the max buffer size values. The minimum and default values can be left at their defaults minimum=4096 and default=87380. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 39: Other Considerations

    On RHEL7 systems, it might be beneficial to disable the tuned service if minimum latency is the main consideration. Users are advised to experiment. The service is controlled via systemctl: systemctl stop tuned systemctl disable tuned Other Considerations This section describes some other considerations. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 40 The lspci command can be used to discover the currently negotiated PCIe lane width and speed: $ sudo lspci -d 10ee: -vv 01:00.0 Ethernet controller: Xilinx Corporation Device 5084 LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported LnkSta: Speed 8GT/s (ok), Width x16 (ok) Note: The supported speed might be returned as 'unknown', due to older lspci utilities not knowing how to determine that a slot supports PCIe Gen.

  • Page 41: Tuning Recommendations

    Leave at default. Interrupt affinity & irqbalance service Interrupt affinity settings are application dependent Stop irqbalance service: /sbin/service irqbalance stop Reload the drivers to use the driver default interrupt affinity. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 42 Tuned service Experiment disabling this with: systemctl stop tuned systemctl disable tuned Memory bandwidth Ensure memory uses all memory channels on system motherboard. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 43 Interrupt affinity & irqbalance service Interrupt affinity. Affinitize each interface to its own CPU (if possible select CPU's on the same package). Refer to Interrupt Affinity. Stop irqbalance service: /sbin/service irqbalance stop UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 44: Chapter 6: Programming The X3522

    • Change the behavior of the X3522 by writing configuration to flash. • Upgrade the X3522 by enabling hybrid mode so that it is programmable. Note: For a description of the initial setup for the Xilinx X3522 and its host machine, see the Alveo X3522 Installation Guide (UG1522).

  • Page 45: Get The Update Files

    Update to Latest Version To update all partitions of the FPGA to the latest version, download one of the following bundles: • XN-201078-LS Alveo X3522 Update Bundle [RPM package], recommended for Red Hat, SUSE or CentOS • XN-201079-LS Alveo X3522 Update Bundle [Debian package], recommended for Ubuntu or Debian.

  • Page 46: Install The Update Files

    • For an RPM package: rpm -ivh <package_name>.rpm • For a DEB package: apt install <package_name>.deb The .update files are installed in the /lib/firmware/xilinx/x3 directory. Program the FPGA The .update file is applied using standard Linux utilities: • The devlink command is preferred.
  • Page 47 The xilinx_efct net driver must be loaded to use either of these commands. Program with devlink To apply a .update file with devlink: 1. List the Xilinx PCIe devices for the X3522: # lspci -D -d 10ee:5084 0000:01:00.0 Ethernet controller: Xilinx Corporation Device 5084 0000:01:00.1 Ethernet controller: Xilinx Corporation Device 5084...
  • Page 48 To apply a .update file with ethtool: 1. Apply the .update file to the first interface on the X3522: # ethtool -f enp1s0f0np0 xilinx/x3/x3-bundle.update The ethtool command hands the filename to the driver, and the driver then calls the firmware to perform the update.

  • Page 49: Confirm Success Of The Update

    Ensure that you are logged in as root before you perform any of these procedures. Confirm Version Information Confirm that the components of the X3522 have the expected versions: 1. Confirm that you can see the Xilinx PCIe devices for the X3522, including one or more Ethernet controllers. # lspci -D -d 10ee: 0000:01:00.0 Ethernet controller: Xilinx Corporation Device 5084...
  • Page 50 You can view the current configuration in the fw.psid field of the output from the devlink dev info command: 1. Confirm that you can see the Xilinx PCIe devices for the X3522, including one or more Ethernet controllers. # lspci -D -d 10ee: 0000:01:00.0 Ethernet controller: Xilinx Corporation Device 5084...
  • Page 51 If devlink is not available, you can instead use lspci running as root: # lspci -d 10ee: -v • If UEFI is enabled then a line is output for the Expansion ROM: 01:00.0 Ethernet controller: Xilinx Corporation Device 5084 Subsystem: Xilinx Corporation Device 000b Flags: bus master, fast devsel, latency 0...

  • Page 52: Chapter 7: Teaming/Bonding

    To set up an adapter team, consult the following documentation: • General: https:/ /www.kernel.org/doc/Documentation/networking/bonding.txt • RHEL 7: https:/ /access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/ networking_guide/ch-configure_network_teaming • RHEL 8: https:/ /access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/ configuring_and_managing_networking/configuring-network-teaming_configuring-and- managing-networking • Ubuntu: https:/ /help.ubuntu.com/community/UbuntuBonding UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 53: Chapter 8: Statistics And Logging

    Total number of transmitted packets dropped by the net driver. port_tx_pause Number of pause frames transmitted with valid pause op_code. port_tx_unicast Number of unicast packets transmitted. Includes flow control packets. port_tx_multicast Number of multicast packets transmitted. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 54 Number of packets received with a length greater than 9000 bytes, but with incorrect CRC value. port_rx_align_error Number of packets received with an align error. port_rx_length_error Number of packets received with a length error. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 55 Number of times the transmit queue has been stopped because space was not available. evq_time_sync_events Number of time sync events received. evq_error_events Number of error events received. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 56: Reading Sensors

    • Alternatively, you can access the raw values via the filesystem. Using the Sensors Command For a formatted view of the sensor values, use the sensors command from the lm_sensors package in your Linux distribution. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 57 • On some platforms, the name for the Ethernet controller instead starts with its interface name (e.g., enp1s0f0np0). • Names usually end with the PCI device and function numbers (e.g.,0100), providing further confirmation. UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 58 So in the above example, the four directories /sys/class/hwmon/hwmon1/device to /sys/ class/hwmon/hwmon4/device (inclusive) contain sensor data from adapters using the xilinx_efct driver. Within these directories, each sensor has the following files: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...
  • Page 59 FPGA, the previous output shows that this information is in the temp2_* files, and so: $ cd /sys/class/hwmon/hwmon1/device $ grep -HT . temp2_* temp2_input: 48000 temp2_label: Ambient (FPGA) temp2_max: temp2_min: UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 60: Driver Logging Levels

    <iface> | grep -A 1 'message level:' Current message level: 0x00007fff (32767) drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol UG1523 (v1.0) October 18, 2022 www.xilinx.com Send Feedback Alveo X3522 User Guide...

  • Page 61: Appendix A: Additional Resources And Legal Notices

    • On Windows, select Start → All Programs → Xilinx Design Tools → DocNav. • At the Linux command prompt, enter docnav. Xilinx Design Hubs provide links to documentation organized by design tasks and other topics, which you can use to learn key concepts and address frequently asked questions. To access the Design Hubs: •...

  • Page 62: Revision History

    Appendix A: Additional Resources and Legal Notices Alveo Documents 1. Alveo X3522 Data Sheet (DS1002) 2. Alveo X3522 Installation Guide (UG1522) 3. Alveo Card Debug Guide (XD027) 4. Alveo Data Center Accelerator Card Platforms User Guide (UG1120) Onload Documents 1. Onload User Guide (UG1586) 2.

  • Page 63: Please Read: Important Legal Notices

    IP cores may be subject to warranty and support terms contained in a license issued to you by Xilinx. Xilinx products are not designed or intended to be fail-safe or for use in any application requiring fail-safe performance; you assume sole risk and liability for...
  • Page 64 Spartan, Versal, Vitis, Virtex, Vivado, Zynq, and other designated brands included herein are trademarks of Xilinx in the United States and other countries. PCI, PCIe, and PCI Express are trademarks of PCI-SIG and used under license. All other trademarks are the property of their respective owners.

Table of Contents