Arria 10 Avalon-ST Interface for PCIe Solutions User Guide Last updated for Quartus Prime Design Suite: 15.1 UG-01145_avst 101 Innovation Drive Subscribe 2015.11.02 San Jose, CA 95134 Send Feedback www.altera.com...
Page 2
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 3
AN 456: PCI Express High Performance Reference Design This example design includes an Avalon-ST interface to the Application Layer. It illustrates chaining DMA performance. You can download this design to an Altera Development Kit and that passes PCI- SIG interoperability tests.
Page 4
The table compares the features for three variants of the Hard IP for PCI Express IP Core. An SR-IOV variant is also available, but not included because it is very specialized product. Consult the Arria 10 Avalon-ST Interface with SR-IOV PCIe Solutions User Guide for features of this IP core.
Page 6
AtomicOp Request FetchAdd The Arria 10 Avalon-ST Interface for PCIe Solutions User Guide explains how to use this IP core and not the PCI Express protocol. Although there is inevitable overlap between these two purposes, use this document only in conjunction with an understanding of the PCI Express Base Specification.
Page 7
Altera verifies that the current version of the Quartus Prime software compiles the previous version of each IP core, if this IP core was included in the previous release. Altera reports any exceptions to this verification in the Altera IP Release Notes or clarifies them in the Quartus Prime IP Update tool. Altera does not verify compilation with IP core versions older than the previous release.
Page 8
• Data Link Layer (DL) • Transaction Layer (TL) The Hard IP supports all memory, I/O, configuration, and message transactions. It is optimized for Altera devices. The Application Layer interface is also optimized to achieve maximum effective throughput. You can customize the Hard IP to meet your design requirements.
Page 9
Debug features allow observation and control of the Hard IP for faster debugging of system-level problems. Related Information Debugging on page 18-1 IP Core Verification To ensure compliance with the PCI Express specification, Altera performs extensive verification. The simulation environment uses multiple testbenches that consist of industry-standard bus functional Datasheet Altera Corporation Send Feedback...
Page 10
Compliance Checklist tests that specifically test the items in the checklist ® • Random tests that test a wide range of traffic patterns Altera provides example designs that you can leverage to test your PCBs and complete compliance base board testing (CBB testing) at PCI-SIG, upon request. Compatibility Testing Environment Altera has performed significant hardware testing to ensure a reliable solution.
Page 11
Table 1-6: Arria 10 Recommended Speed Grades for All Link Widths and Application Layer Clock Frequen‐ cies Altera recommends setting the Quartus Prime Analysis & Synthesis Settings Optimization Technique to Speed when the Application Layer clock frequency is 250 MHz. For information about optimizing synthesis, refer to Setting Up and Running Analysis and Synthesis in Quartus II Help.
Page 12
Aldec RivieraPro, Cadence NCsim, Mentor Graphics ModelSim, and Synopsys VCS and VCS-MX simulators. The Altera testbench and Root Port or Endpoint BFM provide a simple method to do basic testing of the Application Layer logic that interfaces to the variation. However, the testbench and Root Port BFM are not intended to be a substitute for a full verification environment.
Page 13
Creating a Design for PCI Express 2015.11.02 5. Download your design to an Altera development board or your own PCB. Click on the All Develop‐ ment Kits link below for a list of Altera's development boards. 6. Test the hardware. You can use Altera's SignalTap II Logic Analyzer or a third-party protocol ®...
Page 14
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
In this example, the host processor issues single-dword MemRd and MemWr TLPs. The target device contains a 8192 kilobyte (KB) On-Chip Memory. The Endpoint (DUT) and Avalon-ST to Avalon-MM Bridge (Apps) perform the necessary Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 16
However, the testbench and Root Port BFM are not intended to be a substitute for a full verification environment. To thoroughly test your application, Altera suggests that you obtain commercially available PCI Express verification IP and tools, or do your own extensive hardware testing or both.
Page 17
However, the testbench and Root Port BFM are not intended to be a substitute for a full verification environment. To thoroughly test your application, Altera suggests that you obtain commercially available PCI Express verification IP and tools, or do your own extensive hardware testing or both.
Page 18
Arria 10 Development Kit Conduit Interface on page 6-61 Generating the Design Figure 2-7: Procedure Start Parameter Specify IP Variation Select Specify Initiate Editor and Select Device Design Parameters Design Example Design Generation Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 19
Quartus Prime release. Arria 10 production devices will be available in a future Quartus Prime release. 8. Click the Generate Example Design button. The software generates all files necessary to run simulations and hardware tests on the Arria 10 FPGA Development Kit ES2. Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 20
USER_DEFINED_SIM_ OPTIONS="" 3. Type the following commands: a. chmod +x *.sh b. ./my_setup.sh 4. A successful simulation ends with the following message, "Simulation stopped due to successful completion!" Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 21
3. chmod +x *.sh 4. ./my_setup.sh 5. A successful simulation ends with the following message, "Simulation stopped due to successful completion!" Figure 2-10: Partial Transcript from Successful PIO Simulation Testbench Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 22
UG-01145_avst Compiling and Simulating the Design 2015.11.02 Figure 2-11: Partial Transcript from Successful DMA Simulation Testbench Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 23
2. Writes 0x00000000 to the specified BAR at offset 0x00000000 to initialize the memory and read it back. 3. Writes 0xABCD1234 at offset 0x00000000 of the specified BAR. Reads it back and compares. If successful, the test program displays the message 'PASSED' Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 24
Programmer). d. Open the Windows Device Manager and scan for hardware changes. e. Select the Altera FPGA listed as an unknown PCI device and point to the appropriate 32- or 64-bit driver (altera_pice_win_driver.inf) in the Windows_driver directory. f. After the driver loads successfully, a new device named Altera PCI API Device appears in the Windows Device Manager.
Page 25
5. The test displays the message, PASSED, if the test is successful. Related Information Arria 10 Development Kit Conduit Interface • on page 6-61 Arria 10 GX FPGA Development Kit • Arria 10 PCI Express Quick Start Guide Altera Corporation Send Feedback...
Page 26
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 27
Qsys Design Flow 2015.11.02 Altera provides example designs to help you get started with the Arria 10 Hard IP for PCI Express IP Core. You can use example designs as a starting point for your own design. The example designs include scripts to compile and simulate the Arria 10 Hard IP for PCI Express IP Core.
Page 28
, type the following commands: tb/sim/mentor do msim_setup.tcl (This command compiles all design files and elaborates the top-level design without any ld_debug optimization.) run -all Getting Started with the Arria 10 Hard IP for PCI Express Altera Corporation Send Feedback...
Page 29
For a more detailed listing of the directories and files the Quartus II software generates, refer to Files Generated for Altera IP Cores in Compiling the Design in the Qsys Design Flow. Understanding Simulation Log File Generation Starting with the Quartus II 14.0 software release, simulation automatically creates a log file, altpcie_ in your simulation directory.
Page 30
1. Before compiling, you can optionally turn on two parameters in the testbench. The first parameter specifies pin assignments that match those for the Altera Development Kit board I/Os. The second parameter enables the Compliance Base Board (CBB) logic on the development board. In the Gen1 x8 example design, complete the following steps if you want to enable these parameters: a.
Page 31
17.To compile your design using the Quartus II software, on the Processing menu, click Start Compila‐ tion. The Quartus II software then performs all the steps necessary to compile your design. Getting Started with the Arria 10 Hard IP for PCI Express Altera Corporation Send Feedback...
Page 32
UG-01145_avst Compiling the Design in the Qsys Design Flow 2015.11.02 Files Generated for Altera IP Cores Figure 3-3: IP Core Generated Files <Project Directory> <your_ip>.qip or .qsys - System or IP integration file <your_ip>.sopcinfo - Software tool-chain integration file <your_ip> - IP core variation files <your_ip>.bsf - Block symbol schematic file...
Page 33
Parameters and Options in Introduction to Altera IP Cores. For more information about upgrading older IP cores to the current release, refer to Upgrading Outdated IP Cores in Introduction to Altera IP Cores. Getting Started with the Arria 10 Hard IP for PCI Express...
Page 34
UG-01145_avst Files Generated for Altera IP Cores and Qsys Systems 2015.11.02 Related Information • Qsys Design Flow on page 3-2 Introduction to Altera IP Cores • Managing Quartus II Projects • Files Generated for Altera IP Cores and Qsys Systems Figure 3-5: Files generated for IP cores and Qsys Systems <Project Directory>...
Page 35
UG-01145_avst 3-10 Files Generated for Altera IP Cores and Qsys Systems 2015.11.02 Table 3-4: IP Core and Qsys Simulation Generated Files File Name Description <my_ip>.qsys Describes the connections and IP component parameterizations in <system>.sopcinfo your Qsys system. You can parse its contents to get requirements when you develop software drivers for IP components.
Page 36
UG-01145_avst 3-11 Files Generated for Altera IP Cores and Qsys Systems 2015.11.02 File Name Description If the IP contains register information, the .regmap file generates. <my_ip>.regmap The .regmap file describes the register map information of master and slave interfaces. This file complements the .sopcinfo file by providing more detailed register information about the system.
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 38
PCIe link. This option is recommended for typical endpoint applications where most of the PCIe traffic is generated by a DMA engine that is located in the endpoint application layer logic. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 39
Table 4-2: RX Buffer Allocation Selections Available by Interface Type Interface Type Minimum Balanced High Maximum Avalon-ST Available Available Available Available Available Avalon-MM Available Available Available Not Available Not Available Avalon-MM with Available Available Available Not Available Not Available Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 40
Packets per Cycle on the Avalon-ST TX 256-Bit Interface on page 6-28. Enable credit On/Off When you turn on this option, the core includes the tx_cons_ consumed port. This parameter does not apply to the Avalon-MM cred_sel selection port interface. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 41
Arria 10 Transceiver PHY User Guide Provides information about the ADME feature for Arria 10 devices. Base Address Register (BAR) and Expansion ROM Settings The type and size of BARs available depend on port type. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 42
• 128 bytes–2 GBytes or 8 EBytes: Endpoint and Root Port variants • 6 bytes–4 KBytes: Legacy Endpoint variants Expansion Disabled–16 MBytes Specifies the size of the optional ROM. The expansion ROM is only available for the Avalon-ST interface. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 43
Address offset: 0x000. Device ID 0x00000000 Sets the read-only value of the register. Device ID Address offset: 0x000. Revision ID 0x00000000 Sets the read-only value of the register. Revision ID Address offset: 0x008. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 44
Specifies the maximum payload size supported. This payload size parameter sets the read-only value of the max payload 256 bytes size supported field of the Device Capabilities register 512 bytes (0x084[2:0]). Address: 0x084. 1024 bytes 2048 bytes Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 45
50 s to 50 ms. The following values specify the range: • None—Completion timeout programming is not supported • 0001 Range A • 0010 Range B • 0011 Ranges A and B • 0110 Ranges B and C Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 46
• 0111 Ranges A, B, and C • 1110 Ranges B, C and D • 1111 Ranges A, B, C, and D All other values are reserved. Altera recommends that the completion timeout mechanism expire in no less than 10 ms.
Page 47
Root Port uses the same physical reference clock that the system provides on the connector. When Off, the IP core uses an independent clock regardless of the presence of a reference clock on the connector. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 48
1. Throughout this user guide, the terms word, dword and qword have the same meaning that they have in the PCI Express Base Specification. A word is 16 bits, a dword is 32 bits, and a qword is 64 bits. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 49
In combination with the Slot power scale value, specifies the upper 0–255 limit in watts on power supplied by the slot. Refer to Section 7.8.9 of the PCI Express Base Specification for more information. Slot number Specifies the slot number. 0-8191 Arria 10 Parameter Settings Altera Corporation Send Feedback...
Active State Power Management (ASPM). This setting is disabled for Root Ports. The default value of this parameter is 1 µs. This is the safest setting for most designs. Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 51
Hard IP Reconfiguration Interface. tion of PCIe read-only registers Enable Altera When On, you can use the Altera System Console to read and write On/Off Debug Master the embedded Arria 10 Native PHY registers. Endpoint (ADME)
Page 52
Table 4-15: PHY Characteristics Parameter Value Description Gen2 TX de- 3.5dB Specifies the transmit de-emphasis for Gen2. Altera emphasis recommends the following settings: • 3.5dB: Short PCB traces • 6.0dB: Long PCB traces. Arria 10 Example Designs Table 4-16: Example Designs...
Page 53
Development Kit devices. Select Arria 10 FPGA Development Kit ES for engineering sample (ES) or ES2 devices. Select None if you are Arria 10 FPGA targeting your own development board. Development Kit None Arria 10 Parameter Settings Altera Corporation Send Feedback...
Page 54
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 55
(1) Nomenclature of left column bottom transceiver banks always begins with “C” (2) Nomenclature of right column bottom transceiver banks may begin with “C”, “D”, or “E”. Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 56
(1) Nomenclature of left column bottom transceiver banks always begins with “C” (2) Nomenclature of right column bottom transceiver banks may begin with “C”, “D”, or “E”. Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 57
Refer to the Arria 10 Transceiver Layout in the Arria 10 Transceiver PHY User Guide for comprehensive figures for Arria 10 GT, GX, and SX devices. Related Information Arria 10 Transceiver PHY User Guide Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 58
Hard IP Ch0 <txvr_block_N>_TX/RX_CH4N PMA Channel 3 PCS Channel 3 PMA Channel 2 PCS Channel 2 PMA Channel 1 PCS Channel 1 PMA Channel 0 PCS Channel 0 Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 59
Note: In all configurations, physical channel 4 in the PCS connects to logical channel 0 in the hard IP. You cannot change the channel placements illustrated below. Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 60
PCS Channel 2 PMA Channel 1 PCS Channel 1 ATX0 PLL PMA Channel 0 PCS Channel 0 Master indicates the location of the master clock generation block (CGB) Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 61
PCS Channel 2 PMA Channel 1 PCS Channel 1 ATX0 PLL PMA Channel 0 PCS Channel 0 Master indicates the location of the master clock generation block (CGB) Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 62
PCS Channel 2 PMA Channel 1 PCS Channel 1 ATX0 PLL PMA Channel 0 PCS Channel 0 Master indicates the location of the master clock generation block (CGB) Physical Layout of Hard IP In Arria 10 Devices Altera Corporation Send Feedback...
Page 63
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 64
TLP ends in this cycle. Output Indicates the number of empty qwords in . Not used rx_st_empty[1:0] rx_st_data when is 64 bits. Valid only when rx_st_data rx_st_eop asserted in 128-bit and 256-bit modes. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 65
<n> , then <n + > > is a ready cycle, during which readyLatency the Transaction Layer may assert and transfer data. valid The RX interface supports a of 2 cycles. readyLatency Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 66
Altera recommends resetting the Arria 10 Hard IP for PCI Express when an uncorrectable double-bit ECC error is detected. Attention: If you instantiate this IP core as a separate component from the Quartus II IP Catalog, the Message pane reports the following warning messages: pcie_a10.pcie_a10_hip_0.tx.st Interface must have an associated reset...
Page 67
• Bit 0: BAR 0 • Bit 1: BAR 1 • Bit 2: Primary Bus number • Bit 3: Secondary Bus number • Bit 4: Secondary Bus number to Subordinate Bus number window Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 68
2 of the field. This bit is always 0 (aligned to qword boundary) lower address for completion with data TLPs that are for configuration read or I/O read requests. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 69
Note: The Avalon-ST protocol, as defined in Avalon Interface Specifications, is big endian, while the Hard IP for PCI Express packs symbols into words in little endian format. Consequently, you cannot use the standard data format adapters available in Qsys. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 70
The following figure shows the mapping of Avalon-ST RX packets to PCI Express TLPs for TLPs for a four dword header with qword aligned addresses with a 64-bit bus. pld_clk header1 header3 data1 rx_st_data[63:32] rx_st_data[31:0] header0 header2 data0 rx_st_sop rx_st_eop Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 71
Application Layer is able to accept it. rx_st_data pld_clk 000 . 010 . CCCC0002CCCC0001 . CC . CC . CC . CC . CC rx_st_data[63:0] rx_st_sop rx_st_eop rx_st_ready rx_st_valid Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 72
64 bits of rx_st _data pld_clk data3 rx_st_data[127:96] header2 data2 rx_st_data[95:64] header1 data1 data<n> rx_st_data[63:32] header0 data0 data<n-1> rx_st_data[31:0] rx_st_bar[7:0] rx_st_sop rx_st_eop rx_st_empty rx_st_valid Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 73
128 bits in the cycle. rx_st_eop pld_clk rx_st_valid rx_st_data[127:96] Header 3 Data 2 Data n Header 2 Data 1 rx_st_data[95:64] Data n-1 Header 1 Data 0 rx_st_data[63:32] Header 0 Data n-2 rx_st_data[31:0] rx_st_sop rx_st_eop rx_st_empty Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 74
8945 . . . rx_st_data[127:0] rx_st_sop rx_st_eop rx_st_empty rx_st_ready rx_st_valid The following figure illustrates back-to-back transmission on the 128-bit Avalon-ST RX interface with no idle cycles between the assertion of rx_st_eop rx_st_sop Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 75
The following figure illustrates a two-cycle packet with valid data in the lower qword ) and a one-cycle packet where the occur in the same rx_st_data[63:0] rx_st_sop rx_st_eop cycle. pld_clk 0000090 1C0020000F0000000100004 450AC89000012FE0D10004 rx_st_data[127:0] rx_st_sop rx_st_eop rx_st_empty rx_st_ready rx_st_valid Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 76
Avalon-ST interface is 256 bits wide. It reduces the wasted bandwidth for small packets. A comparison of the largest and smallest packet sizes illustrates this point. Large packets using the full 256 bits achieve the following throughput: 256/256*8 = 8 GBytes/sec Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 77
Avalon-ST TX Interface The following table describes the signals that comprise the Avalon-ST TX Datapath. The TX data signal can be 64, 128, or 256 bits. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 78
<n + readyLatency> the Application Layer may assert and transfer data. valid When tx_st_ready tx_st_valid tx_st_data registered (the typical case), Altera recommends a readyLa- of 2 cycles to facilitate timing closure; however, a tency Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 79
For 256-bit data, when you turn on Enable multiple packets per cycle, the bit 0 applies to the entire bus . Bit tx_st_data[255:0] 1 is not used. To facilitate timing closure, Altera recommends that you register both the signals. If no other tx_st_ready tx_st_valid...
Page 80
For example, bit[0] corresponds to tx_st_data , bit[1] corresponds to tx_st_data[7:0] tx_st_data[15:8] and so on. <n> = 8, 16, or 32. Component Specific Signals Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 81
The 6 bits of this vector correspond to the following 6 types of credit types: • [5]: posted headers • [4]: posted data • [3]: non-posted header • [2]: non-posted data • [1]: completion header • [0]: completion data Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 82
TLP type (message request with data payload). For additional information about TLP packet headers, refer to Section 2.2.1 Common Packet Header Fields in the PCI Express Base Specification . Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 85
64 bits of tx_st_data pld_clk Data3 tx_st_data[127:96] Header2 Data 2 tx_st_data[95:64] Header1 Data1 Data(n) tx_st_data[63:32] Header0 Data0 Data(n-1) tx_st_data[31:0] tx_st_sop tx_st_eop tx_st_empty tx_st_valid Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 86
The following figure shows the mapping of 128-bit Avalon-ST TX packets to PCI Express TLPs for a four dword header TLP with non-qword aligned addresses. In this example, is low because the tx_st_empty data ends in the upper 64 bits of tx_st_data Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 87
Application Layer by deasserting . Because the is two cycles, the tx_st_ready readyLatency Application Layer deasserts after two cycles and holds until two cycles after tx_st_valid tx_st_data is reasserted tx_st_ready Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 88
128-bits of the Avalon-ST interface. Although this mode simplifies the Application Layer logic, failure to use the full 256-bit Avalon-ST may slightly reduce the throughput of a design. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 89
Application Layer logic, failure to use the full 256-bit Avalon-ST may slightly reduce the throughput of a design. The following figure illustrates the layout of header and data for a three dword header on a 256-bit bus with aligned and unaligned data. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 90
Figure 6-32: 256-Bit Avalon-ST TX Interface with Multiple Packets Per Cycle tx_st_data[255:0] 12 ... 12... 12... 12... 12... 12... 12... 12... 00... 5A... 5A... 5A... 5A... 5A... 5A... 5A... 5A... tx_st_sop[0] tx_st_eop[0] tx_st_sop[1] tx_st_eop[1] tx_st_empty[1:0] tx_st_ready tx_st_valid Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 91
Related Information Clocks on page 8-4 Reset, Status, and Link Training Signals Refer to Reset and Clocks for more information about the reset sequence and a block diagram of the reset logic. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 92
Reset, Status, and Link Training Signals 2015.11.02 Table 6-6: Reset Signals Signal Direction Description Input Active low reset signal. In the Altera hardware example designs, npor is the coming from the npor pin_perst local_rstn software Application Layer. If you do not drive a soft reset signal from the Application Layer, this signal must be derived from .
Page 93
If pld_clk Hard IP output clock is sourcing the coreclkout_hip pld_ Hard IP input, this input can be connected to the serdes_ output. pll_locked Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 94
Lane Active Mode: This signal indicates the number of lanes that lane_act[3:0] configured during link training. The following encodings are defined: • 4’b0001: 1 lane • 4’b0010: 2 lanes • 4’b0100: 4 lanes • 4’b1000: 8 lanes Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 95
• 11100: Recovery.Equalization, Phase 1 • 11101: Recovery.Equalization, Phase 2 • 11110: Recovery.Equalization, Phase 3 Related Information PCI Express Card Electromechanical Specification 2.0 • Configuration via Protocol (CvP) Implementation in Altera FPGAs User Guide • Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 96
When a correctable ECC error occurs, the Arria 10 Hard IP for PCI Express recovers without any loss of information. No Application Layer intervention is required. In the case of uncorrectable ECC error, Altera recommends that you reset the core. Table 6-8: Error Signals...
Page 97
• The message TLP has been transmitted in Assert_INTA response to the assertion of the app_int_sts • The message TLP has been transmitted in Deassert_INTA response to the deassertion of the signal. app_int_sts Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 98
Hard IP. For a description of the completion rules, the completion header format, and completion status field values, refer to Section 2.2.9 of the PCI Express Base Specification. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 99
Application Layer master block detects an unexpected completion transaction. Many cases of unexpected completions are detected and reported internally by the Transac‐ tion Layer. For a list of these cases, refer to Transaction Layer Errors. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 100
Register as Transactions Pending Device Status defined in Section 7.8.5 of the PCI Express Base Specification. Related Information • Transaction Layer Errors on page 10-3 • PCI Express Base Specification Rev 3.0 Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 101
• 2’b01: Some time later, the parity error is detected by the TX Data Link Layer which drives 2’b01 to indicate the error. Altera recommends resetting the Arria 10 Hard IP for PCI Express when this error is detected. Contact Altera if resetting becomes unworkable.
Page 102
All LMI reads are also held and executed when no configuration TLP requests are pending. The LMI interface supports two operations: local read and local write. The timing for these operations complies with the Avalon-MM protocol described in the Avalon Interface Specifications. LMI reads can be issued at Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 103
Address inputs, [1:0] not used. lmi_addr[11:0] Input Data inputs. Data is driven from LSB, [7:0], to MSB,[31:24]. The LSB lmi_din[7:0] coincides with lim_wren Figure 6-35: LMI Read read_en addr_in[11:0] bits[7:0] bits[15:8] bits[23:16] bits[31:24] dataout[7:0] rd_wr_ack Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 104
Slot capability register is enabled. Refer to the Slot register and Slot capability register parameters in Table 6–9 on hpg_ctrler[4:0] page 6–10. For Endpoint variations the input hpg_ctrler should be hardwired to 0s. The bits have the following meanings: Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 105
• Bit 1: non-fatal error detected • Bit 0: correctable error detected [48] Slot Status Register[8] Data Link Layer state changed Slot Status Register[4] Command completed. (The hot plug [47] controller completed a command.) Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 106
• Bit 15: detected parity error • Bit 14: received system error • Bit 13: received master abort • Bit 12: received target abort • Bit 11: signalled target abort Secondary Status Register[8] Master Data Parity Error Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 107
Information stored in the Configuration Space is accessed in round robin order where indicates which register is being accessed. The following table shows the layout tl_cfg_add of configuration information that is multiplexed on tl_cfg_ctl Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 108
Device Control 2 for the cfg_dev_ctrl2 cfg_dev2ctrl[15:0] PCI Express capability structure. Output is the Slot Status of the PCI cfg_slot_ctrl cfg_slot_ctrl[15:0] Express capability structure. This register is only available in Root Port mode. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 109
Root Port mode. Output Secondary bus number. This register is available cfg_secbus only in Root Port mode. Output Subordinate bus number. This register is available cfg_subbus only in Root Port mode. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 110
Power Management Control cfg_pmcsr cfg_pmcsr[31:16] is the Power Management cfg_pmcsr[15:0] Status register. Output MSI-X message control. cfg_msixcsr Output MSI message control. Refer to the following table cfg_msicsr for the fields of this register. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 111
Application Layer registers. 64-bit address capable. 64-bit address capability • 1: function capable of sending a 64-bit message address • 0: function not capable of sending a 64-bit message address Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 112
Reconfiguration clock. The frequency range for this clock is 100– hip_reconfig_clk 125 MHz. Input Active-low Avalon-MM reset. Resets all of the dynamic reconfi‐ hip_reconfig_rst_n guration registers to their default values as described in Hard IP Reconfiguration Registers. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 113
324 ns after switching to user mode. Input A selector which must be asserted when performing dynamic interface_sel reconfiguration. Drive this signal low 4 clock cycles after the release of ser_shif t_load Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 114
Root Port—This signal is asserted for 1 clock cycle when the Root Port receives the acknowledge message. pme_turn_off Endpoint—This signal is asserted for 1 cycle when the Endpoint receives the message from the Root Port. PME_turn_off Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 115
Power Management Auxiliary Power: This signal can be tied to 0 pm_auxpwr because the L2 power state is not supported. Figure 6-41: Layout of Power Management Capabilities Register data reserved PME_status data_scale data_select PME_EN reserved PM_state register Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 116
Endpoint. First, the Hard pme_to_sr pme_to_cr IP receives the message which causes to assert. Then, the Application Layer PME_turn_off pme_to_sr sends the message to the Root Port by asserting PME_to_ack pme_to_cr pme_to_sr hard pme_to_cr Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 117
Receive input. These signals are the serial inputs of lanes 7–0. rx_in[7:0] Refer to Pin-out Files for Altera Devices for pin-out tables for all Altera devices in .pdf, .txt, and .xls formats. Transceiver channels are arranged in groups of six. For GX devices, the lowest six channels on the left side of the device are labeled GXB_L0, the next group is GXB_L1, and so on.
Page 118
When deasserted indicates half swing. Output Transmit V margin selection. The value for this signal is based txmargin[2:0] on the value from the . Available for Link Control 2 Register simulation only. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 119
Input Receive valid <n>. This signal indicates symbol lock and valid rxvalid0 <n> and <n>. data on rxdata rxdatak Input PHY status <n>. This signal communicates completion of several phystatus0 PHY requests. Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 121
• 3'b110: Absence of Electrical Idle Exit in 2000 UI interval for Gen1 and 16000 UI interval for Gen2 • 3'b111: Absence of Electrical idle exit in 128 us window for Gen1 Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 122
• [6]: Forces entry to compliance mode when a timeout is reached in the polling.active state and not all lanes have detected their exit condition. • [7]: Disable low power state negotiation. Altera recommends setting this bit. • [31:8]: Reserved. Set to all 0s.
Page 123
4'b0100 • devkit_ctrl[6:5]:test_in[6:5] is typically set to 2'b01 • devkit_ctrl[31:7]:test_in[31:7] is typically set to 25'h1 • devkit_ctrl[63:32]:is typically set to 32'b0 • devkit_ctrl[255:64]:is typically set to 192'b0 Interfaces and Signal Descriptions Altera Corporation Send Feedback...
Page 124
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 125
Type 1 Configuration Space Header 0x00C BIST, Header Type, Primary Latency Timer, Type 0 Configuration Space Header Cache Line Size Type 1 Configuration Space Header 0x010 Base Address 0 Base Address Registers 0x014 Base Address 1 Base Address Registers Registers Altera Corporation Send Feedback...
Page 126
Type 0 Configuration Space Header Bridge Control, Interrupt Pin, Interrupt Line Type 1 Configuration Space Header 0x050 MSI-Message Control Next Cap Ptr MSI and MSI-X Capability Structures Capability ID 0x054 Message Address MSI and MSI-X Capability Structures Registers Altera Corporation Send Feedback...
Page 127
Root Error Command Root Error Command Register 0x830 Root Error Status Root Error Status Register 0x834 Error Source Identification Register Correct‐ Error Source Identification Register able Error Source ID Register Related Information PCI Express Base Specification 3.0 Registers Altera Corporation Send Feedback...
Page 128
BAR Registers 0x01C BAR Registers 0x020 BAR Registers 0x024 Reserved 0x028 Subsystem Device ID Subsystem Vendor ID 0x02C Expansion ROM Base Address 0x030 Reserved Capabilities Pointer 0x034 0x038 Reserved 0x03C 0x00 Interrupt Pin Interrupt Line Registers Altera Corporation Send Feedback...
Page 129
Figure 7-3: MSI Capability Structure 24 23 16 15 Message Control 0x050 Configuration MSI Control Status Next Cap Ptr Capability ID Register Field Descriptions 0x054 Message Address 0x058 Message Upper Address 0x05C Reserved Message Data Registers Altera Corporation Send Feedback...
Page 131
Root Status Device Compatibilities 2 0x0A4 0x0A8 Device Status 2 Device Control 2 0x0AC Link Capabilities 2 0x0B0 Link Status 2 Link Control 2 0x0B4 Slot Capabilities 2 0x0B8 Slot Status 2 Slot Control 2 Registers Altera Corporation Send Feedback...
Page 132
Correctable Internal Error Status Register 0x240 Correctable Internal Error Mask Register Table 7-2: Altera‑Defined VSEC Capability Register, 0x200 The Altera-Defined Vendor Specific Extended Capability. This extended capability structure supports Configuration via Protocol (CvP) programming and detailed internal error reporting. Bits Register Description...
Page 133
. This read only register is an additional marker. If A Device Altera Marker you use the standard Altera Programmer software to configure Value the device with CvP, this marker provides a value that the programming software reads to ensure that it is operating with the correct VSEC.
Page 134
• 0x08 for all compressed images [7:3] Reserved. . Request that the FPGA control block 1’b0 CVP_FULLCONFIG reconfigure the entire FPGA including the Arria 10 Hard IP for PCI Express, bring the PCIe link down. Registers Altera Corporation Send Feedback...
Page 135
Lower 32 bits of configuration data to be transferred to the FPGA 0x00000000 control block to configure the device. Table 7-10: CvP Programming Control Register This register is written by the programming software to control CvP programming. Bits Register Description Reset Value Access [31:2] Reserved. 0x0000 Registers Altera Corporation Send Feedback...
Page 136
Mask for data parity errors detected on the RX to Configuration 1b’1 Space Bus interface. Mask for data parity error detected at the input to the RX Buffer. 1b’1 Mask for the retry buffer uncorrectable ECC error. 1b’1 Registers Altera Corporation Send Feedback...
Page 137
Link Layer. When set, indicates a parity error has been detected on the RX RW1CS to Configuration Space bus interface. When set, indicates a parity error was detected at input to the RW1CS RX Buffer. Registers Altera Corporation Send Feedback...
Page 138
Correctable Internal Errors as defined in the PCI Express Base Specification 3.0. This register is for debug only. Only use this register to observe behavior, not to drive logic custom logic. Bits Register Description Reset Access Value [31:7] Reserved. Corrected Internal Error reported by the Application Layer. RW1CS Registers Altera Corporation Send Feedback...
Page 139
[4:2] Reserved. When set, the retry buffer correctable ECC error status RW1CS indicates an error. When set, the RX buffer correctable ECC error status RW1CS indicates an error. Related Information PCI Express Base Specification 3.0 Registers Altera Corporation Send Feedback...
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
3. The Hard IP for PCI Express deasserts the output to the Application Layer. reset_status 4. The deasserts cycles after is released. altpcied_<device>v_hwtcl.sv app_rstn pld_clk reset_status Arria 10 Reset and Clocks Altera Corporation Send Feedback...
Page 142
IP core deasserts the input to the TX transceiver. npor npor_serdes 2. The SERDES reset controller waits for to be stable for a minimum of 127 cycles pll_locked pld_clk before deasserting tx_digitalreset. Arria 10 Reset and Clocks Altera Corporation Send Feedback...
Figure 8-5: Clock Domains and Clock Generation for the Application Layer The following illustrates the clock domains when using to drive the Application Layer coreclkout_hip and the of the IP core. The Altera-provided example design connects to the pld_clk coreclkout_hip . However, this connection is not mandatory.
Avalon-MM interface for Hard IP dynamic reconfi‐ hip_reconfig_clk guration interface which you can use to change the value of read-only configuration registers at run-time. This interface is optional. It is not required for Arria 10 devices. Arria 10 Reset and Clocks Altera Corporation Send Feedback...
Page 146
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 147
For example, in the following figure, the Endpoint requests eight MSIs but is only allocated two. In this case, you must design the Application Layer to use only two allocated messages. Interrupts Altera Corporation Send Feedback...
Page 148
In this timing app_msi_req app_msi_ack diagram can extend beyond before deasserting. However, must app_msi_req app_msi_ack app_msi_req be deasserted before or within the same clock as is deasserted to avoid inferring a new app_msi_ack interrupt. Interrupts Altera Corporation Send Feedback...
Page 149
The BIOS sets up the starting address offsets and BAR associated with the pointer to the starting address of the MSI-X table and PBA registers. The following figure shows the Application Layer modules that implement MSI-X interrupts. Interrupts Altera Corporation Send Feedback...
Page 150
The PBA can use qword or dword accesses. For qword accesses, the IRQ Source calculates the address of the <m > bit using the following formulas: qword address = <PBA base addr> + 8(floor(<m>/64)) qword bit = <m> mod 64 Interrupts Altera Corporation Send Feedback...
Page 151
The following figure illustrates the timing for deassertion of legacy interrupts. The assertion of instructs the Hard IP for PCI Express to send a message. app_int_sts Deassert_INTA Figure 9-9: Legacy Interrupt Deassertion app_int_sts app_int_ack Related Information Correspondence between Configuration Space Registers and the PCIe Specification on page 7-1 Interrupts Altera Corporation Send Feedback...
Page 152
The register is Root Error Status Root Error Status part of the PCI Express AER Extended Capability structure. It is located at offset 0x830 of the Configura‐ tion Space registers. Interrupts Altera Corporation Send Feedback...
Page 153
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 154
This error occurs when the replay number rolls over. Data Link Layer protocol Uncorrectable(fatal) This error occurs when a sequence number specified by the Ack/Nak block in the Data Link Layer ( AckNak_Seq_ does not correspond to an unacknowledged TLP. Num) Error Handling Altera Corporation Send Feedback...
Page 155
• A memory transaction when the Memory Space Enable bit (bit [1] of the PCI Command register at Configuration Space offset 0x4) is set to 0. • A poisoned configuration write request ( CfgWr0 Error Handling Altera Corporation Send Feedback...
Page 156
Transaction Layer using the signal. cpl_err[0] Completer abort Uncorrectable The Application Layer reports this error using the cpl_ (non-fatal) signal when it aborts receipt of a TLP. err[2] Error Handling Altera Corporation Send Feedback...
Page 157
IP block deletes the TLP and it is not presented to the Application Layer. Flow control protocol Uncorrectable This error occurs when a component does not receive error (FCPE) (fatal) update flow control credits with the 200 µs limit. Error Handling Altera Corporation Send Feedback...
Page 158
Configuration Space. • Received poisoned Configuration Write TLPs are not written in the Configuration Space. • The Configuration Space never generates a poisoned TLP; the error/poisoned bit of the header is always set to 0. Error Handling Altera Corporation Send Feedback...
Page 159
ECRC Error Status Malformed TLP Status Receiver Overflow Status Unexpected Completion Status Completer Abort Status Completion Timeout Status Flow Control Protocol Status Poisoned TLP Status Surprise Down Error Status Data Link Protocol Error Status Undefined Error Handling Altera Corporation Send Feedback...
Page 160
16 15 14 13 12 11 9 Rsvd Rsvd Rsvd Header Log Overflow Status Corrected Internal Error Status Advisory Non-Fatal Error Status Replay Timer Timeout Status REPLAY_NUM Rollover Status Bad DLLP Status Bad TLP Status Receiver Error Status Error Handling Altera Corporation Send Feedback...
Page 161
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 163
Application Layer can optimize performance by selecting for transmission only the TLPs that have credits available. Related Information • Avalon-ST RX Interface on page 6-2 Avalon-ST TX Interface • on page 6-15 Avalon Interface Specifications • IP Core Architecture Altera Corporation Send Feedback...
Page 164
Interrupts for Root Ports on page 6-36 PIPE The PIPE interface implements the Intel-designed PIPE interface specification. You can use this parallel interface to speed simulation; however, you cannot use the PIPE interface in actual hardware. IP Core Architecture Altera Corporation Send Feedback...
Page 165
• The Gen1, Gen2, and Gen3 simulation models support PIPE and serial simulation. • For Gen3, the Altera BFM bypasses Gen3 Phase 2 and Phase 3 Equalization. However, Gen3 variants can perform Phase 2 and Phase 3 equalization if instructed by a third-party BFM.
Page 167
LMI access completes. ready Note: Altera does not support the use of the LMI interface to read and write the other registers in function0 of the Hard IP for PCI Express Configuration Space. You must create your own function0 in your application logic.
Page 168
RX interface. The Application Layer must handle the error. If ECRC generation is enabled, the core generates ECRC and appends it to the end of the TX TLP from the Application Layer. Refer to Table 12–1 on page 12–2 and Table 12–2 on page 12–3 for additional information. IP Core Architecture Altera Corporation Send Feedback...
Page 169
Severity Device Control Register signals. The Application Layer must also log these errors in the soft Configuration Space and send error Messages. Related Information PCI Express Base Specification 3.0 IP Core Architecture Altera Corporation Send Feedback...
Page 170
DLLP reception using the retry buffer • Management of the retry buffer • Link retraining requests in case of error through the Link Training and Status State Machine (LTSSM) of the Physical Layer IP Core Architecture Altera Corporation Send Feedback...
Page 171
• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledged packets in the case of NAK DLLP reception. In case of ACK DLLP reception, the retry buffer discards all acknowledged packets. IP Core Architecture Altera Corporation Send Feedback...
Page 172
• Serializing and deserializing data • Operating the PIPE 3.0 Interface • Implementing auto speed negotiation (Gen2 and Gen3) • Transmitting and decoding the training sequence • Providing hardware autonomous speed control • Implementing auto lane reversal IP Core Architecture Altera Corporation Send Feedback...
Page 173
The Physical Layer integrates both digital and analog elements. Intel designed the PIPE interface to separate the MAC from the PHY. The Arria 10 Hard IP for PCI Express complies with the PIPE interface specification. IP Core Architecture Altera Corporation Send Feedback...
Page 174
If all lanes have not detected a COM symbol after seven clock cycles, they are reset and the resynchronization process restarts, or else the RX alignment function recreates a 64-bit data word which is sent to the DLL. IP Core Architecture Altera Corporation Send Feedback...
Page 175
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 176
Root Port sends the message. PME_turn_off • Endpoint: When PME_to_cr asserted, the Endpoint acknowl‐ edges the message PME_turn_off by sending a pme_to_ack message to the Root Port. PME_ — TO_Ack Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 177
MSI is msi_num being sent to the root complex when an error is logged in the AER Capability structure. ERR_ — NONFAT ERR_ — FATAL Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 178
PCI Express Base Specification Revision 3.0 Vendor-Defined Messages Table 12-6: Vendor-Defined Message Generated by Message Root Port Endpoint Comments Core Core (with Layer App Layer input) Vendor Transmit Transmit Defined Receive Receive Type 0 Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 179
PCI Express Base Specification indicator Revision , these messages are not transmitted to the Application Layer. Power_ Transmit Receive Indicato r On Power_ Transmit Receive Indicato r Blink Power_ Transmit Receive Indicato r Off Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 180
Additionally, it can generate MSI requests under the control of the dedicated signals. • In Root Port mode, the Application Layer can issue Type 0 or Type 1 Configuration TLPs on the Avalon-ST TX bus. Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 181
• Y/N: There are no requirements. A device may allow the second transaction to pass the first. • No: The second transaction must not be allowed to pass the first. The following transaction ordering rules apply to the table below. Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 182
Memory Write or Read Request I/O or Cfg Write Req Message Req Spec Hard IP Spec Hard IP Spec Hard IP Spec Hard IP Posted Read Non- Posted Req with data Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 183
However, because the PCI-to-PCI bridge includes a write buffer, the flag may indicate that it is safe to read data while the actual data remains in the PCI-to- PCI bridge posted write buffer. Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 184
In this system, if relax ordering is not enabled, a memory read to the legacy Endpoint is blocked. The legacy Endpoint read is blocked because an earlier posted write cannot be completed as the write buffer is full. . Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 185
5. If your analysis indicates that you can enable relaxed ordering, simulate your system with and without relaxed ordering enabled. Compare the results and performance. 6. If relaxed ordering improves performance without introducing errors, you can enable it in your system. Transaction Layer Protocol (TLP) Details Altera Corporation Send Feedback...
Page 186
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 187
FC Update DLLPs are raised to a high priority under the following three circumstances: a. When the last sent counter minus the amount of received data is less than credit allocated and the current counter is greater than the last sent credit MAX_PAYLOAD credit allocated Throughput Optimization Altera Corporation Send Feedback...
Page 188
However, much of the delay encountered in this loop is well outside the IP core and is very difficult to estimate. PCI Express switches can be inserted in this loop, which makes determining a bound on the delay more difficult. Throughput Optimization Altera Corporation Send Feedback...
Page 189
You can specify 32 or 64 tags though configuration software to restrict the Application Layer to use only 32 tags. In commercial PC systems, 32 tags are usually sufficient to maintain optimal read throughput. Throughput Optimization Altera Corporation Send Feedback...
Page 190
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 191
Express IP Core and Application Layer for a description of the key signals that reset, control dynamic reconfiguration, and link training. Altera recommends separate control of reset signals for the Endpoint and Root Port. Successful reset sequence includes the following steps: 1.
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 193
ECRC. If the ECRC generation option is turned off, no error detection occurs. If the ECRC forwarding For maximum use of the Arria 10 device, Altera recommends that you use the bottom Hard IP for PCI Express on either side to satisfy the 100 ms PCIe wake up time requirement for Gen3 operations.
Page 194
1, the TLP includes an ECRC. is the TL digest bit of the TL packet. field is in the ECRC Check Enable Configuration Space Advanced Error Capabilities and Control Register Optional Features Altera Corporation Send Feedback...
Page 195
=1, with ECRC ECRC Core forwards the ECRC =0, without ECRC without ECRC =1, with =1, with ECRC ECRC field is in the ECRC Generation Enable Configuration Space Advanced Error Capabilities and Control Register Optional Features Altera Corporation Send Feedback...
Page 196
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 197
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 198
Your Application Layer design may need to handle at least the following scenarios that are not possible to create with the Altera testbench and the Root Port BFM: • It is unable to generate or receive Vendor Defined Messages. Some systems generate Vendor Defined Messages and the Application Layer must be designed to process them.
Page 199
• altpcietb_bfm_driver_rp—This module drives transactions to the Root Port BFM. This is the module that you modify to vary the transactions sent to the example Endpoint design or your own design. For more information about this module, see Test Driver Module. Testbench and Design Example Altera Corporation Send Feedback...
Page 200
• Length of the transfer • Address of the source • Address of the destination • Control bits to set the handshaking behavior between the software application or BFM driver and the chaining DMA module Testbench and Design Example Altera Corporation Send Feedback...
Page 201
Table Avalon-MM interfaces Avalon-ST Data Hard IP for DMA Write DMA Read PCI Express PCI Express Root Port DMA Control/Status Register DMA Wr Cntl (0x0-4) Configuration DMA Rd Cntl (0x10-1C) RC Slave Testbench and Design Example Altera Corporation Send Feedback...
Page 202
I/Os. These test ports can be used in your design. • <variation name>.v or <variation name>.vhd— Because Altera provides five sample parameteriza‐ tions, you may have to edit one of the provided examples to create a simulation that matches your requirements.
Page 203
• altpcierd_dma_dt—This module arbitrates PCI Express packets issued by the submodules altpcierd_dma_prg_reg, altpcierd_read_dma_requester, altpcierd_write_dma_requester and altpcierd_dma_descriptor. Testbench and Design Example Altera Corporation Send Feedback...
Page 204
The following table shows the mapping. Table 17-1: BAR Map Memory BAR Mapping 32-bit BAR0 Maps to 32 KByte target memory block. Use the rc_slave module to bypass the chaining DMA. 32-bit BAR1 64-bit BAR1:0 Testbench and Design Example Altera Corporation Send Feedback...
Page 205
Base Address of the Read Descriptor Table (BDT) in the RC Memory– DMA Rd Cntl DW1 Upper DWORD 0x18 Base Address of the Read Descriptor Table (BDT) in the RC Memory– DMA Rd Cntl DW2 Lower DWORD Testbench and Design Example Altera Corporation Send Feedback...
Page 206
To stop the infinite loop, set this bit to The following table defines the DMA status registers. These registers are read only. In this table, Addr specifies the Endpoint byte address offset from BAR2 or BAR3. Testbench and Design Example Altera Corporation Send Feedback...
Page 207
[23:21] The following encodings are defined: Max payload size • 001 128 bytes • 001 256 bytes • 010 512 bytes • 011 1024 bytes • 100 2048 bytes [20:17] Reserved — Testbench and Design Example Altera Corporation Send Feedback...
Page 208
The following table describes the Chaining DMA descriptor table. This table is stored in the BFM shared memory. It consists of a four-dword descriptor header and a contiguous list of <n> four-dword descrip‐ Testbench and Design Example Altera Corporation Send Feedback...
Page 210
Table 17-10: Chaining DMA Descriptor Fields Descriptor Field Endpoint RC Access Description Access Endpoint A 32-bit field that specifies the base address of the Address memory transfer on the Endpoint site. Testbench and Design Example Altera Corporation Send Feedback...
Page 211
Chaining DMA Descriptor Table after finishing • The chaining DMA writes the EPLast the data transfer for the first and last descriptors. • The chaining DMA issues an MSI when the last descriptor has completed. Testbench and Design Example Altera Corporation Send Feedback...
Page 212
BFM shared memory data buffer 0 upper address value 0x81c 0x1800 BFM shared memory data buffer 1 lower address value Data 0x1800 Increment by 1 Data content in the BFM shared memory from address: Buffer 0 from 0x1515_ 0x01800–0x1840 0001 Testbench and Design Example Altera Corporation Send Feedback...
Page 213
2. Sets up the chaining DMA descriptor header and starts the transfer data from the Endpoint memory to the BFM shared memory. The transfer calls the procedure which writes four dwords, dma_set_header DW0:DW3, into the DMA write register module. Testbench and Design Example Altera Corporation Send Feedback...
Page 214
BFM shared memory data buffer 0 upper address value 0x91c 0x8DF0 BFM shared memory data buffer 0 lower address value Data 0x8DF0 Increment by 1 Data content in the BFM shared memory from address: Buffer 0 from 0xAAA0_ 0x89F0 0001 Testbench and Design Example Altera Corporation Send Feedback...
Page 215
DW0:DW3 into the DMA read register module. Table 17-18: DMA Control Register Setup for DMA Read Offset in DMA Control Value Description Registers (BAR2) Number of descriptors and control bits as described in Chaining DMA Control Register Definitions. Testbench and Design Example Altera Corporation Send Feedback...
Page 216
• Test Driver (altpcietb_bfm_driver_rp.v)—the chaining DMA Endpoint test driver which configures the Root Port and Endpoint for DMA transfer and checks for the successful transfer of data. Refer to the Test Driver Modulefor a detailed description. Testbench and Design Example Altera Corporation Send Feedback...
Page 217
(altpcietb_bfm_vc_intf) (variation_name.v) You can use the example Root Port design for Verilog HDL simulation. All of the modules necessary to implement the example design with the variation file are contained in altpcietb_bfm_ep_example_chaining_pipen1b.v. Testbench and Design Example Altera Corporation Send Feedback...
Page 218
The basic Root Port BFM provides Verilog HDL task-based interface for requesting transactions that are issued to the PCI Express link. The Root Port BFM also handles requests received from the PCI Express link. The following figure provides an overview of the Root Port BFM. Testbench and Design Example Altera Corporation Send Feedback...
Page 219
• Storing a data structure that contains the sizes of and the values programmed in the BARs of the Endpoint. A set of procedures is provided to read, write, fill, and check the shared memory from the BFM driver. For details on these procedures, see BFM Shared Memory Access Procedures. Testbench and Design Example Altera Corporation Send Feedback...
Page 220
Before you issue transactions to the Endpoint, you must configure the Root Port and Endpoint Configu‐ ration Space registers. To configure these registers, call the procedure , which is included ebfm_cfg_rp_ep in altpcietb_bfm_driver_rp.v. Testbench and Design Example Altera Corporation Send Feedback...
Page 221
The above algorithm cannot always assign values to all BARs when there are a few very large (1 GByte or greater) 32-bit BARs. Although assigning addresses to all BARs may be possible, a more complex algorithm would be required to effectively assign these addresses. However, such a Testbench and Design Example Altera Corporation Send Feedback...
Page 222
BAR4 read back value after being written with all 1’s BAR5 read back value after being written with all 1’s Expansion ROM BAR read back value after being written with all 1’s Testbench and Design Example Altera Corporation Send Feedback...
Page 223
Largest Unused Endpoint Memory Space BARs Prefetchable 32-bit and 64-bit Assigned Smallest to Largest 0xFFFF FFFF If addr_map_4GB_limit is 0, the resulting memory space map is shown in the following figure. Testbench and Design Example Altera Corporation Send Feedback...
Page 224
Assigned Smallest to Largest 0x0000 0001 0000 0000 Endpoint Memory Space BARs Prefetchable 64-bit Assigned Smallest to Largest BAR-Size Dependent Unused 0xFFFF FFFF FFFF FFFF The following figure shows the I/O address space. Testbench and Design Example Altera Corporation Send Feedback...
Page 225
Issuing Read and Write Transactions to the Application Layer Read and write transactions are issued to the Endpoint Application Layer by calling one of the ebfm_bar procedures in altpcietb_bfm_driver_rp.v. The procedures and functions listed below are available in the Testbench and Design Example Altera Corporation Send Feedback...
Page 226
This routine returns as soon as the last transaction has been accepted by the VC interface module. Location altpcietb_bfm_rdwr.v Syntax ebfm_barwr(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass) Testbench and Design Example Altera Corporation Send Feedback...
Page 227
Traffic class used for the PCI Express transaction. tclass ebfm_barwr_imm Procedure procedure writes up to four bytes of data to an offset from the specified Endpoint ebfm_barwr_imm BAR. Location altpcietb_bfm_driver_rp.v Syntax ebfm_barwr_imm(bar_table, bar_num, pcie_offset, imm_data, byte_len, tclass) Testbench and Design Example Altera Corporation Send Feedback...
Page 228
This procedure waits until all of the completion data is returned and places it in shared memory. Location altpcietb_bfm_driver_rp.v Syntax ebfm_barrd_wait(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass) Testbench and Design Example Altera Corporation Send Feedback...
Page 229
Length, in bytes, of the data to be read. Can be 1 to the byte_len minimum of the bytes remaining in the BAR space or BFM shared memory. Traffic Class to be used for the PCI Express transaction. tclass Testbench and Design Example Altera Corporation Send Feedback...
Page 230
This procedure returns as soon as the VC interface module accepts the transaction, allowing other writes to be issued in the interim. Use this procedure only when successful completion status is expected. Testbench and Design Example Altera Corporation Send Feedback...
Page 231
BFM shared memory. This procedure waits until the read completion has been returned. Location altpcietb_bfm_driver_rp.v Syntax ebfm_cfgrd_wait(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr, compl_status) Testbench and Design Example Altera Corporation Send Feedback...
Page 232
Location altpcietb_bfm_driver_rp.v Syntax ebfm_cfgrd_nowt(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr) Testbench and Design Example Altera Corporation Send Feedback...
Page 233
All Verilog HDL arguments are type and are input-only unless specified otherwise. integer ebfm_cfg_rp_ep Procedure procedure configures the Root Port and Endpoint Configuration Space registers for ebfm_cfg_rp_ep operation. Location altpcietb_bfm_driver_rp.v Syntax ebfm_cfg_rp_ep(bar_table, ep_bus_num, ep_dev_num, rp_max_rd_req_size, display_ep_config, addr_map_4GB_limit) Testbench and Design Example Altera Corporation Send Feedback...
Page 234
4 GByte limit. ebfm_cfg_decode_bar Procedure procedure analyzes the information in the BAR table for the specified BAR ebfm_cfg_decode_bar and returns details about the BAR attributes. Location altpcietb_bfm_driver_rp.v Syntax ebfm_cfg_decode_bar(bar_table, bar_num, log2_size, is_mem, is_pref, is_64b) Testbench and Design Example Altera Corporation Send Feedback...
Page 235
Specifies a data pattern of incrementing 8-bit bytes (0x00, 0x01, SHMEM_FILL_BYTE_INC 0x02, etc.) Specifies a data pattern of incrementing 16-bit words (0x0000, SHMEM_FILL_WORD_INC 0x0001, 0x0002, etc.) Specifies a data pattern of incrementing 32-bit dwords SHMEM_FILL_DWORD_INC (0x00000000, 0x00000001, 0x00000002, etc.) Testbench and Design Example Altera Corporation Send Feedback...
Page 236
Bits 7 downto 0 are read from the location specified by addr bits 15 downto 8 are read from the addr+1 location, etc. Testbench and Design Example Altera Corporation Send Feedback...
Page 237
Initial data value used for incrementing data pattern modes. init This argument is reg [63:0] The necessary least significant bits are used for the data patterns that are smaller than 64 bits. Testbench and Design Example Altera Corporation Send Feedback...
Page 238
Certain message types also stop simulation after the message is displayed. The following table shows the default value determining whether a message type stops simulation. You can specify whether simulation stops for particular messages with the procedure ebfm_log_set_stop_on_msg_mask All of these log message constants type integer Testbench and Design Example Altera Corporation Send Feedback...
Page 239
CONTIN Specifies an error that stops EBFM_ FATAL: simulation because the error MSG_ Cannot Cannot leaves the testbench in a state ERROR_ suppress suppress where further simulation is FATAL not possible. Testbench and Design Example Altera Corporation Send Feedback...
Page 240
Also, because Verilog HDL does not allow variable length strings, this routine strips off leading characters of 8’h00 before displaying the message. Return Applies only to the Verilog HDL routine. always 0 Testbench and Design Example Altera Corporation Send Feedback...
Page 241
[EBFM_MSG_ERROR_CONTINUE:EBFM_ MSG_DEBUG] A 1 in a specific bit position of the causes messages msg_mask of the type corresponding to the bit position to stop the simulation after the message is displayed. Testbench and Design Example Altera Corporation Send Feedback...
Page 242
Return data is type with a of 8:1 range himage2 This function creates a two-digit hexadecimal string representation of the input argument that can be concatenated into a larger message string and passed to ebfm_display Testbench and Design Example Altera Corporation Send Feedback...
Page 243
64:1. range himage16 This function creates a 16-digit hexadecimal string representation of the input argument that can be concatenated into a larger message string and passed to ebfm_display Testbench and Design Example Altera Corporation Send Feedback...
Page 244
Returns a 2-digit decimal representation of the input argument string range that is padded with leading 0s if necessary. Return data is type with a of 16:1. range Returns the letter U if the value cannot be represented. Testbench and Design Example Altera Corporation Send Feedback...
Page 245
This function creates a five-digit decimal string representation of the input argument that can be concatenated into a larger message string and passed to ebfm_display. Location altpcietb_bfm_driver_rp.v Syntax string:= dimage(vec) Argument Input data type with a of 31:0. range range Testbench and Design Example Altera Corporation Send Feedback...
Page 246
Returns the letter <U> if the value cannot be represented. Procedures and Functions Specific to the Chaining DMA Design Example The procedures specific to the chaining DMA design example are in the Verilog HDL module file altpcietb_bfm_driver_rp.v. Testbench and Design Example Altera Corporation Send Feedback...
Page 247
When set, the Root Port uses BFM shared memory polling to Use_eplast detect the DMA completion. dma_wr_test Procedure Use the procedure for DMA writes from the BFM shared memory to the Endpoint memory. dma_wr_test Testbench and Design Example Altera Corporation Send Feedback...
Page 248
Address of the Endpoint structure in BFM shared bar_table bar_table memory. Arguments BAR number to analyze. bar_num dma_set_header Procedure Use the procedure to configure the DMA descriptor table for DMA read or DMA write. dma_set_header Testbench and Design Example Altera Corporation Send Feedback...
Page 249
MSI traffic class which is set Multi_message_enable use_msi by the procedure. dma_set_msi rc_mempoll Procedure Use the procedure to poll a given dword in a given BFM shared memory location. rc_mempoll Location altpcietb_bfm_driver_rp.v Syntax rc_mempoll (rc_addr, rc_data, rc_mask) Testbench and Design Example Altera Corporation Send Feedback...
Page 250
Procedure procedure sets PCI Express native MSI for the DMA read or the DMA write. dma_set_msi Location altpcietb_bfm_driver_rp.v Syntax dma_set_msi(bar_table, bar_num, bus_num, dev_num, fun_num, direction, msi_ address, msi_data, msi_number, msi_traffic_class, multi_message_enable, msi_expected) Testbench and Design Example Altera Corporation Send Feedback...
Page 251
Returns the expected MSI data value, which is msi_expected msi_data modified by the chosen. msi_number find_mem_bar Procedure procedure locates a BAR which satisfies a given memory space requirement. find_mem_bar Location altpcietb_bfm_driver_rp.v Syntax Find_mem_bar(bar_table,allowed_bars,min_log2_size, sel_bar) Testbench and Design Example Altera Corporation Send Feedback...
Page 252
The message string is limited to a maximum of 100 characters. message Also, because Verilog HDL does not allow variable length strings, this routine strips off leading characters of 8'h00 before displaying the message. Testbench and Design Example Altera Corporation Send Feedback...
Page 253
Changing the simulation parameters reduces simulation time and provides greater visibility. Changing Between Serial and PIPE Simulation By default, the Altera testbench runs a serial simulation. You can change between serial and PIPE simulation by editing the top-level testbench file. For Endpoint designs, the top-level testbench file is <working_dir>/<instantiation_name>_tb/<instantiation_name>_tb/sim/<instantiation_name>_tb.v...
Page 254
Disabling 8B/10B Encoding and Decoding for Gen1 and Gen2 Simulations You can disable 8B/10B encoding and decoding to facilitate debugging. For Gen1 and Gen2 variants, you can disable 8B/10B encoding and decoding by setting test_in[2] = 1 in altpcietb_bfm_top_rp.v. Testbench and Design Example Altera Corporation Send Feedback...
Page 255
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 256
UG-01145_avst 18-2 Link Training 2015.11.02 The following sections, describe how to debug the hardware bring-up flow. Altera recommends a systematic approach to diagnosing bring-up issues as illustrated in the following figure. Figure 18-1: Debugging Link Training Issues Successful Does Link...
Page 257
Arria 10 Hard IP for PCI Express IP Core credit calculation is out-of-sink with its link partner. Debugging Altera Corporation Send Feedback...
Page 258
RX buffer. Related Information PIPE Interface Signals • on page 6-55 Avalon Interface Specifications • PCI Express Base Specification 3.0 • • Design Debugging Using the SignalTap II Embedded Logic Analyzer Debugging Altera Corporation Send Feedback...
Page 259
Both FPGA programming (configuration) and the initialization of a PCIe link require time. Potentially, an Altera FPGA including a Hard IP block for PCI Express may not be ready when the OS/BIOS begins enumeration of the device tree. If the FPGA is not fully programmed when the OS/BIOS begins its enumeration, the OS does not include the Hard IP for PCI Express in its device map.
Page 260
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 261
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 262
PCB. Connecting the lanes without lane reversal creates routing problems. Using lane reversal solves the problem. No Lane Reversal With Lane Reversal Results in PCB Routing Challenge Signals Route Easily Root Port Endpoint Root Port Endpoint no lane lane reversal reversal Lane Initialization and Reversal Altera Corporation Send Feedback...
Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
Page 264
Root Ports. It is always 0 for Endpoints. • Corrected link to Arria 10 Avalon-MM DMA Interface for PCIe Solutions User Guide. • Added Enable Altera Debug Master Endpoint (ADME) parameter to support optional Native PHY register programming with the Altera System Console.
Page 265
• Enhanced instructions Compiling the Design to include steps necessary to download to Altera development kits. 2014.08.18 14.0a10 Made the following changes to the Arria 10 Hard IP for PCI Express: • Changed the PIPE interface to 32 bits for all data rates. This change requires you to recompile your 13.1 variant in 14.0.
Page 266
15 bits. lmi_addr • Changed the directory structure for generated files. Refer to Files Generated for Altera IP Cores Targeting Arria 10 for more information. • In the Getting Started with the Arria 10 Hard IP for PCI Express chapter, changed the recommended device to 10AX115R2F40I2LG (Advanced).
Page 267
• Removed port from connect between PHY IP reconfig_busy Core for PCI Express and the Transceiver Reconfiguration Controller in the Altera Transceiver Reconfiguration Controller Connectivity figure. The Transceiver Reconfiguration Controller drives port to the Altera PCIe Reconfig Driver. reconfig_busy •...
Page 268
• Added limitation for when interrupts are rxm_irq_<n>[<m>:0] received on consecutive cycles. • Corrected description of . It is the Base/Primary cfg_prm_cmr Command register for the PCI Configuration Space. • Revised channel placement illustrations. Additional Information Altera Corporation Send Feedback...
Page 269
Core. Offset cancellation is not required for Gen1 or Gen2 operation. 2011.07.30 11.01 Corrected typographical errors. 2011.05.06 11.0 First release. How to Contact Altera To locate the most up-to-date information about Altera products, refer to the following table. Additional Information Altera Corporation Send Feedback...
Website www.altera.com/literature Nontechnical support (general) Email nacomp@altera.com (software licensing) Email authorization@altera.com Note to Table: 1. You can also contact your local Altera sales office or sales representative. Related Information • Technical Support • Technical Training Customer Training • Product Documentation •...
Page 271
The hand points to information that requires special attention. The question mark directs you to a software help system with related information. The feet direct you to another document or website with related information. Additional Information Altera Corporation Send Feedback...
Page 272
The Subscribe button links to the Email Subscription Management Center page of the Altera website, where you can sign up to receive update notifications for Altera documents. The Feedback icon allows you to submit feedback to Altera about the document. Methods for collecting feedback vary as appropriate for each document.
Need help?
Do you have a question about the Arria 10 Avalon-ST Interface and is the answer not in the manual?
Questions and answers