Download
Share
Print this page
Nvidia ConnectX-5 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Computer Hardware
  5. Mellanox ConnectX-5
  6. User manual
Nvidia ConnectX-5 User Manual

Nvidia ConnectX-5 User Manual

Infiniband/ethernet adapter cards for ocp spec 2.0
Hide thumbs Also See for ConnectX-5:
1
Table Of Contents
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990

Advertisement

Quick Links

Download this manual
NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards
for OCP Spec 2.0 User Manual

Advertisement

loading
Need help?

Need help?

Do you have a question about the ConnectX-5 and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia ConnectX-5

Summary of Contents for Nvidia ConnectX-5

  • Page 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 2: Table Of Contents

    Speci cations Speci cations 2728 Monitoring Monitoring 2735 Finding the MAC on the Adapter Card Finding the MAC on the Adapter Card 2736 Document Revision History Document Revision History 2738 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 3 EOL'd (End of Life) Ordering Part Numbers The table below provides the ordering part numbers (OPN) for ConnectX-5 In niBand/Ethernet adapter cards for OCP Spec 2.0. NVIDIA SKU...
  • Page 4 This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with Ethernet network and architecture speci cations. Technical Support Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: https://www.nvidia.com > Support E-mail: enterprisesupport@nvidia.com...
  • Page 5 MLNX _OFED User Manual describing OFED features, Linux performance, band User diagnostic, tools Manua content, and l and con guration. See Releas MLNX_OFED for Linux Documentation. Notes NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 6 (mlxup) Documentation. l and Releas Notes NVIDI Firmw User Manual describing the set of MFT rmware Tools management tools for a (MFT) single node. See MFT User User Manual. Manua NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 7 LinkX NVIDIA LinkX In niBand Interc cables and transceivers onnec are designed to maximize the Soluti performance of High- Performance Computing networks, requiring high-bandwidth, low- latency connections between compute NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 8 When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 9: Introduction

    Type 2 Stacking Stacking Height Height Height Height Type Singl e/Mul Single- Single-Host Multi-Host Host Host Host Mana geme Netw Dual-port belly-to- Conn Single-port QSFP28 belly ector QSFP28 Type NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 10 4120 for Virtual Function (VF) 4122 for Virtual Function (VF) For more detailed information see Speci cations. OCP Spec 2.0 Stacking Heights OCP Spec 2.0 Type 1 Stacking Height Note NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 11 This section applies to MCX542A-ACAN. The ingle-port 100Gb/s adapter card comply with OCP Spec 2.0 Type 1 with 8mm stacking height. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 12 OCP Spec 2.0 Type 2 Stacking Height Note This gure applies to MCX545A-ECAN, MCX545M-ECAN. The single-port 100Gb/s adapter cards follow OCP Spec 2.0 Type 2 with 12mm stacking height. Single-port Card NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 13: Features And Bene Ts

    Features and Bene ts Note This section describes hardware features and capabilities. Please refer to the relevant driver and/or rmware release notes for feature NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 14 PCIe SMBus. Note: A ddress 0x58 is reserved.) Overlay Networks In order to better scale their networks, data center operators often create overlay networks that carry tra c from individual NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 15 • NVMe over Fabric o oads for the target machine • Erasure Coding • T10-DIF Signature Handover ConnectX-5 SR-IOV technology provides dedicated adapter SR-IOV resources and guaranteed isolation and protection for virtual machines (VM) within the server. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 16: Operating Systems/Distributions

    Host technology to deliver direct and independent PCIe connections to each of the four CPUs in the server. The ConnectX-5 PCIe x16 interface is separated into four independent PCIe x4 interfaces. Each interface is connected to a separate host with no performance degradation.
  • Page 17 OpenFabrics Enterprise Distribution (OFED) OpenFabrics Windows Distribution (WinOF-2) Connectivity Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches Passive copper cable with ESD protection Powered connectors for optical and active cable support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 18: Interfaces

    Note The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables. PCI Express Interface The table below describes the supported PCIe interface in ConnectX-5 adapter cards. ConnectX- Supported PCIe Interface Features 5 IC Link Rates: 2.5.

  • Page 19: Led Interface

    PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA PCIe stand-up adapter. For con guring the adapter for the speci c manageability solution in use by the server, please contact NVIDIA Support.

  • Page 20: Hardware Installation

    Hardware Installation Installation and initialization of ConnectX-5 adapter cards for OCP Spec 2.0 require attention to the mechanical attributes, power speci cation, and precautions for electronic equipment. Safety Warnings Info Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety Instructions.

  • Page 21: Installation Procedure Overview

    CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Installation Procedure Overview The installation procedure of ConnectX-5 adapter cards for OCP Spec 2.0 involves the following steps: Step Procedure Direct Link Check the system’s hardware and...

  • Page 22: System Requirements

    A system with a PCI Express x16 slot (two FCI B2B x8 connectors) is required for installing the card. ConnectX-5 cards for OCP are complaint with automatic hosts-count detection NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 23: Software Requirements

    (http://www.opencompute.org/wiki/Server/SpecsAndDesigns). Air ow Requirements ConnectX-5 adapter cards are o ered with two air ow patterns: from the heatsink to the network ports, and vice versa, as shown below. Please refer to Speci cations for air ow numbers for each speci c card model.
  • Page 24 Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock may harm the EMI ngers on the networking connector. Carefully remove the card from the antistatic bag to avoid damaging the EMI ngers. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 25: Cables And Modules

    PCI Express slot until rmly seated. 5. Secure the adapter with the adapter clip or screw. To uninstall the adapter card, see Uninstalling the Card. Cables and Modules Cable Installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 26 4. After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When a logical connection is made, Green LED1 will light. When data is being transferred, Green LED1 will blink. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 27 1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. 2. Expand System Devices and locate your NVIDIA ConnectX-5 adapter card. 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window.
  • Page 28 5. In the Value display box, check the elds VEN and DEV ( elds are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-5) – this is a valid NVIDIA PCI Device ID.
  • Page 29 2. Wait 30 seconds. 3. To remove the card, disengage clip 1 and 2 on connector A side. 4. To disconnect connector A, gently pull the adapter card upwards. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 30 5. Disengage clip 3 and clip 4 on the adapter card on Connector B side. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 31 6. To remove the card, gently pull the adapter card upwards. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...

  • Page 32: Driver Installation

    Please use the relevant driver installation section. Linux Driver Installation Windows Driver Installation VMware Driver Installation Linux Driver Installation This section describes how to install and test the NVIDIA OFED for Linux package on a single server with a ConnectX-5 adapter card installed. Prerequisites Requirements Description...
  • Page 33 1. Verify that the system has an NVIDIA network adapter installed. The following example shows a system with an installed NVIDIA adapter card: Requirements Description A server platform with one of the following adapters installed: Platforms ConnectX®-5 Ex (EN) ( rmware: fw-ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5)
  • Page 34 Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="203cc571-506f-4f4c-b960- b4a327d68f18"><ac:plain-text-body><![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body></ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_LINUX- <ver>-<OS label>.iso Installing MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 35 The installation requires administrator (root) privileges Installer Privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema-version="1" ac:macro- id="41aa4530-f1d1-4506-8492-de3cd0d36b44"> <ac:plain-text-body><![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body></ac:structured-macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 36 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema-version="1" ac:macro- id="41aa4530-f1d1-4506-8492-de3cd0d36b44"> <ac:plain-text-body><![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body></ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 37 Usage Requireme Description A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) Platforms ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw- ConnectX5) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 38 1. lspci - Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="203cc571-506f-4f4c- b960-b4a327d68f18"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 39 To avoid this, make sure to add --skip- unsupport devices- check ag during installation <ac:structu ./mnt/mlnxofedinstall ]]></ac:plain-text-body> red-macro [OPTIONS] </ac:structured-macro> ac:name=" unmigrate d-wiki- markup" ac:schema- version="1" ac:macro- id="ee2f54 c0-5ec8- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 40 MNLX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 41 Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort installation procedure. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 42 ISO image on one of the cluster nodes and then copy it to a shared le system such as NFS. To install on all the cluster nodes, use cluster-aware tools (suchaspdsh). NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 43 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 44 If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupport devices- check ag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 45 ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to use the mlnx_add_ kernel_sup port.sh NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 46 MNLX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 47 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 48 [OPTIONS] </ac:structured-macro> c0-5ec8- 4127- 8227- b4709ab35 0fc"> <ac:plain- text-body> <![CDATA[ Pre- existing con gurati on les will be saved with the extension ".conf.rpms ave". On Redhat and SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 49 MLNX_OFE D kernel modules. If you regenerate kernel modules for a custom kernel (using -- add- kernel- support), packages installation will not involve automatic regeneratio n of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 50 'mdev' already exists with di erent path| The "mlnx_add_kernel_support.sh" script can be executed directly from the mlnxofedinstall script. For further information, please see '--add-kernel- support' option below. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 51 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 52 If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupport devices- check ag during installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 53 ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to use the mlnx_add_ kernel_sup port.sh script. The regular installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 54 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regeneratin g the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 55 Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on the host during MLNX_OFE installation. Therefore, using "mlnx_add_ kernel_sup NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 56 MNLX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 57 D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 58 0fc"> <ac:plain- text-body> <![CDATA[ Pre- existing con gurati on les will be saved with the extension ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 59 (using -- add- kernel- support), packages installation will not involve automatic regeneratio n of the initramfs. In some cases, such as a system NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 60 -f On Ubuntu and Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 61 1. ./MLN X_OF ED_LI NUX- x.x-x- rhel7. x86_6 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make -tgz Note: This progr NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 62 _OFE D_LIN rhel7. under /tmp direct ory. Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro i18nk editor NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 63 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 64 ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation Linux operating system. For the list of supported operating system Operating distributions and kernels, please System refer to the MNLX_OFED Release Notes NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 65 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_fw _bin_ les If the driver detects NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 66 ./mnt/mlnxofedinstall [OPTIONS] </ac:structured-macro> id="ee2f54c 0-5ec8- 4127-8227- b4709ab35 0fc"> <ac:plain- text-body> <![CDATA[ Pre-existing con guratio n les will be saved with the extension ".conf.rpms ave". On Redhat and SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 67 D kernel modules. If you regenerate kernel modules for a custom kernel (using -- add-kernel- support), packages installation will not involve automatic regeneratio n of the initramfs. In some cases, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 68 -f On Ubuntu and Debian distribution s drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 69 Ubuntu and Debian distribution 1. ./MLN X_OFE D_LIN x.x-x- rhel7. x86_6 4/mlnx _add_ kernel _supp ort.sh /tmp/ MLNX _OFED _LINU X-x.x- rhel7. x86_6 4/ -- make- Note: This progra NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 70 /tmp direct ory. Mellan OEM, OFED, Distrib ution packa will be remov image s/plugi ns/ser vlet/co n uen ce/pla cehold er/erro i18nke editor- placeh older- broke n-link- locale- en_us- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 71 Remov OFED RPMs.. Create /tmp/ MLNX _OFED _LINU X-x.x- rhel7. x86_6 ext.tgz ./mlnxofedi nstall --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 72 Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 73 --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupport NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 74 ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to use the mlnx_add_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 75 (using -- add- kernel- support), packages installation will not involve automatic regeneratio n of the initramfs. In some cases, such as a system with a root lesystem mounted over a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 76 -f On Ubuntu and Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on the host during MLNX_OFE NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 77 X_OF ED_LI NUX- x.x-x- rhel7. x86_6 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make -tgz Note: This progr will creat NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 78 /tmp direct ory. Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro i18nk editor place holde broke NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 79 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 80 Operating system distributions and System kernels, please refer to the MNLX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 81 [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 82 ./mnt/mlnxofedinstall ]]></ac:plain-text-body> id="ee2f54 [OPTIONS] </ac:structured-macro> c0-5ec8- 4127- 8227- b4709ab35 0fc"> <ac:plain- text-body> <![CDATA[ Pre- existing con gurati on les will be saved with the extension NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 83 MLNX_OFE D kernel modules. If you regenerate kernel modules for a custom kernel (using -- add- kernel- support), packages installation will not NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 84 In such cases, the installer will recommen d running following command to update initramfs: dracut -f On Ubuntu and Debian distributio ns drivers installation Dynamic Kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 85 Ubuntu and Debian distributio 1. ./MLN X_OF ED_LI NUX- x.x-x- rhel7. x86_6 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 86 Note: This progr will creat MLNX _OFE D_LIN rhel7. under /tmp direct ory. Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 87 /place holde r/erro i18nk editor place holde broke link- locale en_us versio 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 88 MLNX_OFE D_LINUX<v er><OS label> <CPU arch>.iso /mnt /mnt/mlnxo fedinstall Logs dir: /tmp/MLN X_OFED_LI NUX-x.x- x.logs This program will install MLNX_OFE D_LINUX package on your machine. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 89 MLNX_OFE D_LINUX, do not reinstall them. Starting MLNX_OFE D_LINUX- x.x.x installation ....Installation nished successfull Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 90 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 91 If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupport devices- check ag during installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 92 ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to use the mlnx_add_ kernel_sup port.sh script. The regular installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 93 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regeneratin g the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 94 Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on the host during MLNX_OFE installation. Therefore, using "mlnx_add_ kernel_sup NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 95 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make -tgz Note: This progr will creat MLNX _OFE D_LIN NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 96 Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro i18nk editor place holde broke link- locale en_us NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 97 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 ext.tg ./mlnxofedi nstall --h NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 98 D_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distributio n IB packages will be removed. Those packages removed due to con icts with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 99 Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattende installation, use the -- force installation option while running MLNX_OFE installation script: /mnt/mlnxo fedinstall -- force NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 100 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 101 If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupport devices- check ag during installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 102 ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to use the mlnx_add_ kernel_sup port.sh script. The regular installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 103 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regeneratin g the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 104 Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on the host during MLNX_OFE installation. Therefore, using "mlnx_add_ kernel_sup NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 105 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make -tgz Note: This progr will creat MLNX _OFE D_LIN NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 106 Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro i18nk editor place holde broke link- locale en_us NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 107 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 ext.tg ./mlnxofedi nstall --h NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 108 D_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distributio n IB packages will be removed. Those packages removed due to con icts with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 109 Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattende installation, use the -- force installation option while running MLNX_OFE installation script: /mnt/mlnxo fedinstall -- force NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 110 --force For example: ./mlnxofedi nstall -- without- dkms -- add- kernel- support -- kernel 3.13.0-85- generic -- without- fw-update --force Note that the path to kernel sources (- kernel- sources) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 111 (root) privileges Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 112 --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 113 Pre- existing con gurati on les will be saved with the extension ".conf.rpms ave". On Redhat and SLES distributio ns with errata kernel installed there is no need to NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 114 (using -- add- kernel- support), packages installation will not involve automatic regeneratio n of the initramfs. In some cases, such as a system with a root lesystem NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 115 -f On Ubuntu and Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilatio n will take place on the host during NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 116 1. ./MLN X_OF ED_LI NUX- x.x-x- rhel7. x86_6 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make -tgz Note: This progr will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 117 /tmp direct ory. Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro i18nk editor place holde NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 118 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 119 X_OFED_LI NUX-x.x- x.logs This program will install MLNX_OFE D_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distributio n IB packages will be removed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 120 D_LINUX- x.x.x installation ....Installation nished successfull Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattende installation, use the -- force installation option while running MLNX_OFE NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 121 -- kernel <kernel version in chroot> -- without- fw-update --force For example: ./mlnxofedi nstall -- without- dkms -- add- kernel- support -- kernel 3.13.0-85- generic -- without- fw-update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 122 Device #1: ---------- Device Type: ConnectX- X Part Number: MCXXXX- XXX PSID: MT_<versio NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 123 MNLX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci - grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 124 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupport ed cards on the system, it will abort NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 125 [OPTIONS] </ac:structured-macro> c0-5ec8- 4127- 8227- b4709ab35 0fc"> <ac:plain- text-body> <![CDATA[ Pre- existing con gurati on les will be saved with the extension ".conf.rpms ave". On Redhat and SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 126 MLNX_OFE D kernel modules. If you regenerate kernel modules for a custom kernel (using -- add- kernel- support), packages installation will not involve automatic regeneratio n of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 127 In such cases, the installer will recommen d running following command to update initramfs: dracut -f On Ubuntu and Debian distributio ns drivers installation Dynamic Kernel Module Support (DKMS) framework. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 128 Ubuntu and Debian distributio 1. ./MLN X_OF ED_LI NUX- x.x-x- rhel7. x86_6 4/mln x_add _kern el_su pport. sh -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 129 MLNX _OFE D_LIN rhel7. under /tmp direct ory. Mella nox, OEM, OFED, Distri butio n IB packa will remo ved. image s/plug ins/se rvlet/ uence /place holde r/erro NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 130 2.png > /tmp/ mlnx_ ofed_i 2164 2 .log Buildi OFED RPMs Pleas wait... Remo ving OFED RPMs Creat /tmp/ MLNX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 131 <CPU arch>.iso /mnt /mnt/mlnxo fedinstall Logs dir: /tmp/MLN X_OFED_LI NUX-x.x- x.logs This program will install MLNX_OFE D_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 132 Starting MLNX_OFE D_LINUX- x.x.x installation ....Installation nished successfull Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattende installation, use the -- force NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 133 ./mlnxofedi nstall - without- dkms -- add- kernel- support -- kernel <kernel version in chroot> -- without- fw-update --force For example: ./mlnxofedi nstall -- without- dkms -- add- kernel- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 134 Device #1: ---------- Device Type: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 135 Please contact your hardware vendor for help with rmware updates. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 136 Versions: Current Available XX.XX.XXX X Status: matching image found Error message #2: The rmware for this device is distributed inside NVIDIA driver: 0000:01:00 .0 (PSID: IBM21501 10033) To obtain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 137 For the list of supported operating Operatin system distributions and kernels, please System refer to the MNLX_OFED Release Notes The installation Installer requires administrator Privilege (root) privileges on the target machine. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 138 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_O FED_LIN NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 139 If the driver detects unsuppo rted cards on system, it will abort installati procedu re. To avoid this, make sure to add -- skip- unsuppo rted- devices- check during installati NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 140 5ec8- 4127- 8227- b4709ab 350fc"> <ac:plain -text- body><! [CDATA[ Pre- existing con gur ation les will be saved with the extensio ".conf.rp msave". Redhat SLES distribut ions with errata NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 141 MLNX_O kernel modules. If you regenera te kernel modules for a custom kernel (using -- add- kernel- support), package NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 142 Connect X card, regenera ting the initramfs even cause system to fail to reboot. In such cases, installer will recomm running followin comman NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 143 Support (DKMS) framewo rk. Thus, drivers' compilat ion will take place on the host during MLNX_O installati Therefor e, using "mlnx_a dd_kern el_suppo rt.sh" is irrelevan t on Ubuntu NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 144 Debian distribut ions. 1. ./M x.x- ort. x.x- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 145 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 146 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 147 > .log ldin t... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 148 ./mlnxof edinstall host1# mount o ro,loop MLNX_O FED_LIN UX<ver> <OS label> <CPU arch>.iso /mnt NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 149 Mellanox , OEM, OFED, RDMA or Distribut ion IB package s will be removed Those package s are removed due to con icts with MLNX_O FED_LIN UX, do NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 150 ....Installati nished successf ully. Attempti ng to perform Firmwar update... Queryin Mellanox devices rmware unatten installati on, use the -- force installati option while running MLNX_O installati NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 151 ./mlnxof edinstall without- dkms -- add- kernel- support --kernel <kernel version chroot> without- update - -force example: ./mlnxof edinstall without- dkms -- add- kernel- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 152 In case your machine has the latest rmware , no rmware update will occur and the installati on script will print at the end of installati NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 153 X-XXX PSID: MT_<ver sion> Device Name: 0b:00.0 Base MAC: 0000e41 d2d5cf8 Versions: Current Available XX.XX.X Status: Up to date In case your machine has an unsuppo rted network adapter NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 154 Error message Device #1: ------- Device Type: Connect X-X Part Number: MCXXX X-XXX PSID: MT_<ver sion> Device Name: 0b:00.0 Base MAC: 0000e41 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 155 IBM215 011003 3) To obtain rmware for this device, please contact your HW vendor. Action \ Standard Reboot (Soft Cold Reboot Driver Restart Adapter Reset) (Hard Reset) Standar Connect NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 156 The list of the modules that will be loaded automatically upon boot can be found in the /etc/in niband/openib.conf le. Require Description ments NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 157 1. lsp grep Mellanox ci - <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1-4506- 8492-de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 158 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_O FED_LIN <ver>- <OS label>.iso ./mlnxofe dinstall -- image- /tmp/my _fw_bin_ If the driver detects unsuppo rted cards on system, it will abort NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 159 <ac:struc tured- macro ac:name ="unmigr ated- wiki- markup" ac:sche version=" 1" ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro [OPTIONS] </ac:structured-macro> id="ee2f 54c0- 5ec8- 4127- 8227- b4709ab 350fc"> <ac:plain -text- body><! [CDATA[ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 160 _support script. regular installati on can perform ed and weak- updates mechani sm will create symbolic links to NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 161 . In some cases, such as a system with a root lesyste mounted over a Connect X card, regenera ting the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 162 -f Ubuntu Debian distributi drivers installati on use Dynamic Kernel Module Support (DKMS) framewo rk. Thus, drivers' compilati on will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 163 MLNX_O installati Therefor e, using "mlnx_ad d_kernel _support .sh" is irrelevan t on Ubuntu Debian distributi ons. 1. ./M x.x- rhel 7. 3 rnel rt.s NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 164 7. 3 / -- will rhel 7. 3 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 165 IB will /plu ervl et/c er/e rror NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 166 > iso. .log Buil t... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 167 7. 3 ext. ./mlnxofe dinstall -- host1# mount o ro,loop MLNX_O FED_LIN UX<ver> <OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 168 Mellanox , OEM, OFED, RDMA or Distributi on IB package s will be removed. Those package s are removed due to con icts with MLNX_O FED_LIN UX, do NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 169 Attempti ng to perform Firmware update... Querying Mellanox devices rmware unattend installati on, use the -- force installati option while running MLNX_O installati on script: /mnt/mln NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 170 - -kernel <kernel version chroot> - -without- update -- force For example: ./mlnxofe dinstall -- without- dkms -- add- kernel- support - -kernel 3.13.0- generic - -without- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 171 : Device #1: ------- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 172 0000e41 d2d5cf8 Versions: Current Available XX.XX.X Status: Up to date In case your machine has an unsuppo rted network adapter device, rmware update will occur and one of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 173 Error message Device #1: ------- Device Type: Connect X-X Part Number: MCXXXX -XXX PSID: MT_<ver sion> Device Name: 0b:00.0 Base MAC: 0000e41 d2d5cf8 Versions: Current Available XX.XX.X NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 174 HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) Standar Connect 4/Conne ctX-4 Lx or higher Adapter s with Multi- Host Support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 175 ( rmware: fw- ConnectX5) Required Disk Space for Installation Linux operating system. For the list of supported operating system Operating distributions and System kernels, please refer to the MNLX_OFED Release Notes NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 176 1. lspci -v 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="203cc571- 506f-4f4c-b960- b4a327d68f18"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 177 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="ee2f54c0 -5ec8-4127- 8227- b4709ab350f c"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with the extension NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 178 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 179 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 180 - /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 181 -editor- placehol der- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 182 /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 183 Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 184 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no rmware update will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 185 Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages below will be NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 186 Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 187 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 188 '- without fw- update' The rmware version of the adapter device is older than NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 189 To obtain rmware for this device, please contact your HW vendor." Installation Logging While installing MLNX_OFED, the install log for each selected package will be saved in a separate log le. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 190 Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="41aa4530- f1d1-4506-8492- de3cd0d36b44"> <ac:plain-text-body><! [CDATA[86:00.0 Network NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 191 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 192 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 193 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 194 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_L INUX- x.x-x- rhel7. 3 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 195 MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be remove images/ plugins/ servlet/ con ue NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 196 See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 197 RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 198 Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 199 Device #1: ---- ------ Device Type: ConnectX-X Part Number: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 200 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 201 HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 202 (rdma tool) - installed under /opt/Mellanox /iproute2/sbin /rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mlnx -ofa_kernel on RHEL and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 203 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, the automatic rmware update will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 204 .IBMM21501 10033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: 1. Add the following lines to the "/etc/modprobe.d/mlnx.conf" le. Requirement Description...
  • Page 205 Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 206 <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 207 ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 208 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with a root lesystem NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 209 Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 210 /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 211 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 212 This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 213 -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 214 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no rmware update will occur and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 215 XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages below will be printed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 216 Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 217 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 218 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 219 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 220 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib 2. Set "ONBOOT=no" in the "/etc/in niband/openib.conf" le. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 221 MNLX_OFED Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 222 [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 223 [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 224 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 225 Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 226 --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 227 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 228 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 229 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 230 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a message similar to the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 231 Please contact your hardware vendor for help with rmware updates. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 232 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 233 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox /iproute2/sbin /rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 234 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: If an adapter's Flash was NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 235 To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 236 ConnectX®-5 Ex Platforms (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw- ConnectX5) Required Disk Space for Installation Operating Linux operating system. System For the list of supported NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 237 </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="203cc571- 506f-4f4c-b960- b4a327d68f18"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 238 --skip- unsupported -devices- check ag during installation. <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="ee2f54c0 -5ec8-4127- 8227- b4709ab350f c"><ac:plain- text-body><! [CDATA[ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 239 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 240 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 241 OFED_L INUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_kern el_supp ort.sh - /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 242 -editor- placehol der- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 243 /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 244 Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 245 -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their default location. In case your machine has NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 246 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 247 MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 248 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 249 '- without fw- update' The rmware version of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 250 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 251 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 252 '- without fw- update' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 253 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 254 MNLX_OFED Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 255 [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 256 [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 257 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 258 Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 259 --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 260 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 261 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 262 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 263 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a message similar to the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 264 Please contact your hardware vendor for help with rmware updates. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 265 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 266 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox /iproute2/sbin /rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 267 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: If an adapter's Flash was NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 268 To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 269 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox /iproute2/sbin /rdma The kernel modules are installed under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 270 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 271 NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs Uninstalling MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 272 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation Linux operating system. For the list of supported Operating operating system System distributions and kernels, please refer to the MNLX_OFED Release Notes NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 273 [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 274 To avoid this, make sure to add --skip- unsupported -devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- wiki-markup" ac:schema- version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 275 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 276 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f On Ubuntu and Debian NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 277 "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 278 /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- broken- link- locale- en_us- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 279 Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 280 MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 281 --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall -- without- dkms --add- kernel- support -- kernel 3.13.0-85- generic -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 282 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 283 Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 284 Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 285 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 286 '-without update' rmware version of adapter device is older than rmware NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 287 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 288 This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 289 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 290 '-without update' rmware version of adapter device is older than rmware NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 291 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 292 MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 3. Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY- Mellanox Requirement Description A server platform with one of the following adapters installed: ConnectX®-5...
  • Page 293 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 294 ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 295 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 296 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 297 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 298 MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 299 _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 300 RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 301 Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 302 Device #1: --- ------- Device Type: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 303 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: --- ------- Device Type: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 304 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 305 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 306 '-without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 307 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 308 Return Code Meaning The Installation ended successfully The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 309 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 310 '-without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 311 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 312 : 80 ... connected. ]]></ac:plain- http: HTTP request sent, text-body> //www. awaiting response... </ac:structured mellano 200 OK -macro> x.com/d <ac:structured-macro ownloa ac:name="unmigrated ds/ofed -wiki-markup" /RPM- ac:schema- GPG- version="1" ac:macro- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 313 1df9d9-fa9f- 4b0a-a545- 5f05f950ce9b"> <ac:plain-text-body> <![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 4. Install the key. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 314 Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 315 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 316 ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 317 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 318 On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 319 /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 320 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 321 Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 322 Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 323 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 324 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 325 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 326 RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 327 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 328 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 329 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 330 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 331 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 332 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 333 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 334 <ac:plain-text-body> ellanox. <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 335 -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 336 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 337 Requirement Description A server platform with one of the following adapters installed: ConnectX®-5 Platforms Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 338 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 339 /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupported -devices- check ag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 340 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular installation can be performed and weak- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 341 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 342 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 343 MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 344 Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 345 IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 346 ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 347 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 348 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 349 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 350 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 351 '-without update' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 352 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 353 The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 354 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 355 '-without update' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 356 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 357 : 80 ... connected. ]]></ac:plain- http: HTTP request sent, text-body> //www. awaiting response... </ac:structured mellano 200 OK -macro> x.com/d <ac:structured-macro ownloa ac:name="unmigrated ds/ofed -wiki-markup" /RPM- ac:schema- GPG- version="1" ac:macro- KEY- id="ddfd0a12-60b5- 4a59-89e1- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 358 <ac:plain-text-body> <![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 359 Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 360 OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a "> <ac:plai n-text- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 361 ConnectX®-5 Platforms Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation Operating Linux operating System system. For the list of supported operating system NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 362 <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 363 To avoid this, make sure to add --skip- unsupported -devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 364 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 365 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 366 Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 367 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- broken- link- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 368 Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 369 MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 370 -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall -- without- dkms --add- kernel- support -- kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 371 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 372 Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 373 Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 374 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 375 '-without update' rmware version of adapter device is older than NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 376 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 377 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 378 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 379 '-without update' rmware version of adapter device is older than NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 380 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 381 KEY- id="ddfd0a12-60b5- Mellano 4a59-89e1- 7ea7e73df37b"> - 2018 <ac:plain-text-body> 01 25 <![CDATA[Length: 13 : 52 : 1354 ( 1 .3K) 30 - [text/plain] http: ]]></ac:plain-text- //www. body></ac:structured- mellano macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 382 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 383 ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan Technol ogies - Signing Key v2) <suppo rt@mell NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 384 <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a "> <ac:plai n-text- body><! [CDATA [Is this [y/N]: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 385 RPM-GPG-KEY- [CDATA[ Mellanox> gpgcheck= 1 7. Check that the repository was successfully added. Requirement Description Platforms A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 386 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 387 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 388 <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 389 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 390 Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 391 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 392 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 393 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 394 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 395 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 396 Please contact your hardware vendor for help with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 397 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 398 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 399 RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 400 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 401 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 402 MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 403 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 404 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 405 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 406 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 407 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 408 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 409 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 410 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 411 Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation Operating Linux operating System system. For the list of supported operating system distributions and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 412 <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 413 To avoid this, make sure to add --skip- unsupported -devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- wiki-markup" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 414 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 415 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 416 Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 417 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- broken- link- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 418 Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 419 MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 420 -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall -- without- dkms --add- kernel- support -- kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 421 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 422 Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 423 Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 424 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 425 '-without update' rmware version of adapter device is older than NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 426 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 427 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 428 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 429 '-without update' rmware version of adapter device is older than NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 430 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 431 KEY- id="ddfd0a12-60b5- Mellano 4a59-89e1- 7ea7e73df37b"> - 2018 <ac:plain-text-body> 01 25 <![CDATA[Length: 13 : 52 : 1354 ( 1 .3K) 30 - [text/plain] http: ]]></ac:plain-text- //www. body></ac:structured- mellano macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 432 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 433 ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan Technol ogies - Signing Key v2) <suppo rt@mell NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 434 <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a "> <ac:plai n-text- body><! [CDATA [Is this [y/N]: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 435 1 867c36c669 gpgkey= 89"> le: ///<path to the <ac:plain- downloaded key text-body><! RPM-GPG-KEY- [CDATA[ Mellanox> gpgcheck= 1 1. yum repolist Loaded plugins: product -id, security subscri ption- manage NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 436 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount ro,loop MLNX_ OFED_L INUX<v NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 437 MNLX_OFED Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 438 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 439 --skip- unsupported -devices- check ag during installation. <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 440 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 441 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 442 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 443 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 444 --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 445 ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 446 ./mlnxofedins tall -- without- dkms --add- kernel- support -- kernel 3.13.0-85- generic -- without-fw- update -- force Note that the path to kernel sources (- kernel- sources) should be NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 447 Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 448 Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 449 Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 450 The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 451 '-without update' rmware version of adapter device is older than rmware version included with the OFED ISO image Note: If adapter's Flash was originally program NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 452 NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 453 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 454 The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 455 '-without update' rmware version of adapter device is older than rmware version included with the OFED ISO image Note: If adapter's Flash was originally program med with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 456 NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 457 Saving to: ?RPM-GPG- ownloa KEY-Mellanox? ds/ofed <ac:structured-macro /RPM- ac:name="unmigrated GPG- -wiki-markup" KEY- ac:schema- Mellano version="1" ac:macro- id="aa22e978-c6aa- Resolvi 4a5b-81aa- e140bb07d253"> www.m <ac:plain-text-body> ellanox. <![CDATA[ 100 % NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 458 KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 459 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan Technol ogies - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 460 [Is this [y/N]: 1. rpm -q gpg- pubkey grep Mellanox --qf '% gpg-pubkey- {NAME} a9e4b643-520791ba gpg(Mellanox {VERSI Technologies ON}% <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 461 1 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 462 4 , repolist: 8 , 351 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 463 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 464 957.21 . .el7.x86 _64 ... WARNI NG: If you are going con gu re this packag e as a reposit ory, then please note WARNI that it contain unsigne NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 465 -ext.tgz 11. Open the tarball. Requirement Description Platforms A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 466 Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 467 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. To avoid this, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 468 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 469 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 470 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 471 Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 472 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 473 Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 474 /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 475 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a message similar to the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 476 Please contact your hardware vendor for help with rmware updates. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 477 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 478 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 479 Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio n script is NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 480 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. In case your machine has an NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 481 /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed No rmware was found for the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 482 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 483 Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio n script is NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 484 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. In case your machine has an NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 485 - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 1. wget 72.3 . 194.0 : 80 ... connected. ]]></ac:plain- http: text-body> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 486 [================ 194.0 ================= Connec ================> ting to ] 1 , 354 --.-K/s in 0s www.m ]]></ac:plain-text- ellanox. body></ac:structured- macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- version="1" ac:macro- id=" 1df9d9-fa9f- 4b0a-a545- 5f05f950ce9b"> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 487 RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 488 Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 489 [CDATA [Is this [y/N]: 1. rpm -q gpg- pubkey grep Mellanox --qf '% gpg-pubkey- {NAME} a9e4b643-520791ba gpg(Mellanox {VERSI Technologies ON}% <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 490 1 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 491 - dag 4 , repolist: 8 , 351 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 492 /tmp director images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 493 . 0 - 957.21 . .el7.x86 _64 ... WARNI NG: If you are going con gu re this packag e as a reposit ory, then please note WARNI that it contain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 494 2. tar xvf /tmp/MLNX_OFED_LINUX 5.2 - 0.5 . 5.0 -rhel7. 6 -x86_64-ext.tgz| 1. Create a YUM repository con guration le called "/etc/yum.repos.d/mlnx_ofed.repo" with the following content: Requirement Description Platforms A server platform with one of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 495 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 496 Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 497 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 498 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with a root lesystem mounted over a ConnectX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 499 Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 500 LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 501 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 502 _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 503 -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 504 In case your machine has the latest rmware, no rmware update will occur and installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 505 Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages below will be printed. Please contact your NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 506 Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 507 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 508 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 509 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 510 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 511 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 512 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 513 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 514 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 515 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 516 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 517 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 518 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 519 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 520 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 521 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 522 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 523 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 2. Check that the repository was successfully added. Requirement Description Platforms A server platform with one of the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 524 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 525 Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 526 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 527 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with a root lesystem mounted over a ConnectX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 528 Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 529 LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 530 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 531 _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 532 -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 533 In case your machine has the latest rmware, no rmware update will occur and installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 534 Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages below will be printed. Please contact your NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 535 Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 536 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 537 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 538 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 539 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 540 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 541 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 542 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 543 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 544 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 545 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 546 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 547 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 548 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 549 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 550 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 551 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 552 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 553 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 Installing MLNX_OFED Using the YUM Tool NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 554 MNLX_OFED Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 555 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 556 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 557 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not involve automatic regeneration NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 558 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 559 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 560 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 561 <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 562 ..Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 563 -- kernel 3.13.0-85- generic -- without-fw- update -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 564 XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 565 > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 566 OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 567 /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 568 '-without update' rmware version of adapter device is older than rmware version included with the OFED ISO image Note: If adapter's Flash was originally program med with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 569 NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 570 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 571 /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 572 '-without update' rmware version of adapter device is older than rmware version included with the OFED ISO image Note: If adapter's Flash was originally program med with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 573 NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 574 Saving to: ?RPM-GPG- ownloa KEY-Mellanox? ds/ofed <ac:structured-macro /RPM- ac:name="unmigrated GPG- -wiki-markup" KEY- ac:schema- Mellano version="1" ac:macro- id="aa22e978-c6aa- Resolvi 4a5b-81aa- e140bb07d253"> www.m <ac:plain-text-body> ellanox. <![CDATA[ 100 % com... [================ ================= NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 575 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 576 C050 : Userid: "Mellan Technol ogies (Mellan Technol ogies - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 577 1. rpm -q gpg- pubkey grep Mellanox --qf '% gpg-pubkey- {NAME} a9e4b643-520791ba gpg(Mellanox {VERSI Technologies ON}% <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 578 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 579 8 , 351 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 580 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 581 957.21 . .el7.x86 _64 ... WARNI NG: If you are going con gu re this packag e as a reposit ory, then please note WARNI that it contain unsigne NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 582 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="5c39ade le: ///<path to 6-a02e- extracted 4db8-88cd- MLNX_OFED 0f49c8593e package>/RPMS bb"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 583 Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmfor NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 584 : MLNX_ OFED installer packag e (with support mlnx- ofed- all- user- only.no arch : MLNX_ OFED all- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 585 : MLNX_ OFED basic- user- only installer packag e (User Space packag es only) mlnx- ofed- blue el d.noarc MLNX_ OFED blue el installer packag e (with support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 586 : MLNX_ OFED dpdk installer packag e (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 587 (User Space packag es only) mlnx- ofed- dpdk- user- only.no arch : MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 588 : MLNX_ OFED guest installer packag e (with support mlnx- ofed- guest- user- only.no arch : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 589 : MLNX_ OFED hpc- user- only installer packag e (User Space packag es only) mlnx- ofed- hypervi sor.noar ch : MLNX_ OFED hypervi installer packag e (with support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 590 : MLNX_ OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 591 : MLNX_ OFED vma- eth- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 592 (with support mlnx- ofed- vma- vpi- user- only.no arch : MLNX_ OFED vma- vpi- user- only installer packag e (User Space packag es only where: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 593 Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 594 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 595 ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 596 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 597 On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 598 /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 599 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 600 Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 601 Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 602 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 603 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 604 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 605 RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 606 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 607 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 608 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 609 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 610 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 611 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 612 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 613 <ac:plain-text-body> ellanox. <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 614 -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 615 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 616 <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 617 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 618 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 619 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 620 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 621 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 622 You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 623 : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 624 (User Space packag es only) mlnx- ofed- blue el d.noarc MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 625 OFED dpdk installer packag e (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- upstrea m-libs- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 626 : MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- only- user- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 627 : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 628 (User Space packag es only) mlnx- ofed- hypervi sor.noar ch : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 629 : MLNX_ OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 630 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 631 Space packag es only Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 632 RPM packages have the supported kernel version in their package's name. Example: Requirement Description Platforms A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 633 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 634 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 635 <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 636 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 637 Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 638 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 639 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 640 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 641 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 642 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 643 Please contact your hardware vendor for help with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 644 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 645 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 646 RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 647 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 648 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 649 MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 650 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 651 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 652 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 653 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 654 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 655 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 656 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 657 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 658 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 659 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 660 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 661 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 662 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 663 : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 664 MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : MLNX_ OFED blue el d-user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 665 (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- upstrea m-libs- user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 666 MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- only- user- only installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 667 : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 668 : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ OFED hypervi sor- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 669 OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ OFED vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 670 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 671 Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS mlnx-ofed- Installs packages required for HPC NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 672 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 673 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 674 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 675 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 676 'gpgcheck=0' in the repository con guration le. 4. Install the desired group. Requirement Description Platforms A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) ( rmware: fw- ConnectX5) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 677 Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 678 Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 679 [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 680 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even cause the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 681 Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 682 --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 683 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 684 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 685 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 686 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 687 Please contact your hardware vendor for help with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 688 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 689 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 690 RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 691 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 692 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 693 MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 694 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 695 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 696 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 697 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 698 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 699 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 700 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 701 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 702 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 703 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 704 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 705 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 706 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 707 : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 708 MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : MLNX_ OFED blue el d-user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 709 (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- upstrea m-libs- user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 710 MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- only- user- only installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 711 : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 712 : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ OFED hypervi sor- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 713 OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ OFED vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 714 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 715 Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS mlnx-ofed- Installs packages required for HPC NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 716 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 717 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 718 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 719 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 720 3.17 . 4 301 .fc21.x86_64 (without support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscri ption- manage Resolvi Depend encies --> Runnin NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 721 . 2.1 .g832a 737.rhe l7u1 for packag mlnx- ofed-all 3.1 - 0.1 .noarch ........qperf.x 86_64 0 : 0.4 . 9 - 9 rds- devel.x8 6_64 0 : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 722 ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation Operating Linux operating System system. For the list of supported operating system distributions and kernels, please refer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 723 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="203cc571-506f- 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 724 To avoid this, make sure to add --skip- unsupported -devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 725 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 726 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f On Ubuntu and Debian NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 727 "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup port.sh /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 728 /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- broken- link- locale- en_us- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 729 Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 730 MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 731 --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall -- without- dkms --add- kernel- support -- kernel 3.13.0-85- generic -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 732 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 733 Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<version > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 734 Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 735 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 736 '-without update' rmware version of adapter device is older than rmware NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 737 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 738 This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 739 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES Fedora Distributi /lib/modul es/`unam r`/extra/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 740 '-without update' rmware version of adapter device is older than rmware NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 741 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM215011003 3) To obtain NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 742 KEY- id="ddfd0a12-60b5- Mellano 4a59-89e1- 7ea7e73df37b"> - 2018 <ac:plain-text-body> 01 25 <![CDATA[Length: 13 : 52 : 1354 ( 1 .3K) 30 - [text/plain] http: ]]></ac:plain-text- //www. body></ac:structured- mellano macro> x.com/d NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 743 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 744 ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan Technol ogies - Signing Key v2) <suppo rt@mell NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 745 <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a "> <ac:plai n-text- body><! [CDATA [Is this [y/N]: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 746 1 867c36c669 gpgkey= 89"> le: ///<path to the <ac:plain- downloaded key text-body><! RPM-GPG-KEY- [CDATA[ Mellanox> gpgcheck= 1 1. yum repolist Loaded plugins: product -id, security subscri ption- manage NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 747 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount ro,loop MLNX_ OFED_L INUX<v NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 748 /mnt/ Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/ con ue nce/pla ceholde r/error- i18nkey -editor- placeho lder- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 749 Building MLNX_ OFED_L INUX RPMS . Please wait... Creatin metada rpms for 3.10 . 0 - 957.21 . .el7.x86 _64 ... WARNI NG: If you are going NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 750 WARNI NG: by setting 'gpgche ck=0' in reposit conf Created /tmp/M LNX_O FED_LI NUX- 5.2 - 0.5 . 5.0 - rhel7. 6 x86_64 -ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 751 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 752 RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noar ch : MLNX_ OFED installer packag e (with support mlnx- ofed- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 753 : MLNX_ OFED basic installer packag e (with support mlnx- ofed- basic- user- only.no arch : MLNX_ OFED basic- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 754 : MLNX_ OFED blue el d-user- only installer packag e (User Space packag es only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer packag e (with support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 755 : MLNX_ OFED dpdk- upstrea m-libs- user- only installer packag e (User Space packag es only) mlnx- ofed- dpdk- user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 756 MLNX_ OFED eth- only- user- only installer packag e (User Space packag es only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer packag e (with support mlnx- ofed- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 757 : MLNX_ OFED installer packag e (with support mlnx- ofed- hpc- user- only.no arch : MLNX_ OFED hpc- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 758 : MLNX_ OFED hypervi sor- user- only installer packag e (User Space packag es only) mlnx- ofed- kernel- only.no arch : MLNX_ OFED kernel- only installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 759 : MLNX_ OFED vma- installer packag e (with support mlnx- ofed- vma- eth- user- only.no arch : MLNX_ OFED vma- eth- user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 760 (User Space packag es only) mlnx- ofed- vma- vpi.noar ch : MLNX_ OFED vma-vpi installer packag e (with support mlnx- ofed- vma- vpi- user- only.no arch : MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 761 VMA to vma-eth work over Ethernet Installs packages mlnx-ofed- required by VMA to vma-vpi support VPI Installs packages blue eld required for BlueField Installs packages dpdk required for DPDK NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 762 4 301 .fc21.x86_64. noarch : MLNX_OFED basic installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- guest- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED guest installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 763 MLNX_OFED hypervisor installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 764 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED hypervisor installer package for kernel 3.17 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 765 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 301 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 766 Resolvi Depend encies --> Runnin transac tion check --> Packag e mlnx- ofed- all.noar ch 0 : 3.1 0.1 . 2 will be installe -> Process Depend ency: kmod- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 767 1.12 rds- tools.x8 6_64 0 : 2.0 . 7 - 1.12 sdpnet stat.x8 6_64 0 : 1.60 - srptool s.x86_6 4 0 : 1.0 . 2 - 12 Comple NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 768 7. Create an apt-get repository con guration le called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: Requirement Description Platforms A server platform with one of the following adapters installed: ConnectX®-5 Ex (EN) ( rmware: fw- ConnectX5) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 769 Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 770 Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 771 [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 772 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even cause the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 773 Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 774 --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 775 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 776 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 777 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 778 In case your machine has the latest rmware, no rmware update will occur and installation script will print at the end of installation a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 779 Please contact your hardware vendor for help with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 780 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 781 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 782 RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installatio NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 783 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion image. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 784 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 785 MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 786 RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 787 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update will also burn an Expansion NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 788 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS label> <CPU NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 789 <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated -wiki-markup" ac:schema- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 790 </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 791 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 792 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- MLNX_OFED 867c36c669 package>/RPMS 89"> enabled= 1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 793 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 794 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 795 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 796 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 797 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 798 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 799 : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 800 MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : MLNX_ OFED blue el d-user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 801 (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- upstrea m-libs- user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 802 MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- only- user- only installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 803 : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer packag e (with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 804 : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ OFED hypervi sor- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 805 OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ OFED vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 806 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 807 Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS mlnx-ofed- Installs packages required for HPC NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 808 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 809 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 810 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 811 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 812 3.17 . 4 301 .fc21.x86_64 (without support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscri ption- manage Resolvi Depend encies --> Runnin NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 813 . 2.1 .g832a 737.rhe l7u1 for packag mlnx- ofed-all 3.1 - 0.1 .noarch ........qperf.x 86_64 0 : 0.4 . 9 - 9 rds- devel.x8 6_64 0 : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 814 To update the rmware to the version included in MLNX_OFED package, run: # yum install mlnx-fw- updater le:/ <path to extracted MLNX_OFED package >/DEBS ./ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 815 MNLX_OFED Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 816 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 817 ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 818 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 819 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 820 /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 821 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 822 /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 823 Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 824 -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their default location. In case your machine has NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 825 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 826 MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 827 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 828 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 829 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 830 (PSID: IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 831 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 832 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 833 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 834 IBM215011003 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 835 4a5b-81aa- e140bb07d253"> www.m <ac:plain-text-body> ellanox. <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- macro> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 836 ]]></ac:plain-text- rpm -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 837 (Mellan Technol ogies - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 838 {VERSI Technologies ON}% <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 839 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 840 MLNX_ OFED_L INUX<v er><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 841 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 842 WARNI that it contain unsigne d rpms, therefo re, you need to disable NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 843 </ac:structured- ac:schema- macro> version="1" name=MLNX_OFED ac:macro- Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 844 Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 845 (with support mlnx- ofed- all- user- only.no arch : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 846 (User Space packag es only) mlnx- ofed- blue el d.noarc MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 847 : MLNX_ OFED dpdk installer packag e (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 848 : MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 849 (with support mlnx- ofed- guest- user- only.no arch : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 850 (User Space packag es only) mlnx- ofed- hypervi sor.noar ch : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 851 : MLNX_ OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 852 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 853 (User Space packag es only Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 854 Installs packages kernel-only required for a non- default kernel mlnx-ofed- all- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 855 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hpc- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 856 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 857 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 858 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscri ption- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 859 2 will be installe -> Process Depend ency: kmod- isert = 1.0 - OFED. 3.1 . 0.1 . 2.1 .g832a 737.rhe l7u1 for packag mlnx- ofed-all 3.1 - 0.1 .noarch ....NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 860 Installing MLNX_OFED using the "YUM" tool does not automatically update the rmware. To update the rmware to the version included in MLNX_OFED package, run: # yum install mlnx-fw- updater NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 861 Requirement Description A server platform with one of the following adapters installed: ConnectX®-5 Platforms Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) Required Disk Space Installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 862 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 863 /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- unsupported -devices- check ag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 864 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular installation can be performed and weak- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 865 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 866 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ mlnx_a dd_ker nel_sup NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 867 MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue nce/pla ceholde r/error- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 868 Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED _LINUX<ver> <OS label> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 869 IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....Installation nished successfully. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 870 ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 871 Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 872 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: --- ------- Device Type: ConnectX-X Part Number: MCXXXX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 873 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 874 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 875 '-without update' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 876 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 877 The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 878 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam r`/update s on SLES NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 879 '-without update' NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 880 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 881 : 80 ... connected. ]]></ac:plain- http: HTTP request sent, text-body> //www. awaiting response... </ac:structured mellano 200 OK -macro> x.com/d <ac:structured-macro ownloa ac:name="unmigrated ds/ofed -wiki-markup" /RPM- ac:schema- GPG- version="1" ac:macro- KEY- id="ddfd0a12-60b5- 4a59-89e1- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 882 <ac:plain-text-body> <![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body> import NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 883 Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan Technol ogies (Mellan NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 884 OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a "> <ac:plai n-text- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 885 441b-9e1f- enabled= 1 867c36c669 gpgkey= 89"> le: ///<path to the <ac:plain- downloaded key text-body><! RPM-GPG-KEY- [CDATA[ Mellanox> gpgcheck= 1 1. yum repolist Loaded plugins: product -id, security NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 886 MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 887 - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/ con ue nce/pla NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 888 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX RPMS . Please wait... Creatin metada rpms for 3.10 . 0 - 957.21 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 889 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting 'gpgche ck=0' in reposit conf Created /tmp/M LNX_O FED_LI NUX- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 890 <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat Subscri NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 891 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noar ch : MLNX_ OFED installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 892 Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer packag e (with support mlnx- ofed- basic- user- only.no arch : MLNX_ OFED basic- user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 893 : MLNX_ OFED blue el d-user- only installer packag e (User Space packag es only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 894 (with support mlnx- ofed- dpdk- upstrea m-libs- user- only.no arch : MLNX_ OFED dpdk- upstrea m-libs- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 895 : MLNX_ OFED eth- only- user- only installer packag e (User Space packag es only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer packag NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 896 (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer packag e (with support mlnx- ofed- hpc- user- only.no arch : MLNX_ OFED hpc- user- only NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 897 : MLNX_ OFED hypervi sor- user- only installer packag e (User Space packag es only) mlnx- ofed- kernel- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 898 MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ OFED vma- installer packag e (with support mlnx- ofed- vma- eth- user- only.no arch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 899 MLNX_ OFED vma- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- vpi.noar ch : MLNX_ OFED vma-vpi installer packag e (with support mlnx- ofed- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 900 OS mlnx-ofed- Installs packages required by VMA Installs packages mlnx-ofed- required by VMA to vma-eth work over Ethernet Installs packages mlnx-ofed- required by VMA to vma-vpi support VPI NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 901 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED basic installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- guest- 3.17 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 902 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED hypervisor installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma- 3.17 . 4 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 903 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 904 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 905 -id, subscri ption- manage Resolvi Depend encies --> Runnin transac tion check --> Packag e mlnx- ofed- all.noar ch 0 : 3.1 0.1 . 2 will be installe NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 906 9 - 9 rds- devel.x8 6_64 0 : 2.0 . 7 - 1.12 rds- tools.x8 6_64 0 : 2.0 . 7 - 1.12 sdpnet stat.x8 6_64 0 : 1.60 - NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 907 # yum install mlnx-fw- updater le:/ <path to extracted MLNX_OFED package >/DEBS ./ 1. wget - qO - http: //www. mellano x.com/d ownloa ds/ofed /RPM- GPG- KEY- Mellano x | sudo NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 908 10. Update the apt-get cache. Requirement Description A server platform with one of the following adapters installed: ConnectX®-5 Platforms Ex (EN) ( rmware: fw- ConnectX5) ConnectX®-5 (EN) ( rmware: fw-ConnectX5) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 909 4506-8492- de3cd0d36b44"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured- macro NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 910 ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 911 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular installation NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 912 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 913 MLNX_OFED installation. Therefore, using "mlnx_add_k ernel_suppor t.sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX _OFED_ LINUX- x.x-x- rhel7. 3 x86_64/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 914 MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB packag es will remove images/ plugins/ servlet/ con ue NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 915 _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64 -ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 916 RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED _LINUX, do not reinstall them. Starting MLNX_OFED _LINUX-x.x.x installation ....NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 917 Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 918 Device #1: --- ------- Device Type: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 919 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: --- ------- Device Type: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 920 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 921 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 922 '-without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 923 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 924 Return Code Meaning The Installation ended successfully The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 925 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mella nox/iprout e2/sbin/rd The kernel modules are installed under /lib/modul es/`unam NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 926 '-without NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 927 Expansion image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 928 : 80 ... connected. ]]></ac:plain- http: HTTP request sent, text-body> //www. awaiting response... </ac:structured mellano 200 OK -macro> x.com/d <ac:structured-macro ownloa ac:name="unmigrated ds/ofed -wiki-markup" /RPM- ac:schema- GPG- version="1" ac:macro- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 929 1df9d9-fa9f- 4b0a-a545- 5f05f950ce9b"> <ac:plain-text-body> <![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ?RPM-GPG- KEY-Mellanox? saved [ 1354 / 1354 ] NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 930 Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importi ng GPG 0x6224 C050 : Userid: "Mellan NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 931 MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " ac:sche version ="1" ac:macr id="b05 a377e- 3456- 4 5- a4db- c63f43 e5434a NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 932 MLNX_OFED e-4ec4- package>/RPMS 441b-9e1f- enabled= 1 867c36c669 gpgkey= 89"> le: ///<path to the <ac:plain- downloaded key text-body><! RPM-GPG-KEY- [CDATA[ Mellanox> gpgcheck= 1 1. yum repolist Loaded plugins: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 933 You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 934 <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 935 .log Checkin g if all needed packag es are installe d... Building MLNX_ OFED_L INUX RPMS . Please wait... Creatin metada rpms for 3.10 . 0 - 957.21 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 936 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche WARNI NG: by setting 'gpgche ck=0' in reposit conf Created /tmp/M LNX_O FED_LI NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 937 <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: product -id, security subscri ption- manage This system is not register ed to Red Hat NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 938 Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noar ch : MLNX_ OFED installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 939 (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer packag e (with support mlnx- ofed- basic- user- only.no arch : MLNX_ OFED basic- user- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 940 : MLNX_ OFED blue el d-user- only installer packag e (User Space packag es only) mlnx- ofed- dpdk.no arch : MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 941 (with support mlnx- ofed- dpdk- upstrea m-libs- user- only.no arch : MLNX_ OFED dpdk- upstrea m-libs- user- only installer packag e (User Space NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 942 : MLNX_ OFED eth- only- user- only installer packag e (User Space packag es only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 943 (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer packag e (with support mlnx- ofed- hpc- user- only.no arch : MLNX_ OFED hpc- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 944 (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ OFED hypervi sor- user- only installer packag e (User Space packag es only) mlnx- ofed- kernel- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 945 : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ OFED vma- installer packag e (with support mlnx- ofed- vma- eth- user- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 946 : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) mlnx- ofed- vma- vpi.noar ch : MLNX_ OFED vma-vpi installer packag e (with support NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 947 OS mlnx-ofed- Installs packages required by VMA Installs packages mlnx-ofed- required by VMA to vma-eth work over Ethernet mlnx-ofed- Installs packages vma-vpi required by VMA to NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 948 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED basic installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 949 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED hypervisor installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 950 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 951 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 952 (without support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscri ption- manage Resolvi Depend encies --> Runnin transac tion check --> Packag e mlnx- ofed- all.noar NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 953 86_64 0 : 0.4 . 9 - 9 rds- devel.x8 6_64 0 : 2.0 . 7 - 1.12 rds- tools.x8 6_64 0 : 2.0 . 7 - 1.12 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 954 MLNX_OFED package, run: # yum install mlnx-fw- updater le:/ <path to extracted MLNX_OFED package >/DEBS ./ 1. wget - qO - http: //www. mellano x.com/d ownloa ds/ofed NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 955 Setting up MLNX_OFED apt-get Repository Using --add-kernel-support 11. Log into the installation machine as root. 12. Mount the ISO image on your machine and copy its content to a shared location in your network. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 956 Release Notes The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="41aa4530-f1d1- 4506-8492- de3cd0d36b44"> <ac:plain-text-body> NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 957 4f4c-b960- b4a327d68f18"> <ac:plain-text-body> <![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT27620 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED _LINUX- <ver>-<OS label>.iso ./mlnxofedins tall --fw- image-dir NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 958 ]]></ac:plain-text- ac:macro- ./mnt/mlnxofedinstall body></ac:structured- id="ee2f54c0 [OPTIONS] macro> -5ec8-4127- 8227- b4709ab350 fc"> <ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with extension ".conf.rpmsav e". NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 959 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 960 On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_k NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 961 /tmp/M LNX_O FED_LI NUX- x.x-x- rhel7. 3 x86_64/ --make- Note: This progra m will create MLNX_ OFED_L INUX TGZ for rhel7. 3 under /tmp director Mellano NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 962 -editor- placeho lder- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 963 Logs dir: /tmp/MLNX_ OFED_LINUX -x.x-x.logs This program will install MLNX_OFED _LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 964 Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 965 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 966 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 967 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 968 RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 969 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 970 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 971 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 972 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 973 SLES Fedora Distributi /lib/modul es/`unam r`/extra/ mlnx- ofa_kerne l on RHEL and other RedHat like Distributi /lib/modul es/`unam r`/update s/dkms/ Ubuntu The rmware of existing network adapter devices NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 974 OFED ISO image Note: If adapter's Flash was originally program med with Expansion image, automatic rmware update NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 975 3) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX - 4.4 - 1.0 . .IBMM21501 10033.logs 1. mount ro,loop MLNX_ OFED_L INUX<v er><OS NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 976 <ac:plain-text-body> ellanox. <![CDATA[ 100 % com... [================ 72.3 . ================= 194.0 ================> Connec ] 1 , 354 --.-K/s in 0s ting to ]]></ac:plain-text- www.m body></ac:structured- ellanox. macro> <ac:structured-macro ac:name="unmigrated NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 977 -- body> import </ac:structured- RPM- macro> GPG- KEY- Mellano warning rpmts_ HdrFro mFdno: Header DSA/SH Signatu re, key 6224c0 NOKEY Retrievi ng key from ///repos /MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 978 - Signing Key v2) <suppo rt@mell anox.co m>" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:str uctured -macro ac:nam e="unm igrated- wiki- markup " NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 979 <support @mellanox {RELEA .com>) SE}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body> migrated- </ac:structured- wiki-markup" macro> ac:schema- name=MLNX_OFED version="1" Repository ac:macro- baseurl= id="3388df1 le: ///<path to e-4ec4- extracted 441b-9e1f- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 980 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscri ption- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 981 <CPU arch>.is o /mnt 1. /mnt/ml nx_add _kernel _suppor t.sh -- make- <option al -- kmp> - $(unam e -r) -m /mnt/ Note: This progra m will NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 982 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_is o. 4120 .log Checkin g if all needed packag es are installe d... Building MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 983 WARNI that it contain unsigne d rpms, therefo re, you need to disable gpgche NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 984 Repository id="5c39ade baseurl= 6-a02e- le: ///<path to 4db8-88cd- extracted 0f49c8593e MLNX_OFED bb"> package>/RPMS <ac:plain- enabled= 1 text-body><! gpgcheck= 0 [CDATA[ 1. yum repolist Loaded plugins: NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 985 You can subscri ption- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmfor RHEL 6Server RPMfor ge.net - dag 4 , NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 986 : MLNX_ OFED all- user- only installer packag e (User Space packag es only) mlnx- ofed- basic.n oarch : MLNX_ OFED basic installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 987 (User Space packag es only) mlnx- ofed- blue el d.noarc MLNX_ OFED blue el installer packag e (with support mlnx- ofed- blue el d-user- only.no arch : MLNX_ OFED NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 988 OFED dpdk installer packag e (with support mlnx- ofed- dpdk- upstrea libs.noa rch : MLNX_ OFED dpdk- upstrea m-libs installer packag e (with support mlnx- ofed- dpdk- upstrea m-libs- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 989 : MLNX_ OFED dpdk- user- only installer packag e (User Space packag es only) mlnx- ofed- eth- only- user- only.no arch : MLNX_ OFED eth- only- user- NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 990 : MLNX_ OFED guest- user- only installer packag e (User Space packag es only) mlnx- ofed- hpc.noa rch : MLNX_ OFED installer NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 991 (User Space packag es only) mlnx- ofed- hypervi sor.noar ch : MLNX_ OFED hypervi installer packag e (with support mlnx- ofed- hypervi sor- user- only.no arch : MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 992 : MLNX_ OFED kernel- only installer packag e (with support mlnx- ofed- vma.no arch : MLNX_ OFED installer packag e (with support mlnx- ofed- vma- eth.noa rch : MLNX_ NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 993 (User Space packag es only) mlnx- ofed- vma- user- only.no arch : MLNX_ OFED vma- user- only installer packag e (User Space packag es only) NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 994 Space packag es only Installs all available mlnx-ofed-all packages in MLNX_OFED Installs basic mlnx-ofed- packages required basic for running NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 995 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- basic- 3.17 . NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 996 (without support) mlnx-ofed- hpc- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- hypervisor- 3.17 . 4 301 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 997 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 998 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 999 MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 301 .fc21.x86_64 (without support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscri ption- manage Resolvi Depend encies NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...
  • Page 1000 1.0 - OFED. 3.1 . 0.1 . 2.1 .g832a 737.rhe l7u1 for packag mlnx- ofed-all 3.1 - 0.1 .noarch ........qperf.x 86_64 0 : 0.4 . 9 - 9 NVIDIA ConnectX-5 InfiniBand/Ethernet Adapter Cards for OCP Spec 2.0 User Manual...