Nvidia 900-9X657-0058-SB0 User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Adapter
  5. 900-9X657-0058-SB0
  6. User manual
Nvidia 900-9X657-0058-SB0 User Manual

Nvidia 900-9X657-0058-SB0 User Manual

Connectx-6 infiniband/ethernet adapter cards for ocp spec 3.0
Hide thumbs Also See for 900-9X657-0058-SB0:
1
Table Of Contents
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards
for OCP Spec 3.0 User Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the 900-9X657-0058-SB0 and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Nvidia 900-9X657-0058-SB0

Summary of Contents for Nvidia 900-9X657-0058-SB0

  • Page 1 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 2: Table Of Contents

    2158 Finding the GUID_MAC and Serial Number on the Adapter Card Finding the GUID_MAC and Serial Number on the Adapter Card 2171 Document Revision History Document Revision History 2174 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 3 About This Manual This User Manual describes NVIDIA® ConnectX®-6 In niBand/Ethernet adapter cards for Open Compute Project (OCP), Spec 3.0. It provides details as to the interfaces of the board, speci cations, required software and rmware for operating the board, and relevant documentation.
  • Page 4 Dual-port QSFP56, PCIe4.0 x16, Internal Lock 900- MCX653435 ConnectX®-6 9X65 A-HDAI In niBand/Et hernet 0018- adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0, with host management, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 5 Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with In niBand/Ethernet network and architecture speci cations. Technical Support NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 6 Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: https://www.nvidia.com > Support E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should rst seek assistance through their reseller.
  • Page 7 OCP 3.0, with 0018- host management, Single-port QSFP56, PCIe4.0 x16, Internal Lock 900- MCX653435 ConnectX®-6 9X65 A-EDAI In niBand/Et hernet 0016- adapter card, 100Gb/s (HDR100, EDR IB and 100GbE) for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 8 D for band Linux diagnostic, User tools Manu content, and al and con guration Relea . See MLNX_OFED Notes for Linux Documentati WinO User Manual describing WinOF-2 Wind features, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 9 ESXi Drivers Documentati NVIDI NVIDIA rmware Firmw update and query utility Utility used to (mlxu update the rmware. User Refer to Manu Firmware al and Utility Relea (mlxup) Documentati Notes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 10 Relea se 1.3 IEEE IEEE 802.3 Ethernet Specif Speci cation icatio Industry Expre Standard PCI Express Base Specif and Card icatio Electromech anical Speci cation s. Refer to PCI-SIG NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 11 NVIDIA o ers one of the industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100 (100Gb/s), (200Gb/s) and NDR (400Gb/s) cables, including Direct Attach NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 12 Bit Error Rate of less than 1E-15 . Read more at LinkX Cables Transceivers. Open Comp https://www. Projec opencomput t 3.0 e.org/ Specif icatio NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 13 When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to mean PCI Express. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 14: Introduction

    XXX, as the OCP spec 3.0 available at that time did not contain any S&V de nitions. A newer version of the OCP spec 3.0 has de ned S&V speci cations and NVIDIA is in the midst of retesting these cards to comply with OCP spec 3.0.

  • Page 15: Features And Bene Ts

    ConnectX-6 o ers the highest throughput In niBand/Ethernet In niBand/Ethernet adapter, supporting HDR 200Gb/s Adapter In niBand and 200Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate seamlessly over NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 16 100Gb/s. Memory Components EEPROM - The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. (Note: A ddress 0x58 is reserved) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 17 Quality of Service (QoS) application requirements for latency and SLA. ConnectX-6 provides dedicated adapter resources and Hardware-based I/O guaranteed isolation and protection for virtual machines Virtualization within the server. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 18: Operating Systems/Distributions

    Reset-on-LAN (RoL) Supported Operating Systems/Distributions RHEL/CentOS Windows FreeBSD VMware OpenFabrics Enterprise Distribution (OFED) OpenFabrics Windows Distribution (WinOF-2) Connectivity Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches Passive copper cable with ESD protection NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 19 Powered connectors for optical and active cable support NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 20: Interfaces

    There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status. LED0 is bicolor (yellow and green) LED and LED1 is a single color (green) LED. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 21 ConnectX-6 IC. It is attached either by using four spring-loaded push pins that insert into four mounting holes. ConnectX-6 IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 card in case of a high-temperature event, improper thermal coupling or NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 22: Smbus Interface

    OCP 3.0 adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA OCP 3.0 adapter. For con guring the adapter for the speci c manageability solution in use by the server, please contact NVIDIA Support.

  • Page 23: Hardware Installation

    (3 inches) of clearance around the ventila tion openings. During Lightning - Electrical Hazard During periods of lightning activity, do not work on the equipment or connect or dis connect cables. Copper Cable Connecting/Disconnecting NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 24: Installation Procedure Overview

    Jewelry Removal Warning Before you install or remove equipment that is connected to power lines, remove jewelry such as bracelets, necklaces, rings, watches, and so on. Metal NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 25 Equipment Installation This equipment should be installed, replaced, or serviced only by trained and quali ed personnel. Equipment Disposal The disposal of this equipment should be in accordance to all NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 26 Install ConnectX-6 adapter card Refer to Installation Instructions for OCP spec 3.0 in the system Connect cables or modules to Refer to Cables and Modules the card NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 27: System Requirements

    A system with a PCI Express x16 slot for OCP spec 3.0 is required for installing the card. Air ow Requirements ConnectX-6 adapter cards are o ered with two air ow patterns: from the heatsink to the network ports, and vice versa, as shown below. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 28 Please refer to the "Speci cations" chapter for air ow numbers for each speci c card model. Warning All cards in the system should be planned with the same air ow direction. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 29 Step Procedure General Installation Instructions Read all installation instructions before connecting the equipment to the power NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 30 Equipment Disposal The disposal of this equipment should be in accordance to all national laws and regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 31 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 32 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 33 (Optional) Replace the assembled OCP 3.0 bracket with the desired form factor bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 34 Install ConnectX-6 adapter card for OCP spec 3.0 in the system Connect cables or modules to the card NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 35 Identify ConnectX-6 adapter card in the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 36: Safety Precautions

    Remove any metallic objects from your hands and wrists. Make sure to use only insulated tools. Verify that the system is powered o and is unplugged. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 37: Pre-Installation Checklist

    The OCP 3.0 adapter card is shipped assembled either with a thumbscrew (pull-tab) bracket, an internal-lock bracket, or an ejector-latch bracket. If this form factor is suitable for your requirements, you can skip the remainder of this section and move NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 38 The required torx tool type as speci ed in the instructions. Removing the Existing Bracket 1. Using the torx tool type listed in the below table, remove the screws according to the instructions per OCP 3.0 bracket type. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 39 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 40 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 41 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 42 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 43 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 44 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 45 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 46 2. Gently separate the bracket from the OCP 3.0 card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 47 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 48 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 49 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 50 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 51 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 52 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 53 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 54 Warning NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 55 Be careful not to put stress on the LEDs on the adapter card. 3. Save the two screws. Installing the New Bracket 1. Assemble the new bracket onto the card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 56 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 57 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 58 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 59 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 60 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 61 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 62 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 63 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 64 3. Screw on the OCP 3.0 bracket with the supplied screws that came with the new bracket kit. Use the speci ed torx tool type and apply the speci ed torque on the screws per bracket form factor. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 65 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 66 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 67 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 68 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 69 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 70 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 71 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 72 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 73 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 74 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 75 The following gures are for illustration purposes only. 1. Before installing the card, make sure that the system is o and the power cord is not connected to the server. Please follow proper electrical grounding procedures. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 76 2. Open the system case. 3. Align the card with the system rails. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 77 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 78 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 79 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 80 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 81 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 82 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 83 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 84 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 85 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 86 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 87 4. Push the card until connectors are in a full mate. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 88 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 89 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 90 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 91 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 92 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 93 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 94 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 95 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 96 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 97 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 98 5. Secure the card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 99 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 100 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 101 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 102 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 103 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 104 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 105 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 106 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 107 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 108 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 109 A clicking sound is heard once the Turn the captive screw clockwise unt connectors are in a full mate. rmly locked. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 110 Do not apply any torque, up or down, to the connector cage in the adapter card. 4. Make sure that the connector locks in place. Note When installing cables make sure that the latches engage. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 111 “Mellanox Technologies”: lspci |grep -i Mellanox Network controller: Mellanox Technologies MT28800 Family [ConnectX- On Windows 1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 112 ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-6) – this is a valid NVIDIA PCI Device ID.
  • Page 113 The list of NVIDIA PCI Device IDs can be found at the PCI ID repository. Adapter Cards Extraction Instructions Follow the below instructions depending on the card form-factor you have purchased. Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal.
  • Page 114 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 115 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 116 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 117 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 118 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 119 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 120 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 121 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 122 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 123 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 124 A clicking sound is heard once the Turn the captive screw clockwise unt connectors are in a full mate. rmly locked. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 125 While holding the heatsink, gently push the card out of the server. Rotate the captive screw counterclockwise. Warning Careful, the heatsink might be hot. 4. Gently pull out the adapter card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 126 Step Procedure General Installation Instructions Read all installation instructions befo connecting the equipment to the pow NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 127 Equipment Disposal The disposal of this equipment shou be in accordance to all national laws regula tions. Local and National Electrical Codes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 128 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825- 1:1994+A1:1996+ A2:20 Check the system’s hardware and software requirements. Pay attention to the air ow consideration within the host system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 129 Follow the safety precautions Follow the pre-installation checklist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 130 (Optional) Replace the assembled OC 3.0 bracket with the desired form fac bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 131 Install ConnectX-6 adapter card for O spec 3.0 in the system Connect cables or modules to the ca NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 132 Identify ConnectX-6 adapter card in t system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 133 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 134 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 135 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 136 A clicking sound is heard once the Turn the captive screw clockwise unt connectors are in a full mate. rmly locked. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 137 Rotate the captive screw counterclockwise. Warning Careful, the heatsink might be hot. Gently pull out the adapter card from While holding the tab, pull out the the server. adapter card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 138 Some copper cables are heavy and not exible, as such, they should be carefully attached to or detached from the connectors. Refer to the cable manufacturer for spe cial warnings and instructions. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 139 Refer to OCP 3.0 Bracket bracket with the desired form factor Replacement Instructions bracket Install ConnectX-6 adapter card for OCP Refer to Installation spec 3.0 in the system Instructions NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 140: Hardware Requirements

    Your System System Requirements Important Unless otherwise speci ed, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination. The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for...

  • Page 141: Software Requirements

    See Operating Systems/Distributions section under the Introduction section. Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section. Safety Precautions Important NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 142 Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock may harm the EMI ngers on the networking connector. Carefully remove the card from the antistatic bag to avoid damaging the EMI ngers. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 143 1. Using the torx tool type listed in the below table, remove the screws according to the instructions per OCP 3.0 bracket type. Internal Lock Pull-tab (Thumbscrew) Bracket Ejector-Latch Bracket Bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 144 Be careful not to put stress on the LEDs on the adapter card. 3. Save the two screws. Installing the New Bracket 1. Assemble the new bracket onto the card. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 145 Internal Lock Pull-tab Bracket Ejector-Latch Bracket Bracket Note that one screw is at-head 90° screw and the other is socket-head screw. Screw on the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 146 Please follow proper electrical grounding procedures. 2. Open the system case. 3. Align the card with the system rails. Internal Lock Pull-tab (Thumbscrew) Bracket Ejector-Latch Bracket Bracket NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 147: Cables And Modules

    To uninstall the adapter card, see Uninstalling the Card. Cables and Modules Cable Installation 1. All cables can be inserted or removed with the unit powered on. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 148 5. Care should be taken as not to impede the air exhaust ow through the ventilation holes. Use cable lengths that allow for routing horizontally around to the side of the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 149 3. Right click the mouse on your adapter's row and select Properties to display the adapter card properties window. 4. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from the Property pull-down menu. PCI Device (Example) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 150 5. In the Value display box, check the elds VEN and DEV ( elds are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of NVIDIA; and DEV is equal to 1018 (for ConnectX-6) – this is a valid NVIDIA PCI Device ID.
  • Page 151 Pull-tab (Thumbscrew) Internal Lock Bracket Ejector-Latch Bracket Bracket While holding the Rotate the captive screw Open the ejector latch. heatsink, gently push counterclockwise. the card out of the server. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 152 4. Gently pull out the adapter card. Internal Lock Pull-tab (Thumbscrew) Bracket Ejector-Latch Bracket Bracket Gently pull While holding the tab, pull out Open the ejector latch. out the the adapter card. adapter card NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 153 Internal Lock Pull-tab (Thumbscrew) Bracket Ejector-Latch Bracket Bracket from the server. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

  • Page 154: Driver Installation

    Windows Driver Installation For Windows, download and install the latest WinOF-2 for Windows software package available via the NVIDIA website at: WinOF-2 webpage. Follow the installation instructions included in the download package (also available from the download page). The snapshots in the following sections are presented for illustration purposes only. The installation interface may slightly vary, depending on the operating system in use.
  • Page 155 Windows 10 Client (64 bit only) Windows 8.1 Client (64 bit only) echo %PROCESSOR_ARCHI TECTURE% Description Package Windows Server 2022 MLNX_WinOF2-<version>_All_x64.exe Windows Server 2019 Windows Server 2016 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 156 "AMD64". 2. Go to the WinOF-2 web page at: https://www.nvidia.com/en-us/networking/ > Products > Software > In niBand Drivers (Learn More) > Nvidia WinOF-2. 3. Download the .exe image according to the architecture of your machine (see Step 1).
  • Page 157 1. Double click the .exe and follow the GUI instructions to install MLNX_WinOF2. 2. [Optional] Manually con gure your setup to contain the logs option (replace "LogFile" with the relevant directory). Description Package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 158 Installing the incorrect .exe le is prohibited. If you do so, an error message will be displayed. For example, if you install a 64-bit .exe on a 32-bit machine, the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 159 3. [Optional] If you do not want to upgrade your rmware version (i.e., MT_SKIPFWUPGRD default value is False). Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe Windows Server 2019 Windows Server 2016 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 160 For example, if you install a 64-bit .exe on a 32-bit machine, the wizard will display the following (or a similar) error message: "The NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 161 4. [Optional] If you do not want to install the Rshim driver, run. Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe Windows Server 2019 Windows Server 2016 Windows Server 2012 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 162 For example, if you install a 64-bit .exe on a 32-bit machine, the wizard will display the following (or a similar) error message: "The installation package is NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 163 F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe Windows Server 2019 Windows Server 2016 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 164 For example, if you install a 64-bit .exe on a 32-bit machine, the wizard will display the following (or a similar) error message: "The NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 165 F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver installanion will fail if a prior Rshim driver is already NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 166 5. [Optional] If you want to skip the check for unsupported devices, run. Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe Windows Server 2019 Windows Server 2016 Windows Server 2012 Windows 11 Client (64 bit only) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 167 64-bit .exe on a 32-bit machine, the wizard will display the following (or a similar) error message: "The installation package is supported by this processor NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 168 Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver installanion will fail if a prior Rshim driver is already installed. The following fail message will be displayed in the log: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 169 Please remove the following oem inf les from driver store: <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP ORTEDDEVC HECK=1" 6. Click Next in the Welcome screen. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 170 1. Read and accept the license agreement and click Next. 1. Select the target folder for the installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 171 If the user has an OEM card. In this case, the rmware will not be displayed. If the user has a standard NVIDIA® card with an older rmware version, the rmware will be updated accordingly. However, if the user has both an OEM card and a NVIDIA®...
  • Page 172 Management tools - installation tools used for management, such as mlxstat Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd 2. Click Next to install the desired tools. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 173 1. Click Install to start the installation. 1. In case rmware upgrade option was checked in Step 7, you will be noti ed if a rmware upgrade is required (see NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 174 1. Click Finish to complete the installation. Unattended Installation Description Package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 175 Installing the incorrect .exe le is prohibited. If you do so, an error message will be displayed. For example, if you install a 64-bit .exe on a 32-bit machine, the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 176 8"><ac:plain- text-body><! [CDATA[ MLNX_WinOF 2_<revision_v ersion>_All_Ar ch.exe /v " MT_SKIPFWU PGRD=1" MLNX_WinOF 2_<revision_v ersion>_All_Ar ch.exe /v " MT_DISABLE_ RSHIM_INSTA LL=1" The Rshim driver installanion will fail if a NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 177 MLNX_WinOF 2_<revision_v ersion>_All_Ar ch.exe /v " SKIPUNSUPP ORTEDDEVCH ECK=1" If no reboot options are speci ed, the installer restarts the computer whenever necessary without displaying any prompt or NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 178 Windows Server 2019 Windows Server 2016 Windows Server 2012 Windows 11 Client (64 bit only) Windows 10 Client (64 bit only) Windows 8.1 Client (64 bit only) echo %PROCESSO NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 179 "The installation package is supported by this processor type. Contact your vendor" <ac:structure MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> d-macro n>_All_Arch.exe /v"/l*vx </ac:structured-macro> ac:name="un [LogFile]" migrated- wiki-markup" ac:schema- version="1" ac:macro- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 180 Rshim driver is already installed. The following fail message will be displayed in the log: "ERROR!!! Installation failed due to following errors: MlxRshim drivers installation disabled and MlxRshim drivers NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 181 To control the reboots, use /norestart or /forcerestart standard command- line options. <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn migrated- wiki-markup" ac:schema- version="1" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 182 Server 2012 Windows 11 Client (64 bit only) Windows 10 Client (64 bit only) Windows 8.1 Client (64 bit only) echo %PROCESSO R_ARCHITEC TURE% On an x64 (64-bit) machine, the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 183 Contact your vendor" <ac:structure MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> d-macro n>_All_Arch.exe /v"/l*vx </ac:structured-macro> ac:name="un [LogFile]" migrated- wiki-markup" ac:schema- version="1" ac:macro- id="8fba82b 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 184 "ERROR!!! Installation failed due to following errors: MlxRshim drivers installation disabled and MlxRshim drivers Installed, Please remove the following oem inf les NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 185 /norestart or /forcerestart standard command- line options. <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn migrated- wiki-markup" ac:schema- version="1" ac:macro- id="7e2a314 4-c9a8-453c- 8694- 0a0ed32375 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 186 MT_NDPROPERTY default value is True ). Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe Windows Server 2019 Windows Server 2016 Windows Server 2012 Windows 11 Client (64 bit only) Windows 10 Client (64 bit only) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 187 (or a similar) error message: "The installation package is supported by this processor type. Contact your vendor" <ac:structure MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> d-macro n>_All_Arch.exe /v"/l*vx </ac:structured-macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 188 Rshim driver is already installed. The following fail message will be displayed in the log: "ERROR!!! Installation failed due to following errors: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 189 To control the reboots, use /norestart or /forcerestart standard command- line options. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 190 /S /v/qn /v"/l*vx [LogFile]" -ec89-4d11- b755- e3bb6be5f96 a"><ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2- ]]></ac:plain-text-body> d-macro [Driver/Version]_<revision_versi </ac:structured-macro> ac:name="un on>_All_Arch.exe migrated- /vMT_NDPROPERTY= 1 wiki-markup" ac:schema- version="1" ac:macro- id="e8e4ee9 4-ea0e- 45d8-a365- 38dd5f6afab NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 191 Windows 8.1 Client (64 bit only) echo %PROCESSO R_ARCHITEC TURE% On an x64 (64-bit) machine, the output will be "AMD64". Installing the incorrect .exe le is prohibited. If NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 192 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> ac:macro- n>_All_Arch.exe /v"/l*vx </ac:structured-macro> id="8fba82b [LogFile]" 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 193 MlxRshim drivers installation disabled and MlxRshim drivers Installed, Please remove the following oem inf les from driver store: <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 194 MLNX_WinOF2-[Driver/Version] ac:macro- ]]></ac:plain-text-body> <revision_version>_All -Arch.exe id="7e2a314 </ac:structured-macro> /S /v/qn 4-c9a8-453c- 8694- 0a0ed32375 48"> <ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> d-macro <revision_version>_All -Arch.exe </ac:structured-macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 195 /vMT_NDPROPERTY= 1 4-ea0e- 45d8-a365- 38dd5f6afab 5"><ac:plain- text-body><! [CDATA[ <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- MLNX_WinOF2- version="1" [Driver/Version]_<revision_versi ]]></ac:plain-text-body> ac:macro- on>_All_Arch.exe </ac:structured-macro> id="023902e /vMT_SKIPFWUPGRD= 1 8-0815- 4298-a57f- ea14fd3883e 5"><ac:plain- text-body><! [CDATA[ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 196 On an x64 (64-bit) machine, the output will be "AMD64". Installing the incorrect .exe le is prohibited. If you do so, an error message will be displayed. For example, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 197 MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> ac:macro- n>_All_Arch.exe /v"/l*vx </ac:structured-macro> id="8fba82b [LogFile]" 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" MLNX_WinO F2_<revision_ version>_All_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 198 MlxRshim drivers Installed, Please remove the following oem inf les from driver store: <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP ORTEDDEVC HECK=1" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 199 ]]></ac:plain-text-body> <revision_version>_All -Arch.exe id="7e2a314 </ac:structured-macro> /S /v/qn 4-c9a8-453c- 8694- 0a0ed32375 48"> <ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn /v"/l*vx [LogFile]" migrated- wiki-markup" ac:schema- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 200 [CDATA[ <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- MLNX_WinOF2- version="1" [Driver/Version]_<revision_versi ]]></ac:plain-text-body> ac:macro- on>_All_Arch.exe </ac:structured-macro> id="023902e /vMT_SKIPFWUPGRD= 1 8-0815- 4298-a57f- ea14fd3883e 5"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 201 %PROCESSO R_ARCHITEC TURE% On an x64 (64-bit) machine, the output will be "AMD64". Installing the incorrect .exe le is prohibited. If you do so, an error message will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 202 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" MLNX_WinOF2_<revision_versio ]]></ac:plain-text-body> ac:macro- n>_All_Arch.exe /v"/l*vx </ac:structured-macro> id="8fba82b [LogFile]" 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 203 MlxRshim drivers installation disabled and MlxRshim drivers Installed, Please remove the following oem inf les from driver store: <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 204 MLNX_WinOF2-[Driver/Version] ac:macro- ]]></ac:plain-text-body> <revision_version>_All -Arch.exe id="7e2a314 </ac:structured-macro> /S /v/qn 4-c9a8-453c- 8694- 0a0ed32375 48"> <ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn /v"/l*vx [LogFile]" migrated- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 205 45d8-a365- 38dd5f6afab 5"><ac:plain- text-body><! [CDATA[ <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- MLNX_WinOF2- version="1" [Driver/Version]_<revision_versi ]]></ac:plain-text-body> ac:macro- on>_All_Arch.exe </ac:structured-macro> id="023902e /vMT_SKIPFWUPGRD= 1 8-0815- 4298-a57f- ea14fd3883e 5"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 206 <oem inf list>" 7. [Optional] If you want to enable the default con guration for Rivermax, run. Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 207 If you do so, an error message will be displayed. For example, if you install a 64-bit .exe on a 32-bit machine, the wizard will display the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 208 </ac:structured-macro> id="8fba82b [LogFile]" 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 209 <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP ORTEDDEVC HECK=1" If no reboot options are speci ed, the installer restarts the computer whenever necessary NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 210 4-c9a8-453c- 8694- 0a0ed32375 48"> <ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn /v"/l*vx [LogFile]" migrated- wiki-markup" ac:schema- version="1" ac:macro- id="ee8c8d4f -ec89-4d11- b755- e3bb6be5f96 a"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 211 </ac:structured-macro> id="023902e /vMT_SKIPFWUPGRD= 1 8-0815- 4298-a57f- ea14fd3883e 5"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver installanion will fail if a NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 212 Arch.exe /v "MT_RIVERM AX=1 /l*vx C:\Users\ <user>\log.tx t " 8. [Optional] If you want to skip the check for unsupported devices, run/ Description Package Windows MLNX_WinOF2- Server 2022 <version>_All_x64.exe NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 213 If you do so, an error message will be displayed. For example, if you install a 64-bit .exe on a 32-bit machine, the wizard will display the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 214 </ac:structured-macro> id="8fba82b [LogFile]" 7-5ca3-409a- 8c76- f43a9d2bc7b 8"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_SKIPFW UPGRD=1" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 215 <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP ORTEDDEVC HECK=1" If no reboot options are speci ed, the installer restarts the computer whenever necessary NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 216 4-c9a8-453c- 8694- 0a0ed32375 48"> <ac:plain- text-body><! [CDATA[ <ac:structure MLNX_WinOF2-[Driver/Version] ]]></ac:plain-text-body> <revision_version>_All -Arch.exe d-macro </ac:structured-macro> ac:name="un /S /v/qn /v"/l*vx [LogFile]" migrated- wiki-markup" ac:schema- version="1" ac:macro- id="ee8c8d4f -ec89-4d11- b755- e3bb6be5f96 a"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 217 </ac:structured-macro> id="023902e /vMT_SKIPFWUPGRD= 1 8-0815- 4298-a57f- ea14fd3883e 5"><ac:plain- text-body><! [CDATA[ MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " MT_DISABLE _RSHIM_INS TALL=1" The Rshim driver installanion will fail if a NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 218 <oem inf list>" MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v "MT_RIVERM AX=1 /l*vx C:\Users\ <user>\log.tx t " MLNX_WinO F2_<revision_ version>_All_ Arch.exe /v " SKIPUNSUPP ORTEDDEVC HECK=1" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 219 Firmware Upgrade If the machine has a standard NVIDIA® card with an older rmware version, the rmware will be automatically updated as part of the NVIDIA® WinOF-2 package installation. For information on how to upgrade rmware manually, please refer to MFT User Manual.
  • Page 220 1. Verify that the system has a NVIDIA network adapter installed. The following example shows a system with an installed NVIDIA adapter card: Requirements Description A server platform with a ConnectX®-6 (IB, EN) ( rmware: Platforms fw-ConnectX6) installed Required Disk Space for Installation Linux operating system.
  • Page 221 [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body></ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_LINUX- <ver>-<OS label>.iso Installing MLNX_OFED Installation Script The installation script, mlnxofedinstall, performs the following: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 222 <ac:plain-text-body><![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body></ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema-version="1" ac:macro- id="93935fa9-6364-496b-825e-e6057c30b53a"> <ac:plain-text-body><![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 223 Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema-version="1" ac:macro- id="93935fa9-6364-496b-825e-e6057c30b53a"> <ac:plain-text-body><![CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body></ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_LINUX- <ver>-<OS label>.iso NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 224 MLNX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 225 Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 226 Platforms ( rmware: fw-ConnectX6) installed Required Disk Space for Installation Operating Linux operating system. For System the list of supported operating system distributions and kernels, please refer to the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 227 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 228 ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 229 Privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- 8f81-0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 230 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 231 The regular installation can be performed and weak- updates mechanism will create symbolic links NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 232 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- 8f81-0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 233 To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text-body> d-macro [OPTIONS] </ac:structured-macro> ac:name="un migrated- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 234 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 235 -f On some OSs, dracut -f might result in the following error message which can be safely ignore. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 236 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- 8f81-0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 237 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 238 The regular installation can be performed and weak- updates mechanism will create symbolic links NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 239 In such cases, the installer will recommend running the following command to update the initramfs: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 240 Example: The following command will create a MLNX_OFED_LINUX ISO image for RedHat 7.3 under the /tmp directory. Requirement Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw-ConnectX6) installed Required Disk Space for Installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 241 Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="93935fa9-6364-496b- 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 242 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 243 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 244 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 245 3 x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 246 -editor- placehol der- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 247 Operating system distributions and kernels, System please refer to the MLNX_OFED Release Notes le. The installation requires Installer administrator (root) privileges on Privileges the target machine. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 248 Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedinst all --fw-image- /tmp/my_fw_bi n_ les If the driver detects unsupported cards on the system, it will abort the installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 249 ".conf.rpmsave ". On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_ker nel_support.sh script. The regular installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 250 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 251 MLNX_OFED installation. Therefore, using "mlnx_add_ker nel_support.sh " is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. 3 x86_64/ mlnx_ad d_kernel _support. sh -m /tmp/ML NX_OFE NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 252 OEM, OFED, or Distributi on IB packages will be removed. images/p lugins/se rvlet/con uence/p laceholde r/error- i18nkey- editor- placehol der- broken- link- locale- en_us- version- 2.png > See log NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 253 ./mlnxofedinst all --h Installation Procedure This section describes the installation procedure of MLNX_OFED on NVIDIA adapter cards. 1. Log in to the installation machine as root. 2. Mount the ISO image on your machine. Requirement Description A server platform with a ConnectX®-6 (IB, EN)
  • Page 254 Technologies Device 0014 <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="93935fa9-6364-496b- 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 255 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 256 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 257 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 258 3 x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 259 -editor- placehol der- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 260 Linux operating system. For the list of supported operating system Operating distributions and kernels, System please refer to the MLNX_OFED Release Notes Installer The installation requires Privileges administrator (root) privileges NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 261 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 262 [OPTIONS] </ac:structured-macro> id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 263 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with a root lesystem mounted over a ConnectX NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 264 Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 265 /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 266 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 267 Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 268 MLNX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated-wiki- markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- 8f81-0c1e2667e406"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 269 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 270 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ke rnel_support. sh script. The regular installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 271 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 272 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX- x.x-x- rhel7. 3 x86_64/ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 273 MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be remove images/ plugins/ servlet/ con ue nce/plac eholder/ error- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 274 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 275 Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 276 Operating distributions and kernels, System please refer to the MLNX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci -v grep Mellanox NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 277 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 278 [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 279 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 280 Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 281 --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be remove NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 282 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 283 RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 284 MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 285 ( rmware: fw-ConnectX6) installed Required Disk Space for Installation Linux operating system. For the list of supported operating system Operating distributions and kernels, System please refer to the MLNX_OFED Release Notes NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 286 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 287 ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ac:macro- [OPTIONS] </ac:structured-macro> id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 288 (using --add-kernel- support), the packages installation will not involve automatic regeneration of the initramfs. In some cases, such as a system with a root lesystem mounted NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 289 Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 290 -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 291 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 292 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 293 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 294 In case your machine has the latest rmware, no rmware update will occur and the installation script will print at the end of installation a message similar to the following: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 295 Operating distributions and kernels, System please refer to the MLNX_OFED Release Notes The installation requires Installer administrator (root) privileges Privileges on the target machine. 1. lspci -v grep Mellanox NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 296 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 297 [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 298 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 299 Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 300 --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, Distribu tion IB package s will be remove NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 301 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removi OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 302 RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 303 MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 304 Device #1: ---- ------ Device Type: ConnectX-X Part Number: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 305 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 306 Refer to the table below to nd the appropriate action for your speci c card. Requireme Description Platforms A server platform with a ConnectX®-6 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 307 1. lspci - grep Mellanox <ac:structured- macro ac:name="unmigrate d-wiki-markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1- 4564-8f81- 0c1e2667e406"> <ac:plain-text-body> <![CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body> </ac:structured- macro> Subsystem: Mellanox NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 308 </ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFE D_LINUX- <ver>-<OS label>.iso ./mlnxofedi nstall --fw- image-dir /tmp/my_f w_bin_ les If the driver detects unsupporte d cards on system, it NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 309 38-2def- 47c2-9170- 0c9e6df31 252"> <ac:plain- text-body> <![CDATA[ Pre- existing con gurati on les will be saved with the extension ".conf.rpms ave". On Redhat and SLES distribution s with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 310 If you regenerate kernel modules for a custom kernel (using -- add-kernel- support), packages installation will not involve automatic regeneratio n of the initramfs. In some cases, such NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 311 -f On Ubuntu and Debian distribution s drivers installation Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 312 Debian distribution 1. ./MLN X_OFE D_LIN x.x-x- rhel7. x86_6 4/mln x_add _kern el_sup port.s h -m /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 4/ -- make- Note: This progra NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 313 /tmp direct ory. Mella nox, OEM, OFED, Distri bution packa will be remov image s/plug ins/se rvlet/c on ue nce/pl acehol der/er ror- i18nk editor place holder broke NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 314 2.png > /tmp/ mlnx_ ofed_i 21642 .log Buildi OFED RPMs. Please wait... Remo ving OFED RPMs. Creat /tmp/ MLNX _OFE D_LIN x.x-x- rhel7. x86_6 ext.tg NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 315 MLNX_OFE D_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution packages will be removed. Those packages removed due to con icts NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 316 Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFE installation script: /mnt/mlnxo fedinstall -- force MLNX_OFE D for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 317 -- kernel 3.13.0-85- generic -- without-fw- update -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 318 Device #1: - --------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<versio n> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2 d5cf810 Versions: Current Available XX.XX.XXX NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 319 Error message #1: Device #1: ---------- Device Type: ConnectX-X Part Number: MCXXXX- XXX PSID: MT_<versio n> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 320 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/Connect X-4 Lx or higher Adapters with Multi- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 321 /etc/in niband/openib.conf le. Requiremen Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw- ConnectX6) installed Required Disk Space Installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 322 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="93935fa9- 6364-496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 323 To avoid this, make sure to add --skip- unsupporte d-devices- check ag during installation. <ac:structur ./mnt/mlnxofedinstall ]]></ac:plain-text-body> ed-macro [OPTIONS] </ac:structured-macro> ac:name="u nmigrated- wiki- markup" ac:schema- version="1" ac:macro- id="204fa23 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 324 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFE D kernel modules. If you regenerate kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 325 In such cases, the installer will recommend running the following command to update initramfs: dracut -f NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 326 MLNX_OFE installation. Therefore, using "mlnx_add_k ernel_suppo rt.sh" is irrelevant on Ubuntu and Debian distributions 1. ./MLNX _OFED _LINUX -x.x-x- rhel7. x86_6 4/mlnx _add_k ernel_s upport. sh -m /tmp/ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 327 4/ -- make- Note: This progra m will create MLNX_ OFED_ LINUX rhel7. under /tmp directo Mellan OEM, OFED, Distrib ution packag es will remov images /plugin s/servl et/conf luence/ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 328 -2.png > See log /tmp/ mlnx_o fed_iso 21642 .log Buildin OFED RPMs. Please wait... Removi OFED RPMs... Create /tmp/ MLNX_ OFED_ LINUX- x.x-x- rhel7. x86_6 ext.tgz NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 329 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFE D_LINUX, do NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 330 -- force installation option while running the MLNX_OFE installation script: /mnt/mlnxof edinstall -- force MLNX_OFE D for Ubuntu should be installed with the following ags in NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 331 Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 332 Name: 0b:00.0 Base MAC: 0000e41d2 d5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupporte d network adapter device, no rmware update will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 333 > PCI Device Name: 0b:00.0 Base MAC: 0000e41d2 d5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: rmware for this device is not distributed inside NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 334 OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Installation Results Requirements Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw-ConnectX6) installed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 335 Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- id="93935fa9-6364-496b- 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 336 --skip- unsupported- devices-check ag during installation. <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- version="1" ]]></ac:plain-text- ./mnt/mlnxofedinstall ac:macro- body></ac:structured- [OPTIONS] id="204fa238- macro> 2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 337 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add- kernel- support), the packages installation will not involve automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 338 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 339 _support .sh -m /tmp/ML NX_OFE D_LINUX -x.x-x- rhel7. 3 - x86_64/ --make- Note: This program will create MLNX_O FED_LIN UX TGZ rhel7. 3 under /tmp directory Mellanox , OEM, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 340 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/ML NX_OFE D_LINUX -x.x-x- rhel7. 3 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 341 Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 342 -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedinst all -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 343 Device #1: ----- ----- Device Type: ConnectX-X Part Number: MCXXXX-XXX NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 344 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ----- ----- Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 345 Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) Standard ConnectX- 4/ConnectX-4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 346 SLES and Fedora Distributions /lib/modules/`u name - r`/extra/mlnx- ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/`u name - r`/updates/dkm s/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 347 "The rmware for this device is not distributed inside NVIDIA driver: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 348 MLNX_OFED Release Notes le. The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 349 Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedinst all --fw- image-dir /tmp/my_fw_b in_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 350 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_ker nel_support.s h script. The regular installation can be NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 351 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 352 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support.s h" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _support .sh -m /tmp/ML NX_OFE NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 353 Mellanox , OEM, OFED, or Distribut ion IB package s will be removed images/ plugins/s ervlet/co n uence /placehol der/error i18nkey- editor- placehol der- broken- link- locale- en_us- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 354 3 - x86_64- ext.tgz ./mlnxofedinst all --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_O FED_LINUX- x.x-x.logs This program will install the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 355 Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 356 -- kernel 3.13.0- 85-generic -- without-fw- update --force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 357 MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d5 cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 358 Base MAC: 0000e41d2d5 cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 359 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox/ip route2/sbin/rd NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 360 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: If an adapter's Flash was NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 361 .IBMM215011 0033.logs Driver Load Upon System Boot Upon system boot, the NVIDIA drivers will be loaded automatically. To prevent the automatic load of the NVIDIA drivers upon system boot: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 362 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 363 /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 364 On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed and weak- updates NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 365 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the following command to NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 366 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 367 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 368 OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 369 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 370 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 371 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 372 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 373 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 374 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 375 Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 376 1, and will prevent the kernel from loading the modules automatically. mlnxofedinstall Return Codes The table below lists the mlnxofedinstall script return codes and their meanings. Requirements Description Platforms A server platform with a ConnectX®-6 (IB, EN) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 377 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- id="93935fa9-6364-496b- 825e-e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 378 --skip- unsupported- devices-check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- wiki-markup" ac:schema- version="1" ac:macro- id="204fa238- 2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 379 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add- kernel- support), the packages installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 380 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 381 _support .sh -m /tmp/ML NX_OFE D_LINUX -x.x-x- rhel7. 3 - x86_64/ --make- Note: This program will create MLNX_O FED_LIN UX TGZ rhel7. 3 under /tmp directory NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 382 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/ML NX_OFE NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 383 Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 384 /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedinst all -without- dkms --add- kernel- support -- kernel <kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 385 Device #1: ----- ----- Device Type: ConnectX-X NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 386 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ----- ----- Device Type: ConnectX-X Part Number: MCXXXX-XXX NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 387 HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) Standard ConnectX- 4/ConnectX-4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 388 Software /lib/modules/`u name - r`/updates on SLES and Fedora Distributions /lib/modules/`u name - r`/extra/mlnx- ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/`u name - r`/updates/dkm s/ on Ubuntu NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 389 In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is not NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 390 Prerequisites are not met. For example, missing the required software installed or the hardware is not con gured correctly Failed to start the mst driver NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 391 8f81-0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text-body> </ac:structured-macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- id="93935fa9-6364-496b- 825e-e6057c30b53a"> <ac:plain-text-body><! NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 392 To avoid this, make sure to add --skip- unsupported- devices-check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body></ac:structured- ac:name="un macro> migrated- wiki-markup" ac:schema- version="1" ac:macro- id="204fa238- 2def-47c2- 9170- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 393 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using --add- kernel- support), the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 394 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 395 _support .sh -m /tmp/ML NX_OFE D_LINUX -x.x-x- rhel7. 3 - x86_64/ --make- Note: This program will create MLNX_O FED_LIN UX TGZ rhel7. 3 under /tmp NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 396 /placehol der/error i18nkey- editor- placehol der- broken- link- locale- en_us- version- 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 397 LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 398 MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedinst all -without- dkms --add- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 399 Device #1: ----- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 400 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ----- ----- Device Type: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 401 HW vendor. Action \ Standard Reboot (Soft Cold Reboot (Hard Driver Restart Adapter Reset) Reset) Standard ConnectX- 4/ConnectX-4 Lx or higher Adapters with Multi- Host Support NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 402 The kernel modules are installed under /lib/modules/`u name - r`/updates on SLES and Fedora Distributions /lib/modules/`u name - r`/extra/mlnx- ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/`u name - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 403 Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 404 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 405 Fedora Distributions /lib/modules/`u name - r`/extra/mlnx- ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/`u name - r`/updates/dkm s/ on Ubuntu The rmware of existing network adapter devices will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 406 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 407 MLNX_OFED Release Notes le. The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema- version="1" ac:macro- id="a47cf27f-cef1-4564- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 408 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedinst all --fw- image-dir /tmp/my_fw_b in_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 409 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use the mlnx_add_ker nel_support.s h script. The regular installation can be performed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 410 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the following NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 411 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support.s h" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _support .sh -m /tmp/ML NX_OFE D_LINUX NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 412 Mellanox , OEM, OFED, or Distribut ion IB package s will be removed images/ plugins/s ervlet/co n uence /placehol der/error i18nkey- editor- placehol der- broken- link- locale- en_us- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 413 3 - x86_64- ext.tgz ./mlnxofedinst all --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_O FED_LINUX- x.x-x.logs This program will install the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 414 Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 415 -- kernel 3.13.0- 85-generic -- without-fw- update --force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 416 MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d5 cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 417 Base MAC: 0000e41d2d5 cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 418 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox/ip route2/sbin/rd NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 419 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: If an adapter's Flash was NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 420 HW vendor." Logs dir: /tmp/MLNX_O FED_LINUX- 4.4 - 1.0 . 0.0 .IBMM215011 0033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 421 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellanox/ip route2/sbin/rd The kernel modules are installed under /lib/modules/`u name - r`/updates on SLES and Fedora Distributions NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 422 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, the automatic rmware update will also NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 423 Installing MLNX_OFED Using YUM This type of installation is applicable to RedHat/OL and Fedora operating systems. Setting up MLNX_OFED YUM Repository 1. Log into the installation machine as root. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 424 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 425 --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 426 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 427 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 428 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 429 /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 430 Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 431 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 432 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 433 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 434 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 435 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 436 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 437 Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 438 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 439 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 440 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic rmware NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 441 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 442 3. Download and install NVIDIA's GPG-KEY: The key can be downloaded via the following link: http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY- Mellanox Requirement Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw- ConnectX6) installed Required Disk Space for Installation Linux operating system.
  • Page 443 --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 444 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 445 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 446 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 447 /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 448 Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 449 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 450 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 451 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 452 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 453 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 454 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 455 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 456 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 457 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 458 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 459 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 460 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 461 ConnectX6) installed Required Disk Space for Installation Linux operating system. For the list of supported operating Operating system distributions System and kernels, please refer to the MLNX_OFED Release Notes le. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 462 0014 1. lspci -v <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 463 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- ]]></ac:plain-text- version="1" ./mnt/mlnxofedinstall body> ac:macro- [OPTIONS] </ac:structured- id="204fa238 macro> -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 464 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using - -add-kernel- support), the packages installation will not involve automatic regeneration NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 465 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 466 NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 467 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/M LNX_OF NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 468 LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 469 -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 470 In case your machine has the latest rmware, no rmware update will occur and the installation script will print at the end of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 471 Please contact your hardware vendor for help with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 472 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 473 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 474 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 475 NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 476 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 477 The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation script is run in default mode; that is, without the option '- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 478 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 479 01 25 abe2-4bb6-abc6- 13 : 52 : a63afe0f4f7c"> 30 - <ac:plain-text- http: body><! //www.m [CDATA[Length: ellanox.c 1354 ( 1 .3K) om/dow [text/plain] nloads/o ]]></ac:plain-text- fed/RP body> M-GPG- </ac:structured- KEY- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 480 <ac:plain-text- body><![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ? RPM-GPG-KEY- Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 481 ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG 0x6224 C050 : Userid: "Mellano Technol ogies (Mellano NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 482 OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- 7ba6- 4258- 87b5- 7bb476 76aa8a" > <ac:plai n-text- body><! [CDATA[ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 483 Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 484 <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 485 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 486 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 487 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 488 3 under /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 489 Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 490 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 491 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 492 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 493 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 494 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 495 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 496 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 497 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 498 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 499 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 500 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 501 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 502 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 503 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 504 {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' 6. Create a yum repository con guration le called "/etc/yum.repos.d/mlnx_ofed.repo" with the following content: Requirement Description NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 505 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 506 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 507 The regular installation can be performed and weak- updates mechanism will create symbolic links NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 508 In such cases, the installer will recommend running the following command to update the initramfs: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 509 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 510 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- en_us- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 511 Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 512 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 513 -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- force Note that the path to kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 514 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 515 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 516 Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 517 RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 518 Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 519 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 520 /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 521 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 522 /mnt 1. wget 72.3 . 194.0 : 80 ... connected. ]]></ac:plain- http: HTTP request text-body> //www.m sent, awaiting </ac:structured ellanox.c response... 200 OK -macro> om/dow <ac:structured- nloads/o macro NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 523 <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 524 -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 525 - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 526 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository version="1" baseurl= ac:macro- le: ///<path to id="bf78c405 extracted MLNX_OFED -d0c6-41cc- package>/RPMS b66f- enabled= 1 0fc3164758b gpgkey= 0"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 527 (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 528 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 529 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 530 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 531 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 532 TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 533 RPMs. Please wait... Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 534 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 535 --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 536 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 537 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 538 (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 539 /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 540 If an adapter's Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 541 The installation failed No rmware was found for the adapter device Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 542 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 543 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 544 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 545 Mellanox? <ac:structured- www.me macro llanox.co ac:name="unmigra m... 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 546 Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 547 0x6224 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 548 1. rpm -q gpg- pubkey - -qf '% grep Mellanox {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 549 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 550 ConnectX6) installed Required Disk Space for Installation Linux operating system. For the list of supported operating Operating system distributions System and kernels, please refer to the MLNX_OFED Release Notes le. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 551 0014 1. lspci -v <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 552 <ac:structure d-macro ac:name="un migrated- wiki-markup" ac:schema- ]]></ac:plain-text- version="1" ./mnt/mlnxofedinstall body> ac:macro- [OPTIONS] </ac:structured- id="204fa238 macro> -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 553 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using - -add-kernel- support), the packages installation will not involve automatic regeneration NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 554 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 555 NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 556 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/M LNX_OF NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 557 LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to con icts with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 558 -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with the following ags in chroot environment: ./mlnxofedins NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 559 In case your machine has the latest rmware, no rmware update will occur and the installation script will print at the end of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 560 Please contact your hardware vendor for help with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 561 Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, please contact your HW vendor. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 562 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 563 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 564 NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 565 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 566 The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation script is run in default mode; that is, without the option '- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 567 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 568 01 25 abe2-4bb6-abc6- 13 : 52 : a63afe0f4f7c"> 30 - <ac:plain-text- http: body><! //www.m [CDATA[Length: ellanox.c 1354 ( 1 .3K) om/dow [text/plain] nloads/o ]]></ac:plain-text- fed/RP body> M-GPG- </ac:structured- KEY- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 569 <ac:plain-text- body><![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ? RPM-GPG-KEY- Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 570 ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG 0x6224 C050 : Userid: "Mellano Technol ogies (Mellano NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 571 OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- 7ba6- 4258- 87b5- 7bb476 76aa8a" > <ac:plai n-text- body><! [CDATA[ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 572 0fc3164758b le: ///<path to the 0"><ac:plain- downloaded key RPM- text-body><! GPG-KEY-Mellanox> [CDATA[ gpgcheck= 1 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 573 MLNX_ OFED Reposit ory 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 574 Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 575 <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 576 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 577 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 578 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 579 3 under /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 580 Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 581 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 582 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 583 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 584 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 585 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 586 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 587 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 588 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 589 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 590 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 591 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 592 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 593 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 594 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 595 -qf '% grep Mellanox {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 596 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 597 <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 598 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 599 WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 600 MLNX_OFED Release Notes le. The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 601 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 602 9170- 0c9e6df3125 2"><ac:plain- text-body><! [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 603 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 604 Module Support (DKMS) framework. Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 605 Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 606 /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 607 IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 608 ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 609 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 610 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 611 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 612 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 613 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 614 Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed No rmware was found NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 615 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 616 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 617 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 618 Mellano Saving to: ?RPM- GPG-KEY- Resolvin Mellanox? <ac:structured- www.me macro llanox.co ac:name="unmigra m... 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 619 : 30 ( 247 MB/s) - ? RPM-GPG-KEY- Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 620 KEY- Mellano Importin g GPG 0x6224 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 621 <ac:plai n-text- body><! [CDATA[ Is this ok [y/N]: 1. rpm -q grep Mellanox gpg- gpg-pubkey-a9e4b643- pubkey - 520791ba -qf '% gpg(Mellanox {NAME} Technologies <support @mellanox .com>) {VERSIO {RELEAS E}\t% NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 622 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 623 8 , 351 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 624 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 625 G: If you going to con gur e this package as a reposito ry, then please note WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 626 ( rmware: fw- ConnectX6) installed Required Disk Space for Installation Operating Linux operating system. System For the list of supported operating system distributions and kernels, please refer to the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 627 MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 628 --skip- unsupported- devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body> ac:name="un </ac:structured- migrated- macro> wiki-markup" ac:schema- version="1" ac:macro- id="204fa238 -2def-47c2- 9170- 0c9e6df3125 2"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 629 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using - -add-kernel- support), the packages NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 630 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the drivers' NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 631 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 3 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 632 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 633 /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED_ LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 634 Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu should be installed with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 635 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no rmware update will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 636 XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages below will be printed. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 637 Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain rmware for this device, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 638 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 639 '- without fw- update' The rmware version of the adapter device is older than the rmware NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 640 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 641 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 642 `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation script is run in default NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 643 "The rmware for this device is not distributed inside NVIDIA driver: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 644 - 2018 id=" 99241e- 01 25 abe2-4bb6-abc6- 13 : 52 : a63afe0f4f7c"> 30 - <ac:plain-text- http: body><! //www.m [CDATA[Length: ellanox.c 1354 ( 1 .3K) om/dow [text/plain] nloads/o NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 645 </ac:structured- macro> <ac:structured- macro ac:name="unmigra ted-wiki-markup" ac:schema- version="1" ac:macro- id="6dc3c0e3- a747-4d43-8afe- 4185f107057a"> <ac:plain-text- body><![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ? RPM-GPG-KEY- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 646 Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG 0x6224 C050 : Userid: "Mellano NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 647 From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- 7ba6- 4258- 87b5- 7bb476 76aa8a" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 648 -d0c6-41cc- enabled= 1 b66f- gpgkey= 0fc3164758b le: ///<path to the 0"><ac:plain- downloaded key RPM- text-body><! GPG-KEY-Mellanox> [CDATA[ gpgcheck= 1 1. yum repolist Loaded plugins: product -id, security, subscrip NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 649 MLNX_ OFED Reposit ory 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount o ro,loop MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 650 -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 651 OFED_LI RPMS . Please wait... Creating metadat a-rpms for 3.10 . 0 - 957.21 . .el7.x86 _64 ... WARNIN G: If you going to con gur e this package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 652 Created /tmp/M LNX_OF ED_LIN UX- 5.2 - 0.5 . 5.0 -rhel7. 6 x86_64- ext.tgz 1. cd /tmp/ <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 653 MLNX_OFED Release Notes le. The installation requires Installer administrator (root) Privileges privileges on the target machine. 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 654 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 SHA256 MLNX_OFED_ LINUX-<ver>- <OS label>.iso ./mlnxofedins tall --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 655 [CDATA[ Pre-existing con guration les will be saved with the extension ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 656 In some cases, such as a system with a root lesystem mounted over a ConnectX card, not regenerating the initramfs may even NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 657 Thus, the drivers' compilation will take place on the host during MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 658 MLNX_ OFED_LI TGZ for rhel7. 3 under /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 659 _iso. 21642 .log Building OFED RPMs. Please wait... Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 660 IB packages will be removed. Those packages are removed due to con icts with MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 661 ./mlnxofedins tall -without- dkms --add- kernel- support -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 662 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 663 Please contact your hardware vendor for help with rmware updates. Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 664 HW vendor. Action \ Standard Reboot Cold Reboot Driver Restart Adapter (Soft Reset) (Hard Reset) Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 665 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 666 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 667 Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs blacklist mlx5_core blacklist mlx5_ib Return Code Meaning The Installation ended successfully The installation failed No rmware was found NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 668 "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 669 '- without fw- update' The rmware version of the adapter device is older than the rmware version included with the OFED ISO image Note: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 670 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 671 Mellano Saving to: ?RPM- GPG-KEY- Resolvin Mellanox? <ac:structured- www.me macro llanox.co ac:name="unmigra m... 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 672 : 30 ( 247 MB/s) - ? RPM-GPG-KEY- Mellanox? saved [ 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 673 KEY- Mellano Importin g GPG 0x6224 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 674 <ac:plai n-text- body><! [CDATA[ Is this ok [y/N]: 1. rpm -q grep Mellanox gpg- gpg-pubkey-a9e4b643- pubkey - 520791ba -qf '% gpg(Mellanox {NAME} Technologies <support @mellanox .com>) {VERSIO {RELEAS E}\t% NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 675 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 676 8 , 351 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 677 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 678 G: If you going to con gur e this package as a reposito ry, then please note WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 679 Repository ac:macro- baseurl= id="90524ed le: ///<path to 9-998e-430c- extracted MLNX_OFED a1c6- package>/RPMS b2a8c9f685d enabled= 1 c"><ac:plain- gpgcheck= 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 680 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 Installing MLNX_OFED Using the YUM Tool After setting up the YUM repository for MLNX_OFED package, perform the following: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 681 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 682 --fw- image-dir /tmp/my_fw_ bin_ les If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 683 ".conf.rpmsav e". On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 684 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 685 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 686 /tmp director Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 687 Removin g OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 688 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 689 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 690 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 691 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 692 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 693 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 694 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 695 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 696 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 697 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 698 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 699 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 700 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 701 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 702 -qf '% grep Mellanox {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 703 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 704 <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 705 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 706 WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 707 1 c"><ac:plain- gpgcheck= 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 708 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 709 : MLNX_ OFED basic installer package (with support) mlnx- ofed- basic- user- only.noa rch : MLNX_ OFED basic- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 710 : MLNX_ OFED blue el d-user- only installer package (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 711 : MLNX_ OFED dpdk- upstrea m-libs- user- only installer package (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 712 (User Space package s only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 713 MLNX_ OFED installer package (with support) mlnx- ofed- hpc- user- only.noa rch : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 714 (User Space package s only) mlnx- ofed- kernel- only.noa rch : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 715 : MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 716 MLNX_ OFED vma-vpi installer package (with support) mlnx- ofed- vma- vpi- user- only.noa rch : MLNX_ OFED vma- vpi- user- only installer package (User Space package s only where: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 717 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 718 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 719 On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed and weak- updates mechanism NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 720 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the following command to NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 721 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 722 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 723 OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 724 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 725 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 726 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 727 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 728 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 729 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 730 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 731 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 732 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 733 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 734 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 735 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 736 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 737 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 738 -qf '% grep Mellanox {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 739 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 740 <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 741 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 742 WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 743 1 c"><ac:plain- gpgcheck= 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 744 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 745 : MLNX_ OFED basic installer package (with support) mlnx- ofed- basic- user- only.noa rch : MLNX_ OFED basic- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 746 : MLNX_ OFED blue el d-user- only installer package (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 747 : MLNX_ OFED dpdk- upstrea m-libs- user- only installer package (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 748 (User Space package s only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 749 MLNX_ OFED installer package (with support) mlnx- ofed- hpc- user- only.noa rch : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 750 (User Space package s only) mlnx- ofed- kernel- only.noa rch : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 751 : MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 752 : MLNX_ OFED vma-vpi installer package (with support) mlnx- ofed- vma- vpi- user- only.noa rch : MLNX_ OFED vma- vpi- user- only installer package (User Space package s only NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 753 RHEL and SLES. For other operating systems, kernel module RPM packages are provided only for the operating system's default kernel. In this case, the group RPM packages have the supported kernel version in their package's name. Example: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 754 1. lspci -v grep Mellanox <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro-id="a47cf27f- cef1-4564-8f81- 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 755 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 756 On Redhat and SLES distributions with errata kernel installed there is no need to use mlnx_add_ker nel_support.s h script. The regular installation can be performed and weak- updates mechanism NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 757 ConnectX card, not regenerating the initramfs may even cause the system to fail to reboot. In such cases, the installer will recommend running the following command to NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 758 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 759 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 760 OFED RPMs... Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 761 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 762 -- kernel <kernel version in chroot> -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 763 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 764 Error message #1: Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 765 Standard ConnectX- 4/ConnectX- 4 Lx or higher Adapters with Multi- Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 766 SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 767 Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 768 Invalid parameter Not enough free space Not applicable to this system con guration. This can occur when the required hardware is not present on the system NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 769 /opt/Mellano x/iproute2/sb in/rdma The kernel modules are installed under /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 770 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 771 HW vendor." Logs dir: /tmp/MLNX_ OFED_LINUX- 4.4 - 1.0 . 0.0 .IBMM21501 10033.logs 1. mount o ro,loop MLNX_ OFED_LI NUX<ve r><OS label> <CPU arch>.is o /mnt NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 772 72.3 ted-wiki-markup" . 194.0 ac:schema- Connect version="1" ing to ac:macro- www.me id="14eb4058- llanox.co f674-4142-a252- b9eb47ddc32e"> <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 773 1354 / 1354 ] 1. sudo ]]></ac:plain-text- rpm -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 774 C050 : Userid: "Mellano Technol ogies (Mellano Technol ogies - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 775 -qf '% grep Mellanox {NAME} gpg-pubkey-a9e4b643- 520791ba {VERSIO gpg(Mellanox Technologies <support {RELEAS @mellanox .com>) E}\t% {SUMM ARY}\n' <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 776 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 777 <CPU arch>.is o /mnt 1. /mnt/ml nx_add_ kernel_s upport.s h -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 778 2.png > See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 779 WARNIN G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 780 1 c"><ac:plain- gpgcheck= 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 781 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 782 : MLNX_ OFED basic installer package (with support) mlnx- ofed- basic- user- only.noa rch : MLNX_ OFED basic- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 783 : MLNX_ OFED blue el d-user- only installer package (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 784 : MLNX_ OFED dpdk- upstrea m-libs- user- only installer package (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 785 (User Space package s only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 786 MLNX_ OFED installer package (with support) mlnx- ofed- hpc- user- only.noa rch : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 787 (User Space package s only) mlnx- ofed- kernel- only.noa rch : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 788 : MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 789 : MLNX_ OFED vma-vpi installer package (with support) mlnx- ofed- vma- vpi- user- only.noa rch : MLNX_ OFED vma- vpi- user- only installer package (User Space package s only NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 790 DPDK upstream-libs using RDMA-Core Installs packages kernel-only required for a non- default kernel mlnx-ofed-all- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED all installer package for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 791 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- hpc- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 792 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 793 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED vma installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 794 A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw- ConnectX6) installed Required Disk Space for Installation Operating Linux operating system. System For the list of supported operating NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 795 [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller [0207]: Mellanox Technologies MT28908A0 Family NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 796 --skip- unsupported- devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body> ac:name="un </ac:structured- migrated- macro> wiki-markup" ac:schema- version="1" ac:macro- id="204fa238 -2def-47c2- 9170- 0c9e6df3125 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 797 MLNX_OFED kernel modules. If you regenerate kernel modules for a custom kernel (using - -add-kernel- support), the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 798 -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support (DKMS) framework. Thus, the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 799 Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ OFED_LI TGZ for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 800 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. Please wait... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 801 Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED_ LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 802 Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force MLNX_OFED for Ubuntu NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 803 (- kernel- sources) should be added if the sources are not in their default location. In case your machine has the latest rmware, no NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 804 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the error messages NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 805 Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 033) To obtain NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 806 "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed under /opt/Mellano NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 807 '- without fw- update' The rmware version of the adapter device is NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 808 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 809 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 810 RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 811 In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is not NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 812 M-GPG- ted-wiki-markup" KEY- ac:schema- Mellano version="1" ac:macro- - 2018 id=" 99241e- 01 25 abe2-4bb6-abc6- 13 : 52 : a63afe0f4f7c"> 30 - <ac:plain-text- http: body><! //www.m [CDATA[Length: ellanox.c NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 813 ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra ted-wiki-markup" ac:schema- version="1" ac:macro- id="6dc3c0e3- a747-4d43-8afe- 4185f107057a"> <ac:plain-text- body><![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ? NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 814 HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG 0x6224 C050 : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 815 >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- 7ba6- 4258- 87b5- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 816 MLNX_OFED id="bf78c405 package>/RPMS -d0c6-41cc- enabled= 1 b66f- gpgkey= 0fc3164758b le: ///<path to the 0"><ac:plain- downloaded key RPM- text-body><! GPG-KEY-Mellanox> [CDATA[ gpgcheck= 1 1. yum repolist Loaded plugins: product -id, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 817 MLNX_ OFED Reposit ory 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount o ro,loop NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 818 $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 819 Building MLNX_ OFED_LI RPMS . Please wait... Creating metadat a-rpms for 3.10 . 0 - 957.21 . .el7.x86 _64 ... WARNIN G: If you going to con gur NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 820 Created /tmp/M LNX_OF ED_LIN UX- 5.2 - 0.5 . 5.0 -rhel7. 6 x86_64- ext.tgz 1. cd /tmp/ <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 821 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 822 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with support) mlnx- ofed-all- user- only.noa rch : MLNX_ OFED all-user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 823 : MLNX_ OFED basic- user- only installer package (User Space package s only) mlnx- ofed- blue eld .noarch : MLNX_ OFED blue eld installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 824 : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- dpdk- upstrea libs.noar ch : MLNX_ OFED dpdk- upstrea m-libs installer package (with support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 825 : MLNX_ OFED dpdk- user- only installer package (User Space package s only) mlnx- ofed- eth- only- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 826 (with support) mlnx- ofed- guest- user- only.noa rch : MLNX_ OFED guest- user- only installer package (User Space package s only) mlnx- ofed- hpc.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 827 OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc MLNX_ OFED hypervis installer package (with support) mlnx- ofed- hypervis or-user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 828 : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ OFED installer package (with support) mlnx- ofed- vma- eth.noar ch : MLNX_ OFED vma-eth NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 829 (User Space package s only) mlnx- ofed- vma- user- only.noa rch : MLNX_ OFED vma- user- only installer package (User Space package s only) mlnx- ofed- vma- vpi.noar NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 830 Installs basic packages mlnx-ofed- required for running basic NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS mlnx-ofed- Installs packages required for HPC Installs packages mlnx-ofed- required by hypervisor hypervisor NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 831 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- basic- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED basic installer package for kernel 3.17 . 4 .fc21.x86_64 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 832 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED hypervisor installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma- 3.17 . 4 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 833 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 834 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 835 --> Running transact check --> Package mlnx- ofed- all.noarc h 0 : 3.1 0.1 . 2 will be installed -> Processi Depend ency: kmod- isert = 1.0 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 836 1.12 sdpnets tat.x86_ 64 0 : 1.60 - srptools. x86_64 0 : 1.0 . 2 - 12 Complet Requirement Description Platforms A server platform with a ConnectX®-6 (IB, EN) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 837 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 838 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 839 The regular installation can be performed and weak- updates mechanism will create symbolic links NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 840 In such cases, the installer will recommend running the following command to update the initramfs: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 841 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 842 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- en_us- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 843 Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 844 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 845 -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- force Note that the path to kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 846 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 847 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 848 Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 849 RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 850 Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 851 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 852 /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 853 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 854 /mnt 1. wget 72.3 . 194.0 : 80 ... connected. ]]></ac:plain- http: HTTP request text-body> //www.m sent, awaiting </ac:structured ellanox.c response... 200 OK -macro> om/dow <ac:structured- nloads/o macro NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 855 <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 856 -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 857 - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 858 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository version="1" baseurl= ac:macro- le: ///<path to id="bf78c405 extracted MLNX_OFED -d0c6-41cc- package>/RPMS b66f- enabled= 1 0fc3164758b gpgkey= 0"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 859 Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmforg e RHEL NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 860 -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 861 See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . Please wait... Creating metadat a-rpms NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 862 G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf Created /tmp/M LNX_OF ED_LIN UX- 5.2 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 863 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 864 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with support) mlnx- ofed-all- user- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 865 (with support) mlnx- ofed- basic- user- only.noa rch : MLNX_ OFED basic- user- only installer package (User Space package s only) mlnx- ofed- blue eld .noarch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 866 (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- dpdk- upstrea libs.noar ch : MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 867 OFED dpdk- upstrea m-libs- user- only installer package (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 868 Space package s only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : MLNX_ OFED guest- user- only installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 869 : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc MLNX_ OFED hypervis installer package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 870 (User Space package s only) mlnx- ofed- kernel- only.noa rch : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ OFED installer package (with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 871 : MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : MLNX_ OFED vma- user- only NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 872 : MLNX_ OFED vma- vpi- user- only installer package (User Space package s only Installs all available mlnx-ofed-all packages in MLNX_OFED mlnx-ofed- Installs basic packages basic required for running NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 873 Installs packages kernel-only required for a non- default kernel mlnx-ofed-all- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 874 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 875 : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 876 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 .fc21.x86_64 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 877 .fc21.x86_64 (without KMP support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscrip tion- manage Resolvin Depend encies --> Running transact check --> Package mlnx- ofed- all.noarc NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 878 0.4 . 9 - rds- devel.x8 6_64 0 : 2.0 . 7 - 1.12 rds- tools.x8 6_64 0 : 2.0 . 7 - 1.12 sdpnets tat.x86_ 64 0 : 1.60 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 879 Software In niBand Drivers. 7. Create an apt-get repository con guration le called "/etc/apt/sources.list.d/mlnx_ofed.list" with the following content: Requirement Description Platforms A server platform with a ConnectX®-6 (IB, EN) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 880 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 881 If the driver detects unsupported cards on the system, it will abort the installation procedure. To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 882 The regular installation can be performed and weak- updates mechanism will create symbolic links NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 883 In such cases, the installer will recommend running the following command to update the initramfs: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 884 MLNX_OFED installation. Therefore, using "mlnx_add_ke rnel_support. sh" is irrelevant on Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 885 Mellano x, OEM, OFED, or Distribu tion IB package s will be removed images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- placehol der- broken- link- locale- en_us- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 886 Created /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64- ext.tgz ./mlnxofedins tall --h host1# mount o ro,loop MLNX_OFED_ LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 887 MLNX_OFED_ LINUX, do not reinstall them. Starting MLNX_OFED_ LINUX-x.x.x installation ....Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 888 -- without-fw- update -- force For example: ./mlnxofedins tall --without- dkms --add- kernel- support -- kernel 3.13.0- 85-generic -- without-fw- update -- force Note that the path to kernel NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 889 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 890 Device #1: ---- ------ Device Type: ConnectX-X Part Number: MCXXXX-XXX PSID: MT_<version> PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 891 Host Support Socket Direct Cards Installing OFED will replace the RDMA stack and remove existing 3rd party RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 892 RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 893 Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 894 Not applicable to this system con guration. This can occur when the required hardware is not present on the system Prerequisites are not met. For example, missing the required software installed or NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 895 /lib/modules/ `uname - r`/updates on SLES and Fedora Distributions /lib/modules/ `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 896 OFED ISO image Note: If an adapter's Flash was originally programmed with an Expansion ROM image, automatic rmware update will also burn an Expansion ROM image. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 897 /mnt 1. wget 72.3 . 194.0 : 80 ... connected. ]]></ac:plain- http: HTTP request text-body> //www.m sent, awaiting </ac:structured ellanox.c response... 200 OK -macro> om/dow <ac:structured- nloads/o macro NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 898 <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 899 -- body></ac:structured- import macro> RPM- GPG- KEY- Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 900 - Signing Key v2) <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 901 <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> wiki-markup" name=MLNX_OFED ac:schema- Repository version="1" baseurl= ac:macro- le: ///<path to id="bf78c405 extracted MLNX_OFED -d0c6-41cc- package>/RPMS b66f- enabled= 1 0fc3164758b gpgkey= 0"><ac:plain- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 902 Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmforg e RHEL NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 903 -- make- <option al -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 904 See log /tmp/ml nx_iso.4 120_log s/mlnx_ ofed_iso . 4120 .log Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . Please wait... Creating metadat a-rpms NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 905 G: that contains unsigne d rpms, therefor e, you need to disable gpgchec WARNIN G: by setting 'gpgche ck=0' in reposito ry conf Created /tmp/M LNX_OF ED_LIN UX- 5.2 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 906 0 text-body><! [CDATA[ 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 907 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with support) mlnx- ofed-all- user- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 908 (with support) mlnx- ofed- basic- user- only.noa rch : MLNX_ OFED basic- user- only installer package (User Space package s only) mlnx- ofed- blue eld .noarch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 909 (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- dpdk- upstrea libs.noar ch : MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 910 OFED dpdk- upstrea m-libs- user- only installer package (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 911 Space package s only) mlnx- ofed- guest.n oarch : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : MLNX_ OFED guest- user- only installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 912 : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc MLNX_ OFED hypervis installer package NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 913 (User Space package s only) mlnx- ofed- kernel- only.noa rch : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ OFED installer package (with NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 914 : MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : MLNX_ OFED vma- user- only NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 915 : MLNX_ OFED vma- vpi- user- only installer package (User Space package s only Installs all available mlnx-ofed-all packages in MLNX_OFED mlnx-ofed- Installs basic packages basic required for running NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 916 Installs packages kernel-only required for a non- default kernel mlnx-ofed-all- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED all installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 917 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED hpc installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 918 : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 919 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-eth- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-eth installer package for kernel 3.17 . 4 .fc21.x86_64 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 920 .fc21.x86_64 (without KMP support) 1. yum install mlnx- ofed-all Loaded plugins: langpac product -id, subscrip tion- manage Resolvin Depend encies --> Running transact check --> Package mlnx- ofed- all.noarc NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 921 0.4 . 9 - rds- devel.x8 6_64 0 : 2.0 . 7 - 1.12 rds- tools.x8 6_64 0 : 2.0 . 7 - 1.12 sdpnets tat.x86_ 64 0 : 1.60 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 922 >/DEBS ./ 8. Download and install NVIDIA's Technologies GPG-KEY. Requirement Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw- ConnectX6) installed Required Disk Space for Installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 923 <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> <ac:plain-text-body><! [CDATA[86:00.1 Network controller NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 924 To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body> ac:name="un </ac:structured- migrated- macro> wiki-markup" ac:schema- version="1" ac:macro- id="204fa238 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 925 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. If you regenerate kernel modules for a custom NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 926 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 927 Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program will create MLNX_ NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 928 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log Building OFED RPMs. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 929 /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED_ LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 930 Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe dinstall -- force NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 931 -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their default location. In case your machine has NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 932 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will occur and one of the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 933 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 934 Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils iproute2 (rdma tool) - installed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 935 '- without fw- update' The rmware version of NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 936 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) To obtain rmware for this device, please contact your HW vendor." NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 937 Failed to start the mst driver Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 938 RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 939 In case your machine has an unsupported network adapter device, no rmware update will occur and the error message below will be printed. "The rmware for this device is not NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 940 M-GPG- ted-wiki-markup" KEY- ac:schema- Mellano version="1" ac:macro- - 2018 id=" 99241e- 01 25 abe2-4bb6-abc6- 13 : 52 : a63afe0f4f7c"> 30 - <ac:plain-text- http: body><! //www.m [CDATA[Length: ellanox.c NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 941 ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra ted-wiki-markup" ac:schema- version="1" ac:macro- id="6dc3c0e3- a747-4d43-8afe- 4185f107057a"> <ac:plain-text- body><![CDATA[ 2018 01 25 13 : 52 : 30 ( 247 MB/s) - ? NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 942 HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG 0x6224 C050 : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 943 >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- 7ba6- 4258- 87b5- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 944 MLNX_OFED id="bf78c405 package>/RPMS -d0c6-41cc- enabled= 1 b66f- gpgkey= 0fc3164758b le: ///<path to the 0"><ac:plain- downloaded key RPM- text-body><! GPG-KEY-Mellanox> [CDATA[ gpgcheck= 1 1. yum repolist Loaded plugins: product -id, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 945 MLNX_ OFED Reposit ory 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , repolist: 8 , 351 1. mount o ro,loop NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 946 $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/c on uen ce/place holder/e rror- i18nkey- editor- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 947 Building MLNX_ OFED_LI RPMS . Please wait... Creating metadat a-rpms for 3.10 . 0 - 957.21 . .el7.x86 _64 ... WARNIN G: If you going to con gur NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 948 Created /tmp/M LNX_OF ED_LIN UX- 5.2 - 0.5 . 5.0 -rhel7. 6 x86_64- ext.tgz 1. cd /tmp/ <ac:structure [mlnx_ofed] d-macro ]]></ac:plain-text- ac:name="un body></ac:structured- migrated- macro> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 949 This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage r to register. repo id repo name status NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 950 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with support) mlnx- ofed-all- user- only.noa rch : MLNX_ OFED all-user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 951 : MLNX_ OFED basic- user- only installer package (User Space package s only) mlnx- ofed- blue eld .noarch : MLNX_ OFED blue eld installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 952 : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- dpdk- upstrea libs.noar ch : MLNX_ OFED dpdk- upstrea m-libs installer package (with support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 953 : MLNX_ OFED dpdk- user- only installer package (User Space package s only) mlnx- ofed- eth- only- user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 954 (with support) mlnx- ofed- guest- user- only.noa rch : MLNX_ OFED guest- user- only installer package (User Space package s only) mlnx- ofed- hpc.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 955 OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc MLNX_ OFED hypervis installer package (with support) mlnx- ofed- hypervis or-user- only.noa rch : NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 956 : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ OFED installer package (with support) mlnx- ofed- vma- eth.noar ch : MLNX_ OFED vma-eth NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 957 (User Space package s only) mlnx- ofed- vma- user- only.noa rch : MLNX_ OFED vma- user- only installer package (User Space package s only) mlnx- ofed- vma- vpi.noar NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 958 Installs basic packages mlnx-ofed- required for running basic NVIDIA cards mlnx-ofed- Installs packages guest required by guest OS mlnx-ofed- Installs packages required for HPC Installs packages mlnx-ofed- required by hypervisor hypervisor NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 959 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- basic- 3.17 . 4 .fc21.x86_64. noarch : MLNX_OFED basic installer package for kernel 3.17 . 4 .fc21.x86_64 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 960 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED hypervisor installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma- 3.17 . 4 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 961 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- hypervisor- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 962 3.17 . 4 .fc21.x86_64 (without KMP support) mlnx-ofed- vma-vpi- 3.17 . 4 301 .fc21.x86_64. noarch : MLNX_OFED vma-vpi installer package for kernel 3.17 . 4 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 963 --> Running transact check --> Package mlnx- ofed- all.noarc h 0 : 3.1 0.1 . 2 will be installed -> Processi Depend ency: kmod- isert = 1.0 - NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 964 2.0 . 7 - 1.12 sdpnets tat.x86_ 64 0 : 1.60 - srptools. x86_64 0 : 1.0 . 2 - 12 Complet Installing MLNX_OFED using the "YUM" tool does not NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 965 | sudo apt-key add - 9. Verify that the key was successfully imported. Requirement Description A server platform with a ConnectX®-6 (IB, EN) Platforms ( rmware: fw- ConnectX6) installed NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 966 0c1e2667e406"> <ac:plain-text-body><! [CDATA[86:00.0 Network controller [0207]: Mellanox Technologies MT28908A0 Family ]]></ac:plain-text- body></ac:structured- macro> Subsystem: Mellanox Technologies Device 0014 <ac:structured-macro ac:name="unmigrated- wiki-markup" ac:schema-version="1" ac:macro- id="93935fa9-6364- 496b-825e- e6057c30b53a"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 967 To avoid this, make sure to add --skip- unsupported- devices- check ag during installation. <ac:structure ./mnt/mlnxofedinstall ]]></ac:plain-text- d-macro [OPTIONS] body> ac:name="un </ac:structured- migrated- macro> wiki-markup" ac:schema- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 968 The regular installation can be performed and weak- updates mechanism will create symbolic links to the MLNX_OFED kernel modules. If you regenerate NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 969 In such cases, the installer will recommend running the following command to update the initramfs: dracut -f On Ubuntu and Debian distributions drivers installation use Dynamic NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 970 Ubuntu and Debian distributions. 1. ./MLNX_ OFED_LI NUX-x.x- x-rhel7. x86_64/ mlnx_ad d_kernel _suppor t.sh -m /tmp/M LNX_OF ED_LIN UX-x.x- x-rhel7. x86_64/ --make- Note: This program NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 971 2.png > See log /tmp/ml nx_ofed _iso. 21642 .log NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 972 LINUX<ver> <OS label> <CPU arch>.iso /mnt /mnt/mlnxofe dinstall Logs dir: /tmp/MLNX_ OFED_LINUX- x.x-x.logs This program will install the MLNX_OFED_ LINUX package on your machine. Note that all other Mellanox, NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 973 ..Installation nished successfully. Attempting to perform Firmware update... Querying Mellanox devices rmware ... unattended installation, use the -- force installation option while running the MLNX_OFED installation script: /mnt/mlnxofe NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 974 3.13.0- 85-generic -- without-fw- update -- force Note that the path to kernel sources (- kernel- sources) should be added if the sources are not in their default location. NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 975 PCI Device Name: 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: Up to date In case your machine has unsupported network adapter device, no rmware update will NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 976 0b:00.0 Base MAC: 0000e41d2d 5cf810 Versions: Current Available FW XX.XX.XXXX Status: No matching image found Error message #2: The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 977 RDMA connectors. Software Most of MLNX_OFED packages are installed under the "/usr" directory except for the following packages which are installed under the "/opt" directory: fca and ibutils NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 978 The rmware of existing network adapter devices will be updated if the following two conditions are ful lled: installation script is run in default mode; that is, without the option '- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 979 "The rmware for this device is not distributed inside NVIDIA driver: 0000:01:00.0 (PSID: IBM2150110033) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 980 Prerequisites are not met. For example, missing the required software installed or the hardware is not con gured correctly Failed to start the mst driver Software Most of MLNX_OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 981 `uname - r`/extra/mln x-ofa_kernel on RHEL and other RedHat like Distributions /lib/modules/ `uname - r`/updates/d kms/ on Ubuntu The rmware of existing network adapter devices will be updated if NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 982 ROM image, automatic rmware update will also burn an Expansion ROM image. In case your machine has an unsupported network adapter device, no rmware update will occur and the NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 983 //www.m sent, awaiting </ac:structured ellanox.c response... 200 OK -macro> om/dow <ac:structured- nloads/o macro fed/RP ac:name="unmigra M-GPG- ted-wiki-markup" KEY- ac:schema- Mellano version="1" ac:macro- - 2018 id=" 99241e- 01 25 abe2-4bb6-abc6- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 984 <ac:plain-text- body><![CDATA[ 100 % [============= ============== ============== ========>] 1 , 354 --.-K/s in 0s ]]></ac:plain-text- body> </ac:structured- macro> <ac:structured- macro ac:name="unmigra ted-wiki-markup" ac:schema- version="1" ac:macro- id="6dc3c0e3- a747-4d43-8afe- 4185f107057a"> NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 985 Mellano warning: rpmts_ HdrFro mFdno: Header DSA/SH Signatur e, key ID 6224c0 NOKEY Retrievi ng key from ///repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano Importin g GPG NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 986 <suppor t@mella nox.com >" From : /repos/ MLNX_ OFED/< MLNX_ OFED le>/RP M-GPG- KEY- Mellano <ac:stru ctured- macro ac:name ="unmig rated- wiki- markup" ac:sche version= "1" ac:macr id="110 be138- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 987 ///<path to ac:macro- extracted MLNX_OFED id="bf78c405 package>/RPMS -d0c6-41cc- enabled= 1 b66f- gpgkey= 0fc3164758b le: ///<path to the 0"><ac:plain- downloaded key RPM- text-body><! GPG-KEY-Mellanox> [CDATA[ gpgcheck= 1 1. yum repolist NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 988 You can subscrip tion- manage r to register. repo id repo name status mlnx_of MLNX_ OFED Reposit ory 108 rpmforg e RHEL 6Server RPMfor ge.net - dag 4 , NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 989 -- kmp> -k $(unam e -r) -m /mnt/ Note: This program will create MLNX_ OFED_LI TGZ for rhel7. 6 under /tmp director images/ plugins/ servlet/c on uen ce/place NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 990 Checkin g if all needed package s are installed Building MLNX_ OFED_LI RPMS . Please wait... Creating metadat a-rpms for 3.10 . 0 - 957.21 . .el7.x86 _64 ... NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 991 WARNIN G: by setting 'gpgche ck=0' in reposito ry conf Created /tmp/M LNX_OF ED_LIN UX- 5.2 - 0.5 . 5.0 -rhel7. 6 x86_64- ext.tgz NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 992 1. yum repolist Loaded plugins: product -id, security, subscrip tion- manage This system is not register ed to Red Hat Subscri ption Manage ment. You can subscrip tion- manage NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 993 4 , repolist: 8 , 351 1. yum search mlnx- ofed- mlnx- ofed- all.noarc MLNX_ OFED all installer package (with support) mlnx- ofed-all- user- only.noa rch : MLNX_ OFED NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 994 : MLNX_ OFED basic- user- only installer package (User Space package s only) mlnx- ofed- blue eld .noarch : MLNX_ OFED blue eld installer NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 995 (User Space package s only) mlnx- ofed- dpdk.no arch : MLNX_ OFED dpdk installer package (with support) mlnx- ofed- dpdk- upstrea libs.noar ch : MLNX_ OFED dpdk- upstrea m-libs NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 996 (User Space package s only) mlnx- ofed- dpdk- user- only.noa rch : MLNX_ OFED dpdk- user- only installer package (User Space package s only) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 997 : MLNX_ OFED guest installer package (with support) mlnx- ofed- guest- user- only.noa rch : MLNX_ OFED guest- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 998 : MLNX_ OFED hpc- user- only installer package (User Space package s only) mlnx- ofed- hypervis or.noarc MLNX_ OFED hypervis installer package (with support) NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 999 : MLNX_ OFED kernel- only installer package (with support) mlnx- ofed- vma.noa rch : MLNX_ OFED installer package (with support) mlnx- ofed- NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...
  • Page 1000 MLNX_ OFED vma- eth- user- only installer package (User Space package s only) mlnx- ofed- vma- user- only.noa rch : MLNX_ OFED vma- user- only installer package (User Space NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP Spec 3.0 User Manual...

This manual is also suitable for:

Mcx653436a-hdab

Table of Contents