Nvidia DGX-2 SYSTEM User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Computer Hardware
  5. DGX-2 SYSTEM
  6. User manual

Nvidia DGX-2 SYSTEM User Manual

Hide thumbs Also See for DGX-2 SYSTEM:
Table of Contents

Advertisement

Quick Links

DGX-2 SYSTEM
DU-09130-001_v03 | November 2018
User Guide

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the DGX-2 SYSTEM and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia DGX-2 SYSTEM

Summary of Contents for Nvidia DGX-2 SYSTEM

  • Page 1 DGX-2 SYSTEM DU-09130-001_v03 | November 2018 User Guide...

  • Page 2: Table Of Contents

    TABLE OF CONTENTS Introduction to the NVIDIA DGX-2 System ......5 About this Document ................. 6 Hardware Overview .................. 6 Network Ports ..................11 Recommended Ports to Use for External Storage ..........12 DGX OS Software ................... 13 Additional Documentation ................. 13 Customer Support ..................
  • Page 3 Using DGX-2 System in KVM Mode ........66 Overview ................... 66 Preliminary Setup - Converting the DGX-2 System to a DGX-2 KVM Host ....69 Launching a Guest GPU VM Instance ............70 Stopping, Restarting, and Deleting a Guest GPU VM ........72 Connecting to Your Guest GPU VM .............
  • Page 4 C.10. Vietnam ..................103 C.11. South Korea ................... 104 C.12. Taiwan ..................107 DGX-2 System User Guide...

  • Page 5: Introduction To The Nvidia Dgx-2 System

    The NVIDIA® DGX-2™ System is the world’s first two-petaFLOPS system that engages 16 fully interconnected GPUs for accelerated deep learning performance. The DGX-2 System is powered by NVIDIA® DGX™ software stack and an architecture designed for Deep Learning, and High-Performance Computing and analytics.

  • Page 6: About This Document

    Introduction to the NVIDIA DGX-2 System ABOUT THIS DOCUMENT This document is for users and administrators of the DGX-2 System. It is organized as follows: Chapters 1-4: Overview of the DGX-2 System, including basic first-time setup and  operation Chapters 5-6: Network and storage configuration instructions.
  • Page 7 Introduction to the NVIDIA DGX-2 System Component Description 8x NVIDIA® Tesla V100 6x NVSwitches 512GB total HBM2 memory NVSwitch 2.4 TB/s bi-section bandwidth Network (cluster) EDR Infiniband or 100 GbE (1600 Gb/s total bi-directional bandwidth) Dual Intel Xeon Platinum 8168, 2.7 GHz, 24-cores System Memory 1.5 TB...
  • Page 8 Additional SSDs available for purchase to expand to 16. Motherboard tray fans Front console board: USB 3.0 (2x) VGA (1x) Power and ID buttons: Top: Power button and LED Press to turn the DGX-2 System on or off. DGX-2 System User Guide...
  • Page 9 Introduction to the NVIDIA DGX-2 System Description Green steady: Power is On Red steady: Power is Off Red blinking: Status warning/error Bottom: ID button Press to cause an LED on the back of the unit to flash as an identifier during servicing.
  • Page 10 Introduction to the NVIDIA DGX-2 System With EMI Shield Removed Description NVIDIA NVLink™ plane card 1.2.8 Motherboard Tray Ports and Controls Description (Optional) High profile PCI card slot (for network storage) (Default) QSFP28 network ports (for network storage) Left side port designation: enp134s0f0...

  • Page 11: Network Ports

    Introduction to the NVIDIA DGX-2 System Description IPMI port (for out-of-band management (BMC)) VGA port Serial port (DB-9) System ID LED Blinks blue when ID button is pressed from the front of the unit as an aid in identifying the unit needing servicing...

  • Page 12: Recommended Ports To Use For External Storage

    Introduction to the NVIDIA DGX-2 System Connectivity Uses Number of Port Type Cable Type Ports (Right): (QSFP28 to SFP28 enp134s0f1 or SFP+) ConnectX-5 Clustering InfiniBand EDR InfiniBand mode Storage QSFP28 Ethernet 100GbE Ethernet mode RECOMMENDED PORTS TO USE FOR EXTERNAL...

  • Page 13: Dgx Os Software

     NVIDIA System Management (NVSM) ● Provides active health monitoring and system alerts for NVIDIA DGX nodes in a data center. It also provides simple commands for checking the health of the DGX-2 SYSTEM from the command line. Data Center GPU Management (DCGM) ●...

  • Page 14: Customer Support

    Contact NVIDIA Enterprise Support for assistance in reporting, troubleshooting, or diagnosing problems with your DGX-2 System . Also contact NVIDIA Enterprise Support for assistance in installing or moving the DGX-2 System. You can contact NVIDIA Enterprise Support in the following ways.

  • Page 15: Connecting To The Dgx-2 Console

    DGX OS Server software installs Docker CE which uses the 172.17.xx.xx subnet by default for Docker containers. If the DGX-2 System is on the same subnet, you will not be able to establish a network connection to the DGX-2 System.

  • Page 16: Direct Connection

    Connecting to the DGX-2 Console DIRECT CONNECTION At either the front or the back of the DGX-2 System, connect a display to the VGA connector, and a keyboard to any of the USB ports. DGX-2 Server Front DGX-2 Server Back...

  • Page 17: Remote Connection Through The Bmc

    “<system-username>”, in which case the credentials are Username: <administrator-username> Password: <new-bmc-password> Make sure you have connected the BMC port on the DGX-2 System to your LAN. Open a browser within your LAN and go to: https://<ipmi-ip-address>/ Make sure popups are allowed for the BMC address.
  • Page 18 From the left-side navigation menu, click Remote Control. The Remote Control page allows you to open a virtual Keyboard/Video/Mouse (KVM) on the DGX-2 System, as if you were using a physical monitor and keyboard connected to the front of the system.

  • Page 19: Ssh Connection

    Connecting to the DGX-2 Console SSH CONNECTION You can also establish an SSH connection to the DGX-2 System through the network port. See the section Network Ports to identify the port to use, and the section Configuring Static IP Addresses for the Network Ports if you need to configure a static IP address.

  • Page 20: Setting Up The Dgx-2 System

    SETTING UP THE DGX-2 SYSTEM While NVIDIA service personnel will install the DGX-2 System at the site and perform the first boot setup, the first boot setup instructions are provided here for reference and to support any re-imaging of the server.
  • Page 21 Setting Up the DGX-2 System Using the Remote BMC ● The system will take a few minutes to boot. You are presented with end user license agreements (EULAs) for the NVIDIA software. DGX-2 System User Guide...
  • Page 22 Create a user account with your name, username, and password. ● You will need these credentials to log in to the DGX-2 System as well as to log in to the BMC remotely. When logging in to the BMC, enter your username for both the User ID as well as the password.
  • Page 23 Setting Up the DGX-2 System sudo apt update Upgrade to the latest version. sudo apt full-upgrade  Note: RAID 1 Rebuild in Progress - When the system is booted after restoring the image, software RAID begins the process of rebuilding the RAID 1 array - creating a mirror of (or resynchronizing) the drive containing the software.

  • Page 24: Quick Start Instructions

    REGISTRATION Be sure to register your DGX-2 System with NVIDIA as soon as you receive your purchase confirmation e-mail. Registration enables your hardware warranty and allows you to set up an NVIDIA GPU Cloud for DGX account.

  • Page 25: Installation And Configuration

    NVIDIA GPU Cloud (NGC) provides simple access to GPU-optimized software tools for deep learning and high-performance computing (HPC) that take full advantage of NVIDIA GPUs. An NGC account grants you access to these tools as well as the ability to set up a private registry to manage your customized tools.

  • Page 26: Verifying Basic Functionality

    Select Registry from the left side menu. Select a CUDA container tag. Click the cuda repository (under the nvidia registry space). In the Tag section, scroll down to find the latest ‘-runtime’ version. For example, ‘10.0-runtime’.
  • Page 27 Quick Start Instructions Verify connection to the NVIDIA repository and that the NVIDIA Driver is installed. sudo docker container run --runtime=nvidia --rm nvcr.io/nvidia/cuda:<cuda-tag-obtained-from-previous- section> nvidia-smi Docker pulls the nvidia/cuda container image layer by layer, then runs nvidia-smi. When completed, the output should show the NVIDIA Driver version and a description of each installed GPU.

  • Page 28: Network Configuration

     CONFIGURING NETWORK PROXIES If your network requires use of a proxy server, you will need to set up configuration files to ensure the DGX-2 System communicates through the proxy. 5.2.1 For the OS and Most Applications Edit the file /etc/environment and add the following proxy addresses to the file, below the PATH line.

  • Page 29: Configuring Docker Ip Addresses

    Docker, https://docs.docker.com/engine/admin/systemd/#http-proxy CONFIGURING DOCKER IP ADDRESSES To ensure that the DGX-2 System can access the network interfaces for Docker containers, Docker should be configured to use a subnet distinct from other network resources used by the DGX-2 System.
  • Page 30 Docker IP address range, then no changes are needed and you can skip this section. However, if your network uses the addresses within this range for the DGX-2 System, you should change the default Docker network addresses.

  • Page 31: Opening Ports

    If port 443 is proxied through a corporate firewall, then WebSocket protocol traffic must be supported CONNECTIVITY REQUIREMENTS To run NVIDIA NGC containers from the NGC container registry, your network must be able to access the following URLs: http://archive.ubuntu.com/ubuntu/ ...

  • Page 32: Configuring Static Ip Address For The Bmc

    This section describes how to set a static IP address for the BMC from the Ubuntu command line.  Note: If you cannot access the DGX-2 System remotely, then connect a display (1440x900 or lower resolution) and keyboard directly to the DGX-2 System. To view the current settings, enter the following command.
  • Page 33 System BIOS This section describes how to set a static IP address for the BMC when you cannot access the DGX-2 System remotely. This process involves setting the BMC IP address during system boot. Connect a keyboard and display (1440 x 900 maximum resolution) to the DGX-2 System, then turn on the DGX-2 System.
  • Page 34 Scroll to Configuration Address Source and press Enter, then at the Configuration Address source pop-up, select Static and then press Enter. Set the addresses for the Station IP address, Subnet mask, and Router IP address as needed by performing the following for each: DGX-2 System User Guide...
  • Page 35 IPv6 fields. Log into the BMC, then click Settings->Network Settings->Network IP Settings. Clear the Enable IPv4 DHCP check box, then enter the appropriate values for the IPv4 Address, IPv4 Subnet, and IPv4 Gateway fields . DGX-2 System User Guide...

  • Page 36: Configuring Static Ip Addresses For The Network Ports

    CONFIGURING STATIC IP ADDRESSES FOR THE NETWORK PORTS During the initial boot setup process for the DGX-2 System, you had an opportunity to configure static IP addresses for a single network interface. If you did not set this up at that time, you can configure the static IP addresses from the Ubuntu command line using the following instructions.
  • Page 37 1. When finished with your edits, press ESC to switch to command mode, then save the file to the disk and exit the editor. Apply the changes. $ sudo netplan apply DGX-2 System User Guide...

  • Page 38: Switching Between Infiniband And Ethernet

    SWITCHING BETWEEN INFINIBAND AND ETHERNET The NVIDIA DGX-2 System is equipped with eight QSFP28 network ports on the I/O board, typically used for cluster communications. By default these are configured as InfiniBand ports, but you have the option to convert these to Ethernet ports.
  • Page 39 /dev/mst/mt4119_pciconf6 - PCI configuration cycles access. domain:bus:dev.fn=0000:e1:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 /dev/mst/mt4119_pciconf7 - PCI configuration cycles access. domain:bus:dev.fn=0000:e6:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 /dev/mst/mt4119_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:86:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 DGX-2 System User Guide...
  • Page 40 Map the Device bus numbers from your output to the device name from the mst status output on your system. For example, this example output shows that the device name for bus bd is /dev/mst/mt4119_pciconf5. You will need the device name when changing the configuration. DGX-2 System User Guide...
  • Page 41 ETH (1) Device #4: Device type: ConnectX4 Device: 0000:e1:00.0 LINK_TYPE_P1 ETH (1) Device #5: Device type: ConnectX4 Device: 0000:35:00.0 LINK_TYPE_P1 ETH (1) Device #6: Device type: ConnectX4 Device: 0000:5d:00.0 LINK_TYPE_P1 ETH (1) Device #7: Device type: ConnectX4 DGX-2 System User Guide...
  • Page 42 Device #1: Device type: ConnectX4 Device: 0000:bd:00.0 LINK_TYPE_P1 IB(1) Device #2: Device type: ConnectX4 Device: 0000:b8:00.0 LINK_TYPE_P1 IB(1) Device #3: Device type: ConnectX4 Device: 0000:3a:00.0 LINK_TYPE_P1 IB(1) Device #4: Device type: ConnectX4 Device: 0000:e1:00.0 LINK_TYPE_P1 IB(1) DGX-2 System User Guide...
  • Page 43 Device type: ConnectX4 Device: 0000:5d:00.0 LINK_TYPE_P1 IB(1) Device #7: Device type: ConnectX4 Device: 0000:e6:00.0 LINK_TYPE_P1 IB(1) Device #8: Device type: ConnectX4 Device: 0000:58:00.0 LINK_TYPE_P1 IB(1) Device #9: Device type: ConnectX5 Device: 0000:86:00.0 LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 ETH(2) DGX-2 System User Guide...

  • Page 44: Configuring Storage - Nfs Mount And Cache

    SSDs are intended for application caching, so you must set up your own NFS storage for long term data storage. The following instructions describe how to mount the NFS onto the DGX-2 System, and how to cache the NFS using the DGX-2 SSDs for improved performance.
  • Page 45 /mnt is an example mount point. Verify caching is enabled. cat /proc/fs/nfsfs/volumes Look for the text FSC=yes in the output. The NFS will be mounted and cached on the DGX-2 System automatically upon subsequent reboot cycles. DGX-2 System User Guide...

  • Page 46: Restoring The Dgx-2 Software Image

    DGX-2 software image to its original factory condition from a pristine copy of the image. The process for restoring the DGX-2 software image is as follows: Obtain an ISO file that contains the image from NVIDIA Enterprise Support as explained in Obtaining the DGX-2 Software ISO Image and Checksum File Restore the DGX-2 software image from this file either remotely through the BMC or locally from a bootable USB flash drive.

  • Page 47: Obtaining The Dgx-2 Software Iso Image And Checksum File

    To ensure that you restore the latest available version of the DGX-2 software image, obtain the current ISO image file from NVIDIA Enterprise Support. A checksum file is provided for the image to enable you to verify the bootable installation medium that you create from the image file.

  • Page 48: Creating A Bootable Installation Medium

    Retaining the RAID Partition While Installing the OS more information. Press Enter. The DGX-2 System will reboot from ISO image and proceed to install the image. This can take approximately 15 minutes.  Note: The Mellanox InfiniBand driver installation may take up to 10 minutes.
  • Page 49 Obtain the device name of the USB flash drive by running the fdisk command. sudo fdisk -l You can identify the USB flash drive from its size, which is much smaller than the size of the SSDs in the DGX-2 System. Create a mount point. Example: sudo mkdir /media/usb Mount the USB flash drive.
  • Page 50 Under File system, select FAT32. Click Start. Because the image is a hybrid ISO file, you are prompted to select whether to write the image in ISO Image (file copy) mode or DD Image (disk image) DGX-2 System User Guide...

  • Page 51: Re-Imaging The System From A Usb Flash Drive

    Plug the USB flash drive containing the OS image into the DGX-2 System. Connect a monitor and keyboard directly to the DGX-2 System. Boot the system and press F11 when the NVIDIA logo appears to get to the boot menu.

  • Page 52: Retaining The Raid Partition While Installing The Os

    OS disk as well as the RAID disks. Since the RAID array on the DGX-2 System is intended to be used as a cache and not for long-term data storage, this should not be disruptive. However, if you are an advanced...
  • Page 53 Restoring the DGX-2 Software Image systemctl start cachefilesd These changes are preserved across system reboots. DGX-2 System User Guide...

  • Page 54: Updating The Dgx Os Software

    UPDATING THE DGX OS SOFTWARE You must register your DGX-2 System in order to receive email notification whenever a new software update is available. These instructions explain how to update the DGX-2 software through an internet connection to the NVIDIA public repository. The process updates a DGX-2 System image to the latest QA’d versions of the entire DGX-2 software stack, including the...

  • Page 55: Update Instructions

    Other questions will depend on what other packages were installed before the update and how those packages interact with the update. Typically, you can accept the default option when prompted. Reboot the system. DGX-2 System User Guide...

  • Page 56: Updating Firmware

    UPDATING FIRMWARE This section provides instructions for updating firmware for the NVIDIA® DGX server BIOS and BMC using a Docker container. GENERAL FIRMWARE UPDATE GUIDELINES Before updating the firmware, do the following to prevent corrupting the firmware  due to a system crash or disruption to the update process.

  • Page 57: Obtaining The Firmware Update Container

    Updating Firmware OBTAINING THE FIRMWARE UPDATE CONTAINER Obtain the container tarball from the NVIDIA Enterprise Support portal and transfer it to the DGX-2 System. The container is provided in the tarball <image-name>.tar.gz From the directory where you copied the tarball file, enter the following command.

  • Page 58: Updating The Firmware

    The command will scan the specified firmware components and update any that are down-level. See the section for an explanation of the [ ] and [ Additional Options -e auto=1 options. DGX-2 System User Guide...
  • Page 59 The container will scan the components and then prompt for confirmation before starting the update. Following components will be updated with new firmware version: IMPORTANT: Firmware update is disruptive and may require system reboot. Stop system activities before performing the update. DGX-2 System User Guide...

  • Page 60: Additional Options

    To update the firmware regardless of whether it is down-level, use the option as follows. sudo docker run --rm --privileged -ti -v /:/hostfs <image-name> update_fw -f <target> The container will not check the onboard versions against the manifest. DGX-2 System User Guide...

  • Page 61: Command Summary

    Remove the container and image from the DGX server when it is no longer needed. To remove the container and image, enter the following: $ sudo docker rmi -f <image-name> In this case, specify only the container repository and not the tag. DGX-2 System User Guide...

  • Page 62: Using The Bmc

    It monitors system sensors and other parameters. CONNECTING TO THE BMC Make sure you have connected the BMC port on the DGX-2 System to your LAN. Open a browser within your LAN and go to: https://<ipmi-ip-address>/ Make sure popups are allowed for the BMC address.

  • Page 63: Overview Of Bmc Controls

    Using the BMC Log in. OVERVIEW OF BMC CONTROLS The left-side navigation menu on the BMC dashboard contains the primary controls. DGX-2 System User Guide...
  • Page 64 Provides, chassis, board, and product information for each FRU device. 10.2.4 Logs & Reports Lets you view, and if applicable, download and erase, the IPMI even log, and system, audit, video and POST Code logs. 10.2.5 Settings Configure the following settings DGX-2 System User Guide...
  • Page 65 10.2.8 Maintenance  Note: While you can update the BMC firmware from this page, NVIDIA recommends using the NVIDIA Firmware Update Container instead (see section Updating Firmware for instructions). If you need to update from this page, click Dual Firmware Update and then select the Both Images from the Images to be Updated menu.

  • Page 66: Using Dgx-2 System In Kvm Mode

    GPU multi-tenancy. Since the KVM Hypervisor is part of the Linux kernel on the DGX-2 System, it contains the system-level components necessary to support multi-tenancy on the DGX-2 System, such as a memory manager, process scheduler, input/output (I/O) stack, device drivers, security manager, and a network stack.
  • Page 67  Note: Unlike the-bare metal DGX-2 system or the KVM host OS, the guest VM OS is configured for English-only, and cannot be configured for other languages. Running NVIDIA containers on the VM is just like running containers on a DGX-2 bare metal system with DGX OS software installed.
  • Page 68 DGX KVM package. nvidia-vm simplifies the process of creating guest GPU VMs and allocating resources. In addition, you can use nvidia-vm to modify default options to suit your needs for the VM and manage VM images installed on the system.

  • Page 69: Preliminary Setup - Converting The Dgx-2 System To A Dgx-2 Kvm Host

    Your DGX-2 System is now ready for you to create VMs. Restoring to Bare Metal After setting up the DGX-2 System as a KVM host, you can restore the server to a bare metal system. CAUTION: Reverting the server back to a bare metal system destroys all guest GPU VMs that were created as well as any data.

  • Page 70: Launching A Guest Gpu Vm Instance

    Reboot the system. sudo reboot LAUNCHING A GUEST GPU VM INSTANCE To create and delete guest GPU VMs, use the NVIDIA utility nvidia-vm which simplifies the complex process of these tasks. For other VM management tasks, use virsh where indicated in these instructions.
  • Page 71 Acceptable values: 1, 2, 4, 8, 16 --gpu-index For the purposes of the KVM, GPUs on the DGX-2 System are distinguished by a zero-based, sequential index. gpu_index specifies the starting index value for the group of sequentially indexed GPUs to be assigned to the VM.

  • Page 72: Stopping, Restarting, And Deleting Aguest Gpu Vm

    This command creates a VM with 2 GPUs, starting with index 8, named mydgx2vm- 2g8-9. Specifying an image  nvidia-vm create --gpu-count 2 --gpu-index 2 --image dgx-kvm- image-4-0-1 This command creates a VM with 2 GPUs, starting with index 2, named jsmithTue1308-2g2-3, and based on the image dgx-kvm-image-4-0-1.
  • Page 73 Deleting a VM Like the process of creating a guest GPU VM, deleting a VM involves several virsh commands. For this reason, NVIDIA provides a simple way to delete a VM using nvidia-vm. Deleting a VM using nvidia-vm does the following: Stops the VM if it is running ...

  • Page 74: Connecting To Your Guest Gpu Vm

    Using DGX-2 System in KVM Mode Use nvidia-vm as explained in About nvidia-vm. Syntax nvidia-vm delete --domain <vm-domain> Command Help nvidia-vm delete --help Command Examples Deleting an individual VM  nvidia-vm delete --domain dgx2vm-labTue1308-4g12-15 Deleting all the VMs on the system ...
  • Page 75 11.5.4 Changing Login Credentials When the guest GPU VM is created, the default login credentials are nvidia/nvidia. As a security practice, use the standard Ubuntu methods to create a new user account and then delete the nvidia user account from the GPU VM. The basic commands are provided below for convenience.

  • Page 76: Managing Images

    To check available DGX KVM images, enter the following. apt-cache policy dgx-kvm-image This returns a list of images in the repository. To install a KVM image from the list, use the nvidia-vm image install command. Syntax nvidia-vm image install <vm-image>...

  • Page 77: Using The Guest Os Drives And Data Drives

    11.6.4 Uninstalling Images If you convert the DGX-2 System from a KVM OS back to the bare metal system, you need to uninstall all the dgx-kvm images that were installed. Perform the following for each installed image.
  • Page 78 KVM Host for Data Drives on the Guests. The Data drive is automatically carved, by nvidia-vm tool, out of the Storage Pool and allocated to each GPU VM as a Data Drive (/dev/vdb1) which is automatically mounted on /raid. The Data Drive size is pre- configured according to the size of the GPU VM.
  • Page 79 Since the Data Drive is created on the Host RAID-0 array, data is not intended to be persistent. Therefore, when the GPU VM is destroyed, the Data Drive is automatically deleted and data is not preserved. Using the nvidia-vm tool, a system administrator can change the default Data Drive size. 11.7.3 Storage Pool Demonstration This section shows how to view the storage pool, and how disk space is assigned to a VM from the storage pool.

  • Page 80: Updating The Software

    Using DGX-2 System in KVM Mode Connect to the guest GPU VM. $ virsh console dgx2vm-labTue1209-1g0 Connected to domain dgx2vm-labTue1209-1g0 List the virtual storage on the guest GPU VM. nvidia@dgx2vm-labTue1209-1g0:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 252:0 0 disk └─vda1 252:1...
  • Page 81 Other questions will depend on what other packages were installed before the ― update and how those packages interact with the update. If a message appears indicating that nvidia-docker.service failed to start, you can ● disregard it and continue with the next step. The service will start normally at that time.

  • Page 82: Supplemental Information

    You can use command options to customize memory allocation, OS disk size, and  number of vCPUs to assign. 11.9.2 Resource Management NVIDIA KVM optimizes resources to maximize the performance of the VM. vCPU  vCPUs are pinned to each VM to be NUMA-aware and to provide better VM performance.
  • Page 83 Creating VMs with the DGX-2 System in Degraded Mode You can still create guest GPU VMs on a DGX-2 System in degraded mode as long as you do not try to assign a failed GPU. If you attempt to create a VM with a failed GPU...
  • Page 84 Using DGX-2 System in KVM Mode after its state has been marked as ‘bad’ by the system, the VM will fail to start and an appropriate error message is returned. Restarting an existing VM after a GPU fails will result in the same failure and error message.
  • Page 85 Using DGX-2 System in KVM Mode The server must be powered off when performing the replacement. After GPU replacement and upon powering on the server, the KVM software runs a health scan to add any new GPUs to the health database.

  • Page 86: Appendix A. Installing Software On Air-Gapped Dgx-2 Systems

    Docker containers as well. A.1. Installing NVIDIA DGX-2 Software One method for updating DGX-2 software on an air-gapped DGX-2 System is to download the ISO image, copy it to removable media and then re-image the DGX-2 System from the media.

  • Page 87: Re-Imaging The System

    This process destroys all data and software customizations that you CAUTION: have made on the DGX-2 System. Be sure to back up any data that you want to preserve, and push any Docker images that you want to keep to a trusted registry.

  • Page 88: Installing Docker Containers

    Using DGX-2 System in KVM Mode A.4. Installing Docker Containers This method applies to Docker containers hosted on the NVIDIA NGC Container Registry, and requires that you have an active NGC account. On a system with internet access, log in to the NGC Container Registry by entering the following command and credentials.

  • Page 89: Appendix B. Safety

    Use only the described, regulated components specified in this guide. Use of other products I components will void the UL Listing and other regulatory approvals of the product, and may result in noncompliance with product regulations in the region(s) in which the product is sold. DGX-2 System User Guide...

  • Page 90: Safety Warnings And Cautions

    Using DGX-2 System in KVM Mode B.2. Safety Warnings and Cautions To avoid personal injury or property damage, before you begin installing the product, read, observe, and adhere to all of the following safety instructions and information. The following safety symbols may be used throughout the documentation and may be marked on the product and/or the product packaging.

  • Page 91: Intended Application Uses

    Using DGX-2 System in KVM Mode B.3. Intended Application Uses This product was evaluated as Information Technology Equipment (ITE), which may be installed in offices, schools, computer rooms, and similar commercial type locations. The suitability of this product for other product categories and environments (such as medical, industrial, residential, alarm systems, and test equipment), other than an ITE application, may require further evaluation.

  • Page 92: Electrical Precautions

    Using DGX-2 System in KVM Mode B.6. Electrical Precautions Power and Electrical Warnings Caution: The power button, indicated by the stand-by power marking, DOES NOT completely turn off the system AC power; standby power is active whenever the system is plugged in. To remove power from system, you must unplug the AC power cord from the wall outlet.

  • Page 93: System Access Warnings

    Using DGX-2 System in KVM Mode B.7. System Access Warnings Caution: To avoid personal injury or property damage, the following safety instructions apply whenever accessing the inside of the product: Turn off all peripheral devices connected to this product. ...

  • Page 94: Rack Mount Warnings

    Using DGX-2 System in KVM Mode B.8. Rack Mount Warnings Note: The following installation guidelines are required by UL for maintaining safety compliance when installing your system into a rack. The equipment rack must be anchored to an unmovable support to prevent it from tipping when a server or piece of equipment is extended from it.

  • Page 95: Electrostatic Discharge (Esd)

    NICKEL NVIDIA Bezel. The bezel’s decorative metal foam contains some nickel. The metal foam is not intended for direct and prolonged skin contact. Please use the handles to remove, attach or carry the bezel. While nickel exposure is unlikely to be a problem, you should be aware of the possibility in case you’re susceptible to nickel-related reactions.
  • Page 96 Using DGX-2 System in KVM Mode Dispose of batteries according to local ordinances and regulations. Do not attempt to recharge a battery. attempt to disassemble, puncture, or otherwise damage a battery. 更換電池警告: Cooling and Airflow Caution: Carefully route cables as directed to minimize airflow blockage and cooling problems.

  • Page 97: Appendix C. Compliance

    APPENDIX C. COMPLIANCE The NVIDIA DGX-2 is compliant with the regulations listed in this section. C.1. United States Federal Communications Commission (FCC) FCC Marking (Class A) This device complies with part 15 of the FCC Rules. Operation is subject to the following...

  • Page 98: United States / Canada

    Using DGX-2 System in KVM Mode C.2. United States / Canada cULus Listing Mark C.3. Canada Innovation, Science and Economic Development Canada (ISED) CAN ICES-3(A)/NMB-3(A) The Class A digital apparatus meets all requirements of the Canadian Interference- Causing Equipment Regulation.

  • Page 99: Japan

    The full text of EU declaration of conformity is available at the following internet address: www.nvidia.com/support A copy of the Declaration of Conformity to the essential requirements may be obtained directly from NVIDIA GmbH (Floessergasse 2, 81369 Munich, Germany). C.5. Japan Voluntary Control Council for Interference (VCCI) This is a Class A product.
  • Page 100 A Japanese regulatory requirement, defined by specification JIS C 0950, 2008, mandates that manufacturers provide Material Content Declarations for certain categories of electronic products offered for sale after July 1, 2006. To view the JIS C 0950 material declaration for this product, visit www.nvidia.com Japan RoHS Material Content Declaration 日本工業規格JIS C 0950:2008により、2006年7月1日以降に販売される特定分野の電気および電子機器について、製造者による含有物質の表示が義務付けられま...

  • Page 101: Australia And New Zealand

    Using DGX-2 System in KVM Mode A Japanese regulatory requirement, defined by specification JIS C 0950: 2008, mandates that manufacturers provide Material Content Declarations for certain categories of electronic products offered for sale after July 1, 2006. Product Model Number: DGX-2...

  • Page 102: China

    Using DGX-2 System in KVM Mode C.7. China China RoHS Material Content Declaration 产品中有害物质的名称及含量 The Table of Hazardous Substances and their Content 根据中国《电器电子产品有害物质限制使用管理办法》 as required by China’s Management Methods for Restricted of Hazardous Substances Used in Electrical and Electronic Products 有害物质...

  • Page 103: Israel

    Using DGX-2 System in KVM Mode C.8. Israel C.9. Russia/Kazakhstan/Belarus EurAsian Customs Union This device complies with the technical regulations of the Customs Union (CU TR) C.10. Vietnam Phù hợp quy chuẩn kỹ thuật, tiêu chuẩn: TCVN 7189:2009 - Thiết bị công nghệ thông tin –...
  • Page 104 Using DGX-2 System in KVM Mode C.11. South Korea Korean Agency for Technology and Standards (KATS) Class A Equipment (Industrial Broadcasting & Communication Equipment). This equipment Industrial (Class A) electromagnetic wave suitability equipment and seller or user should take notice of it, and this equipment is to be used in the places except for home.
  • Page 105 Using DGX-2 System in KVM Mode 세부모델명(번호): 해당없음 제품출시일 해당없음 제품의 중량 해당없음 제조, 수입업자 앤비디아 엔비디아의 그래픽 카드제품은 전기 전자제품 및 자동차의 자원순환에 관한 법률 시행령 제 11조 제 1항에 의거한 법 시행행규칙 제 3조에에따른 유해물질함유 기준을 확인 및 평가한...
  • Page 106 NVIDIA Corporation and/or Importer: This for is publicly certify That NVIDIA Company has undergone the confirmation and evaluation procedures for the acceptable amounts of hazardous materials contained in graphic card according to the regulations stipulated in Article 3 of the ‘Status on the Recycling of Electrical and Electronic Products, and Automobiles’...
  • Page 107 Using DGX-2 System in KVM Mode C.12. Taiwan Bureau of Standards, Metrology & Inspection (BSMI) 報驗義務人: 香港商輝達香港控股有限公司台灣分公司 統一編號：80022300 臺北市內湖區基湖路8號. Taiwan RoHS Material Content Declaration DGX-2 System User Guide...
  • Page 108 Using DGX-2 System in KVM Mode 限用物質含有情况標示聲明書 Declaration of the presence condition of the Restricted Sustances Marking 設備名稱：DGX-2 Equipment Name: DGX-2 限用物質及其化學符號 Restricted substances and its chemical symbols 單元 Parts 铅 汞 镉 六價铬 多溴聯苯 多溴二苯醚 (Pb ) (Hg) (Cd)
  • Page 109 NVIDIA does not accept any liability related to any default, damage, costs or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this guide, or (ii) customer product designs.

Table of Contents