1. Manuals
  2. Brands
  3. Cray Manuals
  4. Server
  5. Urika-GX
  6. Administration manual

Cray Urika-GX Administration Manual

Hide thumbs
Table of Contents

Advertisement

Quick Links

Download this manual
Urika®-GX System Administration Guide
(2.2.UP00)
S-3016

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the Urika-GX and is the answer not in the manual?

Questions and answers

Subscribe to Our Youtube Channel

Related Manuals for Cray Urika-GX

Summary of Contents for Cray Urika-GX

  • Page 1 Urika®-GX System Administration Guide (2.2.UP00) S-3016...

  • Page 2: Table Of Contents

    Contents ® 1 About the Urika -GX System Administration Guide....................7 2 The Urika-GX System.............................9 2.1 Administrative Components of Urika-GX....................9 2.2 Network Components..........................10 2.3 File Systems............................11 2.4 System Nodes............................12 2.5 Restrictions on Use..........................12 3 System Management............................15 3.1 Check the Current Service Mode......................
  • Page 3 3.9 Power Up the Urika-GX System......................46 3.10 Power Down the Urika-GX System....................... 49 3.11 Urika-GX CLI Commands for Managing Services................. 51 3.12 Remote HDFS Remote Access and Multihoming on Urika-GX.............54 3.13 Update the InfluxDB Data Retention Policy...................54 3.14 Service to Node Mapping........................55 3.15 Image Management with Docker and Kubernetes................
  • Page 4 5.5 Manage Long Running Services Using Marathon................. 133 5.6 Manage the Spark Thrift Server as a Non-Admin User................. 136 5.7 Manage Jobs Using the Cray Application Management UI..............137 5.7.1 Overview of the Cray Application Management UI..............138 6 Cray DVS................................140 6.1 Introduction to DVS..........................
  • Page 5 7.9.5 Change LDAP Password on Urika-GX..................224 7.9.6 Reset a Forgotten Password for the Cray Application Management UI........224 7.9.7 Reset an Administrator LDAP Password on Systems Using Urika-GX 1.2UP01 and Earlier Releases..........................225 7.9.8 Reset an Administrator LDAP Password when the OLC Schema Password is Unknown..226 7.9.9 Reset an Administrator LDAP Password when the OLC Scheme Password is Known...
  • Page 6 Contents 8.1 System Management Log File Locations....................245 8.2 Default Log Settings..........................246 8.3 Analytic Applications Log File Locations....................248 8.4 Security Related Troubleshooting Information..................250 8.4.1 Save and Restore Tenant Information..................254 8.4.2 LDAP Server Start-up Issues....................256 8.5 Modify the Secret of a Mesos Framework..................... 256 8.6 Clean Up Log Data..........................

  • Page 7: Gx System Administration Guide

    ® About the Urika -GX System Administration Guide About the Urika -GX System Administration Guide ® This publication contains administrative information about using the Cray ® Urika ® -GX system. Typographic Conventions Monospace Indicates program code, reserved words, library functions, command-line prompts, screen output, file/path names, key strokes (e.g., Enter and Alt-Ctrl-F), and...
  • Page 8 LIBSCI, NODEKARE. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners.

  • Page 9: The Urika-Gx System

    The Urika-GX platform provides the tools required for capturing and organizing a wide variety of data types from different sources and enables analyzing big data and discovering hidden relationships.

  • Page 10: Network Components

    48-port GigE switch that provides dual 1GigE and/or dual 10GigE interfaces to the site network. Urika-GX's login nodes do not route through this switch and need to be directly connected to the site network. The operational network allows node connectivity externally from Urika-GX to the site network.

  • Page 11: File Systems

    Urika-GX also features tiered HDFS storage. HDFS data is transferred over the Aries network. ○ Network File System (NFS) - The Urika-GX SMW hosts NFS, which is made available to every node via the management network. ○...

  • Page 12: System Nodes

    The Urika-GX System System Nodes Each Urika-GX node is a logical grouping of a processor, memory, and a data routing resource. Nodes can be categorized as compute, I/O, service and login nodes. Table 1. Node Types and Descriptions Node Type...
  • Page 13 NOTE: Contact Cray Support if it is required to modify additional software configurations. Security Considerations If the Urika GX system is running in the secure mode in production, Cray does not recommend toggling back to the default mode while in production because, in the default mode, the security assurances provided by secure mode are not in place, and the security of data that was protected by secure mode may be compromised while running in the default mode.
  • Page 14 The Urika-GX System ● It is recommended not to make any changes to the default set of Kubernetes and Kerberos configurations without consulting Cray Support, as doing so can adversely affect the functionality of the system in the secure service mode. ●...

  • Page 15: System Management

    System Management Check the Current Service Mode Prerequisites This procedure requires root privileges on the SMW. About this task Urika-GX supports two service modes, which dictate the list of services available. These modes include: ● Default ● Secure Use the following instructions to determine the service mode the system is currently running in.
  • Page 16 System Management Table 2. Urika-GX Component Naming Conventions Component/Subject Naming Pattern Range Wild card, similar to s0. Wild card, which refers to all the all_comp compute nodes Wild card, which refers to all the all_serv service nodes Machine partition Rack R:0 to 161 Sub-rack.

  • Page 17: System Management Workstation (Smw)

    The System Management Workstation (SMW) is the system administrator's console for managing a Cray system. The SMW is a server that runs the CentOS (version 7.3) operating system, Cray developed software, and third- party software. The SMW is also a point of control for the Hardware Supervisory System (HSS). The HSS data is stored on an internal hard drive of the SMW.
  • Page 18 System Management 3.3.3 Control System Management Workstation (SMW) Power with the iDRAC8 Web Console Prerequisites Ensure that the SMW is up and running. About this task Use the iDRAC's web console to start up and shut down the System Management Workstation (SMW). Procedure 1.
  • Page 19 System Management Figure 1. iDRAC Login Screen 3. On the Quick Launch Tasks section of the iDRAC UI, click on Power ON/ OFF link to control the SMW's power. S3016...

  • Page 20: Synchronize The System Management Workstation (Smw) To The Site Ntp Server

    This procedure requires root privileges. About this task The components of the Cray system synchronize time with the System Management Workstation (SMW) through Network Time Protocol (NTP). By default, the NTP configuration of the SMW is configured to stand alone;...

  • Page 21: Synchronize Time Of Day On System Nodes

    Prerequisites This procedure needs to be carried out as root. About this task Follow this procedure to configure Urika-GX compute nodes to synchronize to a site NTP server. This procedure is specific to a 48 node system. Procedure 1. Stop the NTP server by issuing the systemctl stop ntpd command.

  • Page 22: Reboot A Stopped System Management Workstation (Smw)

    HSS is an integrated system of hardware and software that monitors the hardware components of the system and proactively manages the health of the system. HSS communicates with nodes and management processors over an internal (private) Ethernet network that operates independently of the Cray Aries High Speed Network (HSN). HSS includes the following components: ●...
  • Page 23 Dual Aries Network Card (dANC) Controllers and Rack Controllers A dANC control processor is hierarchically the lowest component of the monitoring system. The dANC Cray network card contains two Aries ASICs and an ANCC. There are 2 dANC cards per sub-rack, and hence 4 Aries ASICs, which support 16 nodes.

  • Page 24: Hardware Supervisory System (Hss) Architecture Overview

    VLANs to connect the SMW to the ANCCs, RC, and iSCBs. The Urika-GX system can consist of 1, 2 or 3 sub-racks per rack, and 2 dANCs per sub-rack, resulting in a maximum of 6 dANCs per rack. Each dANC has 2 Aries ASICs, each of which has 4 NICs to support a single node per NIC connected by PCIe Gen 3.

  • Page 25: The Xtdiscover Command

    3.4.2 The xtdiscover Command The xtdiscover command automatically discovers the hardware components on a Cray system and creates entries in the system database to reflect the current hardware configuration. The xtdiscover command identifies missing or non-responsive cabinets and empty or non-functioning Dual Aries Network Cards (dANCs).

  • Page 26: Hardware Supervisory System (Hss) Daemons

    Thus, the dynamic system state persists between SMW boots. The state manager uses the Lightweight Log Manager (LLM). The log data from state manager is written to: /var/opt/cray/log/sm-yyyymmdd. The default setting for state manager is to enable LLM logging. The state manager performs the following functions: ●...

  • Page 27: Hardware Supervisory System (Hss) Administration And Diagnostic Commands Supported On Urika-Gx

    NID assignment file, nids.ini. CAUTION: The nids.ini file can have a major impact on the functionality of a Cray system and should only be used or modified at the recommendation of Cray support personnel. Setting up this file incorrectly can make the Cray system unroutable.
  • Page 28 Displays HSS events. xtdaemonconfig Configures HSS daemons dynamically. xtdiscover Discovers and configures the Cray system hardware. This command is also used to populate the HSS database and set up HSS IP networking. For more information, see the xtdiscover(8) man page.
  • Page 29 Generates a summary of PCIe link errors. There are a number of HSS diagnostics commands supported on Urika-GX. These commands need to be run from a compute node. WARNING: All HSS diagnostics commands are intended for use by Cray Service Personnel only.

  • Page 30: Hardware Supervisory System (Hss) Environments

    Hardware Supervisory System (HSS) Environments The HSS infrastructure environment is composed of a number of daemons, processes, and commands that help control and monitor physical system components. HSS daemons supported on Urika-GX are listed in the following tables: Table 5. ANCC HSS Daemons Daemon...
  • Page 31 System Management Daemon Description ANCC Network Daemon (anccnwd) Monitors the Aries HSN link status, and reports soft and fatal errors. ANCC PCIe Monitor Daemon (anccpcimond) Monitors the Aries PCIe errors and status. ANCC User-space Driver Daemon Acts as the ANCC JTAG/MMR/Node Memory access driver.

  • Page 32: High Speed Network (Hsn) Management

    3.4.7 High Speed Network (HSN) Management The Cray HSN is composed of a number of custom components developed by Cray that provide high-bandwidth, low-latency communication between all the compute processing elements of the system. CAUTION: xtbounce should never be executed when nodes are up as this command will not gracefully shut nodes down, and will cause them to crash.

  • Page 33: Disable Hardware Components

    For detailed information about using the xtcli disable command, see the xtcli(8) man page. 3.4.10 Enable Hardware Components If links, nodes, or Cray ASICs that have been disabled are later fixed, the system administrator can add them back to the system with the xtcli enable command.

  • Page 34: Set Hardware Components To Empty

    System Management IMPORTANT: The -n option with the xtcli disable command must be used carefully because this may create invalid system state configurations. The state of empty components does not change when using the xtcli enable command, unless the force option (-f) is used.

  • Page 35: Unlock Hardware Components

    3.4.14 Capture and Analyze System-level and Node-level Dumps The xtdumpsys command collects and analyzes information from a Cray system that is failing or has failed, has crashed, or is hung. Analysis is performed on, for example, event log data, active heartbeat probing, voltages, temperatures, health faults, in-memory console buffers, and high-speed interconnection network errors.

  • Page 36: Collect Debug Information From Hung Nodes Using The Xtnmi Command

    System Management The following example shows usage of the xtdumpsys command: crayadm@smw> xtdumpsys --add r0s2c1 The xtdumpsys command is written in Python and supports plug-ins written in Python. A number of plug-in scripts are included in the software release. Call xtdumpsys --list to view a list of included plug-ins and their respective directories.

  • Page 37: Request And Display System Routing

    System Management Find the physical ID for node 12 smw# xtnid2str 12 node id 0xc = 'r0s0c1n4' Find the physical ID for nodes 0, 1, 2, and 3 smw# xtnid2str 0 1 2 3 node id 0x0 = 'r0s0c0n0' node id 0x1 = 'r0s0c0n1' node id 0x2 = 'r0s0c0n2' node id 0x3 = 'r0s0c0n3' Find Node Information Using the nid2nic Command...

  • Page 38: Initiate A Network Discovery Process

    System Management CAUTION: Be sure the Aries ASCIs have been initialized using the xtbounce command and that the HSN links have been initialized. 3.4.18 Initiate a Network Discovery Process Use the HSS rtr --discover command to initiate a network discovery process. crayadm@smw>...

  • Page 39: Check Compute Node High Speed Network (Hsn) Connection

    The xtpcimon command runs on the SMW and is started when the system is initialized. Any PCIe-related errors are reported to stdout, unless directed to a log file. If the optional /opt/cray/hss/default/etc/xtpcimon.ini initialization file is present, the xtpcimon command uses the settings provided in the file.

  • Page 40: View Component Alert, Warning, And Location History

    System Management Check the state manager crayadm@smw> xtalive -l smw -a sm s0 3.4.24 View Component Alert, Warning, and Location History Use the xtcli comp_hist command to display component alert, warning, and location history. Either an error history, which displays alerts or warnings found on designated components, or a location history may be displayed.

  • Page 41: Display Component State Information

    Alerts, reserves, and warnings must be cleared before a component can operate. Clearing an alert on a component frees its state. For more information, see the xtclear(8) man page. 3.4.29 Flash Management on Urika-GX ® HSS is responsible for flashing all devices in the Urika -GX system via an out-of-band (OOB) mechanism.

  • Page 42: Create And Modify The Authorized_Keys File Using The Xtcc-Ssh-Keys Command

    The xtcc-ssh-keys command takes no options. When run, it invokes the user-selected text editor (specified by the VISUAL or EDITOR environment variables and defaulting to vi) on a file maintained by Cray HSS. This file has the format described in sshd(8) under the heading "AUTHORIZED_KEYS FILE FORMAT". Adding a public key to this file permits a user authenticating using the corresponding private key to connect to the RCs and dANCCs without using a password.

  • Page 43: Gather Troubleshooting Information Using The Xtdumpsys Command

    If the two entered passwords match exactly, a salted hash of the password is written to a file maintained by Cray HSS. Within one minute, all booted controllers in the Urika-GX system will be using the new password.

  • Page 44: Analyze Node Memory Dump Using The Kdump And Crash Utilities On A Node

    SMW for crash analysis as well. NOTE: Cray recommends executing the kdump utility only if a node has panicked or is hung, or if a dump is requested by Cray.

  • Page 45: Cray Lightweight Log Management (Llm) System

    Cray Lightweight Log Management (LLM) System The Cray Lightweight Log Management (LLM) system is the log infrastructure for Cray systems and must be enabled for systems to successfully log events. At a high level, a library is used to deliver messages to rsyslog utilizing the RFC 5424 protocol;...

  • Page 46: Power Up The Urika-Gx System

    ® The instructions documented in the procedure can be used for powering up the Urika -GX system. For detailed information, or for troubleshooting issues, please contact Cray Support. Procedure 1. Physically power on the Power Distribution Units (PDUs). 2. Turn on the System Management Workstation (SMW).
  • Page 47 If alerts are discovered and cleared, wait a few seconds and execute the xtcli -t l0 s0 command again to ensure the alerts did not return. If they do return, contact Cray Support. If the errors do not return, the underlying transient problem can be ignored.
  • Page 48 System Management 13. Optional: Start the tenant VMs. For more information, see the ux-tenant-start man page. 14. Resolve any issues related to the Romana service. root@smw:~ main_romana_df_gateway.yml 15. Verify that the compute nodes have been turned on. root@smw:~ xtcli status s0 All the nodes should send a response.

  • Page 49: Power Down The Urika-Gx System

    The instructions documented in the procedure should be following in the order listed to power off the Urika system. For detailed information, or for troubleshooting issues, please contact Cray Support. It is assumed that the instructions in procedure are being carried out on a 48 node system. Please use node IDs according to the actual system configuration.
  • Page 50 System Management root@smw:~ ux-tenant-status -R -F name,state|grep -v =notfound|awk \ '{print $1}'|awk -F= '{print $2}'|xarg 8. Execute the ux-nid-power-off command to power off all the nodes. root@smw:~ ux-nid-power-off all For more information about the ux-nid-power-off command, see the ux-nid-power-off man page. 9.

  • Page 51: Urika-Gx Cli Commands For Managing Services

    The urika-start command is used to start analytic services on the Urika-GX system. This command can be used to start services on one node, a number of nodes, or on all the system nodes. To specify more than one node, specify the node IDs separated by commas.
  • Page 52 The urika-stop command is used to stop analytic services running on one node, a number of nodes or on all the Urika-GX system nodes. To specify more than one node, specify the node IDs separated by commas. To stop a specific service, use the -s or --service option to specify the name of that service. This command stops all the running analytic services if used without any options.
  • Page 53 Display status of services: # urika-state Sequence of Execution of Scripts The urika-state command can be used to view the state services running on Urika-GX. Use this command to ensure that the following dependencies are met: ● Before executing the start -s mesos_master command, ensure that the ZooKeeper service is up and running.

  • Page 54: Remote Hdfs Remote Access And Multihoming On Urika-Gx

    3.12 Remote HDFS Remote Access and Multihoming on Urika-GX ● Multihoming - Urika-GX compute nodes can be multihomed between the Aries HSN and the operational network. ● Remote HDFS Access - HDFS can be set up on Urika-GX to enable external access.

  • Page 55: Service To Node Mapping

    5. Update the data retention policy according to requirements. In this example the data retention duration is changed from 0 (forever) to 2 weeks (504 hours). > alter retention policy default on "Cray Urika GX" Duration 2w > show retention policies on "Cray Urika GX"...
  • Page 56 ● kubelet nid00014 (Login node 1) ● ● HAProxy ● Collectl ● Urika-GX Applications Interface UI ● Cray Application Management UI ● Jupyter Notebook ● Service for flexing a YARN cluster ● Documentation and Learning Resources UI ● nrpe...
  • Page 57 Service for flexing a YARN cluster ● Grafana ● InfluxDB ● nrpe ● kubelet Table 10. Urika-GX Service to Node Mapping (3 Sub-rack System) Node ID(s) Service(s) Running on Node /Role of Node nid00000 ● ZooKeeper ● ncmd ● Mesos Master ●...
  • Page 58 ● nrpe ● kubelet nid00030 (Login node 1) ● ● HA Proxy ● Collectl ● Urika-GX Applications Interface UI ● Jupyter Notebook ● Service for flexing a YARN cluster ● Documentation and Learning Resources UI ● nrpe ● kubelet ●...

  • Page 59: Image Management With Docker And Kubernetes

    Images shipped with the system are managed by Docker when Urika-GX is operating under the default service mode. The SMW hosts Urika-GX's container repository, which is a container itself and can only host Cray developed containers currently.

  • Page 60: Execute Spark Jobs On Kubernetes

    For more information, visit https://kubernetes.io/. About the Cray Spark Image In order to run Spark on Kubernetes, Urika-GX ships with customized Spark images, which are based on the Spark version used on the system. 3.15.1 Execute Spark Jobs on Kubernetes Spark jobs run inside containers, which are managed via Kubernetes on the Urika-GX system.
  • Page 61 System Management pxz79 node name: N/A start time: N/A container images: N/A phase: Pending status: [] ..The output of the Spark Driver can be viewed by executing kubectl logs pod_name and looking at the pod's logs. The pod's name is displayed near the top of the console output, as shown in the preceding example.
  • Page 62 System Management $ kubectl logs spark-triangles-1520878896538-driver | grep "riangles:" numTriangles: 10624230 Number of triangles: 3541410 Resource Configuration for Spark Jobs The following Spark configuration settings may be used to control the amount of resources that Spark will request from Kubernetes for any job launched using spark-submit under the secure service mode, i.e., under Kubernetes: Table 12.

  • Page 63: Multi-Tenant Spark Thrift Server On Urika-Gx

    Limitations The current version of Spark on Urika-GX lacks support for long running spark jobs with secure HDFS. As a result, a Spark Thrift Server instance can only run as long as the HDFS delegation tokens are valid for. Currently this period is 1 day.
  • Page 64 System Management root@login1# kubectl get pods -n TENANT-VM thrift-server-TENANT-VM --show-all To see if the Spark Thrift Server has fully initialized, assuming the previous command showed the POD was "Running" root@login1# kubectl logs -n TENANT-VM thrift-server-TENANT-VM | grep 'Started ThriftHttpCLIService To see the state of the Kubernetes POD containing the Metastore server: root@login1# kubectl get pods -n <TENANT-VM>...

  • Page 65: System Monitoring

    System Monitoring System Monitoring System Monitoring Tools Urika-GX provides many tools for monitoring system components, resources and services. Table 13. Monitoring Tools Monitoring Tool Monitored Component/Service/Resource Helps monitor physical system components, such as PCIe channels and Dual Aries Network Card (dance).

  • Page 66: Monitor Resource Utilization And Node Status Using Nagios

    Speed Network (HSN) network. Also provides per node and aggregated bandwidth for the Aries High Speed network. ○ Lustre, HDD and SSD storage (per node and aggregated) In addition, Nagios can be used on Urika-GX to create and send alert notifications. For more information, refer to Tenant Management on page 189 Procedure 1.

  • Page 67: Configure Ssl/Tls For Nagios Core

    System Monitoring Figure 3. Nagios Services Page Select the Documentation link on the Nagios UI and visit and https://www.nagios.org pnp4nagios.org more information. 4.2.1 Configure SSL/TLS for Nagios Core Prerequisites ● This procedure requires root privileges. ● Ensure that the mod_ssl package is installed on the system. If it is not, install it by executing the following as root on the SMW: # yum install -y mod_ssl About this task...
  • Page 68 System Monitoring Procedure 1. Log on to the SMW as root. All of the remaining steps will be performed from within the root user's home directory to ensure the created files are not accessible to anyone except the root user. 2.
  • Page 69 System Monitoring Provide the CA with everything returned by the above, including the following lines: -----BEGIN CERTIFICATE REQUEST----- and -----END CERTIFICATE REQUEST----- After receiving the signed certificate, copy the certificate into a new file called certfile.crt. The certificate received will contain a lot of random text. Paste that into the new file, which can be modified using any editor, such as vi.

  • Page 70: Configure The Nagios Server To Send Email Notifications

    System Monitoring 8. Restart the Apache web server for the new certificate key to be used. ===== CentOS 5.x / 6.x | RHEL 5.x / 6.x | Oracle Linux 5.x / 6.x ===== # service httpd restart ===== CentOS 7.x | RHEL 7.x | Oracle Linux 7.x ===== # systemctl restart httpd.service 9.
  • Page 71 System Monitoring 2. Stop the Nagios service. # service nagios stop 3. Add the user's contact name, alias, and Email to the /usr/local/nagios/etc/objects/contacts.cfg file using the following format: define contact { contact_name Contact1 alias ContactNameAlias email email-address service_notification_period 24x7 service_notification_options w,u,c,r,f,s service_notification_commands notify-service-by-email...
  • Page 72 System Monitoring 4. Add the details (as shown in the following example) to add a group to which the mail needs to be sent: define contactgroup{ contactgroup_name Group1 alias GroupAlias members Contact1,Contact2 5. Specify the contact name and contact group name for the services for which the notification needs to be sent in the /usr/local/nagios/etc/objects/templates.cfg file.

  • Page 73: Change The Default Log File Path And Rotation Interval

    System Monitoring 6. Edit the /usr/local/nagios/etc/objects/commands.cfg file to add $NOTIFICATIONCOMMENT$\n after the $SERVICEOUTPUT$\n option in notify-service-by-email and in the notify-host-by- email command definition. # 'notify-service-by-email' command definition define command{ command_name notify-service-by-email command_line /usr/bin/printf "%b" "***** Nagios *****\n \nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n $NOTIFICATIONCOMMENT$ \n"...

  • Page 74: Configure Email Alerts

    System Monitoring 3. Switch to the /usr/local/nagios/etc directory. # cd /usr/local/nagios/etc 4. Use an editor to modify the nagios.cfg configuration file as desired. ● Locate the line log_file=/usr/local/nagios/var/nagios.log This line specifies the default path to the log file. Modify the path as desired. ●...
  • Page 75 System Monitoring A contact group will be defined by default in the contacts.cfg file. define contactgroup{ contactgroup_name admins alias Nagios Administrators members admin1 5. Add as many members as needed. Multiple contact groups can be defined if needed. A contact is defined in this file by default. define contact{ contact_name admin1...

  • Page 76: Modify Nagios Plug-In Threshold

    This procedure requires root privileges. About this task Each Cray plugin requires thresholds for the metrics they monitor. All plugins require a warning and critical level, set via parameters. Some require a third parameter. These plugins include cray_check_disk, cray_check_disk_aggr, and cray_check_network. Executing the plugin with the -h option will list the available options and the arguments to pass in for these options.

  • Page 77: Get Started With Using Grafana

    System Monitoring There is a define service block as shown below for the aggregate CPU plug-in. define service{ local-service host_name localhost service_description Aggregate CPU Stats check_command check_cpu_aggr!0.5!0.8 Here, a custom command name is defined, the name being arbitrary. In this example it it called check_cpu_aggr, which is the last line shown above.
  • Page 78 Dashboard. ● Query Editor - Each Panel provides a Query Editor for the data source, which is InfluxDB on the Urika-GX system. Information retrieved from the Query Editor is displayed on the Panel.

  • Page 79: Default Grafana Dashboards

    Since the time displayed on the Grafana UI uses the browser's timezone and that displayed on the Spark History server's UI uses the Urika-GX system's timezone, the timezones displayed on the two UIs may not be the same. By default, Grafana's refresh rate is turned off on the Urika-GX system. Sometimes the Hadoop and Spark Grafana dashboards take longer to load than expected.
  • Page 80 ● Basic Metrics - Displays graphs representing statistical data related to network, CPU, and I/O utilization for the Urika-GX system, as a whole. TIP: It is recommended that administrators use the Basic Metrics dashboard before using other dashboards to retrieve a summary of the system's health.
  • Page 81 System Monitoring ○ CPU AND MEMORY ▪ Used and File Cached Memory - Displays the used and file cached memory for each node. ▪ CPU Utilization User + System - Displays the CPU utilization by the user and system for each node ○...
  • Page 82 System Monitoring Figure 6. Basic Metrics Dashboard ● Compute Node Performance Statistics - Displays graphs representing statistical data related to network, CPU, and I/O utilization for all Urika-GX compute nodes. This dashboard contains the following graphs: ○ ▪ CPU MEMORY UTILIZATION ▪...
  • Page 83 System Monitoring ▪ Aries HSN Bytes/Sec In/Out - Displays the Aries network TCP traffic information for compute nodes. Note that non-TCP Aries traffic, including most traffic generated by CGE, is not shown here. ▪ Operational network Bytes/sec In/Out - Displays the overall operational network traffic information for compute nodes.
  • Page 84 System Monitoring Figure 8. Hadoop Applications Metrics Dashboard ● Hadoop Cluster Metrics - Displays graphs representing statistical data related to Hadoop components, such as HDFS Data Nodes and HDFS Name Nodes. This dashboard contains the following sections: ○ Cluster BlockReceivedAndDeletedAvgTime - Displays the average time in milliseconds for the hdfs cluster to send and receive blocks.
  • Page 85 Figure 9. Hadoop Cluster Metrics Dashboard ● Non-compute Node Performance Statistics - Displays graphs representing statistical data related to network, CPU, and I/O utilization for all the non-compute (I/O and login) nodes of Urika-GX. This dashboard contains the following graphs: ○...
  • Page 86 ● Per Node Performance Statistics - Displays graphs representing statistical data related to network, CPU, and I/O utilization for individual Urika-GX nodes. The node's hostname can be selected using the hostname drop down provided on the UI. This dashboard contains the following graphs: ○...
  • Page 87 System Monitoring ○ Lustre Read/Writes Bytes/Second - Displays the number of Lustre reads/writes by the selected node. ○ Aries HSN Bytes/Sec In/Out - Displays the Aries network TCP traffic information for the selected node. Note that non-TCP Aries traffic, including most traffic generated by CGE, is not shown here. ○...
  • Page 88 System Monitoring ▪ Root File System Hard Drive (/dev/sda) Reads/Writes Bytes/Sec - Displays information about the usage of memory on the root file system of the SMW ▪ Root File System Percent Used - Displays the percentage for used SMW root file system space. ○...
  • Page 89 System Monitoring Figure 13. Grafana Spark Metrics Dashboard Graphs displayed on this dashboard are grouped into the following sections: ○ READ/WRITE: Displays statistics related to the file system statistics of a Spark executor. Results in the graphs of this section are displayed per node for a particular Spark Job. The Y-axis displays the number in bytes, whereas the X-axis displays the start/stop time of the task for a particular Spark Job.

  • Page 90: Update Influxdb Security Settings

    Urika-GX Service Modes on page 177. About this task The default configuration for InfluxDB as shipped with Urika-GX is: 1. InfluxDB UI on socket login2:8083 is disabled 2. No authorization is enabled. S3016...

  • Page 91: Update The Influxdb Data Retention Policy

    This procedure requires root privileges. Before performing this procedure, use the urika-state command to ensure that the system is operating in the service mode that supports using InfluxDB. For more information, see the urika-state man page and refer to Urika-GX Service Modes on page 177. S3016...
  • Page 92 5. Update the data retention policy according to requirements. In this example the data retention duration is changed from 0 (forever) to 2 weeks (504 hours). > alter retention policy default on "Cray Urika GX" Duration 2w > show retention policies on "Cray Urika GX"...

  • Page 93: Configuration Settings Of Grafana

    177. About this task Urika-GX ships with UTC as the default timezone displayed on the Grafana UI. This procedure can be used to change the timezone for newly created dashboards or copies of the predefined dashboards shipped with the system.
  • Page 94 System Monitoring Figure 14. Grafana Login Screen 3. Select Home >admin > Preferences Figure 15. Preferences Interface S3016...

  • Page 95: Create A New Grafana Dashboard

    This procedure requires administrative privileges. Before performing this procedure, use the urika-state command to ensure that the system is operating in the service mode that supports using Grafana. For more information, see the urika-state man page and refer to Urika-GX Service Modes on page 177. About this task In addition to the default set of dashboards, administrators can add additional dashboards to the Grafana UI, as described in the following instructions.
  • Page 96 System Monitoring Procedure 1. Access the Grafana UI using the Urika-GX Applications Interface by pointing a browser at http:// hostname-login1 (which is the recommended way of access) or at http://hostname-login2:3000. 2. Log on to Grafana by entering LDAP credentials, or credentials of the default Grafana account (username: admin, password: admin) to log on to Grafana.

  • Page 97: Add A New Graph To The Grafana Dashboard

    System Monitoring Figure 18. New Dashboard Pop-up 4. Select Dashboard from the left navigation menu 5. Add panels/graphs/rows to the new dashboard and customize as needed. 6. Optional: Update the dashboard's default settings using Settings from the configuration gear at the top of the interface.
  • Page 98 UI, as described in the following instructions. Procedure 1. Access the Grafana UI using the Urika-GX Applications Interface by pointing a browser at http:// hostname(which is the recommended method) or at http://hostname:3000. 2. Log on to Grafana by entering LDAP credentials, or credentials of the default Grafana account (username: admin, password: admin) to log on to Grafana.
  • Page 99 System Monitoring Figure 21. Default Grafana UI on Urika-GX 4. Select Add Panel > graph from the green menu on the left side of the Dashboard. Figure 22. Add a New Graph to Grafana Dashboard 5. Click on the title of the new panel and select edit as shown in the following figure:...

  • Page 100: Start Influxdb Before Hadoop Services

    This procedure requires root privileges. Before performing this procedure, use the urika-state command to ensure that the system is operating in the service mode that supports using InfluxDB. For more information, see the urika-state man page and refer to Urika-GX Service Modes on page 177. About this task If the Hadoop Cluster Metrics Grafana dashboard takes too long to populate, it may because the InfluxDB service was started after Hadoop services.

  • Page 101: Monitor Subrack Attributes

    System Monitoring Procedure 1. Log on to the SMW as root. 2. Stop YARN. # urika-stop --service yarn 3. Stop the HDFS services. # urika-stop --service hdfs 4. Verify that the InfluxDB service is running or start it if it is not already started. # urika-start --service influxdb 5.

  • Page 102: Analyze Node Memory Dump Using The Kdump And Crash Utilities On A Node

    SMW for crash analysis as well. NOTE: Cray recommends executing the kdump utility only if a node has panicked or is hung, or if a dump is requested by Cray.

  • Page 103: Retrieve System Status Information Using The Urika-Check-Platform Command

    Get revision info of Rack Controller, iSCBs, dANCs, and compute nodes and HSS using the --rev option: # urika-check-platform --rev If nodes are not listed then revision information is unavailable. Make sure /global/cray/mfg/self_update_fw/intel_fw_chk.sh is installed properly on each node. ● Check if Aries is routed: # urika-check-platform --aries ●...

  • Page 104: Iscb Description

    Cray Support), as they may interfere with performance of the system management software. For more information, contact Cray support. Each Urika-GX iSCB module has a single Gigabit Ethernet port and a dedicated logical serial port assigned for command and power management access. The iSCB module is typically managed over a secure private management network.

  • Page 105: Iscb Command Reference

    System Monitoring Log Off the iSCB Log off the iSCB: [r0s0i0 root]# exit logout Connection to r0s0i0 closed. [r0s0i0 root]# IP Address When iscb starts up, the value of the SCB IP Source configured in NVRAM is used obtain the IP address: either static or dhcp.
  • Page 106 System Monitoring 4.15.2 iSCB Command Reference Command List Display the BMC status of specified node bootmode Display or set boot mode Display information of all or specified CFUs console Display established cli connections or kill the specified connection crash Create a core dump? danc Display status and power on/off/cycle all or specified dANCs Display CFU status or set fan PWM duty value...
  • Page 107 System Monitoring shutdown Shutdown all or specified nodes status Display Node, PSU and FAN status stop Stop the iSCB service Display status of all established remote console sessions or kill specified connection updatefw Update iSCB firmware Display the current firmware version and build date Command Syntax Display BMC data from all or specified nodes Syntax: bmc [node]...
  • Page 108 System Monitoring [r0s0i0 /]# Display information of all or specified CFUs Syntax: cfu [cfuno] Parameters: cfuno CFU number Example: [r0s0i0 /]# iscb cfu No Stat Fan0 Fan1 Duty Demand Margin 3432 3446 3450 3417 4090 4112 3446 3428 3443 3439 3439 3450 [r0s0i0 /]#...
  • Page 109 System Monitoring Display danc status or set specified danc power on/off/cycle. Syntax: danc [status|on|off|cycle] [0|1|all] Parameters: status Display status of specific dANC Power on specific dANC Power off specific dANC cycle Power cycle specific dANC Example: [r0s0i0 /]# iscb danc # PMDev Pwr +12V Watt Ver Stat dANC Ars0 Ars1 AOC0 AOC1 AOC2 AOC3 AOC4...
  • Page 110 System Monitoring Fan2: 3474rpm 3483rpm 3497rpm 6013rpm 6040rpm 3486rpm Duty: 100% 100% [r0s0i0 /]# iscb fan 05 normal [r0s0i0 /]# iscb fan CFU: Status: Fan1: 3504rpm 3504rpm 3513rpm 5394rpm 5454rpm 4365rpm Fan2: 3483rpm 3495rpm 3504rpm 6000rpm 6000rpm 4591rpm Duty: 100% 100% [r0s0i0 /]# Display power usage of CFUs through Fan Control Board...
  • Page 111 System Monitoring Syntax: history Parameters: None Example: [r0s0i0 /]# history help history [r0s0i0 /]# lasterr Display recently occurred event log or clear the last log Syntax: lasterr [clear] Parameters: clear Clear the last log and reset the red LED on the iSCB front panel Example: [r0s0i0 /]# iscb lasterr ALERT: CFU02 failed...
  • Page 112 System Monitoring Syntax: log [info|critical|warning] Parameters: info Display notice, and info and higher level event logs critical Display err, crit, alert and emerg level event logs warning Display warning and higher level event logs Example: [r0s0i0 /]# iscb log critical 1: Jan 6 23:06:26 iscb:iscbd.sh: Detected location from IDSW: 0-0-0 2: Jan...
  • Page 113 System Monitoring sensor: 4+0 gpu: - [r0s0i0 /]# nvram Display or update the configuration parameters stored in NVRAM Syntax: nvram [save|clear] Parameters: save Store current configuration parameters into the NVRAM clear Remove current configuration parameters and load default parameters. Parameters are loaded but not saved until the save command is used. Example: [r0s0i0 /]# iscb nvram Versions...
  • Page 114 System Monitoring node10: off node11: on node12: on node13: on node14: on node15: on node16: on [r0s0i0 /]# pmnode Display powerman style power status Syntax: pmnode [node|all] Parameters: node Display status for a specified node, 1 through 16 Display status for all nodes Example: [r0s0i0 /]# iscb pmnode 7 node07: on...
  • Page 115 System Monitoring 0xffff: Bit fields for multiple specific nodes 0x3: Node 1,2 0x5: Node 1,3 0xff00: Node 9-16 1,2,..n: slot number separated by comma(,) 1,2,4: Node 1,2,4 1-n: slot number scope by dash(-) 1-4: Node 1,2,3,4 cn000[1,2,..-n]: specified blade name by name with dash and comma cn000[1,4-6]: Node 1,4,5,6 interval Interval time for power on operation of multiple nodes.
  • Page 116 System Monitoring 10665 0 48781 blade14 10666 0 48782 10665 0 48782 blade15 10666 0 48784 10665 0 48783 PSU0 47337 0 48784 PSU1 47337 0 48784 PSU2 47337 0 48784 PSU3 47338 0 48784 PSU4 47338 0 48784 PSU5 47338 0 48784 FCB0...
  • Page 117 System Monitoring psuno PSU number Example: [r0s0i0 /]# iscb psumodel No Pwr Stat Model Serial Number 00 on PSSH162202A H/W Rev.:1.0, F/W Rev.:10.5.0 01234567890123456789 01 on PSSH162202A H/W Rev.:1.0, F/W Rev.:10.5.0 01234567890123456789 02 on PSSH162202A H/W Rev.:1.0, F/W Rev.:10.5.0 01234567890123456789 03 on PSSH162202A H/W Rev.:1.0, F/W Rev.:10.5.0 01234567890123456789...
  • Page 118 [r0s0i0 /]# Display or set iSCB parameters Syntax: set <param> [value] Parameters: ace [on|off] Support ACE cluster. Not implemented for Urika-GX. Default is off. acelog [level] If ace is set, generate ace related log of defined level: info, warning, critical activefan [on| Dynamic Fan Control by node’s health status.
  • Page 119 System Monitoring BMC reports CPU temperature to -60’C, the actual temperature is 90-60 = 30’C. Default is 90. danc [on|off] dANC module power on|off. debug [on|off] Debug option.* email [level [id Event level, e-mail address, mail server/user/passwd of E- [server [user Mail notification.
  • Page 120 System Monitoring nm [nm [unitid]] iSCB Netmask (255.255.0.0).* ns [ns] iSCB NameServer IP address.* ntp [host] IP address of NTP server.* ntpserver [host] Hostname of NTP server. For example, time.nist.gov.* passwd [pass] set iSCB password. polling [seconds] polling interval. port [port] TCP port number of the CLI and Terminal Service.
  • Page 121 System Monitoring Example: [r0s0i0 /]# iscb set ip NVRAM : 10.11.0.10 Current: 192.168.2.10 [r0s0i0 /]# iscb set ip 10.10.1.11 [r0s0i0 /]# iscb set ip NVRAM : 10.10.1.11 Current: 10.10.1.9 [r0s0i0 /]# iscb nvram save [r0s0i0 /]# iscb reboot [$ /]# Connection closed by foreign host. shutdown Shutdown all or specified nodes Syntax: shutdown [node|all]...
  • Page 122 System Monitoring [r0s0i0 /]# stop and start Stop and start the iscb service Syntax: stop|start Parameters: None Example: [r0s0i0 /]# iscb stop [r0s0i0 /]# iscb ver Can't connect iSCB service. err: No such file or directory [r0s0i0 /]# iscb start [r0s0i0 /]# iscb ver iSCB Ver 1.1 (11:53:51 02/18/16 PST) passive (100h) [r0s0i0 /]#...
  • Page 123 System Monitoring Example: [r0s0i0 /]# scp athenaiscb-1.1.bin root@r0s0i0:/tmp athenaiscb-1.0.bin 100% 17MB 8.8MB/s 00:02 [r0s0i0 /]# ssh root@r0s0i0 iscb updatefw [r0s0i0 /]# Display firmware version and build date Syntax: ver Parameters: None Example: [r0s0i0 /]# iscb ver iSCB Ver 1.1 (11:53:51 02/18/16 PST) active (0h) [r0s0i0 /]# S3016...

  • Page 124: Resource Management

    Marathon is registered as a single framework with Mesos. Marathon provides a mechanism to launch non- framework applications to run under Mesos. Marathon enables long-running services under Mesos such as databases, streaming engines etc. Cray has developed: ● the mrun command, which sets up resources for CGE and HPC jobs. For more information, see the mrun man page.
  • Page 125 Additional points to note: ● On Urika-GX, the Mesos cluster runs in High Availability mode, with 3 Mesos Masters and Marathon instances configured with Zookeeper. ● Unlike Marathon, Mesos does not offer any queue. Urika-GX scripts for flexing clusters and the mrun command do not submit their jobs unless they know the resource requirement is satisfied.

  • Page 126: Use Apache Mesos On Urika-Gx

    By default, Urika-GX ships with three Mesos masters with a quorum size of two. At least two Mesos masters must be running at any given time to ensure that the Mesos cluster is functioning properly. Administrators can use the urika-state and urika-inventory commands to check the status of Mesos masters and slaves.
  • Page 127 CGE and HPC jobs. On Urika-GX, all tasks launched directly from Marathon need to be run as user marathon, and cannot be run as any other user ID. If a user tries to launch applications/tasks as non-Marathon user, the application will fail with error “Not authorized to launch as userID”.

  • Page 128: Access The Apache Mesos Web Ui

    The Mesos web UI can be used to monitor components of the Mesos cluster, such as the Mesos slaves, aggregated resources and frameworks. Do not launch applications through Mesos directly because all the frameworks are pre-configured on Urika-GX. Only a few frameworks (Spark and Marathon) are pre-configured to authenticate with Mesos.

  • Page 129: Use Mrun To Retrieve Information About Marathon And Mesos Frameworks

    Mesos Frameworks and Marathon applications and enables specifying how mrun should redirect STDIN. It provides extensive details on running Marathon applications and also enables cancelling/stopping currently active Marathon applications. The Cray Graph Engine (CGE) uses mrun to launch jobs under the Marathon framework on the Urika ® system.
  • Page 130 Resource Management and thus will ask Mesos for ALL the CPUs on the node, not just the number of CPUs per node the user requested to run on. Retrieve a summary of running Marathon applications Use the --brief option of the mrun command to obtain a more concise report on just the running Marathon applications and node availability.
  • Page 131 Resource Management $ mrun --cancel /mrun/cge/user.3750-2016-133-20-01-07.394582 App '/mrun/cge/user.3750-2016-133-20-01-07.394582' does not exist CAUTION: The root user is allowed to use mrun --cancel to kill any Marathon-started job. All other users can only kill the Marathon jobs they launched using the mrun command. If a non-root user tries to use mrun -- cancel to cancel any Marathon job that was not launched by that user using mrun, the system returns the following message: mrun: error: Users may only cancel their own mrun jobs...
  • Page 132 When mrun is invoked, it sets up some internal default values for required settings. mrun will then check if any system defaults have been configured in the /etc/mrun/mrun.conf file. An example mrun.conf file is shown below: # (c) Copyright 2016 Cray Inc. All Rights Reserved. # Anything after an initial hashtag '#' is ignored # Blank lines are ignored.

  • Page 133: Launch An Hpc Job Using Mrun

    YARN nodes. On the Urika-GX system, there are always three Mesos Masters and three Marathon instances running, while one of them is the active leader. Requests received by the login node are proxied to the currently active leader. If a leader runs into issues, one of the backup leaders take over and the requests are proxied to the current leader.
  • Page 134 Resource Management accessed at the port it runs on, i.e. at http://hostname-login1:8080 or http://hostname- login2:8080 Figure 24. Urika-GX Applications Interface S3016...
  • Page 135 Resource Management Figure 25. Marathon UI Marathon also enables creating applications from the UI via the Create Application button, which displays the New Application pop up: Figure 26. Create an Application Using the Marathon UI S3016...

  • Page 136: Manage The Spark Thrift Server As A Non-Admin User

    If using a MAC, the following procedure requires version 10.11 of the operating system. About this task Cray recommends to have the Spark Thrift to be started up by administrators, however, users can use the following instructions if they need to start up their own Spark Thrift server.

  • Page 137: Manage Jobs Using The Cray Application Management Ui

    Manage Jobs Using the Cray Application Management UI Prerequisites Ensure that the system is running in the service mode that allows use of the Cray Application Management UI. Execute the urika-state or urika-service-mode commands to check the service mode. For more...

  • Page 138: Overview Of The Cray Application Management Ui

    5.7.1 Overview of the Cray Application Management UI The Cray Application Management UI is shown in the following figure: Figure 28. Cray Application Management UI The Search field and Quick Filters drop down facilitate searching and filtering submitted jobs, based on the specified criteria.
  • Page 139 Resource Management Figure 29. Filtering by Type ● User- Displays the name of the user who submitted the job. ● Start Time - Displays the time the job started executing. Jobs can be filtered based on starting time using the filter icon provided on the UI.

  • Page 140: Cray Dvs

    The Cray Data Virtualization Service (Cray DVS) is a distributed network service that provides transparent access to file systems residing on the service I/O nodes and remote servers in the data center. Cray DVS provides a service analogous to NFS™. It projects local file systems resident on I/O nodes or remote file servers to compute and service nodes within the Cray system.

  • Page 141: Use Cray Dvs On Urika-Gx

    Cray DVS Administration of Cray DVS is very similar to configuring and mounting any Linux file system. For more information, see the dvs(5) man page. Here is a system administrator's view of Cray DVS. Figure 31. Cray DVS In a Cray System...
  • Page 142 Cray DVS Variable Name Argument Type/Size Purpose DVS_GET_FILE_ATOMIC / signed 16-bit (must be 0 or 1 for Retrieves/sets the atomic option DVS_SET_FILE_ATOMIC SET) value for a file on a DVS mount. DVS_GET_FILE_BLK_SIZE / signed 32-bit (must be > 0 for SET) Retrieves/sets the DVS block size for a file on a DVS mount.

  • Page 143: Dvs Client Mount Point Options

    Cray DVS Variable Name Argument Type/Size Purpose passed through DVS to a remote server. 6.1.3 DVS Client Mount Point Options atomic / noatomic atomic enables atomic stripe parallel mode. This ensures that stripe parallel requests adhere to POSIX read/write atomicity rules. DVS clients send each I/O request to a single DVS server to ensure that the bytes are not interleaved with other requests from DVS clients.
  • Page 144 Associated environment variable: DVS_CACHE ● Additional notes: Cray DVS is not a clustered file system; no coherency is maintained among multiple DVS client nodes reading and writing to the same file. If cache is enabled and data consistency is required, applications must take care to synchronize their accesses to the shared file.
  • Page 145 Cray DVS ● Additional notes: datasync can significantly impact performance. deferopens deferopens defers DVS client open requests to DVS servers for a given set of conditions. When a file is open in stripe parallel mode or atomic stripe parallel mode, DVS clients send the open request to a single DVS server only.
  • Page 146 Cray DVS described by the corresponding retry or noretry option specified for the mount point. ● Default setting: failover or 1 ● Associated environment variable: none ● Additional notes: The failover option cannot be specified at the same time as the noretry option. If all servers fail, operations for the mount point behave as described by the retry option until at least one server is rebooted and has loaded DVS.
  • Page 147 All clients subsequently verify that the server is configured correctly and include the server for that mount point. Many file system magic values are defined in the /usr/include/linux/magic.h file. Commonly used magic values on Cray systems are: 0x6969 GPFS...
  • Page 148 Cray DVS ● Associated environment variable: none ● Additional notes: none maxnodes maxnodes is used in configuring DVS modes. See About DVS Modes page 150. ● Default setting: number of nodes available (nnodes) ● Associated environment variable: DVS_MAXNODES ● Additional notes: none mds specifies which DVS server meta-data operations are sent to.

  • Page 149: Dvs Environment Variables

    Cray DVS ● Default setting: retry or 1 ● Associated environment variable: none ● Additional notes: none ro_cache / no_ro_cache ro_cache enables read-only caching for files on writable mount points. Files opened with read-only permissions in ro_cache mode are treated as if they are on a DVS read-only cached mount point.

  • Page 150: Modes

    6.1.5.1 About DVS Modes There are two primary ways to use Cray DVS: in serial mode or parallel mode, as indicated in the following table. In serial mode, one DVS server on a Cray service node projects a file system to multiple compute node clients. In parallel mode, multiple DVS servers—in configurations that vary in purpose, layout, and performance—project a...
  • Page 151 The following example mount entry contains the mount options essential for serial mode: a single nodename and maxnodes=1. mount -o nodename=server1, maxnodes=1 DVS serial mode adheres to POSIX read/write atomicity rules. Figure 32. Cray DVS Serial Access Mode Cray System Application Application...
  • Page 152 Loadbalance mode is a client access mode for DVS used to more evenly distribute loads across servers. The clients, Cray system compute nodes, automatically select the server based on a DVS-internal node ID (NID) from the list of available server nodes specified on the /etc/fstab line. Loadbalance mode is valid only for read-only mount points.
  • Page 153 This allows attribute-only file system operations to use local attribute data instead of sending the request to the DVS server. This is useful in loadbalance mode because with a read-only file system, attributes are not likely to change. Figure 34. Cray DVS Loadbalance Mode Cray System Application...

  • Page 154: Resiliency And Diagnostics

    This ensures that all clients are informed of server failures and reboots in the same manner at the same time, which reduces the underlying file system coherency traffic associated with rerouting I/O operations away from downed servers and back to rebooted servers. Cray DVS supports failover and failback for parallel modes: S3016...
  • Page 155 Cray DVS ● For cluster, stripe, and atomic stripe parallel modes, add the failover option to the mount line or /etc/fstab entry to specify failover and failback. ● For loadbalance mode, failover and failback are specified by default. DVS failover and failback are done in an active-active manner. Multiple servers must be specified in the /etc/fstab entry for failover and failback to function.
  • Page 156 Cray DVS DVS periodic sync makes the DVS closesync mount option redundant. Periodic sync is more efficient than closesync because it is aware of which files may have dirty pages. Use the following three /proc files to tune DVS periodic sync behavior (echo desired value into each file):...

  • Page 157: Caveats

    Cray DVS ● The third section contains fields for the smallest and largest request sizes (in bytes) for read and write operations, and the number of successful and failed interprocess communication (IPC) requests, IPC asynchronous requests, and IPC replies. In addition, the /proc/fs/dvs/ipc/stats file displays DVS IPC statistics such as bytes transferred and received, NAK counts, and so forth.

  • Page 158: Administrative Tasks

    6.1.7.3 Caveat: Expanded File System Support Setting up and mounting target file systems on Cray service nodes is the sole responsibility of the customer or an agent of the customer. Cray Custom Engineering is available to provide a tailored file system solution. Please contact a Cray service representative for more information.
  • Page 159 Cray DVS 2. Edit /etc/modprobe.d/dvs.conf file on all the DVS server NIDs configured in the cluster to enter the following line : options dvsipc_single_msg_queue=1 3. Restart DVS on all servers. 6.1.8.2 Force a Cache Revalidation on a DVS Mount Point...
  • Page 160 Cray DVS Procedure 1. Edit the /etc/exports file on the DVS server node and add the appropriate export: /xfs/scratch *(rw,no_root_squash,no_subtree_check) 2. Find the attached XFS devices. nid00006# ls -l /dev/disk/by-id/dm-uuid-mpath-360080e50002ff41a000004* lrwxrwxrwx 1 root root 10 May 3 10:29 /dev/disk/by-id/dm-uuid- mpath-360080e50002ff41a0000040b517a2609 \ ->...
  • Page 161 Cray DVS a. Edit the /etc/lvm/lvm.conf file. node/6# vi -n 6 /etc/lvm/lvm.conf b. Add the following custom filter to the configuration file: filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r/.*/" ] c. Remove any previous filters from the configuration file. 5. Edit the /etc/sysconfig/lvm file to append the following string.
  • Page 162 /proc file. Changes to a modprobe.d file are made prior to booting the affected nodes, and the changes take effect at boot. The dvs.conf file is one of many files that are generated automatically and controlled by the Cray Configuration Management Framework (CMF). Such files can be identified by the warning statement in the file header.
  • Page 163 Cray DVS Procedure 1. Create and change to the directory structure that is to be replicated on the target node. smw# mkdir -p etc/modprobe.d smw# cd etc/modprobe.d 2. Create and change to the directory structure that is to be replicated on the target node.
  • Page 164 Cray DVS hostname# echo 0 > /proc/fs/dvs/request_log hostname# echo 1 > /proc/fs/dvs/request_log hostname# echo 2 > /proc/fs/dvs/request_log The value "2" resets the log. dvs_request_log_size_kb Size (KB) of the request log buffer. ● Default value: 16384 KB (16384 * 1024 bytes) ●...
  • Page 165 Cray DVS Field Definition init_free_qhdrs Low water mark for the pool of unused qhdrs. When the pool falls below this number, more are allocated. Used only if single_msg_queue = 0. max_free_qhdrs Maximum number of unused qhdrs that can exist before the system starts freeing them. Used only if single_msg_queue = 0.
  • Page 166 Cray DVS dvsipc_config_type Forces DVS to load in a mode optimized for DVS clients (0) or servers (1). This parameter can be used to make DVS load in a non-default manner. Frequently used for repurposed compute nodes. ● Default value: 0 for compute nodes, 1 for service nodes ●...
  • Page 167 Cray DVS ● To change prior to boot, add these lines to <nid_of_choice>/etc/modprobe.d/dvs-local.conf: # Disable concurrent reads #options dvs dvsof_concurrent_reads=-1 # Set number of threads able to do concurrent # reads = number of cores on CPU options dvs dvsof_concurrent_reads=0 # Set number of threads able to do concurrent # reads = a positive number (e.g., 3)
  • Page 168 Cray DVS # Enable DVS statistics #options dvsproc dvsproc_stat_control=1 ● To change dynamically: This is root writable at /sys/module/dvsproc/parameters/dvsproc_stat_control, but changes should be made only through the /proc/fs/dvs/stats interface, as shown in this example. hostname# echo 0 > /proc/fs/dvs/stats hostname# echo 1 > /proc/fs/dvs/stats hostname# echo 2 >...
  • Page 169 Cray DVS ● To change dynamically (example changes estale_max_retry to 40 for illustration only): hostname# echo 40 > /proc/fs/dvs/estale_timeout_secs estale_timeout_secs Controls the time to wait between retries of an operation after it returns ESTALE. ● Default value: 300 seconds ●...
  • Page 170 Cray DVS Field Definition max_free_qhdrs Maximum number of unused qhdrs that can exist before the system starts freeing them. Used only if single_msg_queue = 0. Interactions among fields: thread_min <= thread_max <= thread_limit (set all three equal for best DataWarp performance) thread_concurrent_creates <= thread_limit...
  • Page 171 Cray DVS sync_dirty_timeout_secs On DVS servers, specifies the number of seconds that must have passed since the file was written before DVS syncs it. The objective is to reduce unnecessary sync operations for files actively being updated. Decreasing this number increases the likelihood that the file is in use when it is synced.
  • Page 172 Cray DVS On DVS clients, specifies the number of seconds between checks for dirty files that need to request the last sync time from the server. This parameter is part of the periodic sync feature. ● Default value: 300 (server), 600 (client) ●...
  • Page 173 Cray DVS When a DVS server receives a request from a client, DVS checks the request path against the list of quiesced directories. The comparison between the path name in the request and the quiesced directory is a simple string compare to avoid any access of the underlying file system that has been quiesced.
  • Page 174 Cray DVS 2. Ensure that the directory was properly quiesced and see if there are any outstanding requests. Repeat this occasionally to know when all outstanding requests have been cleared. dvs1# cat /proc/fs/dvs/quiesce /gpfs/test/foo/: Outstanding_Requests 3 3. Unquiesce the directory when finished with it.

  • Page 175: Security

    GX. A user without an entry, even if that user can pass the standard Linux authentication checks, will not be permitted to log into a Urika-GX. One mode of access defined within the Urika-GX authorized user list is access to tenant VMs.
  • Page 176 Urika-GX login nodes. If a relaxed access user is authorized as a member of a tenant, that user may also log into that tenant environment, where, as a tenant user, that user may also gain access to selected physical node services through the tenant proxy mechanism.

  • Page 177: Urika-Gx Service Modes

    In the simple authorization mode, the HDFS network protocol provides no authentication mechanism, so it is subject to spoofing by code that can emulate the HDFS client protocol on the internal Urika-GX network. Adding the Kerberos authentication layer to the HDFS protocol adds strong authentication, but limits use of HDFS to applications written to use Kerberos with HDFS.
  • Page 178 Table 18. List of Services Available Under the Default and Secure Service Modes Service Available in Default Service Available in Secure Service Mode Mode Cray Programming Environment SELinux Analytic Applications and Resource Management Tools ZooKeeper Spark Mesos Master Mesos Slave...
  • Page 179 Connectivity to Tableau Docker Any additional services installed on the system will use their own security mechanisms and will not be affected by Urika-GX's default and secure modes. Table 19. Relationship Between Access Levels and Service Modes Mode Restricted Access...

  • Page 180: Modify The Service Mode

    Urika-GX services before it changes the service mode. The admin will then need to start Urika-GX services using the urika-start command, unless the -- restart option is provided. For more information, refer to the urika-service-mode man page.

  • Page 181: User Interface Access In The Secure Service Mode

    ● 7.2.2 User Interface Access in the Secure Service Mode The Urika-GX system returns a number of error messages when users attempt to access application UIs in the secure mode. Direct access to a UI Each service is designated a specific port number that it runs on, while the system is operating in the default service mode.

  • Page 182: Set Up Passwordless Ssh

    ● User access level - Determines if a given user is constrained to working within a tenant Virtual Machine (restricted access) or permitted access to physical nodes on the Urika-GX (relaxed access) ● System service mode - Determines if the system is strictly enforcing policies (secure mode) or relaxing...

  • Page 183: Tenancy

    $ utp-launch true; echo $? Tenancy On Urika-GX, tenancy refers to the ability to host users inside of a virtualized environment that is isolated from physical cluster resources, while providing access to selected physical node services through a proxy mechanism. The intent of tenancy is to contain users who are leasing time or resources on Urika-GX and keep them separated from site users who are permitted access to physical nodes, such as a login node.
  • Page 184 IMPORTANT: The env command does not permit any command line arguments, so it does not support command execution, and only displays the environment. All the Cray Graph Engine (CGE) CLI commands, mrun and yam (commands for flexing the cluster) commands cannot be executed from within tenants in this release.

  • Page 185: Configure A Bridge Port

    The Spark and HDFS commands are all provided on the tenant VMs using a wrapper around the utp-launch command that is used to invoke commands through the Urika-GX tenant proxy, so users can use these commands as though they were logged onto a physical node. The two additional commands are not...
  • Page 186 NETMASK=255.255.0.0 About Bridge Port 1 (Operational Network) br1 does not exist by default when Urika-GX tenant management software is installed. The administrator can create a bare-bones network configuration for br1 or produce a fully functioning bridge configuration associated with network port 1's configuration.
  • Page 187 Execution of the utm-host-net command with the --opns-enable-dryrun option is shown below: $ utm-host-net --opns-enable-dryrun \ 172.30.51.152,xx,255.255.240.0,172.30.48.1,172.30.84.40,172.30.84.40,us.cray.com Urika Tenant Management network configuration dryrun mode: no changes will be made to the system...
  • Page 188 ------------------------ DEVICE=br1 TYPE=Bridge BOOTPROTO=static ONBOOT=yes NM_CONTROLLED=no IPADDR=172.30.51.152 NETMASK=255.255.240.0 GATEWAY=172.30.48.1 DNS1=172.30.84.40 DNS2=172.30.84.40 DOMAIN=us.cray.com ifcfg-enp7s0f1 ------------------------ NAME="enp7s0f1" DEVICE="enp7s0f1" ONBOOT=yes BOOTPROTO=static TYPE=Ethernet NM_CONTROLLED=no BRIDGE=br1 The --opns-enable-dryrun option is used in the preceding example only to indicate that the files requested by the user will be generated by the utm-host-net command. This option should be replaced with --opns-enable during the actual execution of the utm-host-net command.

  • Page 189: Tenant Management

    Urika-GX ships with a sample configuration for a tenant named default. While default can be deployed on the system as-is, it is not deployed on Urika-GX when the system is shipped. Use the default tenant's configuration as a template for setting up, configuring and naming a tenant as needed.
  • Page 190 In the preceding code block: ○ UXTENANT_TENANT_MGMT_IP_ADDR is the address of the tenant VM on the Urika-GX management Ethernet. This should be an address in the range 10.142.150.1 to 10.142.255.254 and must be unique among all tenant configurations on a single Urika-GX.
  • Page 191 UXTENANT_MOUNT_SERVER is the IP address (either named or numeric) of the NFS server exporting this mount point. If the server is not a local Urika-GX node, it is best to use a numeric address in case a DNS outage causes Urika-GX to lose the ability to resolve the server name, which can cause NFS outages. If...
  • Page 192 (if one does not already exist) and set up a specific export to each tenant VM for this mount point on the server. If root on Urika-GX does not have privileges on the server, this must be set to NO or creation of tenants using this mount point will fail.
  • Page 193 Unlike other host-names or IP addresses, host-os is specifically interpreted by Urika-GX tenant management to refer to the host where the VM is defined so that operations can be done locally on that server to set up access to the NFS export across the virtual host (VHOST) network on that node.
  • Page 194 A tenant VM that is no longer required can be removed by executing the ux-tenant-remove command as root on the SMW. This only removes the tenant VM and its related changes to Urika-GX, it does not remove the tenant configuration for the tenant VM. Use the ux-tenant-create command to recreate a deleted tenant. To remove...

  • Page 195: Tenant Virtual Machine States

    VM. It is worth noting that the tenant VM itself has limited storage resources, with the expectation that user data will be stored either within Urika-GX (Lustre or HDFS) or within user home directories. As long as user data are not stored on the tenant VM, and administrators keep careful track of local customization of existing tenant VMs, then it is fairly straightforward to re-deploy (using ux-tenant-remove and ux-tenant- create) a tenant that becomes damaged or badly out of date.

  • Page 196: Tenant Management Cli Commands

    Security From Command Used for performing the transition / Due to running crashed Kernel or VM software failure crashed shutoff ux-tenant-stop shutoff running ux-tenant-start shutoff notfound ux-tenant-remove inshutdown shutoff This transition occurs automatically. CAUTION: Tenant VMs are not robust to node redeployment or replacement. Therefore, it is importunate to note that tenant VMs will go into the notfound state if a node is redeployed or replaced.
  • Page 197 Security Table 22. Tenant Management CLI Tools Classification/Purpose Command Name Description Tenant creation ux-tenant-create Creates a VM tenant based on a configuration template. Tenant management ux-tenant-alter-vm Alters the tenant virtual machine (VM) resources, such as memory and CPU allocation size. Can also be used to grow the an existing VM's root disk image size.

  • Page 198: Execution Of Lustre Sub-Commands Inside Tenant Vms

    Security Classification/Purpose Command Name Description Tenant Monitoring ux-tenant-status Provides a view of the tenant VM state ux-tenant-validate Ensures that the physical VM host node is functional User Management ux-tenant-add-user Adds a user to a VM tenant or group of tenants ux-tenant-restrict Configures a user to use a restricted shell...

  • Page 199: Get Started With Tenant Management

    Security These commands are subject to the following set of rules to ensure the isolation of tenants within the Urika-GX multi tenant environment. ● The lfs command may not be used in its interactive mode, normally reached by invoking lfs without any sub-commands.
  • Page 200 Security Figure 38. Tenant Management Workflows Each command serves a different purpose. For example, to create a new tenant VM, the administrator would log on the SMW as root, execute the ux-tenant-create command to create the new tenant and then execute the ux-tenant-status command to verify the tenant was created.
  • Page 201 Security 6. Add/remove users (that have already been added to the site’s LDAP directory via site specific procedures) to the tenant if required by performing the following steps: The steps to add users depend on the security mode the system is running under. Execute the urika- service-mode command to identify the system's service mode.

  • Page 202: Multi-Tenancy

    7.5.7 Multi-Tenancy Urika-GX 2.1 provides secure multi-tenant operation in addition to default mode operation Although Urika-GX enables users to use a single tenant infrastructure, users can also opt to use multi-tenancy, which features: ● File system and data isolation - All relevant file systems are available to tenants, including but not limited to NFS, Lustre and HDFS.

  • Page 203: Multi-Tenant Hdfs

    Security Securing Tenant Isolated Directories To secure a tenant isolated directory it is necessary to secure the directory both against unauthorized detection and against unauthorized use. Securing tenant isolated directories against unauthorized observation involves securing the parent of the tenant isolated directory against reading by tenants while permitting the parent directory to be traversed in a pathname lookup.
  • Page 204 In Urika-GX's multi-tenant set up, each tenant has a dedicated NameNode, which includes the HDFS configuration specific to that tenant NameNode. For a restricted user in secure mode, the Urika-GX tenant proxy is responsible for injecting the correct configuration parameters. Tenants can only interact with their own designated NameNode.

  • Page 205: Authorized User Management

    ACL with the tenant membership reported by Urika-GX tenant management. Authorized User Management Urika-GX Login Authorization Once authenticated, a user must be present in the Urika-GX authorized user list to be permitted any mode of access to the Urika-GX. NOTE:...
  • Page 206 - remove a user from tenant membership or from the authorized user list. ● usm-sync-users - enable Urika-GX to discover new users (that have been added via site-specific procedures) and create Mesos and Kerberos credentials for them. This command also removes secrets of user accounts that are no longer authorized.
  • Page 207 Once a user is entered in the authorized user list and granted either tenant membership or relaxed access to Urika-GX, the user is able to log into Urika-GX , either in a tenant VM or on a physical login node. The user still will not have the ability to launch jobs under Mesos or (in the secure service mode) to access HDFS.

  • Page 208: Guidance On Ldap Forwarding

    The Urika-GX cluster generally relies on LDAP to make user authentication and basic Linux authorization data available to physical nodes and to the tenant VM. Urika-GX ships with a local LDAP server running on login1, with a minimal set of users needed for sanity testing. If the local LDAP is examined or edited with an LDAP browser, it is important to note that the two organizational units that contain standard posix user and posix group information are the ou=users and ou=groups trees located under dc=urika,dc=com.
  • Page 209 Sites using a central LDAP server for standard Linux user authentication and authorization will likely want an efficient way to import users from that LDAP into their Urika-GX authorized user list. This can be scripted using the ux-tenant-add-user, ux-tenant-remove-user, ux-tenant-restrict, and ux-tenant-relax commands.
  • Page 210 Security port="-p ${3}" if [ ! -z "${4}" ]; then query="-t ${4}" user_field="${4}" search="ldapsearch ${host} ${port} -x ${search_dn} ${query}" grep="grep ^${user_field}:[[:space:]]" sed="sed -e s/^${user_field}:[[:space:]][[:space:]]*//" out="$(${search} | ${grep} | ${sed})" if [ $? -ne 0 ]; then return 1 echo ${out} return 0 # Read the shell attribute for the specified user from LDAP and return # the value as a string...
  • Page 211 "" echo " The following example command will import users (in dry-run mode) from" echo " the LDAP server at cfdcg02.us.cray.com using the table with the DN" echo " 'ou=people,dc=datacenter,dc=cray,dc=com' assuming that the field name" echo " for the user name is the default ('uid') and the field name for login"...
  • Page 212 Security usage "Host must be specified using the -H option" if [ -z "${search_dn}" ]; then usage "Search DN must be specified using the -b option" if ! users="$(get_users "${host}" "${search_dn}" "${port}" "${user_field}")"; then echo "ERROR: looking up user list from '${host}' at '${search_dn}' failed." >&2 exit 1 for u in ${users};...
  • Page 213 If the site LDAP has a way to distinguish users with access to Urika-GX using some user attribute, group membership, etc., querying might return all the users in a particular group or set of groups instead of simply taking the complete list of users.
  • Page 214 Security search="ldapsearch ${host} ${port} -x ${search_dn} ${query}" grep="grep ^${shell_field}:[[:space:]]" sed="sed -e s/${shell_field}:[[:space:]][[:space:]]*//" if ! out="$(${search} | ${grep} | ${sed})"; then return 1 echo ${out} return 0 Again, the parameters to the function specify the information needed to set up the ldapsearch command. The ldapsearch results are then filtered and processed by grep and sed to produce the loginShell value.
  • Page 215 The preceding form of the command is required to remove a user entirely from the Urika-GX authorized user list.

  • Page 216: Authentication Mechanisms

    Authentication Mechanisms Table 23. Authentication Mechanisms Application Authentication Mechanism Cray Application Management UI LDAP. Users can also log in with the default account shipped with the system. This account has the following credentials: username: admin...

  • Page 217: Change Default Passwords

    Change the Default System Management Workstation (SMW) Passwords on page 223. Change the iDRAC's Password Follow the instructions documented in Change the Default iDRAC8 Password on page 221 Change the Password of the Cray Application management UI Default credentials: ● username: admin ● password: admin Select Change Password from the admin drop down menu to change the default password.
  • Page 218 Table 24. Enabled System Accounts Account Name Default Password builder initial0 marathon initial0 Passwords for both the builder and marathon accounts can be changed using the passwd command. LDAP Accounts Credentials for Urika-GX's internal LDAP accounts are: ● Admin account credentials: S3016...

  • Page 219: Default Urika-Gx System Accounts

    Account Name Password root initial0 crayadm crayadm The SMW account should not be set up with the local Urika-GX LDAP for security. The SMW can be connected directly to the corporate LDAP. Table 26. Default iDRAC Account Account Name Password root...

  • Page 220: Change The Default Nagios Password

    LDAP Credentials for Urika-GX's internal LDAP accounts are: ● Admin account credentials: ○ username: crayadm ○...

  • Page 221: Change The Default Idrac8 Password

    Security 7.9.2 Change the Default Nagios Password Prerequisites This procedure requires root privileges. About this task The default credentials for login on to the Nagios UI on the Urika-GX system are: ● User name: crayadm ● Password: initial0 Procedure 1. Log on to the SMW as root.
  • Page 222 Security Procedure 1. Bring up a web browser. 2. Go to: https://cray-drac, where cray-drac is used as an example for the iDRAC's name, such as https:// system-smw-ras The iDRAC's login screen appears. 3. Enter root and initial0 as the default user name and password on the iDRAC's log in screen.

  • Page 223: Change The Default System Management Workstation (Smw) Passwords

    Prerequisites Ensure that the SMW is accessible. This procedure requires root access. About this task After logging on to the SMW for the first time, Cray recommends changing the default passwords, as described in the following instructions. Procedure 1. Log in to SMW as root.

  • Page 224: Change Ldap Password On Urika-Gx

    3. Update the password in the same manner on all the nodes. For rack-mount SMWs, such as that used in the Urika-GX system, it is also necessary to change the default iDRAC password. 7.9.5...

  • Page 225: Reset A Forgotten Password For The Cray Application Management Ui

    ● requires root privileges. ● assumes that the system is running Urika-GX 1.2UP01 or an earlier release. ● assumes that the LDAP host server is running on nid00030, which is login node 1 on a 48 node system. About this task This procedure provides instructions for updating the LDAP admin server password for the root domain name, i.e., cn=crayadm,dc=urika,dc=com.

  • Page 226: Reset An Administrator Ldap Password When The Olc Schema Password Is Unknown

    Security Procedure 1. Log on to the LDAP host server as root. 2. Generate a new hashed password. # slappasswd New password: Re-enter new password: {SSHA}ZNYj4jyMpTo3xfln0lxpirj0ZyuKVa24 3. Make a back up copy of the /usr/local/openldap/etc/openldap/slapd.conf file. 4. Edit the /usr/local/openldap/etc/openldap/slapd.conf file, replacing values for the two 'rootpw' entries with the hashed password output of the slappasswd command.
  • Page 227 LDAP schema to allow for roots uid and gid to perform the necessary ldif operations, however this is not configured in the default Urika-GX installation, therefore, setting this up for existing systems requires knowing the OLC admin password.

  • Page 228: Reset An Administrator Ldap Password When The Olc Scheme Password Is Known

    The cn=admin,cn=config LDAP OLC schema password needs to be known while carrying out this procedure. ● This procedure requires Urika-GX 2.0UP00 or latter installed on the system. About this task This procedure provides instructions for updating the LDAP admin server password for the root domain name, i.e., cn=crayadm,dc=urika,dc=com.

  • Page 229: Tableau Authorization And Authentication Mechanisms

    SSL authentication for Tableau can be set up using instructions documented in Enable SSL on Urika-GX. Urika- GX ships without authorization enabled. To enable storage based authorization for connecting to HiveServer2, follow the instructions documented by Hive, visit https://cwiki.apache.org...
  • Page 230 3. Edit the settings for the Urika Applications Interface by uncommenting some settings to enable SSL. In the following instructions, it is assumed that the SSL certificate is being installed on Urika-GX system containing 48 nodes and nid00030 is used as the node ID of login node 1.
  • Page 231 Security 2. Uncomment the following lines in /etc/haproxy/haproxy.cfg configuration file and replace all occurrences of /etc/ssl/certs/filename.pem with the full path to the SSL certificate. frontend hive_ssl bind *:29207 ssl crt /etc/hive/conf/server.pem mode tcp option forwardfor reqadd X-Forwarded-Proto:\ https default_backend hive_ssl_backend backend hive_ssl_backend mode tcp balance source...
  • Page 232 Security option http-server-close option httpclose reqadd X-Forwarded-Proto:\ https default_backend jupyter_ssl_backend backend jupyter_ssl_backend mode http balance source server server4 192.168.0.31:7800 frontend urika-app-magmt_ssl bind *:29203 ssl crt /etc/ssl/certs/filename.pem mode http option forwardfor option http-server-close option httpclose reqadd X-Forwarded-Proto:\ https reqadd X-Forwarded-Protocol:\ https default_backend urika-app-magmt_ssl_backend backend urika-app-magmt_ssl_backend mode http...
  • Page 233 This allows the Urika-GX Applications Interface page to load secure URLs (configured in the preceding steps) when the HUE/Grafana/Jupyter/Urika-GX Application Management UIs are accessed from the Urika-GX Applications Interface page. If there is any change in the HAProxy port numbers, the following URLs in settings.py need to be updated:...

  • Page 234: Enable Ssl For Spark Thrift Server Of A Tenant

    Security apache> Require all granted </Directory> <VirtualHost *:80> Redirect /applications https://hostname-login1.us.cray.com:29203/ applications </VirtualHost> Listen 8000 <VirtualHost *:8000> </VirtualHost> <Directory /opt/cray/ui-application-management/default/application_management/ app_monitoring/static> Require all granted </Directory> 11. Restart Apache on login node 1. # service httpd restart 12. SSH on to the SMW.

  • Page 235: Install A Trusted Ssl Certificate On Urika-Gx

    This procedure provides instructions for installing a SSL certificate that has been issued by a trusted Certificate Authority (CA). In the following instructions, nid00030 is used as an example for login node 1's ID. Replace 'hostname-login1.us.cray.com' in the following examples with the FQDN of login node 1. Procedure 1.

  • Page 236: Enable Ldap Authentication On Urika-Gx

    This procedure requires root access. ● Ensure that the storage LDAP client points at login node 1, which is the LDAP server on Urika-GX. This ensures that the Urika-GX system and storage are authenticating to the same source. NOTE: This examples used in this procedure are intended for a 3 sub-rack system. Replace node IDs as needed when executing the following commands if using a system containing less than 3 sub-racks.
  • Page 237 -GX node. In the following instructions, login-1 is used as an example for the name of login node 1, which is where the LDAP service runs. CAUTION: Cray Support should be involved for all LDAP related procedures and changes. Procedure 1.

  • Page 238: Enable Ldap For Connecting Tableau To Hiveserver2

    Verify that all LDAP users are listed when the ldapsearch command is executed for all entries: [root@nid00030 ~]# ldapsearch -h 127.0.0.1 -D "cn=crayadm,dc=local" -w initial0 -b "dc=local" For information about internal LDAP or the LDAP server, contact Cray Support. For advanced configuration settings, see the OpenLDAP Software Administrator’s guide at http://www.openldap.org.

  • Page 239: Enable Sql Standard Based Authorization For Hiveserver2

    Prerequisites This procedure requires root privileges. About this task SQL standard based authentication for connecting to HiverServer2 is not enabled by default on Urika-GX. Follow the instructions in this procedure to enable it. Procedure 1. Log on to the SMW as root.

  • Page 240: File System Permissions

    Network File System (NFS) - The SMW provides the NFS mount, which has the same groups and user permissions as those of the host operating system. ● External file system (Lustre) - If Lustre is used on Urika-GX, it inherits Linux group permissions from the login node. S3016...

  • Page 241: Urika-Gx Security Quick Reference Information

    Configuration of LDAP Settings on Urika-GX The Open LDAP server on Urika-GX runs on login node 1. For example, if the system's hostname is hostnatme, then the Open LDAP service would run on hostname-login1. The particular ID of the login node (such as nid00030 on a 48 node system) would change, based on the number of nodes in the system i.e.

  • Page 242: Port Assignments

    Security 7.18 Port Assignments Table 30. Services Running on the System Management Workstation (SMW) Service Name Default Port Table 31. Services Running on the I/O Nodes Service Name Default Port Table 32. Services Running on the Compute Nodes Service Name Default Port YARN Node Managers 8040, 8042, 45454, and 13562...
  • Page 243 8086 on login2. InfluxDB runs on nid00046 on three sub-rack, and on nid00030 on a two sub-rack system. InfluxDB port for listening for 2003. InfluxDB runs on login node 2 on the Urika-GX system. collectl daemons on compute nodes InfluxDB cluster communication...
  • Page 244 Security Service Port Kafka (not configured by default) 9092 Flume (not configured by 41414 default) Port for SSH S3016...

  • Page 245: Troubleshooting

    Located on the SMW syslogs, Controller (RC) at /var/opt/cray/log/controller/rackname forwarded by syslog and HSS Logs of SMW HSS Located at var/opt/cray/log/p0-default on the SMW. The All HSS monitor daemons daemons and following log files are stored in this directory: processes log ●...

  • Page 246: Default Log Settings

    Logging for individual pods/jobs exists in the associated containers. Default Log Settings The following table lists the default log levels of various Urika-GX analytic components. If a restart of the service is needed, please first stop services using the urika-stop command, change the log level, and then restart services using the urika-start command.
  • Page 247 CGE is shut down. Flex scripts: INFO. Changing the log level for these scripts is not supported. ● urika-yam-status ● urika-yam-flexup ● urika-yam-flexdown ● urika-yam-flexdown-all Spark Thrift server INFO. HiverServer2 INFO. Tenant proxy logs DEBUG Urika-GX security manager logs INFO S3016...

  • Page 248: Analytic Applications Log File Locations

    These files can also be accessed via the web UI of the slave daemon. The location of the Spark logs is determined by the cluster resource manager that it runs under, which is Mesos on Urika-GX. Grafana /var/log/grafana/grafana.log InfluxDB /var/log/influxdb/influxd.log...
  • Page 249 Troubleshooting Application/Script Log File Location ● urika-yam-status ● urika-yam- flexdown ● urika-yam- flexdown-all ● urika-yam-flexup ZooKeeper /var/log/zookeeper Hive Metastore /var/log/hive HiveServer2 /var/log/hive /var/log/hue Spark Thrift Server /var/log/spark Spark Audit Logs A per-user Spark audit log that details start and stop of applications is located at /var/log/spark/k8s/username.log with entries of the following form: Tue Apr 03 07:54:05 CDT 2018 username spark-test-1522760043061-driver START \ Application Started with 1 driver plus 5.0 executors using 6.0 cores and 496.0GB memory...

  • Page 250: Security Related Troubleshooting Information

    Security Related Troubleshooting Information User Authorization Issues While logging on to Urika-GX (either a tenant or a physical node), the system may return the following message: You are not authorized to log into this system -- to obtain access please contact your system administrator This can mean several things.
  • Page 251 GX services by executing urika-stop and then executing urika-start to sync up the created Mesos secrets and give the user access to Mesos for job launch. Since stopping and starting Urika-GX services can cause running jobs to be killed, schedule that action carefully.
  • Page 252 Troubleshooting version of the software, the system will return a message of the following form when the ux-tenant-* commands are executed: The following configuration settings are no longer used and can be removed from the Urika Tenant Management configuration: UXTENANT_GLOBAL_LDAP_DEFAULT_TENANT_USER_PASS UXTENANT_GLOBAL_LDAP_GROUP_OU Clean up all references to these to avoid seeing this warning again.
  • Page 253 TYPE=Bridge BOOTPROTO=static ONBOOT=yes NM_CONTROLLED=no IPADDR=172.30.51.237 NETMASK=255.255.240.0 GATEWAY=172.30.48.1 DNS1=172.30.84.40 DNS2=172.31.84.40 DOMAIN=us.cray.com Then look at ifcfg-enp8s0f1. This is the configuration for the Ethernet device plugged into the public network and should look something like this: NAME="enp8s0f1" DEVICE="enp8s0f1" ONBOOT=yes BOOTPROTO=static TYPE=Ethernet NM_CONTROLLED=no BRIDGE=br1 Notice a few different things about this interface config: ●...

  • Page 254: Save And Restore Tenant Information

    About this task When making changes to Urika-GX that result in swapping, re-deploying or wiping out the contents of tenant VM host nodes or the node on which HDFS name nodes reside (NID 0), critical tenant data can be destroyed, resulting in loss of the tenant VM environment or users' HDFS data.
  • Page 255 4. Back up data/files. While the HDFS data on Urika-GX is spread across data nodes and is stored redundantly to prevent loss, the name nodes for both tenant and non-tenant data store the metadata on NID 0. This means that if NID 0 loses its data, the user data may persist, but the ability to identify and retrieve it will be lost.

  • Page 256: Ldap Server Start-Up Issues

    This procedure requires root privileges. About this task The urika-mesos-change-secret command is used to change the secret of Urika-GX service-level Mesos secrets, such as, marathon, haproxy, etc. The following list of items, which is subject to change, needs to be...

  • Page 257: Clean Up Log Data

    Troubleshooting ● All lowercase and uppercase letters and numbers are allowed in secrets. -, _, and . are allowed at any position in the secret ● ! can be used, but should not be the first character ● Other punctuation characters can cause authentication issues and are not recommended ●...

  • Page 258: Diagnose And Troubleshoot Orphaned Mesos Tasks

    Log files are rolled daily by default, but if space is critical, logs can be deleted manually. ● Spark logs - Shuffle data files on the SSDs is automatically deleted on Urika-GX. Spark logs need to be deleted manually and are located at the following locations: ○...

  • Page 259: Troubleshoot Common Analytic And System Management Issues

    Mesos tasks if the CURL call returns a higher number of CPUs used than that returned by the UI. Cray-developed scripts for flexing YARN sub-clusters use curl calls, and hence do not allow flexing up if there are not enough resources reported.
  • Page 260 Troubleshooting Error Message Description Notes/Resolution 'Urika ® -GX System Administration Guide'. Error message: ERROR: tag(s) Description: User has specified a Resolution: Use the correct name of service that does not exist to the the services by selecting one of the not found in playbook: urika-stop or urika-start options listed in the error message.
  • Page 261 Troubleshooting Error Message Description Resolution Lost executor # on host Something has caused the Spark Increase the memory allocated to executor to die. One of the reasons executors via one of the following may be that there is not enough parameters: memory allocated to the executors.
  • Page 262 Troubleshooting Error Message Description Resolution Invalid app name. Your app name This error is seen when the Follow the rules mentioned there can consist of a series of names identifier provided by user for the and re-submit a new flex up separated by slashes.
  • Page 263 Mesos and Marathon daemons are up and running. If any of these daemons are not running for some reason, report the logs to Cray Support and restart the Mesos cluster using the urika-start command. For more information, see the urika-start man page.
  • Page 264 For more information, the system is in the secure service refer to the Urika-GX System mode, instead of in the default Administration Guide. mode. Table 41. Marathon/Mesos/mrun Error Messages...
  • Page 265 Troubleshooting Error Message Description Resolution ○ error("mrun: Getting mrund state threw exception - %s" ○ error("getting marathon controller state threw exception - %s" % ○ error("Unexpected 'apps' data from Marathon") ○ error("mrun: Launching mrund threw exception - %s" % (str(e))) ○...
  • Page 266 For more while the system is in the secure information, refer to the Urika-GX service mode, instead of in the System Administration Guide. default mode. S3016...
  • Page 267 HDFS space may have reached full To identify the used capacity by RemoteException(java.io.IOExcepti capacity. Even though Urika-GX has storage type, use the following on): \ File /tmp/test could only be a heterogeneous file system, the commands:For both DISK and SSD,...

  • Page 268: Troubleshoot Mrun Issues

    In rare cases, switching from the secure to default mode may result in some Romana network policy information that is not translated into the appropriate IP table rules. This allows a recently created pod to ping a pod in a different Kubernetes name space. Contact Cray support if this problem is encountered. Troubleshoot mrun Issues...
  • Page 269 Troubleshooting ○ error("Unexpected 'frameworks' data from Mesos") ○ error("mrun: Getting mrund state threw exception - %s" % ) ○ error("getting marathon controller state threw exception - %s" %) ○ error("Unexpected 'apps' data from Marathon") ○ error("mrun: Launching mrund threw exception - %s" % (str(e))) ○...

  • Page 270: Troubleshoot: Application Hangs As A Result Of Nfs File Locking

    About this task If DVS fails after the Cray system's data store is moved to a shared external Lustre file system, verify that DVS has the correct lnd_name that uniquely identifies the Cray system to the LNet router. The default value for lnd_name on a single-user Lustre file system is gni.

  • Page 271: Troubleshoot: Dvs Ignores User Environment Variables

    To define a large number of user environment variables, Cray recommends that users include those definitions in the user's shell so that they are available at startup and stored where DVS can always locate them.

  • Page 272: Remove Temporary Spark Files From Ssds

    Urika-GX checks for any idle nodes once per hour, and cleans up any left over temporary files. This is handled by a cron job running on one of the login nodes that executes the /usr/sbin/cleanupssds.sh script once per hour.

Table of Contents