Do you have a question about the 380578-B21 and is the answer not in the manual?
Questions and answers
Summary of Contents for TruCluster 380578-B21
Page 1
TruCluster Software Products Hardware Configuration Technical Update for Fibre Channel for TruCluster Software Products Version 1.6 July 2000 Product Version: TruCluster Production Server Software Version 1.6 and TruCluster Available Server Software Version 1.6 Operating System and Version: Tru64 UNIX Version 4.0F (with NHD-2) This technical update is an update to the Fibre Channel information that is provided in the TruCluster™...
Page 5
Mirroring Storage Units with LSM in a Cluster ....1–6 Single Switch Configuration with Full Complement ... . 1–7 Arbitrated Loop Configuration with One Storage Array ..
About This Technical Update The material in this technical update provides an update to the information that is provided in Chapter 6 (Fibre Channel) of the Version 1.6 TruCluster Software Products Hardware Configuration manual. It also updates the previous version of this technical update dated 9 February 2000. This update is required because of new support for Fibre Channel arbitrated loop.
• Chapter 2: Fibre Channel Hardware, Requirements, and Restrictions — Provides information regarding the supported Fibre Channel hardware and firmware, and the configuration requirements and restrictions. • Chapter 3: Using the emx Manager (emxmgr) Utility — Provides instruction on the use of the emxmgr utility. •...
Page 9
Hardware Support, but because no formal documentation is provided with the NHD-2 release, the refererence page is included in Appendix A of this update. See the following Compaq StorageWorks documentation for switch and HSG80 RAID array controller configuration information: •...
______________________ The TruCluster Software Products support 500 meters between the KGPSA and the switch or hub, and 500 meters between the switch or hub and the HSG80 array controller. See Figure 1–4 for an example cluster configuration. 1.1 Basic Fibre Channel Terminology...
Page 12
(worldwide name) that is built into the node when it is manufactured. Fabric A switch, or multiple interconnected switches, that route frames between the originator node (transmitter) and destination node (receiver). Fabrics do not originate nor are they the final recipient of frames;...
A link may be optical fibre, coaxial cable, or shielded twisted pair. E_Port An expansion port on a switch that connects two switches in the fabric. 1.2 Fibre Channel Topologies Fibre Channel supports three different interconnect topologies: •...
A fabric may consist of a single switch, or several interconnected switches. Each switch contains two or more fabric ports (F_Port) that are interconnected by the fabric switching function, which routes the frame from one F_Port to another F_Port within the fabric.
Figure 1–2: Fabric Topology Node 1 Transmit Transmit N_Port Receive Receive Node 2 Transmit Transmit N_Port Receive Receive 1.2.3 Arbitrated Loop Topology In an arbitrated loop topology, frames are routed around a loop that is set up by the links between the nodes. The hub maintains loop continuity by bypassing a node when the node or its cabling fails, when the node is powered down, or when the node is removed for maintenance.
Arbitrated loop reduces Fibre Channel cost by substituting a lower-cost, often nonintelligent and unmanaged hub, for a more expensive switch. The hub operates by collapsing the physical loop into a logical star. The cables, associated connectors, and allowable cable lengths are similar to those of a fabric.
Although more expensive, the fabric topology provides both increased connectivity and higher performance; switches provide a full-duplex 100 (200) MB/sec point-to-point connection to the fabric. Switches also provide improved performance and scaling because nodes on the fabric see only data destined for themselves, and individual nodes are isolated from reconfiguration and error recovery of other nodes within the fabric.
HSG80 controllers (with the limit of a total of 128 storage units). You cannot achieve no-single-point-of-failure (NSPOF) with the configuration shown in Figure 1–4 because the switch represents a single-point-of-failure. You can, however, add the hardware for a second bus (another KGPSA,...
Figure 1–5: Mirroring Storage Units with LSM in a Cluster Member Memory System Channel Interface Memory Channel KGPSA KGPSA Fibre Channel Switch HSG 80 Controller A Port 1 Port 2 HSG 80 Controller B Port 1 Port 2 RA8000/ESA12000 Member...
Figure 1–6 shows a cluster with a full complement of eight targets on one Fibre Channel switch. Each member system represents one target, and each dual-redundant HSG80 port pair represents one target. Figure 1–6: Single Switch Configuration with Full Complement...
Figure 1–7 shows a two-node Fibre Channel arbitrated loop cluster with a single RA8000 or ESA12000 storage array with dual-redundant HSG80 controllers and an SWXHB-07 Fibre Channel hub. Figure 1–7: Arbitrated Loop Configuration with One Storage Array Member Member Memory System System Channel...
Figure 1–8 shows the maximum supported arbitrated loop configuration of a two-node Fibre Channel cluster with two RA8000 or ESA12000 storage arrays, each with dual-redundant HSG80 controllers and two SWXHB-07 Fibre Channel hubs. Figure 1–8: Arbitrated Loop Maximum Configuration Member System Memory Channel...
Zoning allows partitioning of resources for management and access control. In some configurations, it may provide for more efficient use of hardware resources by allowing one switch to serve multiple clusters or multiple operating systems. Figure 1–9 provides an example configuration using zoning. This configuration consists of two independent zones with each zone containing an independent cluster.
______________________ Only static zoning is supported; zones can only be changed when all connected systems are shut down. For information on setting up zoning, see the SAN Switch Zoning documentation that is provided with the switch. 1.5.2 Cascaded Switches Multiple switches may be connected to each other. When cascading switches, a maximum of three switches is supported, with a maximum of two hops between switches.
KGPSA-CA adapter is supported in both fabric and arbitrated loop configurations. • A host is only allowed to have one KGPSA connected to each switch, switch zone, or hub. • The TruCluster Software Products support a fabric configuration of up to three cascaded switches with a maximum of two hops between switches.
Page 27
SCSI targets allocated to the HSG80 ports, depending on the application requirements. • A storage array with dual-redundant HSG80 controllers is two targets and consumes four ports on a switch or hub. • Each KGPSA is one target. •...
Page 28
right). (The HSG80 documentation refers to these ports as Port 1 and Port 2, respectively.) In transparent failover mode, only one left port (Port 1) and one right port (Port 2) are active at any given time. With transparent failover enabled, assuming that the left port of the top controller and the right port of the bottom controller are active, if the top controller fails, its functions will automatically fail over to the bottom controller and vice versa.
Using the emx Manager (emxmgr) Utility This section introduces the emx manager (emxmgr) utility. You use this utility to synchronize the worldwide name-to-target ID mapping in Chapter 4. The emx manager (emxmgr) utility provides you with the ability to: • Display the presence of KGPSA Fibre Channel adapters •...
Page 30
SCSI tgt id 7 : portname 1000-0000-C921-0D00 The previous example shows four Fibre Channel devices on this SCSI bus. The Fibre Channel adapter in question, emx0, at SCSI ID 7, is denoted by the presence of the emx0 designation. Use the emxmgr -t command to display the Fibre Channel topology for the adapter.
Page 31
Status of the emx1 link. The connection is a point-to-point fabric (switch) connection, and the link is up. The adapter is on SCSI bus 3 at SCSI ID 7. The port name and node name of the adapter are provided.
SCSI ID (3) is assigned to the Fibre Channel device with this worldwide name. This device, and any device that is listed below the dotted line, is not present. This situation can occur when an adapter has been replaced, or disconnected, or when the system powered down. If an adapter has been replaced, you need to modify the worldwide name to target ID mapping to give the new adapter a SCSI ID of 3.
The Fibre Channel DID number is the physical Fibre Channel address that the N_Port uses. 3.2 Using the emxmgr Utility to Modify Worldwide Name-to-Target ID Mapping You can use the emxmgr utility to modify worldwide name-to-target ID mapping in either of two ways: •...
Page 34
If you use the interactive mode to modify worldwide name-to-target ID mapping, you can provide the information interactively or cause the emxmgr utility to read a file from which to obtain the mapping data. When you start the emxmgr utility with no command-line options, the default device used is the first Fibre Channel adapter it finds.
______________________ If you selected f to obtain the new target ID mappings from a file, the emxmgr utility prompts you for the name of the file to use. You can save time if you create a file to modify worldwide name-to-target ID mappings.
Page 36
# /usr/sbin/emxmgr -c /tmp/mapping-modifications Continue with the change in id mappings (y/n) [n] ? y Change in Target id mappings has been performed You can avoid the verification message by using the -f (force) option with the -c option. 3–8 Using the emx Manager (emxmgr) Utility...
• Determining the device special file name for a disk (Section 4.8) Ensure that the member systems, the Fibre Channel switch or hub, and the HSG80 array controllers are placed within the lengths of the optical cables you will be using.
Page 38
HSG80 array controller. Each switch, except the DS-DSGGB-AB, has a front panel display and four push buttons that you can use to manage the switch. There are four menus that allow you to configure, operate, obtain status, or test the switch. You manage the DS-DSGGB-AB through a Telnet session after the IP address has been set (from a PC or terminal).
When you plan the switch location, ensure that you provide access to the front of the switch. All cables plug into the front of the switch. Also, for those switches with a control panel, the display and switches are on the front of the switch.
Before you can make a Telnet connection or access the switch via the Web, you must assign an IP address and subnet mask to the Ethernet connection using the front panel or from a PC or terminal (DS-DSGGB-AA). 4.1.2.1 Using the Switch Front Panel The switch front panel consists of a display and four buttons.
Page 41
Press the Tab/Esc button repeatedly to get back to the configuration menu in the main menu. ____________________ After changing any configuration menu settings, you must reboot the switch for the change to take effect. Press the down button to select the Operation Menu: Select Menu: Operation Menu...
Press the Enter button. You are allowed to change your mind and not reboot: Reboot Accept? 10. Use the Tab/Esc button to select Yes to reboot the switch. Press the Enter button to reboot the switch and execute the POST tests. See the switch documentation for information on other switch configuration settings.
Telnet session is started. You can use a Telnet session to log in to the switch at one of three security levels. The default user names, shown from lowest security level to highest security level, are listed in Table 4–1.
HSG80 array controller. Unlike the DSGGA switch, the SWXHB-07 hub does not have any controls or even a power-on switch. Simply plug in the hub to power it on. The hub has a green power indicator on the front panel.
The GBIC modules and MMF optical cables are not provided with the hub. If you need to obtain them, contact your authorized Compaq service representative. 4.2.1 Installing the Hub Ensure that you place the hub within 500 meters of the member systems (with KGPSA-CA PCI-to-Fibre Channel adapter) and the HSG80 array controllers.
RA8000 or ESA1200 storage arrays. If you use an external cache battery (ECB), ensure that it is connected to the controller cache modules. Install the fiber-optic cables between the switch and the HSG80. 4–10 Installing Fibre Channel Hardware and Configuring the emx Driver...
Page 47
Connect the RA8000 or ESA12000 to the power source and apply power. ____________________ For the HSG80 to see the connection to the KGPSAs, the KGPSA host bus adapters must be cabled to the switch, with the system power applied before you turn power on to the RA8000/ESA12000.
Page 48
> set this CACHE_UPS ____________________ Setting the controller variable CACHE_UPS for one controller sets it for both controllers. 10. From the maintenance terminal, use the show this and show other commands to verify that controllers have the current firmware version. See the HSG80 Array Controller ACS Version 8.4 Maintenance and Service Guide for information on upgrading the firmware.
Specifies that the failover mode is transparent failover. With transparent failover, controller A (the upper controller) port 1 is active for units D0-D99. Controller A port 2 is passive for units D100-D199. Controller B port 1 is passive for units D0-D99 and controller B port 2 is active for units D100-D199.
Page 50
For more information on installing the HSG80 in an arbitrated loop topology, see the HSG80 Array Controller ACS Version 8.5 Configuration Guide. To set up an HSG80 for TruCluster arbitrated loop operation, use the maintenance terminal to set the controller values as follows: Set the PORT_x_TOPOLOGY value to LOOP_HARD.
of a failover. Because the ports are not active at the same time, the AL_PA addresses do not conflict. Do not make the AL_PA address that is assigned to Port 1 the same as that assigned to Port 2 because they are distinct devices on the Fibre Channel loop.
4.5 Installing and Configuring the KGPSA PCI-to-Fibre Channel Adapter Module When you install the KGPSA PCI-to-Fibre Channel adapter module, you have to ensure that it is configured symmetrically. The following sections discuss KGPSA installation and configuration. For an arbitrated loop configuration, you need to configure the link type of the adapter for loop operation, as described in Section 4.5.3.
Connect the fiber-optic cables to the DSGGA switch or hub. 4.5.3 Configuring the KGPSA-CA Adapter for Loop Mode If you do not want to use the KGPSA-CA adapter in loop mode, you can skip this section.
Page 54
KGPSA functions as expected. ____________________ If a KGPSA in loop mode is connected to a Fibre Channel switch, the results are unpredictable. The same is true for a KGPSA in fabric mode that is connected to a loop. Therefore, determine the topology setting before you use the adapter.
The SCSI adapters have to be connected symmetrically, and the KGPSA Fibre Channel adapters (emxn) must also be connected symmetrically. For example, all emx0 adapters must be connected to the same switch, and all emx1 adapters must be connected to the same switch (which is not the same switch to which the emx0 adapters are connected).
Additionally, make sure that all HSG80 controllers have storage sets defined, are powered up, and properly connected to a Fibre Channel switch or hub. Boot all cluster member systems but one to single-user mode as follows: >>> boot -fl s...
Page 57
____________________ Booting all cluster members but one to single-user mode is critical because the member that will be booted to multi-user mode must see the KGPSA adapters of every other member system to create target IDs for them. Boot the remaining cluster member system to multi-user mode. The initial target ID mapping takes place and is written to the emx drivers data and backup files, /etc/emx.db and /etc/emx.db.bak (binary files).
Each member system is cabled symmetrically to each shared storage bus (for example, the emx0 adapter on each member system is cabled to a common switch and emx1 adapter on each member is also cabled to a common switch). The worldwide name-to-target ID mapping thus established persists across member and cluster reboots.
Page 59
# /sbin/lsmbstartup # /sbin/bcheckrc # /sbin/init.d/inet start ____________________ You only have to execute /sbin/lsmbstartup if you are using the Logical Storage Manager (LSM). Make backup copies of the following emx driver files: • /etc/emx.db • /etc/emx.db.bak • /etc/emx.info. Use the emxmgr utility to display the topology for the emx for which changes were made.
Page 60
The new KGPSA adapter has the wrong SCSI ID. The port and node names of the new adapter are provided. You need the new port name and node name for the remapping. The old KGPSA adapter still has worldwide name-to-target ID mapping information in the kernel and database files, but it is not seen.
Continue with the change in id mappings (y/n) [n] ? y Change in Target id mappings has been performed 4.7 Verifying the Fibre Channel Device Configuration After you have synchronized the target ID mapping across cluster members, verify that the Fibre Channel storage has been correctly configured. The following are some suggested methods: •...
Page 62
Thus, if an HSG80 is at target 1 (as determined by /etc/emx_info) on scsi6 (as determined by the system configuration file, which could assign scsi6 to device emx0, which is the first KGPSA adapter), then its LUN 0 is /dev/rrz49 and its LUN 1 is /dev/rrzb49. These device assignments are seldom changed throughout the life of the installation.
The Fibre Channel switch represents a single point of failure (SPOF) for each shared SCSI bus that it serves. A total failure of one switch will cause the entire shared bus to fail. Depending on how the storage is configured, this failure can result in the loss of access to some or all of the data.
In some instances, the switch port may end up stuck in this state, depending on the failure mechanism of the device on that port, such as the device rebooting repeatedly due to power problems, and so on, and the port can be individually reset by entering the portDisable command followed by the portEnable command.
• If you replace one controller of a dual-redundant pair, the NVRAM from the remaining controller retains the configuration information (including WWID). When you install the replacement controller, the existing controller transfers configuration information to the replacement controller. • If you have to replace the HSG80 controller in a single controller configuration, or if you must replace both HSG80 controllers in a dual-redundant configuration simultaneously, you have two options: –...
• Fibre Channel switch configuration requirements: For each additional Fibre Channel based shared bus, an additional Fibre Channel switch is required. Each Fibre Channel switch will require a 10Mbit/sec twisted-pair Ethernet connection to the network, an IP address, and sufficient ports to support the configuration.
5.3.2 Adding or Moving Storage Units You create and manipulate storage units (in the form of storagesets) using ACS commands at the HSG80 array controller. Each unit has a unique LUN, which the operating system uses to address the target of an I/O request. For the most part, you can manipulate existing storage units in a cluster as long as you take care to coordinate any changes with the LSM, file system, or application use of those storagesets.
Page 69
Link is SCSI bus 3 (e.g. scsi3) SCSI target id 7 portname is 1000-0000-C921-07C4 nodename is 1000-0000-C921-07C4 N_Port at FC DID 0x210013 - SCSI tgt id 5 : portname 5000-1FE1-0001-8932 nodename 5000-1FE1-0001-8930 Present, Logged in, FCP Target, FCP Logged in, N_Port at FC DID 0x210113 - SCSI tgt id 1 : portname 5000-1FE1-0001-8931 nodename 5000-1FE1-0001-8930...
An additional KGPSA adapter will be added to each cluster member as well, along with a Fibre Channel switch or hub, HSG80 controllers, and cabling required for the storage.
(see Section 4.1.2.2), and after the switch is up, you can access it via Telnet or the Web. Then, you can enter other attributes, such as the switch name, and verify the connections to it by using the switchShow command and other commands.
Page 72
(upper controller) and Port 2 (right port) on Controller B (lower controller) are the only two active ports. If all four ports of the HSG80 are connected to the Fibre Channel switch or hub, then, if a controller module fails, the surviving controller assumes control of both the Port 1 and Port 2 ports.
The new member must be connected to the cluster’s shared storage in the same way as the other cluster members (for example, its emx0 adapter must be connected to the same switch as the emx0 adapters of other cluster members, and its emx1 adapter must be connected to the same switch as the emx1 adapters of other cluster members).
Before you add the new member system to the cluster (or ASE), boot it to single-user mode: >>> boot -fl s Execute the following commands to mount the file systems, set the root file system writeable, and start the network on the new member system: # /sbin/lsmbstartup # /sbin/bcheckrc # /sbin/init.d/inet start...
Page 75
If the new system has new KGPSA adapters, the worldwide name-to-target ID mapping has to be synchronized to remove the worldwide name for the old KGPSA adapters and assign their target IDs to the new worldwide name. To replace a cluster member system with a new member system that has new KGPSA host bus adapters, follow these steps: Remove the member system from the cluster (that is, delete it from the ASE membership).
Page 76
# /usr/sbin/emxmgr -c /tmp/new-system-remapping Continue with the change in id mappings (y/n) [n] ? y Change in Target id mappings has been performed Copy /etc/emx.info to each of the other cluster member systems. 10. Modify the worldwide name-to-target ID mapping on each of the other cluster member systems by executing the emxmgr utility using the /etc/emx.info file: # /usr/sbin/emxmgr -c /etc/emx.info...
Page 77
emxmgr Reference Page Because there is no formal documentation provided with the NHD-2 release, the emxmgr utility reference page is provided here. emxmgr Reference Page A–1...
Page 78
emxmgr(8) NAME emxmgr – Maintains the Fibre Channel worldwide name-to-target ID mappings for emx devices SYNOPSIS emxmgr [-h] [-s] [-d] [-m emx ? | -t emx ?] [ [-f] -c mapping_file ] OPTIONS To invoke the emxmgr utility interactively, enter the command with no options.
Page 79
Displays information on an adapter’s current Fibre Channel topology. The emx? argument specifies which adapter instance, such as emx0, to display. Use the -d option to obtain the instance numbers for all host adapters that are currently installed on the system. DESCRIPTION The emxmgr utility enables you to maintain the mappings of Fibre Channel worldwide names to device target IDs (identifiers) for the emx device driver.
Page 80
emxmgr(8) When you modify the target ID mappings using emxmgr, the following system changes occur: • The kernel immediately updates the emx driver with the changes. • Target IDs that do not correspond to the IDs of local host adapters are changed immediately.
Page 81
FC Port Is the N*_port’s port name, specified in the Name XXXX-XXXX-XXXX-XXXX byte-by-byte form of notation. FC Node Is the N*_port’s node name, specified in the Name XXXX-XXXX-XXXX-XXXX byte-by-byte form of notation. Syntax 2 This syntax accepts mappings in the form that is specified in the /etc/emx.info and /sys/data/emx_data.c files.
Page 82
emxmgr(8) EXAMPLES The following examples demonstrate the use of the emxmgr utility: Use the following command to update the mappings that specify the mapping_file named newmappings as a data source: force99> emxmgr -c newmappings The following mapping_file entry assigns the target ID 5 on emx0 to the Fibre Channel N*_Port with port name 5000-1FE1-0000-0CB2, and the node name 5000-1FE1-0000-0CB0: emx0 5 5000-1FE1-0000-0CB2 5000-1FE1-0000-0CB0...
Need help?
Do you have a question about the 380578-B21 and is the answer not in the manual?
Questions and answers