Page 2
All other trademarks referenced herein are the property of their respective owners. The GPG fingerprint of the security@redhat.com key is: CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E...
Table of Contents Introduction........................i 1. Audience ......................i 2. Document Conventions..................i 3. More to Come ....................iv 3.1. Send in Your Feedback ............... iv 4. Activate Your Subscription ..................v 4.1. Provide a Red Hat Login...............v 4.2. Provide Your Subscription Number ............ vi 4.3.
Page 5
6. Using GNBD with Red Hat GFS ................59 6.1. GNBD Driver and Command Usage ..............59 6.1.1. Exporting a GNBD from a Server ...........60 6.1.2. Importing a GNBD on a Client............61 6.2. Running GFS on a GNBD Server Node ............61 A. Upgrading GFS ......................63 Index..........................67 Colophon...........................71...
Managing a Cluster. HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes are available online at http://www.redhat.com/docs/. 1. Audience This book is intended primarily for Linux system administrators who are familiar with the following activities: Linux system administration procedures, including kernel configuration...
Page 8
Introduction file in your home directory contains bash shell definitions and aliases .bashrc for your own use. file contains information about different system devices and file /etc/fstab systems. Install the RPM if you want to use a Web server log file analysis program. webalizer application This style indicates that the program is an end-user application (as opposed to system...
Page 9
Introduction button on a GUI screen or window This style indicates that the text can be found on a clickable button on a GUI screen. For example: Click on the Back button to return to the webpage you last viewed. computer output Text in this style indicates text displayed to a shell prompt such as error messages and responses to commands.
Introduction Note Remember that Linux is case sensitive. In other words, a rose is not a ROSE is not a rOsE. The directory contains additional documentation for packages installed /usr/share/doc/ on your system. Important If you modify the DHCP configuration file, the changes do not take effect until you restart the DHCP daemon.
If you spot a typo in the Red Hat GFS Administrator’s Guide, or if you have thought of a way to make this manual better, we would love to hear from you! Please submit a report in Bugzilla (http://www.redhat.com/bugzilla) against the component rh-gfsg. Be sure to mention the manual’s identifier: rh-gfsg(EN)-6.1-Print-RHI (2007-01-05T17:28)
You can provide your subscription number when prompted during the Setup Agent or by visiting http://www.redhat.com/register/. 4.3. Connect Your System The Red Hat Network Registration Client helps you connect your system so that you can begin to get updates and perform systems management.
Page 13
Introduction Topic Reference Comment Storage Area Networks Designing Storage Area Provides a concise (SANs) Networks: A Practical summary of Fibre Channel Reference for Implementing and IP SAN Technology. Fibre Channel and IP SANs, Second Edition by Tom Clark. Addison-Wesley, 2003. Building SANs with Best practices for building Brocade Fabric Switches by...
Chapter 1. GFS Overview Red Hat GFS is a cluster file system that is available with Red Hat Cluster Suite. Red Hat GFS nodes are configured and managed with Red Hat Cluster Suite configuration and management tools. Red Hat GFS provides data sharing among GFS nodes in a Red Hat cluster.
Page 16
Chapter 1. GFS Overview Cluster infrastructure provided by Red Hat Cluster Suite — This release of Red Hat • GFS uses the cluster infrastructure of Red Hat Cluster Suite, taking advantage of the Red Hat Cluster Suite configuration file and cluster graphical user interface (GUI), .
Chapter 1. GFS Overview New and Changed Features with Red Hat GFS 6.1 for Red Hat Enterprise Linux 4 Update 2 Red Hat GFS 6.1 for Red Hat Enterprise Linux 4 Update 2 supports iSCSI and multipath iSCSI. That is, device mapper multipath ( ) can use iSCSI.
Chapter 1. GFS Overview Applications Fabric Shared Files Figure 1-1. GFS with a SAN 1.2.2. Performance, Scalability, Moderate Price Multiple Linux client applications on a LAN can share the same SAN-based data as shown in Figure 1-2. SAN block storage is presented to network clients as block storage devices by GNBD servers.
Chapter 1. GFS Overview Clients Applications GNBD servers Fabric Shared Files Figure 1-2. GFS and GNBD with a SAN 1.2.3. Economy and Performance Figure 1-3 shows how Linux client applications can take advantage of an existing Ethernet topology to gain shared access to all block storage devices. Client data files and file systems can be shared with GFS on each client.
Chapter 1. GFS Overview Clients Applications GNBD servers Shared Files Disk Disk Disk Disk Disk Disk Figure 1-3. GFS and GNBD with Directly Connected Storage 1.3. GFS Functions GFS is a native file system that interfaces directly with the VFS layer of the Linux kernel file-system interface.
Chapter 1. GFS Overview GFS provides the following main functions: Making a File System • Mounting a File System • Unmounting a File System • GFS Quota Management • Growing a File System • Adding Journals to a File System •...
Page 22
Chapter 1. GFS Overview Software Components Description Subsystem Command that adds journals to a gfs_jadd mounted GFS file system. Command that creates a GFS file gfs_mkfs system on a storage device. Command that manages quotas on a gfs_quota mounted GFS file system. Command that configures or tunes a gfs_tool GFS file system.
Chapter 1. GFS Overview Software Components Description Subsystem A server daemon that allows a node to gnbd_serv export local storage over the network. Table 1-1. GFS Software Subsystem Components 1.5. Before Setting Up GFS Before you install and set up GFS, note the following key characteristics of your GFS file systems: Number of file systems Determine how many GFS file systems to create initially.
Chapter 2. System Requirements This chapter describes the system requirements for Red Hat GFS Release 6.1 and consists of the following sections: Section 2.1 Platform Requirements • Section 2.2 Red Hat Cluster Suite • Section 2.3 Fencing • Section 2.4 Fibre Channel Storage Network •...
Chapter 2. System Requirements 2.3. Fencing You must configure each GFS node in your Red Hat cluster for at least one form of fencing. Fencing is configured and managed in Red Hat Cluster Suite. For more information about fencing options, refer to Red Hat Cluster Suite Configuring and Managing a Cluster. 2.4.
Chapter 2. System Requirements Requirement Description Device Type FC RAID array or JBOD Note: Make sure that the devices can operate reliably when heavily accessed simultaneously from multiple initiators. Note: Make sure that your GFS configuration does not exceed the number of nodes an array or JBOD supports.
Chapter 3. Installing GFS Installing GFS consists of installing Red Hat GFS RPMs on nodes in a Red Hat cluster. Before installing the RPMs, make sure of the following: The cluster nodes meet the requirements as described in Chapter 2 System Requirements. •...
Red Hat Network Web-based interface. For more information on using Red Hat Network, visit the following URL: http://rhn.redhat.com You can install Red Hat Cluster Suite and Red Hat GFS RPMs using either of the following methods: Automatic RPM installation —...
Chapter 3. Installing GFS 3.1.1. Automatic RPM Installation Automatic RPM installation consists of running the utility at each node for the up2date Red Hat Cluster Suite and Red Hat GFS products. Note If you are installing the GFS RPMs, you must run for Red Hat Cluster Suite up2date before running it for Red Hat GFS.
Page 32
Chapter 3. Installing GFS Note If you are installing the GFS RPMs, you must install Red Hat Cluster Suite before Red Hat GFS. 3.1.2.1. Determining RPMs To Install Determining which RPMs to install is based on the following criteria: The lock manager Red Hat Cluster Suite is using — either DLM or GULM •...
Page 33
Chapter 3. Installing GFS To determine which RPMs to include in the installation, perform the following steps: 1. Determine whether you are installing Red Hat Cluster Suite with DLM or Red Hat Cluster Suite with GULM. a. If you are installing Red Hat Cluster Suite with DLM, refer to Table 3-1 to identify which RPMs are required, optional, and for development.
Page 34
Chapter 3. Installing GFS RPMs Inclusion Depends Purpose Kernel Type? A library that iddev-ver-rel.arch identifies device contents A cluster/lock magma-ver-rel.arch manager API abstraction library Cluster manager magma-plugins-ver-rel.arch plugins for magma The Grand Unified gulm-ver-rel.arch Note: The module is required with Lock Manager gulm DLM because the...
Page 35
Chapter 3. Installing GFS RPMs Inclusion Depends Purpose Kernel Type? kernel header dlm-kernheaders-ver-rel.arch files development iddev-devel-ver-rel.arch iddev libraries A cluster/lock magma-devel-ver-rel.arch manager API abstraction library Table 3-1. RPM Selection Criteria: Red Hat Cluster Suite with DLM RPMs Inclusion Depends Purpose Kernel Type? The Cluster...
Page 36
Chapter 3. Installing GFS RPMs Inclusion Depends Purpose Kernel Type? Net-Telnet Perl perl-Net-Telnet-ver-rel.arch module GUI to manage system-config-cluster-ver-rel.arch cluster configuration Utility to administer ipvsadm-ver-rel.arch the Linux Virtual Server Cluster piranha-ver-rel.arch administration tools CCS static library ccs-devel-ver-rel.arch libraries gulm-devel-ver-rel.arch gulm development iddev-devel-ver-rel.arch iddev libraries...
Page 37
If a Question dialog box is displayed that asks if you want to run , click No. autorun 3. Copy all the RPM files from the CD (located in /media/cdrom/RedHat/RPMS/ to the temporary directory created earlier. For example: $ cp /media/cdrom/RedHat/RPMS/*.rpm /tmp/RPMS/...
Page 38
6. Copy all the RPM files from the CD (located in /media/cdrom/RedHat/RPMS/ to the temporary directory created earlier. For example: $ cp /media/cdrom/RedHat/RPMS/*.rpm /tmp/RPMS/ Note If your local computer is running a version of Red Hat Enterprise Linux that is earlier than Red Hat Enterprise Linux 4, the path to the RPMs on the CD may be different.
Page 39
Chapter 3. Installing GFS 11. Copy the RPMs from the temporary directory in the local computer to directories in the cluster nodes using the command. For example, to copy the RPMs to node rhcs-node-01, run the following command at the local computer: $ scp /tmp/RPMS/*.rpm root@rhcs-node-01:/tmp/node-RPMS/ 12.
Chapter 4. Getting Started This chapter describes procedures for initial setup of GFS and contains the following sec- tions: Section 4.1 Prerequisite Tasks • Section 4.2 Initial Setup Tasks • 4.1. Prerequisite Tasks Before setting up Red Hat GFS, make sure that you have noted the key characteristics of the GFS nodes (refer to Section 1.5 Before Setting Up GFS) and have loaded the GFS modules into each GFS node.
Page 42
Chapter 4. Getting Started Note You can use scripts included with Red Hat Cluster Suite to automate acti- init.d vating and deactivating logical volumes. For more information about scripts, init.d refer to Red Hat Cluster Suite Configuring and Managing a Cluster . 2.
Chapter 5. Managing GFS This chapter describes the tasks and commands for managing GFS and consists of the following sections: Section 5.1 Making a File System • Section 5.2 Mounting a File System • Section 5.3 Unmounting a File System •...
Chapter 5. Managing GFS 5.1.1. Usage gfs_mkfs -p LockProtoName -t LockTableName -j Number BlockDevice Warning Make sure that you are very familiar with using the LockProtoName and LockTableName LockProtoName parameters. Improper LockTableName parameters may cause file system or lock space corruption. LockProtoName Specifies the name of the locking protocol (for example, ) to use.
Chapter 5. Managing GFS In this example, a second file system is made, which can be used in cluster lock_dlm . The file-system name is . The file system contains eight journals and is created alpha gfs2 /dev/vg01/lvol1 gfs_mkfs -p lock_dlm -t alpha:gfs2 -j 8 /dev/vg01/lvol1 5.1.3.
Chapter 5. Managing GFS Flag Parameter Description MegaBytes Specifies the size of the resource groups in megabytes. Default resource group size is 256 megabytes. Blocks Specifies the journal-segment size in file-system blocks. LockTableName This parameter has two parts separated by a colon (no spaces) as follows: ClusterName:FSName.
Chapter 5. Managing GFS Mounting With ACL Manipulation mount -t gfs -o acl BlockDevice MountPoint GFS-specific option to allow manipulating file ACLs. BlockDevice Specifies the block device where the GFS file system resides. MountPoint Specifies the directory where the GFS file system should be mounted. 5.2.2.
Page 48
Chapter 5. Managing GFS Option Description Allows manipulating file ACLs. If a file system is mounted without the mount option, users are allowed to view ACLs (with ), but are not allowed to set them (with getfacl setfacl This field provides host (the computer on which hostdata=HostIDInfo the file system is being mounted) identity information to the lock module.
Chapter 5. Managing GFS Option Description This option allows a GFS node to not panic oopses_ok when an oops occurs. (By default, a GFS node panics when an oops occurs, causing the file system used by that node to stall for other GFS nodes.) A GFS node not panicking when an oops occurs minimizes the failure on other GFS nodes using the file system that the failed...
Chapter 5. Managing GFS 5.4. GFS Quota Management File-system quotas are used to limit the amount of file-system space a user or group can use. A user or group does not have a quota limit until one is set. GFS keeps track of the space used by each user and group even when there are no limits in place.
Chapter 5. Managing GFS User A user ID to limit or warn. It can be either a user name from the password file or the UID number. Group A group ID to limit or warn. It can be either a group name from the group file or the GID number.
Page 52
Chapter 5. Managing GFS Displaying Quota Limits for a Group gfs_quota get -g Group -f MountPoint Displaying Entire Quota File gfs_quota list -f MountPoint User A user ID to display information about a specific user. It can be either a user name from the password file or the UID number.
Chapter 5. Managing GFS Value The actual amount of disk space used by the user or group. 5.4.2.3. Comments When displaying quota information, the command does not resolve UIDs and gfs_quota GIDs into names if the option is added to the command line. Space allocated to GFS’s hidden files can be left out of displayed values for the root UID and GID by adding the option to the command line.
Chapter 5. Managing GFS 5.4.3.1. Usage Synchronizing Quota Information gfs_quota sync -f MountPoint MountPoint Specifies the GFS file system to which the actions apply. Tuning the Time Between Synchronizations gfs_tool settune MountPoint quota_quantum Seconds MountPoint Specifies the GFS file system to which the actions apply. Seconds Specifies the new time period between regular quota-file synchronizations by GFS.
Chapter 5. Managing GFS 5.4.4.1. Usage gfs_tool settune MountPoint quota_enforce {0|1} MountPoint Specifies the GFS file system to which the actions apply. quota_enforce {0|1} 0 = disabled 1 = enabled 5.4.4.2. Comments A value of 0 disables enforcement. Enforcement can be enabled by running the command with a value of 1 (instead of 0) as the final command line parameter.
Page 56
Chapter 5. Managing GFS 5.4.5.1. Usage gfs_tool settune MountPoint quota_account {0|1} MountPoint Specifies the GFS file system to which the actions apply. quota_account {0|1} 0 = disabled 1 = enabled 5.4.5.2. Comments To enable quota accounting on a file system, the parameter must be set quota_account back to 1.
Chapter 5. Managing GFS 5.5. Growing a File System command is used to expand a GFS file system after the device where the gfs_grow file system resides has been expanded. Running a command on an existing GFS gfs_grow file system fills all spare space between the current end of the file system and the end of the device with a newly initialized GFS file-system extension.
Chapter 5. Managing GFS In this example, the state of the mounted file system is checked. gfs_grow -Tv /gfs1 5.5.4. Complete Usage gfs_grow [Options] {MountPoint | Device} [MountPoint | Device] MountPoint Specifies the directory where the GFS file system is mounted. Device Specifies the device node of the file system.
Chapter 5. Managing GFS To verify that the changes were successful, use the command with the (test) gfs_jadd (verbose) flags. Running the command with those flags displays the current state of the mounted GFS file system. 5.6.1. Usage gfs_jadd -j Number MountPoint Number Specifies the number of new journals to be added.
Chapter 5. Managing GFS gfs_jadd -Tv /gfs1 5.6.4. Complete Usage gfs_jadd [Options] {MountPoint | Device} [MountPoint | Device] MountPoint Specifies the directory where the GFS file system is mounted. Device Specifies the device node of the file system. Table 5-4 describes the GFS-specific options that can be used when adding journals to a GFS file system.
Chapter 5. Managing GFS Flag Parameter Description Turns up the verbosity of messages. Table 5-4. GFS-specific Options Available When Adding Journals 5.7. Direct I/O Direct I/O is a feature of the file system whereby file reads and writes go directly from the applications to the storage device, bypassing the operating system read and write caches.
Chapter 5. Managing GFS 5.7.2.1. Usage Setting the directio Flag File gfs_tool setflag directio Clearing the Flag directio File gfs_tool clearflag directio File Specifies the file where the flag is assigned. directio 5.7.2.2. Example In this example, the command sets the flag on the file named directio datafile...
Chapter 5. Managing GFS 5.7.3.2. Example In this example, the command sets the flag on the directory named inherit_directio /gfs1/data/ gfs_tool setflag inherit_directio /gfs1/data/ 5.8. Data Journaling Ordinarily, GFS writes only metadata to its journal. File contents are subsequently written to disk by the kernel’s periodic sync that flushes file-system buffers.
Chapter 5. Managing GFS File Specifies the zero-length file where the flag is set or cleared. 5.8.2. Examples This example shows setting the flag on a directory. All files created in inherit_jdata the directory or any of its subdirectories will have the flag assigned automatically.
Chapter 5. Managing GFS 5.9.1.1. Usage BlockDevice MountPoint mount -t gfs -o noatime BlockDevice Specifies the block device where the GFS file system resides. MountPoint Specifies the directory where the GFS file system should be mounted. 5.9.1.2. Example In this example, the GFS file system resides on the and is mounted on /dev/vg01/lvol0 directory...
Chapter 5. Managing GFS MountPoint Specifies the directory where the GFS file system is mounted. Changing the Parameter Value atime_quantum MountPoint Seconds gfs_tool settune atime_quantum MountPoint Specifies the directory where the GFS file system is mounted. Seconds Specifies the update period in seconds. 5.9.2.2.
Chapter 5. Managing GFS MountPoint Specifies the file system. 5.10.2. Examples This example suspends writes to file system /gfs gfs_tool freeze /gfs This example ends suspension of writes to file system /gfs gfs_tool unfreeze /gfs 5.11. Displaying Extended GFS Information and Statistics You can use the command to gather a variety of details about GFS.
Chapter 5. Managing GFS File Specifies the file from which to get information. command provides additional action flags (options) not listed in this sec- gfs_tool tion. For more information about other flags, refer to the man page. gfs_tool gfs_tool 5.11.2. Examples This example reports extended file-system usage about file system /gfs gfs_tool df /gfs...
Chapter 5. Managing GFS Refer to the man page, , for additional information about other gfs_fsck gfs_fsck(8) command options. 5.12.1. Usage BlockDevice gfs_fsck flag causes all questions to be answered with . With the flag specified, command does not prompt you for an answer before making changes. gfs_fsck BlockDevice Specifies the block device where the GFS file system resides.
Page 70
Chapter 5. Managing GFS Target Specifies an existing file or directory on a file system. LinkName Specifies a name to represent the real file or directory on the other end of the link. For a Variable Symbolic Link Variable LinkName Variable Specifies a special reserved name from a list of values (refer to Table 5-5) to represent one of multiple existing files or directories.
Chapter 5. Managing GFS Variable Description This variable resolves to a real file or directory named with @gid the group ID string produced by the output of the following command: echo ‘id -g‘ Table 5-5. CDPN Variable Values 5.13.2. Example In this example, there are three nodes with hostnames .
Chapter 6. Using GNBD with Red Hat GFS GNBD (Global Network Block Device) provides block-level storage access over an Eth- ernet LAN. GNBD components run as a client in a GFS node and as a server in a GNBD server node. A GNBD server node exports block-level storage from its local storage (either directly attached storage or SAN storage) to a GFS node.
Chapter 6. Using GNBD with Red Hat GFS (for clients) — User program for importing and managing GNBDs on a • gnbd_import GNBD client. 6.1.1. Exporting a GNBD from a Server daemon must be running on a node before it can export storage as a gnbd_serv GNBD.
Chapter 6. Using GNBD with Red Hat GFS Note If you have been using GFS 5.2 or earlier and do not want to change your GNBD setup you should specify the option. Before GFS Release 5.2.1, Linux caching was enabled by default for .
Page 76
Chapter 6. Using GNBD with Red Hat GFS 6.2. Running GFS on a GNBD Server Node You can run GFS on a GNBD server node, with some restrictions. In addition, running GFS on a GNBD server node reduces performance. The following restrictions apply when running GFS on a GNBD server node.
Appendix A. Upgrading GFS To upgrade a node to Red Hat GFS 6.1 from earlier versions of Red Hat GFS, you must convert the GFS cluster configuration archive (CCA) to a Red Hat Cluster Suite cluster con- figuration system (CCS) configuration file ( ) and convert /etc/cluster/cluster.conf volumes to LVM2 volumes.
Page 78
Appendix A. Upgrading GFS d. Deactivate pools; at each node, deactivate GFS volumes as follows: pool # service pool stop e. Uninstall Red Hat GFS RPMs. 2. Install new software: a. Install Red Hat Enterprise Linux version 4 software (or verify that it is in- stalled).
Page 79
GFS 6.1 nodes are pool using LVM2 for other purposes (root file system) there may be problems activating volumes under GFS 6.1. That is because of static minor conflicts. Refer pool to the following Bugzilla report for more information: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=146035...
Index features, new and changed, 1 feedback, v fencing activating your subscription, v system requirements, 12 adding journals to a file system, 44 fibre channel network requirements table, 12 atime, configuring updates, 50 fibre channel storage device requirements table, mounting with noatime, 50 tuning atime quantum, 51 fibre channel storage devices audience, i...
Page 82
making a file system, 29 atime, configuring updates, 50 managing GFS, 29 mounting with noatime, 50 mount table, 34 tuning atime quantum, 51 mounting a file system, 32 direct I/O, 47 directory attribute, 48 file attribute, 47 O_DIRECT, 47 network power switches displaying extended information and statis- system requirements, 13 tics, 53...
Page 83
GFS-specific options for adding journals, 46 GFS-specific options for expanding file sys- recommended references table, vi tems, 44 Red Hat Cluster Suite, 15 gfs_mkfs command options, 31 custom installation of software mount options, 34 with the rpm utility, 23 platform requirements, 11 installation, 15 recommended references, vi determining RPMs to install, 18...
Colophon The manuals are written in DocBook SGML v4.1 format. The HTML and PDF formats are produced using custom DSSSL stylesheets and custom jade wrapper scripts. The DocBook SGML files are written using Emacs with the help of PSGML mode. Garrett LeSage created the admonition graphics (note, tip, important, caution, and warn- ing).
Page 86
Runa Bhattacharjee — Bengali translations Chester Cheng — Traditional Chinese translations Verena Fuehrer — German translations Kiyoto Hashida — Japanese translations N. Jayaradha — Tamil translations Michelle Jiyeen Kim — Korean translations Yelitza Louze — Spanish translations Noriko Mizumoto — Japanese translations Ankitkumar Rameshchandra Patel —...
Need help?
Do you have a question about the GFS 6.1 and is the answer not in the manual?
Questions and answers