Hat Cluster Suite refer to Red Hat Cluster Suite Overview and Configuring and Managing a Red Hat Cluster. HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes are available online at http://www.redhat.com/docs/. 1. Audience This book is intended primarily for Linux system administrators who are familiar with the following activities: •...
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML, http:// PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at www.redhat.com/docs/. 3. Document Conventions This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information.
Pull-quote Conventions This denotes words or phrases encountered on a system, including application names; dialog box text; labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example: Choose System Preferences Mouse from the main menu bar to launch Mouse Preferences.
4. Feedback If you spot a typo, or if you have thought of a way to make this manual better, we would love to hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the component Documentation-cluster. Be sure to mention the manual's identifier:...
Feedback Bugzilla component: Documentation-cluster Book identifier: Global_File_System(EN)-5 (2010-03-11T15:10) By mentioning this manual's identifier, we know exactly which version of the guide you have. If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.
Chapter 1. GFS Overview The Red Hat GFS file system is a native file system that interfaces directly with the Linux kernel file system interface (VFS layer). When implemented as a cluster file system, GFS employs distributed metadata and multiple journals. Red Hat supports the use of GFS file systems only as implemented in Red Hat Cluster Suite.
Chapter 1. GFS Overview Note When you configure a GFS file system as a cluster file system, you must ensure that all nodes in the cluster have access to the shared file system. Asymmetric cluster configurations in which some nodes have access to the file system and others do not are not supported.
Superior Performance and Scalability Note The deployment examples in this chapter reflect basic configurations; your needs might require a combination of configurations shown in the examples. 1.2.1. Superior Performance and Scalability You can obtain the highest shared-file performance when applications access storage directly. The Figure 1.1, “GFS with a SAN”...
Chapter 1. GFS Overview Figure 1.2. GFS and GNBD with a SAN Figure 1.3, “GFS and GNBD with Directly Connected Storage” shows how Linux client applications can take advantage of an existing Ethernet topology to gain shared access to all block storage devices.
Before Setting Up GFS Software Component Description Kernel module that implements the GFS file system and is gfs.ko loaded on GFS cluster nodes. A lock module that implements DLM locking for GFS. It plugs lock_dlm.ko into the lock harness, lock_harness.ko and communicates with the DLM lock manager in Red Hat Cluster Suite.
Chapter 2. Getting Started This chapter describes procedures for initial setup of GFS and contains the following sections: Section 2.1, “Prerequisite Tasks” • Section 2.2, “Initial Setup Tasks” • 2.1. Prerequisite Tasks Before setting up Red Hat GFS, make sure that you have noted the key characteristics of the GFS Section 1.4, “Before Setting Up GFS”).
Chapter 2. Getting Started mkfs -t gfs -p lock_dlm -t LockTableName -j NumberJournals BlockDevice You can use either of the following formats to create a local GFS file system: gfs_mkfs -p lock_nolock -j NumberJournals BlockDevice mkfs -t gfs -p lock_nolock -j NumberJournals BlockDevice Section 3.1, “Creating a File System”.
Chapter 3. Managing GFS This chapter describes the tasks and commands for managing GFS and consists of the following sections: Section 3.1, “Creating a File System” • Section 3.2, “Mounting a File System” • Section 3.3, “Unmounting a File System” •...
Chapter 3. Managing GFS Note Once you have created a GFS file system with the gfs_mkfs command, you cannot decrease the size of the file system. You can, however, increase the size of an existing Section 3.7, “Growing a File file system with the gfs_grow command, as described in System”.
Examples NumberJournals Specifies the number of journals to be created by the gfs_mkfs command. One journal is required for each node that mounts the file system. (More journals than are needed can be specified at creation time to allow for future expansion.) BlockDevice Specifies a volume.
Chapter 3. Managing GFS Complete Options Table 3.1, “Command Options: gfs_mkfs” describes the gfs_mkfs command options. Flag Parameter Description Sets the file system block size to BlockSize. Default BlockSize block size is 4096 bytes. Enables debugging output. Help. Displays available options. Specifies the size of the journal in megabytes.
Mounting a File System 3.2. Mounting a File System Section 3.1, “Creating Before you can mount a GFS file system, the file system must exist (refer to a File System”), the volume where the file system exists must be activated, and the supporting Chapter 2, Getting Started and Configuring clustering and locking systems must be started (refer to...
Chapter 3. Managing GFS (for example, -r). For information about other Linux mount command options, see the Linux mount man page. Table 3.2, “GFS-Specific Mount Options” describes the available GFS-specific -o option values that can be passed to GFS at mount time. Note This table includes descriptions of options that are used with local file systems only For the RHEL 5.5 release and later Red Hat does not support the use of GFS as a single-...
Unmounting a File System Option Description Note: This option is turned on automatically if lock_nolock locking is specified; however, you can override it by using the ignore_local_fs option. Upgrade the on-disk format of the file system so that it upgrade can be used by newer versions of GFS.
Chapter 3. Managing GFS • If a GFS file system has been mounted manually with the mount command, be sure to unmount the file system manually with the umount command before rebooting or shutting down the system. If your file system hangs while it is being unmounted during system shutdown under these circumstances, perform a hardware reboot.
Chapter 3. Managing GFS Usage Setting Quotas, Hard Limit gfs_quota limit -u User -l Size -f MountPoint gfs_quota limit -g Group -l Size -f MountPoint Setting Quotas, Warn Limit gfs_quota warn -u User -l Size -f MountPoint gfs_quota warn -g Group -l Size -f MountPoint User A user ID to limit or warn.
Displaying Quota Limits and Usage gfs_quota get -u User -f MountPoint Displaying Quota Limits for a Group gfs_quota get -g Group -f MountPoint Displaying Entire Quota File gfs_quota list -f MountPoint User A user ID to display information about a specific user. It can be either a user name from the password file or the UID number.
Chapter 3. Managing GFS Space allocated to GFS's hidden files can be left out of displayed values for the root UID and GID by adding the -d option to the command line. This is useful when trying to match the numbers from gfs_quota with the results of a du command.
Disabling/Enabling Quota Enforcement gfs_tool settune MountPoint quota_quantum Seconds MountPoint Specifies the GFS file system to which the actions apply. Seconds Specifies the new time period between regular quota-file synchronizations by GFS. Smaller values may increase contention and slow down performance. Examples This example synchronizes the quota information from the node it is run on to file system /gfs.
Chapter 3. Managing GFS Examples This example disables quota enforcement on file system /gfs. gfs_tool settune /gfs quota_enforce 0 This example enables quota enforcement on file system /gfs. gfs_tool settune /gfs quota_enforce 1 3.6.5. Disabling/Enabling Quota Accounting By default, quota accounting is enabled; therefore, GFS keeps track of disk usage for every user and group even when no quota limits have been set.
Growing a File System Examples This example disables quota accounting on file system /gfs on a single node. gfs_tool settune /gfs quota_account 0 This example enables quota accounting on file system /gfs on a single node and initializes the quota file.
Chapter 3. Managing GFS • Expand the underlying cluster volume with LVM. For information on administering LVM volumes, see the LVM Administrator's Guide The gfs_grow command provides a -T (test) option that allows you to see the results of executing the command without actually expanding the file system.
Adding Journals to a File System 3.8. Adding Journals to a File System The gfs_jadd command is used to add journals to a GFS file system after the device where the file system resides has been expanded. Running a gfs_jadd command on a GFS file system uses space between the current end of the file system and the end of the device where the file system resides.
Chapter 3. Managing GFS Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 100% metadata data 4980077 4979899 After running the gfs_jadd command, you can run the gfs_tool df MountPoint command again to check that the new journals have been added to the file system.
Direct I/O Flag Parameter Description Help. Displays short usage message. Specifies the size of the new journals in megabytes. MegaBytes Default journal size is 128 megabytes. The minimum size is 32 megabytes. To add journals of different sizes to the file system, the gfs_jadd command must be run for each size journal.
Chapter 3. Managing GFS To cause the O_DIRECT flag to be defined with recent glibc libraries, define _GNU_SOURCE at the beginning of a source file before any includes, or define it on the cc line when compiling. 3.9.2. GFS File Attribute The gfs_tool command can be used to assign (set) a direct I/O attribute flag, directio, to a GFS file.
Data Journaling Usage Setting the inherit_directio flag gfs_tool setflag inherit_directio Directory Clearing the inherit_directio flag gfs_tool clearflag inherit_directio Directory Directory Specifies the directory where the inherit_directio flag is set. Example In this example, the command sets the inherit_directio flag on the directory named /mnt/gfs/ data.
Chapter 3. Managing GFS Usage Setting and Clearing the inherit_jdata Flag gfs_tool setflag inherit_jdata Directory gfs_tool clearflag inherit_jdata Directory Setting and Clearing the jdata Flag gfs_tool setflag jdata File gfs_tool clearflag jdata File Directory Specifies the directory where the flag is set or cleared. File Specifies the zero-length file where the flag is set or cleared.
Mount with noatime If atime updates are enabled as they are by default on GFS and other Linux file systems then every time a file is read, its inode needs to be updated. Because few applications use the information provided by atime, those updates can require a significant amount of unnecessary write traffic and file-locking traffic.
Chapter 3. Managing GFS The gfs_tool settune command is used to change the atime_quantum parameter value. It must be set on each node and each time the file system is mounted. The setting is not persistent across unmounts. Usage Changing the atime_quantum Parameter Value gfs_tool settune MountPoint atime_quantum Seconds MountPoint Specifies the directory where the GFS file system is mounted.
Displaying Extended GFS Information and Statistics This example ends suspension of writes to file system /gfs. gfs_tool unfreeze /gfs 3.13. Displaying Extended GFS Information and Statistics You can use the gfs_tool command to gather a variety of details about GFS. This section describes typical use of the gfs_tool command for displaying space usage, statistics, and extended status.
Chapter 3. Managing GFS 3.13.2. Displaying GFS Counters You can use the counters flag of the gfs_tool to display statistics about a file system. If the -c option is used, the gfs_tool command continues to run, displaying statistics once per second. Note The majority of the GFS counters reflect the internal operation of the GFS filesystem and are for development purposes only.
Displaying GFS Counters log wraps The number of times journal has wrapped around. outstanding LM calls obsolete outstanding BIO calls obsolete fh2dentry misses The number of times an NFS call could not find a dentry structure in the cache. glocks reclaimed The number of glocks which have been reclaimed.
Chapter 3. Managing GFS block I/O writes obsolete Usage gfs_tool counters MountPoint MountPoint Specifies the file system to which the action applies. Example This example reports statistics about the file system mounted at /mnt/gfs. [root@tng3-1 gfs]# gfs_tool counters /mnt/gfs locks 165 locks held 133 freeze count 0 incore inodes 34...
Displaying Extended Status Note The information that the gfs_tool stat command displays reflects internal file system information. This information is intended for development purposes only. Usage gfs_tool stat File File Specifies the file from which to get information. Example This example reports extended file status about file /gfs/datafile. [root@tng3-1 gfs]# gfs_tool stat /gfs/datafile mh_magic = 0x01161970 mh_type = 4...
Chapter 3. Managing GFS 3.14. Repairing a File System When nodes fail with the file system mounted, file system journaling allows fast recovery. However, if a storage device loses power or is physically disconnected, file system corruption may occur. (Journaling cannot be used to recover from storage subsystem failures.) When that type of corruption occurs, you can recover the GFS file system by using the gfs_fsck command.
Example gfs_fsck -y BlockDevice The -y flag causes all questions to be answered with yes. With the -y flag specified, the gfs_fsck command does not prompt you for an answer before making changes. BlockDevice Specifies the block device where the GFS file system resides. Example In this example, the GFS file system residing on block device /dev/gfsvg/gfslv is repaired.
Chapter 3. Managing GFS Syncing the device. Freeing buffers. 3.15. Context-Dependent Path Names Context-Dependent Path Names (CDPNs) allow symbolic links to be created that point to variable destination files or directories. The variables are resolved to real files or directories each time an application follows the link.
Example Variable Description This variable resolves to a real file or directory named with the operating-system name string produced by the output of the following command: echo `uname -s` This variable resolves to a real file or directory named with the @sys combined machine type and OS release strings produced by the output of the following command: echo `uname -m`_`uname -s`...
Chapter 3. Managing GFS perform file system recovery with the gfs_fsck command. The GFS withdraw function is less severe than a kernel panic, which would cause another node to fence the node. An example of an inconsistency that would yield a GFS withdraw is an incorrect block count. When the GFS kernel module deletes a file from a file system, it systematically removes all the data and metadata blocks associated with that file.
Appendix A. Revision History Revision 5.5-1 Thu Mar 18 2010 Steven Levine email@example.com Resolves: #568179 Adds note clarifying support policy for single-node system. Resolves: #562199 Adds note clairfying 16-node limitation. Resolves: #515348 Documents new -o errors mount option. Resolves: #573750 Documents memory requirements for gfs_fsck.
Index displaying quota limits, 18 setting quotas, 17 synchronizing quotas, 20 repairing, 38 suspending activity, 32 adding journals to a file system, 25 unmounting, 15, 15 atime, configuring updates, 30 mounting with noatime, 31 tuning atime quantum, 31 audience, v atime, configuring updates, 30 mounting with noatime, 31 tuning atime quantum, 31...