ggd2(1)
ggd2 - GRIO bandwidth management daemon
Showing IRIX 6.5.30 (default release). Added in IRIX 6.5.30.
This page could not be fully tidied (no-header); it is shown as printed.
NAME ggd2 - GRIO bandwidth management daemon SYNOPSIS ggd2 [-d level] [-f] [-m bw] [-s bw] [-r percent] [-u milliseconds] DESCRIPTION ggd2 is a user-level process, started at boot, that manages the I/O bandwidth of a collection of GRIO-managed storage volumes. At startup, ggd2 forks a child worker process that communicates with the kernel and is responsible for the bulk of the GRIO processing. The parent process monitors the activity of this child and if necessary, during a server relocation for example, will restart it. ggd2 is responsible for activating and deactivating the GRIO kernel I/O scheduler, processing client requests to reserve and release bandwidth, tracking bandwidth utilisation, managing unreserved bandwidth, and enforcing the GRIO software licenses. OPTIONS ggd2 takes the following options: -d level This option enables debug mode and sets the debug level. ggd2 has 5 debug levels: 0-4 inclusive. Level 0 is the default at which only serious error conditions are logged. Normally level 0 messages are logged to the syslog. When debug mode is enabled ggd2 creates an additional log file called /var/tmp/ggd2log<pid> and logs to both the syslog and this file. level is an integer value specifying the maximum debug level that should be printed to this log file. -f By default ggd2 is started as a daemon. The -f option indicates that it should run in the foreground. -m bw If a volume has spare bandwidth not actively reserved by guaranteed rate streams, ggd2 will temporarily allow this bandwidth to be allocated to non-guaranteed user and system I/O. The -m option takes an integer argument bw that specifies the minimum amount of bandwidth in KB/sec that ggd2 will allocate for non-GRIO I/O per GRIO managed volume. All nodes issuing non-GRIO I/O will receive a fair share of this minimum bandwidth. For example, passing '-m2048' causes ggd2 to allocate a minimum of 2048 KB/sec to each GRIO managed volume. This bandwidth becomes permanently allocated to non-GRIO I/O and cannot be reserved for guaranteed rate I/O. A suffix K or M can be added to explicitly specify bandwidth in KB/sec or MB/sec. For example passing '-m3M' causes ggd2 to allocate a minimum of 3 MB/sec to each GRIO managed volume. -s bw If a volume has spare bandwidth not actively reserved by guaranteed rate streams, ggd2 will temporarily allow this bandwidth to be allocated to non-guaranteed user and system I/O. The -s option takes an integer argument bw that specifies the minimum amount of bandwidth in KB/sec that ggd2 will allocate for non-GRIO I/O per node. A node will be allocated the bigger value specified by this option or option -m. For example, passing '-s2048' causes ggd2 to allocate a minimum of 2048 KB/sec to each node accessing a GRIO managed volume. This bandwidth becomes permanently allocated to non-GRIO I/O and cannot be reserved for guaranteed rate I/O. A suffix K or M can be added to explicitly specify bandwidth in KB/sec or MB/sec. For example passing '-s3M' causes ggd2 to allocate a minimum of 3 MB/sec to each node accessing a GRIO managed volume. -r percent By default ggd2 will allow any unreserved bandwidth to be allocated for servicing non-GRIO I/O. This maximizes the total throughput of the system. However, as ggd2 only makes adjustments to these allocations periodically, a new reservation may block until ggd2 can reclaim the requested bandwidth. The -r option disables this behaviour for a proportion of each volume's available qualified bandwidth. Reservation requests can be serviced directly from this pool of 'cached' free bandwidth without blocking. percent is the percentage of each volume's qualified bandwidth that ggd2 attempts to keep unallocated, expressed as an integer in the range 0-100. Its value should be chosen based on the expected I/O utilisation levels, the importance of minimizing the stream creation latency, and the expected rate at which reservation requests will be made. Using this option causes a proportion of the unreserved I/O capacity to remain unused and reduces the total throughput and efficiency of the system for non-GRIO I/O. It should only be used if minimizing reservation latency is a priority. For example, given a volume with a qualified bandwidth of 200 MB/sec, passing -r20 will instruct ggd2 to try to keep upto 20% or 40 MB/sec of any remaining unreserved bandwidth cached and available for servicing reservation requests directly. ggd2 adjusts this cache of free bandwidth every time the Distributed Bandwidth Allocator runs (discussed below), which defaults to once every two seconds. Thus with these settings ggd2 will be able to grant an additional 40MB/sec every two seconds without blocking any reservation requests. -u milliseconds The ggd2 daemon periodically adjusts the amount of bandwidth allocated to the individual non-GRIO streams for its managed filesystems. This functionality is referred to as the distributed bandwidth allocator (DBA). The DBA is responsible for determining how unreserved bandwidth is distributed between the nodes with access to the filesystem. By default, the DBA runs every two seconds, constantly allocating free bandwidth to nodes based on a range of dynamically monitored demand and utilization metrics. The -u option is used to change DBA allocation interval from the default value of two seconds (2000 milliseconds) to any value in the range 250 to 100000. Note that the rate at which the DBA runs affects the delay that an application or node that does not have a GRIO reservation might experience when it starts doing I/O. The longer the interval, the longer a node may have to wait (with its I/O paused) before ggd2 will increase its allocation. LOCAL VERSUS SHARED VOLUMES Following XVM, a managed volume can be either local to a node or shared amongst all nodes in the cluster. A local volume is said to be in the local volume domain, shared volumes in the cluster volume domain. With one exception, each distinct volume domain is managed by a unique instance of the ggd2 daemon. Local volumes are always managed by the ggd2 running on the node to which they are attached. Shared volumes are managed by a single instance of ggd2 running on one of the server-capable nodes within the cluster. This node is referred to as the GRIO server. It is either elected automatically or selected by the administrator. If a node has both local volumes and has also been selected as the GRIO2 server the ggd2 running on that node will serve dual-duty and manage both its local and the cluster volume domains. A user process can request guaranteed rate I/O to a volume using the grio_reserve(3X) and grio_reserve_fd(3X) library calls. Requests are forwarded to the ggd2 actively managing the target volume domain. Requests to volumes in the local domain are sent to the local instance of ggd2. Requests to cluster volumes are forwarded to the GRIO server, which may be running on a different node in the cluster. ENCAPSULATION & DISTRIBUTED BANDWIDTH ALLOCATION When ggd2 begins managing a volume, every node with access to that volume is notified. Each node in turn creates a dedicated system stream for that volume termed the nongrio stream. From that point on, all user and system I/O that doesn't have an explicit GRIO reservation is encapsulated by this stream and managed by the GRIO scheduler. For a locally-attached volume there is a single nongrio stream. For a shared volume there is a nongrio stream on each node with access to the volume. To keep the total throughput of the volume high even when there are active GRIO streams, ggd2 attempts to allocate the unreserved portion of the qualified filesystem bandwidth for use by non-GRIO applications. This bandwidth is effectively lent for short periods of time until ggd2 receives a new request for guaranteed rate bandwidth, at which point it is reclaimed. ggd2 periodically adjusts the amount of bandwidth allocated to individual nongrio streams for its managed volumes. This functionality is referred to as the Distributed Bandwidth Allocator or DBA. The DBA is responsible for determining how unreserved bandwidth is distributed between the nodes with access to the volume. The DBA runs by default every two seconds, constantly allocating free bandwidth to nodes based on a range of dynamically monitored demand and utilisation metrics. Calls to reserve bandwidth may block until the next DBA cycle, applications must therefore be prepared for delays when setting up guaranteed rate streams. Refer to grio_reserve(3X) for more information. To help manage this, the -r option can be used to cause ggd2 to put aside a pool of otherwise unreserved bandwidth (i.e. not allotated for non-GRIO use), from which new reservations can be processed directly. GRIO VERSION 1 & 2 With the introduction of GRIO Version 2 IRIX kernels now support two guaranteed rate I/O subsystems. Both the Version 1 (GRIOv1) and Version 2 (GRIOv2) subsystems can be installed together, however, undefined behaviour may result if both are enabled simultaneously. As a result, mixed operation is forcibly prevented with a kernel interlock. Only one subsystem may be active at a time and once a subsystem has been activated it remains active until the next reboot. GRIO Version 2 is inactive by default. The following sequence of commands can be used to activate it: # chkconfig grio off # chkconfig grio2 on # /etc/init.d/grio2 start Note that if GRIO Version 1 has been run at any point the system must be rebooted to allow the kernel to be reinitialized with the GRIOv2 scheduler. This is done to ensure a user does not inadvertently activate both real-time schedulers by mistake. API calls disabled in the running kernel (eg. v2 calls to a v1-enabled kernel) return ENOSYS. v2 calls return EAGAIN if the GRIOv2 subsystem has not yet been initialized. For both subsystems the switch to a particular mode is triggered by startup of the userspace daemons. ggd2 enables the v2 subsystem and ggd (the v1 bandwidth manager) enables the v1 subsystem. The daemons are in turn controlled by chkconfig variables (grio2 and grio respectively) along with updated init scripts. The init scripts also make checks to ensure that only one userspace is running at a time. CONFIGURING GGD2 Local volumes are configured using the information provided in /etc/griotab. This file lists the volumes that should be managed by GRIO and the maximum qualified bandwidth they can deliver. This file is read at startup and whenever ggd2 receives SIGHUP (i.e. killall -HUP ggd2). Refer to griotab(4) for more information. If the GRIO shared volume support has been installed, ggd2 will query the cluster configuration database (CDB) for GRIO volume configuration information. ggd2 dynamically tracks updates to CDB. The standard cluster administration tools such as cmgr(1M) are used to configure shared GRIO volumes. The cmgr commands to mark a filesystem as GRIO-managed, and set its qualified bandwidth are shown below: $ /usr/cluster/bin/cmgr Welcome to SGI Cluster Manager Command-Line Interface cmgr> modify cxfs_filesystem <fs_name> in cluster <cluster_name> cmgr> set grio_managed to true cmgr> set grio_qualified_bandwidth to <qualified_bandwidth> cmgr> done To show the current status of a shared filesystem: cmgr> show cxfs_filesystem <fs_name> in cluster <cluster_name> ... GRIO Managed Filesystem: true GRIO Managed Bandwidth: <qualified_bandwidth> ... Note that currently cmgr requires that a filesystem be disabled, and therefore unmounted, before it can be modified. This means that a GRIO volume must be taken offline while its parameters are adjusted. LICENSING The GRIO licensing regime controls a number of configuration parameters including the total number of active streams and the total aggregate qualified bandwidth of volumes under management. Separate license types are provided for the local and cluster volume domains, and license constraints are enforced for each volume domain separately. GRIO uses FLEXlm licensing, see license.dat(5) ggd2 checks the license at startup, whenever it detects a configuration change, or when it is explicitly signalled with a SIGHUP. License enforcement for streams is straightforward. The license for a given volume domain specifies a maximum number of active streams. All reservation requests above this limit are denied. In the case of bandwidth, the license specifies the maximum total aggregate qualified bandwidths for all volumes within the volume domain. For the local domain, this would be the sum of the qualified bandwidths listed in /etc/griotab. ggd2 validates the configuration at startup and whenever the configuration is changed. If the configuration of a volume domain is altered and becomes unlicensed ggd2 enters a passive mode in which all further requests pertaining to that domain, with the exception of release requests are denied. A message is sent to the syslog and the volume domain remains deactivated until the configuration returns to a licensed state. Another message will then be logged indicating the domain is again active. FILES /etc/griotab /var/flexlm/license.dat SEE ALSO grio2(5), grio_avail(3X), grio_bind(3X), grio_get_stream(3X), grio_modify(3X), grio_release(3X), grio_reserve(3X), grio_unbind(3X)