intro_ffio(3F)
INTRO_FFIO - Describes performance options available with the FFIO layers
As shipped in IRIX 6.5.19. Last changed in IRIX 6.5.19.
NAME INTRO_FFIO - Describes performance options available with the FFIO layers DESCRIPTION The Flexible File I/O (FFIO) system lets the user specify a comma- separated list of layers through which I/O data is to be passed. This is done by providing a value for the spec argument to the -F option on the assign(1) command. This specifies a class of processing to be done on the data. FFIO on IRIX systems The default layer for direct access on IRIX systems is the cache layer and it does not have the coherency to handle multiple processes doing I/O to the same file. The user must assign the direct access file to either the system or global layer for programs to work as expected with more than one process. On IRIX systems, the FFIO library calls aio_sgi_init the first time it issues an asynchronous I/O call. It passes the following parameters to aio_sgi_init: aio_numusers=MAX(64,sysconf(_SC_NPROC_CONF)) aio_threads=5 aio_locks=3 If a program is using multiple threads and asychronous I/O, it is important that the value of aio_numusers be at least as large as the number of sprocs and pthreads that the application contains. For more information, see the aio_sgi_init man page on IRIX systems. Users can change these values by setting the following environment variables to the desired value: * change FF_IO_AIO_THREADS to modify aio_threads * change FF_IO_AIO_LOCKS to modify aio_locks * change FF_IO_AIO_NUMUSERS to modify aio_numusers The following example causes aio_threads to be set to 8 when the FFIO routines call aio_sgi_init: setenv FF_IO_AIO_THREADS 8 Users can also supersede the FFIO library's call to aio_sgi_init by calling it themselves, before the first I/O statement in their program. The following layers can issue asynchronous I/O calls on IRIX systems: * cos: see the later description on this man page for a description of how the cos layer uses asynchronous I/O. * cachea and bufa: users should assume that these layers may issue asynchronous I/O calls. * system or syscall: these layers may issue asynchronous I/O calls if called from a BUFFER IN or BUFFER OUT statement, or from the cos or cachea layer. The system and syscall layer may also issue asynchronous I/O calls if called via ffreada(3C), ffwritea(3C), or fflistio(3C) (all deferred on IRIX systems). Specifying FFIO Layers The spec argument of the -F option of the assign(1) command comprises a list of layers or filters that are used to manipulate the data file as it is being read or written. The available layers include performance options (such as memory-resident and SDS-resident files) and the capability to read and write files in a variety of different vendors' blocking formats. Each layer spec is of the general form: class[.type[.subtype]][:num1]:[num2]:[num3]] Many of the layers also allow you to specify the numeric parameters with a keyword. For more information about FFIO, see the MIPSpro Application Programmer's I/O Guide. The spec argument can have the following values: Class Explanation bufa Asynchronous buffering layer. The bufa layer provides asynchronous buffering. It allows efficient sequential-access I/O. The num1 field represents the size in 4096-byte blocks of each buffer. The maximum value for num1 on IRIX systems is 32,767. The num2 field selects the number of buffers to be used. You can specify the numeric parameters with this alternate keyword syntax: bufa[.bufsize=num1][.num_buffers=num2] cache Cached file. The cache layer allows efficient random-access I/O, even when file access is clustered in several regions of a file. During reads and writes to the layer, cache buffers frequently must be preempted. The buffer chosen for preemption is always the least recently accessed buffer at the time of preemption. The options available are .mem, which specifies that buffers reside in memory or .sds, which specifies that buffers reside in the SDS. (.sds is not supported on Cray T3E systems or IRIX systems). Memory-resident buffers are the default. The numeric fields are as follows: * num1 is the size in 4096-byte blocks of each cache page. The maximum value for num1 on IRIX systems is 32,767. * num2 selects the number of cache pages to be used. * num3 is the size in 4096-byte blocks at which the cache layer attempts to bypass cache layer buffering. If a user's I/O request is larger than num3, the request might not be copied to a cache page. The default on IRIX systems is num3=num1. You can specify the numeric parameters with this alternate keyword syntax: cache[.type][.page_size=num1][.num_pages=num2] [.bypass_size=num3] cachea Asynchronously cached file. This type of processing usually performs well whenever the cache layer might be used. In addition, any sequential forward and sequential backward access through the file is detected. When sequential access patterns are detected while reading, asynchronous read-ahead is performed provided that the numbers of pages to read ahead has been specified. When writing, selective asynchronous write-behind is performed. The values for type are .mem, which specifies that buffers reside in memory, or .sds, which specifies that buffers reside in the SDS (.sds is not supported on Cray T3E systems or on IRIX systems). Memory-resident buffers are the default. The numeric fields are as follows: * num1 is the size in 4096-byte blocks of each cache page. The maximum value for num1 on IRIX systems is 32,767. * num2 selects the number of cache pages to be used. * num3 selects the number of pages to read ahead asynchronously. The default is 0. * num4 selects a shared cache number in the range of 1 to 15. If num4 is 0, a private cache is indicated. You can specify the numeric parameters with this alternate keyword syntax: cachea[.type][.page_size=num1][.num_pages=num2] [.max_lead=num3][.shared_cache=num4] Stacked shared cachea layers are not supported. cos or blocked COS blocking. Available on IRIX systems. If specified, type must be one of the following: type Action sync Uses a single buffer in the blocking and deblocking process. I/O is done strictly synchronously. async Divides the buffer into two parts and uses asynchronous I/O to transfer the blocked data between the buffer(s) and the logical device. When reading, asynchronous read-ahead is performed, and when writing, asynchronous write-behind is performed. To effectively use async, the buffer size should be at least twice the record length. auto Default (if type is not specified). Chooses either synchronous or asynchronous behavior depending on the buffer size. If the buffer size is less than 64 blocks, synchronous behavior is selected. If it is greater than or equal to 64 blocks, asynchronous behavior is selected. For num1, enter the desired buffer size in 4096-byte blocks (for example, -F cos:42 requests COS blocking and a 42-block buffer). The num1 value also determines the record size for underlying layers which perform record blocking. The underlying record size is num1 blocks if in synchronous mode and num1/2 or num1/2+1 blocks if in asynchronous mode. For an underlying tape layer, the record size is the tape block size. If not specified, the default buffer size is the larger of the following: the preferred I/O block size (see the stat(2) man page), or 48 blocks. Furthermore, if type is auto (the default), then the default buffer size is doubled if asynchronous mode is used (the cos layer automatically switches to asynchronous mode). You can specify the numeric parameters with this alternate keyword syntax: cos[.type][.bufsize=num1] event I/O layer monitoring. The event layer monitors I/O occurring between two layers on a per-file basis. This layer generates statistics in an ASCII log file; users can specify what type of report is generated. The event layer is enabled by default. Users do not have to relink their programs to study I/O performance. To generate information, rerun the program with the event layer specified on the assign command. Statistics are reported to stderr by default. The FF_IO_LOGFILE environment variable can be used to name a file to which statistics are written by the event layer. The default action is to overwrite the existing file. To append information to an existing file, specify a plus sign (+) before the file name. The event layer reports counts for read, read, write, and writea. These counts represent the number of calls made to to an FFIO layer entry point. In some cases, the system layer may actually use a different I/O system call, or multiple system calls. For example, the reada system call does not exist on IRIX systems, and the system layer reada entry point will use aio_read(). The report that is generated may include mention of the "lock" layer, even though the lock layer may not have been specified by the user. Enter one of the following for type: value Information reported nostat No statistical information is reported. summary Information on event types that occur at least once are reported. brief A one-line summary for layer activities is reported. f77 FORTRAN 77/UNIX Fortran record blocking. This is the common blocking format used by most FORTRAN 77 compilers on UNIX systems. Enter one of the following for type: type Format nonvax Default. Control words in a format common to machines such as the MC68000. vax VAX format (byte-swapped) control words. Not available on IRIX systems. The specification of vax or nonvax is not relevant to data conversion. For num1, enter the maximum record size in bytes. For num2, enter the working buffer size, in bytes. You can specify the numeric parameters with this alternate keyword syntax: f77[.type][.recsize=num1][.bufsize=num2] fd Connect to a specific file descriptor. Field num1 is the decimal value of the file descriptor. Classes named stdin, stdout, and stderr exist as alternate names for fd:0, fd:1, and fd:2. You can specify the numeric parameters with this alternate keyword syntax: fd[.file_descriptor=num1] global File global to all processes. Available on IRIX systems. This is a caching layer which distributes the cache across multiple SHMEM or MPI processes. Open and close operations are collective (require participation by all processes which access the file). All other operations are independently performed by one or more processes. The library allows multiple processes to concurrently access the file while maintaining coherency of buffered data. Specify one of the following options for type: type Description privpos Default. The file position is private to a process. All processes may seek to independent locations in the file. globpos (Deferred). The file position is global to all processes. A seek or I/O operation by any process will affect the position for all processes. The numeric fields are as follows: num1 The size in 4096-byte blocks of each cache page. num2 Selects the number of cache pages to be used on each process. If there are n processes, then n * num2 cache pages are used. num2 buffer pages are allocated on every process which shares access to a global file. File pages are direct- mapped onto processes such that page n of the file will always be cached on process (n mod NPES), where NPES is the total number of processes sharing access to the global file. Once the process is identified where caching of the file page will occur, a least-recently- used method is used to assign the file page to a cache page within the caching process. You can specify the numeric parameters with this alternate keyword syntax: global[.type][.page_size=num1][.num_pages=num2] ibm Record blocking for common record types on IBM operating systems. Specify one of the following record formats for type: type Format f Fixed-length record format. For num1, enter the logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes; if specified, num2 must be equal to num1. fb Fixed-length, blocked record format. For num1, enter the logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes; num2 must be an exact multiple of num1. u Undefined record format. For num1, enter the maximum record size, in 8-bit bytes. v Variable length record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes. vb Variable length blocked record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes. vbs Variable length blocked, spanned record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes. No subtype is accepted for the ibm class record formats. num1 does not need to be smaller than num2. You can specify the numeric parameters with this alternate keyword syntax: ibm[.type][.recsize=num1][.mbs=num2] null Syntactic no-op. No optional fields are accepted. null may be specified where syntax demands a value, but no function is desired. This does not perform the same function as /dev/null. site Site layer. This option lets a site add custom I/O handlers for specific needs at load time. See the MIPSpro Application Programmer's I/O Guide, for details. stdin, stdout, or stderr Connects to specific file descriptors 0, 1, and 2, respectively. (See the fd class.) system Generic system I/O layer. Selects bmx, er90, or syscall as appropriate. syscall System I/O call. Each I/O operation results in a corresponding system call. It has one optional parameter as follows: syscall[.cboption] The cbption can be one of the following values: aiocb The syscall layer is notified, via a signal, when the asynchronous I/O completes. noaiocb The syscall layer polls the completion status word to determine asynchronous I/O completion. This is the default value. text Special character terminated records. Enter one of the following for type: type Format nl Newline-separated records eof Newline-separated records and the special character sequence ~e on a line by itself delimiting EOF 205 CYBER 205-style text file ctss CTSS-style text format This class accepts num specifications. If specified, num1 represents the decimal value of the ASCII character to use to delimit records; the default varies depending on the type. For num2, enter the requested working buffer size in bytes. You can specify the numeric parameters with this alternate keyword syntax: text[.type][.newline=num1][.bufsize=num2] user User layer. This option allows a user to add custom I/O handlers for specific needs at load time. See the MIPSpro Application Programmer's I/O Guide, for details. vms Provides record blocking for common record types on VMS/MS operating systems. For type, enter one of the following record formats: type Format f Fixed-length records v Variable length records s Segmented variable records Each type accepts the following subtypes to specify the blocking format within the record type: subtype Format tape ANSI standard record format. This subtype should be used with labeled VAX/VMS tapes. bb Binary blocked format. This subtype should be used with files that are to be fetched or disposed with a BB or TB format, or with unlabeled magnetic tapes. This subtype requires an enclosing blocking; for example, vms.s.bb,bmx or vms.s.bb,cos. tr Transparent format. This class accepts num1 and num2 fields; they have a similar meaning to ibm class. For type s, num1 is ignored. For type f, num2 need not be a multiple of num1. You can specify the numeric parameters with this alternate keyword syntax: vms[.type][.subtype][.recsize=num1][.mbs=num2] EXAMPLES The following are example FFIO specifications. Example: The following example specifies FORTRAN 77 Fortran blocking with a working buffer of 128,000 bytes to allow efficient creation of logical records up to 127,992 bytes: f77::128000 Example: The following example specifies VMS V records with a maximum record size of 1000 bytes: vms.v.tr:1000 See the MIPSpro Application Programmer's I/O Guide, for more detailed information about FFIO specifications. NOTES Some FFIO specification requirements are not obvious. For example, CYBER 205 R-type records must be requested with blx.ctss,text.205. The Fortran I/O library checks for conflicting attributes when file name and unit attributes are both present during OPEN processing for a Fortran unit. The existence of an assign attribute for both the file and the unit results in an error condition. SEE ALSO MIPSpro Application Programmer's I/O Guide acptbad(3F), assign(3F), ffopen(3C), openms(3F), opendr (see openms(3F)), skipbad(3F) df(1), ln(1), setf(1), tpmnt(1), write(1) asgcmd(1), assign(1) ialloc(2), open(2) SDSALLOC(3F)