INTRO_FFIO(3F) Last changed: 3-11-98
INTRO_FFIO - Describes performance options available with the FFIO
layers
See individual man pages for implementation details.
The Flexible File I/O (FFIO) system lets the user specify a commaseparated
list of layers through which I/O data is to be passed. This
is done by providing a value for the spec argument to the -F option on
the assign(1) command. This specifies a class of processing to be
done on the data.
The following C routines are used with FFIO:
* fffcntl(3C): performs functions on files opened using FFIO
* fflistio(3C): initiates a list of I/O requests using FFIO
* ffiolock(3C): locks and unlocks function calls
* ffopen(3C): opens or closes a file using FFIO
* ffpos(3C): positions files opened using FFIO
* ffread(3C): reads from a file using FFIO
* ffreada(3C): provides asynchronous read using FFIO
* ffseek(3C): repositions a FFIO file
* ffwritea(3C): provides asynchronous write using FFIO
* ffsetsp(3C): initiates EOV processing for files opened using FFIO
* ffwrite(3C): writes to a file using FFIO
FFIO on IRIX systems
On IRIX systems, the FFIO library calls aio_sgi_init the first time it
issues an asynchronous I/O call. It passes the following parameters
to aio_sgi_init:
aio_numusers=MAX(64,sysconf(_SC_NPROC_CONF))
aio_threads=5
aio_locks=3
If a program is using multiple threads and asychronous I/O, it is
important that the value of aio_numusers be at least as large as the
number of sprocs and pthreads that the application contains. For more
information, see the aio_sgi_init man page on IRIX systems.
Users can change these values by setting the following environment
variables to the desired value:
* change FF_IO_AIO_THREADS to modify aio_threads
* change FF_IO_AIO_LOCKS to modify aio_locks
* change FF_IO_AIO_NUMUSERS to modify aio_numusers
The following example causes aio_threads to be set to 8 when the FFIO
routines call aio_sgi_init:
setenv FF_IO_AIO_THREADS 8
Users can also supersede the FFIO library's call to aio_sgi_init by
calling it themselves, before the first I/O statement in their
program.
The following layers can issue asynchronous I/O calls on IRIX systems:
* cos: see the later description on this man page for a description of
how the cos layer uses asynchronous I/O.
* cachea and bufa: users should assume that these layers may issue
asynchronous I/O calls.
* system or syscall: these layers may issue asynchronous I/O calls if
called from a BUFFER IN or BUFFER OUT statement, or from the cos or
cachea layer. The system and syscall layer may also issue
asynchronous I/O calls if called via ffreada(3C), ffwritea(3C), or
fflistio(3C) (all deferred on IRIX systems).
Specifying FFIO Layers [Toc] [Back]
The spec argument of the -F option of the assign(1) command comprises
a list of layers or filters that are used to manipulate the data file
as it is being read or written. The available layers include
performance options (such as memory-resident and SDS-resident files)
and the capability to read and write files in a variety of different
vendors' blocking formats. Each layer spec is of the general form:
class[.type[.subtype]][:num1]:[num2]:[num3]]
Many of the layers also allow you to specify the numeric parameters
with a keyword. On UNICOS and UNICOS/mk systems, this requires that
your application be linked with Cray Research's CrayLibs 3.0 or later
releases. The format for these specifications is given in the
description of each layer where the specification is available.
Selected layers are available on IRIX systems. The available
platforms are detailed in each layer's description.
For more information about FFIO, see the Application Programmer's I/O
Guide, publication SG-2168.
The spec argument can have the following values:
Class Explanation
blankx or blx
Blank compression filter. This layer is used as a
filter to compress or decompress blanks in character
data.
Not available on IRIX systems.
type can be one of the following:
cos COS-style blank compression (blxchr=27 or Ox10)
ctss CTSS-style blank compression (blxchr=48 or 0x30)
c205 CYBER 205-style blank compression (blxchr=48 or
0x30)
num1 is the decimal value of the ASCII character used as
the escape code to control the blank compression
(usually ESC or 27). num2 is the decimal value of the
ASCII character that is the object of the compression
(usually BLANK or 32).
You can specify the numeric parameters with this
alternate keyword syntax:
blx[.type][.blxchr=num1][.blnk=num2]
blankx[.type][.blxchr=num1][.blnk=num2]
bmx Tape I/O. Each logical record requested is a physical
tape block.
Deferred implementation on IRIX systems.
The num1 field represents the size of each buffer, in
4096-byte blocks. The num2 field represents the number
of buffers.
On UNICOS systems, files on ER90 volumes that have been
mounted in blocked mode may use this layer (see the Tape
Subsystem User's Guide, publication SG-2051, for
information about restrictions on record sizes when
using ER90 block mode).
You can specify the numeric parameters with this
alternate keyword syntax:
bmx[.bufsize=num1][.num_buffers=num2]
bufa Asynchronous buffering layer.
Available on IRIX systems.
The bufa layer provides asynchronous buffering. It
allows efficient sequential-access I/O.
The num1 field represents the size in 4096-byte blocks
of each buffer. The maximum value for num1 on IRIX
systems is 32,767. The maximum allowed value on UNICOS
and UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount of
memory may not be available).
The num2 field selects the number of buffers to be used.
You can specify the numeric parameters with this
alternate keyword syntax:
bufa[.bufsize=num1][.num_buffers=num2]
c205 CDC CYBER 205/ETA records.
Not available on IRIX systems.
The CYBER 205-type W record can be selected by
specifying c205.w. The num1 field is not permitted.
The num2 field represents the size of the working
buffer, in bytes, used in record blocking. If any
logical records written exceed this size, a major
performance penalty can result.
The type field should be specified as w. The best
performance is currently obtained when num2 is greater
than the largest record to be processed plus 16 bytes.
You can specify the numeric parameters with this
alternate keyword syntax:
c205[.w][.bufsize=num2]
cache Cached file. The cache layer allows efficient
random-access I/O, even when file access is clustered in
several regions of a file.
Available on IRIX systems.
During reads and writes to the layer, cache buffers
frequently must be preempted. The buffer chosen for
preemption is always the least recently accessed buffer
at the time of preemption.
The options available are .mem, which specifies that
buffers reside in memory or .sds, which specifies that
buffers reside in the SDS. (.sds is not supported on
CRAY T3E systems or IRIX systems). Memory-resident
buffers are the default.
The numeric fields are as follows:
* num1 is the size in 4096-byte blocks of each cache
page. The maximum value for num1 on IRIX systems is
32,767. The maximum allowed value on UNICOS and
UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount
of memory may not be available).
* num2 selects the number of cache pages to be used.
* num3 is the size in 4096-byte blocks at which the
cache layer attempts to bypass cache layer buffering.
If a user's I/O request is larger than num3, the
request might not be copied to a cache page. The
default on IRIX systems is num3=num1. The default on
UNICOS and UNICOS/mk systems is num3=num1*num2.
You can specify the numeric parameters with this
alternate keyword syntax:
cache[.type][.page_size=num1][.num_pages=num2]
[.bypass_size=num3]
cachea Asynchronously cached file.
Available on IRIX systems.
This type of processing usually performs well whenever
the cache layer might be used. In addition, any
sequential forward and sequential backward access
through the file is detected. When sequential access
patterns are detected while reading, asynchronous
read-ahead is performed provided that the numbers of
pages to read ahead has been specified. When writing,
selective asynchronous write-behind is performed.
The values for type are .mem, which specifies that
buffers reside in memory, or .sds, which specifies that
buffers reside in the SDS (.sds is not supported on
CRAY T3E systems or on IRIX systems). Memory-resident
buffers are the default.
The numeric fields are as follows:
* num1 is the size in 4096-byte blocks of each cache
page. The maximum value for num1 on IRIX systems is
32,767. The maximum allowed value on UNICOS and
UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount
of memory may not be available).
* num2 selects the number of cache pages to be used.
* num3 selects the number of pages to read ahead
asynchronously. The default is 0.
* num4 selects a shared cache number in the range of 1
to 15. If num4 is 0, a private cache is indicated.
You can specify the numeric parameters with this
alternate keyword syntax:
cachea[.type][.page_size=num1][.num_pages=num2]
[.max_lead=num3][.shared_cache=num4]
On IRIX systems, stacked shared cachea layers are not
supported.
On UNICOS and UNICOS/mk systems, stacked shared cachea
layers are supported, but in multitasked programs,
different files must not mix the order of the shared
caches.
Examples:
The following specifications cannot both be used by a
multitasked program:
* assign -F cachea::::1,cachea::::2 u:1
* assign -F cachea::::2,cachea::::1 u:2
The following specifications can both be used by a
multitasked program on a UNICOS system:
* assign -F cachea::::1,cachea::::2 u:1
* assign -F cachea::::1,cachea::::2 u:2
cdc CDC 60-bit format type.
Not available on IRIX systems.
No numeric fields are accepted.
The supported values for type are as follows:
type Format
iw I-type blocks, W-type records
cw C-type blocks, W-type records
cs C-type blocks, S-type records
cz C-type blocks, Z-type records
The subtype field accepts one of the following values
that indicate the presence of block trailers and other
low-level characteristics:
subtype Value
disk Disk type structure, for use with station
transfers of CYBER datasets
i NOS internal tape format
si System internal or SCOPE internal tape format
cos or blocked
COS blocking.
Available on IRIX systems.
If specified, type must be one of the following:
type Action
sync Uses a single buffer in the blocking and
deblocking process. I/O is done strictly
synchronously.
async Divides the buffer into two parts and uses
asynchronous I/O to transfer the blocked data
between the buffer(s) and the logical device.
When reading, asynchronous read-ahead is
performed, and when writing, asynchronous
write-behind is performed. To effectively
use async, the buffer size should be at least
twice the record length.
auto Default (if type is not specified). Chooses
either synchronous or asynchronous behavior
depending on the buffer size. If the buffer
size is less than 64 blocks, synchronous
behavior is selected. If it is greater than
or equal to 64 blocks, asynchronous behavior
is selected.
For num1, enter the desired buffer size in 4096-byte
blocks (for example, -F cos:42 requests COS blocking and
a 42-block buffer). The num1 value also determines the
record size for underlying layers which perform record
blocking. The underlying record size is num1 blocks if
in synchronous mode and num1/2 or num1/2+1 blocks if in
asynchronous mode. For an underlying tape layer, the
record size is the tape block size.
You can specify the numeric parameters with this
alternate keyword syntax:
cos[.type][.bufsize=num1]
er90 ER90 I/O.
Not available on IRIX systems or on CRAY T3E systems.
Each I/O operation results in a corresponding system
call. Adjacent records are not delimited from one
another. No subfields are accepted. The byte-stream
mode of the device must be used. See tpmnt(1).
event I/O layer monitoring.
Available on Cray PVP systems and CRAY T3E systems.
Deferred implementation on IRIX systems.
The event layer monitors I/O occurring between two
layers on a per-file basis. This layer generates
statistics in an ASCII log file; users can specify what
type of report is generated. The event layer is enabled
by default. Users do not have to relink their programs
to study I/O performance. To generate information,
rerun the program with the event layer specified on the
assign command.
Statistics are reported to stderr by default. The
FF_IO_LOGFILE environment variable can be used to name a
file to which statistics are written by the event layer.
The default action is to overwrite the existing file.
To append information to an existing file, specify a
plus sign (+) before the file name.
On CRAY T3E systems, if more than one process is using
the event layer, and you set the FF_IO_LOGFILE
environment variable, you must use the plus sign (+)
before the file name to prevent process a from
overwriting the information written by process b. Using
the plus sign also means that the information will be
appended to an existing file.
On CRAY T3E systems you can also use the FF_IO_LOGFILEPE
environment variable to name a file to which statistics
are written. The file name will be x.n, where x is the
name specified by the environment variable and n is the
number of the PE which wrote the file. The default
action is to overwrite the existing file. To append
information to the existing file, specify a plus sign
(+) before the file name.
Enter one of the following for type:
value Information reported
nostat No statistical information is reported.
summary Information on event types that occur at
least once are reported.
brief A one-line summary for layer activities
is reported.
f77 FORTRAN 77/UNIX Fortran record blocking. This is the
common blocking format used by most FORTRAN 77 compilers
on UNIX systems.
Available on IRIX systems.
Enter one of the following for type:
type Format
nonvax Default. Control words in a format common to
machines such as the MC68000.
vax VAX format (byte-swapped) control words. Not
available on IRIX systems.
The specification of vax or nonvax is not
relevant to data conversion.
For num1, enter the maximum record size in bytes. For
num2, enter the working buffer size, in bytes.
You can specify the numeric parameters with this
alternate keyword syntax:
f77[.type][.recsize=num1][.bufsize=num2]
fd Connect to a specific file descriptor.
Available on IRIX systems.
Field num1 is the decimal value of the file descriptor.
Classes named stdin, stdout, and stderr exist as
alternate names for fd:0, fd:1, and fd:2.
You can specify the numeric parameters with this
alternate keyword syntax:
fd[.file_descriptor=num1]
global File global to all PEs.
Available on IRIX systems.
This is a caching layer which distributes the cache
across multiple SHMEM or MPI processes. Open and close
operations are collective (require participation by all
processes which access the file). All other operations
are independently performed by one or more processes.
The library allows multiple processes to concurrently
access the file while maintaining coherency of buffered
data.
Specify one of the following options for type:
type Description
privpos Default. The file position is private to
a process. All processes may seek to
independent locations in the file.
globpos (Deferred). The file position is global
to all processes. A seek or I/O
operation by any process will affect the
position for all processes.
The numeric fields are as follows:
num1 The size in 4096-byte blocks of each cache page.
num2 Selects the number of cache pages to be used on
each process. If there are n processes, then n *
num2 cache pages are used.
You can specify the numeric parameters with this
alternate keyword syntax:
global[.type][.page_size=num1][.num_pages=num2]
ibm Record blocking for common record types on IBM operating
systems.
Deferred implementation on IRIX systems.
Specify one of the following record formats for type:
type Format
f Fixed-length record format. For num1, enter the
logical record size in 8-bit bytes. For num2,
enter the maximum physical block size in 8-bit
bytes; if specified, num2 must be equal to num1.
fb Fixed-length, blocked record format. For num1,
enter the logical record size in 8-bit bytes.
For num2, enter the maximum physical block size
in 8-bit bytes; num2 must be an exact multiple of
num1.
u Undefined record format. For num1, enter the
maximum record size, in 8-bit bytes.
v Variable length record format. For num1, enter
the maximum logical record size in 8-bit bytes.
For num2, enter the maximum physical block size
in 8-bit bytes.
vb Variable length blocked record format. For num1,
enter the maximum logical record size in 8-bit
bytes. For num2, enter the maximum physical
block size in 8-bit bytes.
vbs Variable length blocked, spanned record format.
For num1, enter the maximum logical record size
in 8-bit bytes. For num2, enter the maximum
physical block size in 8-bit bytes.
No subtype is accepted for the ibm class record formats.
num1 does not need to be smaller than num2.
You can specify the numeric parameters with this
alternate keyword syntax:
ibm[.type][.recsize=num1][.mbs=num2]
mr or sds Memory-resident and SDS-resident files. mr is a
deferred implementation on IRIX systems. sds is not
available on CRAY T3E systems or on IRIX systems.
The mr and sds layers let you declare a byte-stream
file, but the given file should reside in the specified
medium as much as possible; this can improve
performance.
Enter one of the following for type:
type Load Specification
save Default. Loads the file into the specified
medium when the file is opened and writes the
data back out when closed.
scr Scratch file. Does not attempt to load at
open and discards data on close.
Enter one of the following for subtype:
subtype Overflow Specification
ovfl Default. Writes excess data that does not
fit in the specified medium to the next lower
layer.
novfl If data does not fit in the specified medium,
subsequent write(1) operations fail.
The mr and sds layers accept numeric fields. Generally,
the numeric field specification represents best-efforts
values. They are used for tuning purposes and usually
do not cause errors if they are not satisfied precisely
as specified. The fields are as follows:
Field Value
num1 Initial size of the memory or SDS allocation.
Specified in 4096-byte blocks. Default is 0.
num2 Maximum size of the memory or SDS allocation.
Specified in 4096-byte blocks. Default is
46
s -1.
num3 Increment size of the memory or SDS allocation.
Specified in 4096-byte blocks. This value is
used when allocating additional memory or SDS
space. The default is 256 for SDS files and 32
for memory resident files.
For example, if the SDS limit for a process is 1000
blocks, and sds.scr:1500 is specified, an initial SDS
allocation of 1000 blocks is used.
Similarly, if sds:500:10000:1000 was specified, the
first request for SDS space would result in the
allocation of 500 blocks. If possible, this is
allocated contiguously, but it is not guaranteed. If a
file size exceeds 500 blocks, an attempt is made to
increase the file by an increment of 1000 blocks. If
this attempt fails, whatever can be allocated is given
to the file. Assuming no other SDS use, this is 500
blocks. At this point, the file assumes an overflow
status.
When using memory-resident files, you must ensure that
not too much data is written to the file without
limiting its size. Unrestrained and unmanaged growth of
such a file can cause heap fragmentation and program
abort. If this growth has used all available memory,
the error message processor may be unable to issue a
message. Therefore, such an abort may be difficult to
diagnose and correct. It is recommended that the
maximum memory size field be specified in all cases.
(For example, mr.scr::10000 would limit such a file to
approximately 5 Mwords of main memory.)
You can specify the numeric parameters with this
alternate keyword syntax:
mr[.type[.subtype]][.start_size=num1] [.max_size=num2]
[.inc_size=num3]
sds[.type[.subtype]][.start_size=num1] [.max_size=num2]
[.inc_size=num3]
null Syntactic no-op.
Available on IRIX systems.
No optional fields are accepted. null may be specified
where syntax demands a value, but no function is
desired. This does not perform the same function as
/dev/null.
nosve CDC NOS/VE record formats.
Not available on IRIX systems.
The following values are allowed for the type and num
fields:
type Value
d ANSI D format (variable-length) records. For
num1, enter the maximum logical record size in
8-bit bytes. For num2, enter the maximum
physical block size in 8-bit bytes.
f ANSI F fixed-length records. For num1, enter
the logical record size in 8-bit bytes. For
num2, enter the maximum physical block size in
8-bit bytes; num2 must be an exact multiple of
num1.
s ANSI S format (segmented) records. For num1,
enter the maximum logical record size. For
num2, enter the maximum physical block size used
in the lower-level layers.
u Undefined records. For num1, enter the maximum
logical record size, in 8-bit bytes. For num2,
enter the maximum physical block size in 8-bit
bytes.
v NOS/VE V format records. The num1 field is not
permitted. For num2, specify the size of the
working buffer used in record blocking; if any
logical records that are written exceed this
size, a major performance penalty can result.
You can specify the numeric parameters with this
alternate keyword syntax (as noted previously, .recsize
may not be permitted for some types):
nosve[.type][.recsize=num1][.mbs=num2]
site Site layer. This option lets a site add custom I/O
handlers for specific needs at load time. See the
Application Programmer's I/O Guide, publication SG-2168,
for details.
Available on IRIX systems.
stdin, stdout, or stderr
Connects to specific file descriptors 0, 1, and 2,
respectively. (See the fd class.)
system Generic system I/O layer. Selects bmx, er90, or syscall
as appropriate.
Available on IRIX systems.
syscall System I/O call. Each I/O operation results in a
corresponding system call.
Available on IRIX systems.
tape Tape I/O. Each logical record requested is a physical
tape block. This is the same as bmx.
Deferred implementation on IRIX systems.
The num1 field represents the size of each buffer, in
4096-byte blocks. The num2 field represents the number
of buffers.
You can specify the numeric parameters with this
alternate keyword syntax:
tape[.bufsize=num1][.num_buffers=num2]
text Special character terminated records.
Available on IRIX systems.
Enter one of the following for type:
type Format
nl Newline-separated records
eof Newline-separated records and the special
character sequence ~e on a line by itself
delimiting EOF
205 CYBER 205-style text file
ctss CTSS-style text format
This class accepts num specifications. If specified,
num1 represents the decimal value of the ASCII character
to use to delimit records; the default varies depending
on the type. For num2, enter the requested working
buffer size in bytes.
You can specify the numeric parameters with this
alternate keyword syntax:
text[.type][.newline=num1][.bufsize=num2]
user User layer. This option allows a user to add custom I/O
handlers for specific needs at load time. See the
Application Programmer's I/O Guide, publication SG-2168,
for details.
Available on IRIX systems.
vms Provides record blocking for common record types on
VMS/MS operating systems.
Deferred implementation on IRIX systems.
For type, enter one of the following record formats:
type Format
f Fixed-length records
v Variable length records
s Segmented variable records
Each type accepts the following subtypes to specify the
blocking format within the record type:
subtype Format
tape ANSI standard record format. This subtype
should be used with labeled VAX/VMS tapes.
bb Binary blocked format. This subtype should be
used with files that are to be fetched or
disposed with a BB or TB format, or with
unlabeled magnetic tapes. This subtype requires
an enclosing blocking; for example, vms.s.bb,bmx
or vms.s.bb,cos.
tr Transparent format. This subtype should be used
with files that are transferred between the
VAX/VMS system by using fetch or dispose and the
TR format. Any other method that precisely
transfers the disk image, including all VMS
control words, will also work. Note that ftp
does not correctly transfer nontext,
variable-length record files.
This class accepts num1 and num2 fields; they have a
similar meaning to ibm class. For type s, num1 is
ignored. For type f, num2 need not be a multiple of
num1.
You can specify the numeric parameters with this
alternate keyword syntax:
vms[.type][.subtype][.recsize=num1][.mbs=num2]
The following are example FFIO specifications.
Example 1: The following example specifies FORTRAN 77/UNICOS Fortran
blocking with a working buffer of 128,000 bytes to allow efficient
creation of logical records up to 127,992 bytes:
f77::128000
Example 2: The following example specifies IBM VBS records with a
block size of 16,384 bytes:
ibm.vbs::16384
Example 3: The following example specifies a memory-resident layer
with an initial memory size of 100 4096-byte blocks:
mr:100
Example 4: The following example specifies VMS V records with a
maximum record size of 1000 bytes:
vms.v.tr:1000
Example 5: The following example specifies the SS-resident layer that
loads from disk on open and saves to disk on close, does not allow
overflow, has an initial size of 10 4096-byte blocks in SDS, grows to
a maximum size of 559 blocks in minimum increments of 37 blocks:
sds.save.novfl:10:559:37
See the Application Programmer's I/O Guide, publication SG-2168, for
more detailed information about FFIO specifications.
Some FFIO specification requirements are not obvious. For example,
CYBER 205 R-type records must be requested with blx.ctss,text.205.
The Fortran I/O library checks for conflicting attributes when file
name and unit attributes are both present during OPEN processing for a
Fortran unit. The existence of an assign attribute for both the file
and the unit results in an error condition.
Application Programmer's I/O Guide, publication SG-2168
acptbad(3F), assign(3F), ffopen(3C), openms(3F), opendr (see
openms(3F)), skipbad(3F)
df(1), ln(1), setf(1), tpmnt(1), write(1)
asgcmd(1), assign(1)
ialloc(2), open(2)
SDSALLOC(3F)
Application Programmer's Library Reference Manual, publication
SR-2165, for the printed version of this man page.
INTRO_FFIO(3F) Last changed: 3-11-98
INTRO_FFIO - Describes performance options available with the FFIO
layers
See individual man pages for implementation details.
The Flexible File I/O (FFIO) system lets the user specify a commaseparated
list of layers through which I/O data is to be passed. This
is done by providing a value for the spec argument to the -F option on
the assign(1) command. This specifies a class of processing to be
done on the data.
The following C routines are used with FFIO:
* fffcntl(3C): performs functions on files opened using FFIO
* fflistio(3C): initiates a list of I/O requests using FFIO
* ffiolock(3C): locks and unlocks function calls
* ffopen(3C): opens or closes a file using FFIO
* ffpos(3C): positions files opened using FFIO
* ffread(3C): reads from a file using FFIO
* ffreada(3C): provides asynchronous read using FFIO
* ffseek(3C): repositions a FFIO file
* ffwritea(3C): provides asynchronous write using FFIO
* ffsetsp(3C): initiates EOV processing for files opened using FFIO
* ffwrite(3C): writes to a file using FFIO
FFIO on IRIX systems
On IRIX systems, the FFIO library calls aio_sgi_init the first time it
issues an asynchronous I/O call. It passes the following parameters
to aio_sgi_init:
aio_numusers=MAX(64,sysconf(_SC_NPROC_CONF))
aio_threads=5
aio_locks=3
If a program is using multiple threads and asychronous I/O, it is
important that the value of aio_numusers be at least as large as the
number of sprocs and pthreads that the application contains. For more
information, see the aio_sgi_init man page on IRIX systems.
Users can change these values by setting the following environment
variables to the desired value:
* change FF_IO_AIO_THREADS to modify aio_threads
* change FF_IO_AIO_LOCKS to modify aio_locks
* change FF_IO_AIO_NUMUSERS to modify aio_numusers
The following example causes aio_threads to be set to 8 when the FFIO
routines call aio_sgi_init:
setenv FF_IO_AIO_THREADS 8
Users can also supersede the FFIO library's call to aio_sgi_init by
calling it themselves, before the first I/O statement in their
program.
The following layers can issue asynchronous I/O calls on IRIX systems:
* cos: see the later description on this man page for a description of
how the cos layer uses asynchronous I/O.
* cachea and bufa: users should assume that these layers may issue
asynchronous I/O calls.
* system or syscall: these layers may issue asynchronous I/O calls if
called from a BUFFER IN or BUFFER OUT statement, or from the cos or
cachea layer. The system and syscall layer may also issue
asynchronous I/O calls if called via ffreada(3C), ffwritea(3C), or
fflistio(3C) (all deferred on IRIX systems).
Specifying FFIO Layers [Toc] [Back]
The spec argument of the -F option of the assign(1) command comprises
a list of layers or filters that are used to manipulate the data file
as it is being read or written. The available layers include
performance options (such as memory-resident and SDS-resident files)
and the capability to read and write files in a variety of different
vendors' blocking formats. Each layer spec is of the general form:
class[.type[.subtype]][:num1]:[num2]:[num3]]
Many of the layers also allow you to specify the numeric parameters
with a keyword. On UNICOS and UNICOS/mk systems, this requires that
your application be linked with Cray Research's CrayLibs 3.0 or later
releases. The format for these specifications is given in the
description of each layer where the specification is available.
Selected layers are available on IRIX systems. The available
platforms are detailed in each layer's description.
For more information about FFIO, see the Application Programmer's I/O
Guide, publication SG-2168.
The spec argument can have the following values:
Class Explanation
blankx or blx
Blank compression filter. This layer is used as a
filter to compress or decompress blanks in character
data.
Not available on IRIX systems.
type can be one of the following:
cos COS-style blank compression (blxchr=27 or Ox10)
ctss CTSS-style blank compression (blxchr=48 or 0x30)
c205 CYBER 205-style blank compression (blxchr=48 or
0x30)
num1 is the decimal value of the ASCII character used as
the escape code to control the blank compression
(usually ESC or 27). num2 is the decimal value of the
ASCII character that is the object of the compression
(usually BLANK or 32).
You can specify the numeric parameters with this
alternate keyword syntax:
blx[.type][.blxchr=num1][.blnk=num2]
blankx[.type][.blxchr=num1][.blnk=num2]
bmx Tape I/O. Each logical record requested is a physical
tape block.
Deferred implementation on IRIX systems.
The num1 field represents the size of each buffer, in
4096-byte blocks. The num2 field represents the number
of buffers.
On UNICOS systems, files on ER90 volumes that have been
mounted in blocked mode may use this layer (see the Tape
Subsystem User's Guide, publication SG-2051, for
information about restrictions on record sizes when
using ER90 block mode).
You can specify the numeric parameters with this
alternate keyword syntax:
bmx[.bufsize=num1][.num_buffers=num2]
bufa Asynchronous buffering layer.
Available on IRIX systems.
The bufa layer provides asynchronous buffering. It
allows efficient sequential-access I/O.
The num1 field represents the size in 4096-byte blocks
of each buffer. The maximum value for num1 on IRIX
systems is 32,767. The maximum allowed value on UNICOS
and UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount of
memory may not be available).
The num2 field selects the number of buffers to be used.
You can specify the numeric parameters with this
alternate keyword syntax:
bufa[.bufsize=num1][.num_buffers=num2]
c205 CDC CYBER 205/ETA records.
Not available on IRIX systems.
The CYBER 205-type W record can be selected by
specifying c205.w. The num1 field is not permitted.
The num2 field represents the size of the working
buffer, in bytes, used in record blocking. If any
logical records written exceed this size, a major
performance penalty can result.
The type field should be specified as w. The best
performance is currently obtained when num2 is greater
than the largest record to be processed plus 16 bytes.
You can specify the numeric parameters with this
alternate keyword syntax:
c205[.w][.bufsize=num2]
cache Cached file. The cache layer allows efficient
random-access I/O, even when file access is clustered in
several regions of a file.
Available on IRIX systems.
During reads and writes to the layer, cache buffers
frequently must be preempted. The buffer chosen for
preemption is always the least recently accessed buffer
at the time of preemption.
The options available are .mem, which specifies that
buffers reside in memory or .sds, which specifies that
buffers reside in the SDS. (.sds is not supported on
CRAY T3E systems or IRIX systems). Memory-resident
buffers are the default.
The numeric fields are as follows:
* num1 is the size in 4096-byte blocks of each cache
page. The maximum value for num1 on IRIX systems is
32,767. The maximum allowed value on UNICOS and
UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount
of memory may not be available).
* num2 selects the number of cache pages to be used.
* num3 is the size in 4096-byte blocks at which the
cache layer attempts to bypass cache layer buffering.
If a user's I/O request is larger than num3, the
request might not be copied to a cache page. The
default on IRIX systems is num3=num1. The default on
UNICOS and UNICOS/mk systems is num3=num1*num2.
You can specify the numeric parameters with this
alternate keyword syntax:
cache[.type][.page_size=num1][.num_pages=num2]
[.bypass_size=num3]
cachea Asynchronously cached file.
Available on IRIX systems.
This type of processing usually performs well whenever
the cache layer might be used. In addition, any
sequential forward and sequential backward access
through the file is detected. When sequential access
patterns are detected while reading, asynchronous
read-ahead is performed provided that the numbers of
pages to read ahead has been specified. When writing,
selective asynchronous write-behind is performed.
The values for type are .mem, which specifies that
buffers reside in memory, or .sds, which specifies that
buffers reside in the SDS (.sds is not supported on
CRAY T3E systems or on IRIX systems). Memory-resident
buffers are the default.
The numeric fields are as follows:
* num1 is the size in 4096-byte blocks of each cache
page. The maximum value for num1 on IRIX systems is
32,767. The maximum allowed value on UNICOS and
UNICOS/mk systems is 1,073,741,823 (you may not be
able to use a value of this size because that amount
of memory may not be available).
* num2 selects the number of cache pages to be used.
* num3 selects the number of pages to read ahead
asynchronously. The default is 0.
* num4 selects a shared cache number in the range of 1
to 15. If num4 is 0, a private cache is indicated.
You can specify the numeric parameters with this
alternate keyword syntax:
cachea[.type][.page_size=num1][.num_pages=num2]
[.max_lead=num3][.shared_cache=num4]
On IRIX systems, stacked shared cachea layers are not
supported.
On UNICOS and UNICOS/mk systems, stacked shared cachea
layers are supported, but in multitasked programs,
different files must not mix the order of the shared
caches.
Examples:
The following specifications cannot both be used by a
multitasked program:
* assign -F cachea::::1,cachea::::2 u:1
* assign -F cachea::::2,cachea::::1 u:2
The following specifications can both be used by a
multitasked program on a UNICOS system:
* assign -F cachea::::1,cachea::::2 u:1
* assign -F cachea::::1,cachea::::2 u:2
cdc CDC 60-bit format type.
Not available on IRIX systems.
No numeric fields are accepted.
The supported values for type are as follows:
type Format
iw I-type blocks, W-type records
cw C-type blocks, W-type records
cs C-type blocks, S-type records
cz C-type blocks, Z-type records
The subtype field accepts one of the following values
that indicate the presence of block trailers and other
low-level characteristics:
subtype Value
disk Disk type structure, for use with station
transfers of CYBER datasets
i NOS internal tape format
si System internal or SCOPE internal tape format
cos or blocked
COS blocking.
Available on IRIX systems.
If specified, type must be one of the following:
type Action
sync Uses a single buffer in the blocking and
deblocking process. I/O is done strictly
synchronously.
async Divides the buffer into two parts and uses
asynchronous I/O to transfer the blocked data
between the buffer(s) and the logical device.
When reading, asynchronous read-ahead is
performed, and when writing, asynchronous
write-behind is performed. To effectively
use async, the buffer size should be at least
twice the record length.
auto Default (if type is not specified). Chooses
either synchronous or asynchronous behavior
depending on the buffer size. If the buffer
size is less than 64 blocks, synchronous
behavior is selected. If it is greater than
or equal to 64 blocks, asynchronous behavior
is selected.
For num1, enter the desired buffer size in 4096-byte
blocks (for example, -F cos:42 requests COS blocking and
a 42-block buffer). The num1 value also determines the
record size for underlying layers which perform record
blocking. The underlying record size is num1 blocks if
in synchronous mode and num1/2 or num1/2+1 blocks if in
asynchronous mode. For an underlying tape layer, the
record size is the tape block size.
You can specify the numeric parameters with this
alternate keyword syntax:
cos[.type][.bufsize=num1]
er90 ER90 I/O.
Not available on IRIX systems or on CRAY T3E systems.
Each I/O operation results in a corresponding system
call. Adjacent records are not delimited from one
another. No subfields are accepted. The byte-stream
mode of the device must be used. See tpmnt(1).
event I/O layer monitoring.
Available on Cray PVP systems and CRAY T3E systems.
Deferred implementation on IRIX systems.
The event layer monitors I/O occurring between two
layers on a per-file basis. This layer generates
statistics in an ASCII log file; users can specify what
type of report is generated. The event layer is enabled
by default. Users do not have to relink their programs
to study I/O performance. To generate information,
rerun the program with the event layer specified on the
assign command.
Statistics are reported to stderr by default. The
FF_IO_LOGFILE environment variable can be used to name a
file to which statistics are written by the event layer.
The default action is to overwrite the existing file.
To append information to an existing file, specify a
plus sign (+) before the file name.
On CRAY T3E systems, if more than one process is using
the event layer, and you set the FF_IO_LOGFILE
environment variable, you must use the plus sign (+)
before the file name to prevent process a from
overwriting the information written by process b. Using
the plus sign also means that the information will be
appended to an existing file.
On CRAY T3E systems you can also use the FF_IO_LOGFILEPE
environment variable to name a file to which statistics
are written. The file name will be x.n, where x is the
name specified by the environment variable and n is the
number of the PE which wrote the file. The default
action is to overwrite the existing file. To append
information to the existing file, specify a plus sign
(+) before the file name.
Enter one of the following for type:
value Information reported
nostat No statistical information is reported.
summary Information on event types that occur at
least once are reported.
brief A one-line summary for layer activities
is reported.
f77 FORTRAN 77/UNIX Fortran record blocking. This is the
common blocking format used by most FORTRAN 77 compilers
on UNIX systems.
Available on IRIX systems.
Enter one of the following for type:
type Format
nonvax Default. Control words in a format common to
machines such as the MC68000.
vax VAX format (byte-swapped) control words. Not
available on IRIX systems.
The specification of vax or nonvax is not
relevant to data conversion.
For num1, enter the maximum record size in bytes. For
num2, enter the working buffer size, in bytes.
You can specify the numeric parameters with this
alternate keyword syntax:
f77[.type][.recsize=num1][.bufsize=num2]
fd Connect to a specific file descriptor.
Available on IRIX systems.
Field num1 is the decimal value of the file descriptor.
Classes named stdin, stdout, and stderr exist as
alternate names for fd:0, fd:1, and fd:2.
You can specify the numeric parameters with this
alternate keyword syntax:
fd[.file_descriptor=num1]
global File global to all PEs.
Available on IRIX systems.
This is a caching layer which distributes the cache
across multiple SHMEM or MPI processes. Open and close
operations are collective (require participation by all
processes which access the file). All other operations
are independently performed by one or more processes.
The library allows multiple processes to concurrently
access the file while maintaining coherency of buffered
data.
Specify one of the following options for type:
type Description
privpos Default. The file position is private to
a process. All processes may seek to
independent locations in the file.
globpos (Deferred). The file position is global
to all processes. A seek or I/O
operation by any process will affect the
position for all processes.
The numeric fields are as follows:
num1 The size in 4096-byte blocks of each cache page.
num2 Selects the number of cache pages to be used on
each process. If there are n processes, then n *
num2 cache pages are used.
You can specify the numeric parameters with this
alternate keyword syntax:
global[.type][.page_size=num1][.num_pages=num2]
ibm Record blocking for common record types on IBM operating
systems.
Deferred implementation on IRIX systems.
Specify one of the following record formats for type:
type Format
f Fixed-length record format. For num1, enter the
logical record size in 8-bit bytes. For num2,
enter the maximum physical block size in 8-bit
bytes; if specified, num2 must be equal to num1.
fb Fixed-length, blocked record format. For num1,
enter the logical record size in 8-bit bytes.
For num2, enter the maximum physical block size
in 8-bit bytes; num2 must be an exact multiple of
num1.
u Undefined record format. For num1, enter the
maximum record size, in 8-bit bytes.
v Variable length record format. For num1, enter
the maximum logical record size in 8-bit bytes.
For num2, enter the maximum physical block size
in 8-bit bytes.
vb Variable length blocked record format. For num1,
enter the maximum logical record size in 8-bit
bytes. For num2, enter the maximum physical
block size in 8-bit bytes.
vbs Variable length blocked, spanned record format.
For num1, enter the maximum logical record size
in 8-bit bytes. For num2, enter the maximum
physical block size in 8-bit bytes.
No subtype is accepted for the ibm class record formats.
num1 does not need to be smaller than num2.
You can specify the numeric parameters with this
alternate keyword syntax:
ibm[.type][.recsize=num1][.mbs=num2]
mr or sds Memory-resident and SDS-resident files. mr is a
deferred implementation on IRIX systems. sds is not
available on CRAY T3E systems or on IRIX systems.
The mr and sds layers let you declare a byte-stream
file, but the given file should reside in the specified
medium as much as possible; this can improve
performance.
Enter one of the following for type:
type Load Specification
save Default. Loads the file into the specified
medium when the file is opened and writes the
data back out when closed.
scr Scratch file. Does not attempt to load at
open and discards data on close.
Enter one of the following for subtype:
subtype Overflow Specification
ovfl Default. Writes excess data that does not
fit in the specified medium to the next lower
layer.
novfl If data does not fit in the specified medium,
subsequent write(1) operations fail.
The mr and sds layers accept numeric fields. Generally,
the numeric field specification represents best-efforts
values. They are used for tuning purposes and usually
do not cause errors if they are not satisfied precisely
as specified. The fields are as follows:
Field Value
num1 Initial size of the memory or SDS allocation.
Specified in 4096-byte blocks. Default is 0.
num2 Maximum size of the memory or SDS allocation.
Specified in 4096-byte blocks. Default is
46
s -1.
num3 Increment size of the memory or SDS allocation.
Specified in 4096-byte blocks. This value is
used when allocating additional memory or SDS
space. The default is 256 for SDS files and 32
for memory resident files.
For example, if the SDS limit for a process is 1000
blocks, and sds.scr:1500 is specified, an initial SDS
allocation of 1000 blocks is used.
Similarly, if sds:500:10000:1000 was specified, the
first request for SDS space would result in the
allocation of 500 blocks. If possible, this is
allocated contiguously, but it is not guaranteed. If a
file size exceeds 500 blocks, an attempt is made to
increase the file by an increment of 1000 blocks. If
this attempt fails, whatever can be allocated is given
to the file. Assuming no other SDS use, this is 500
blocks. At this point, the file assumes an overflow
status.
When using memory-resident files, you must ensure that
not too much data is written to the file without
limiting its size. Unrestrained and unmanaged growth of
such a file can cause heap fragmentation and program
abort. If this growth has used all available memory,
the error message processor may be unable to issue a
message. Therefore, such an abort may be difficult to
diagnose and correct. It is recommended that the
maximum memory size field be specified in all cases.
(For example, mr.scr::10000 would limit such a file to
approximately 5 Mwords of main memory.)
You can specify the numeric parameters with this
alternate keyword syntax:
mr[.type[.subtype]][.start_size=num1] [.max_size=num2]
[.inc_size=num3]
sds[.type[.subtype]][.start_size=num1] [.max_size=num2]
[.inc_size=num3]
null Syntactic no-op.
Available on IRIX systems.
No optional fields are accepted. null may be specified
where syntax demands a value, but no function is
desired. This does not perform the same function as
/dev/null.
nosve CDC NOS/VE record formats.
Not available on IRIX systems.
The following values are allowed for the type and num
fields:
type Value
d ANSI D format (variable-length) records. For
num1, enter the maximum logical record size in
8-bit bytes. For num2, enter the maximum
physical block size in 8-bit bytes.
f ANSI F fixed-length records. For num1, enter
the logical record size in 8-bit bytes. For
num2, enter the maximum physical block size in
8-bit bytes; num2 must be an exact multiple of
num1.
s ANSI S format (segmented) records. For num1,
enter the maximum logical record
|
