DPLACE(5) DPLACE(5)
dplace - NUMA placement specification
Placement specifications are described in placement files or invoked from
the command language library. A valid placement specification contains
descriptions for memory and thread placement. Memory topologies as well
as affinities to devices may also be specified. Shared virtual address
ranges can be mapped to a specified memory. Page size and page migration
thresholds can also be controlled.
An example placement file describing two memories and two threads might
look like:
# placement specification file
# set up 2 memories which are close
memories 2 in topology cube
# set up 2 threads
threads 2
# run the first thread on the 2nd memory
run thread 0 on memory 1
# run the 2nd thread on the first memory
run thread 1 on memory 0
This specification, when used for initial placement, would request 2
nearby memories from the operating system. At creation, the threads are
requested to run on an available cpu which is local to the specified
memory. As data and stack space is touched or faulted in, physical memory
is allocated from the memory which is local to the thread which initiated
the fault.
Commands are newline terminated.
Characters following the comment delimiter '#' are ignored.
Tokens are separated by optional whitespace which is ignored.
Line continuation is a \, and must occur between tokens.
In what follows k,l,m,n,t0,t1,dt,m0,m1 and dm are arithmetic expressions
that can contain environment variables preceded by a '$'.
Page 1
DPLACE(5) DPLACE(5)
A legal statement could be:
memories ( $MP_SET_NUMTHREADS + 1)/2 in cube
The first example can be written in a scalable fashion as follows:
# scalable placement specification file
# set up memories which are close
memories $MP_SET_NUMTHREADS in topology cube
# set up threads
threads $MP_SET_NUMTHREADS
# run reversed threads across the memories
distribute threads $MP_SET_NUMTHREADS-1:0:-1 across memories
# set the page placement policy to RoundRobin for data pages
policy data placement "PlacementRoundRobin"
Static specifications may occur in a placement file or be called later
from dplace(3) library routines. Dynamic specifications may only be
called from dplace(3) library routines. Here is a summary of the
grammar.
Static specifications:
memories m [[in] [topology] cube|none|physical|cpucluster] [near
[/hw/*]+]
threads n
run thread n on memory m [using cpu k]
distribute threads [t0:t1[:dt]] across memories [m0:m1[:dm]] [block
[m]]|[cyclic [n]]
place range k to l on memory m [[with] pagesize n [k|K|m|M] ]
policy stack|data|text pagesize n [k|K|m|M]
policy stack|data|text pagewait [off|on]
policy stack|data|text placement s
policy migration n [%]
Page 2
DPLACE(5) DPLACE(5)
mode verbose [on|off|toggle]
mode signal expr
Dynamic specifications:
migrate range k to l to memory m
move thread|pid n to memory m
In the above, the specification: threads t0:t1[:dt] means to use threads
numbered t0 through t1 with an optional stride of dt. The default stride
value is 1.
Similarly, the specification: memories [m0:m1[:dm]] means to use memories
m0 through m1 with an optional stride of dm. The default stride value is
1.
The qualifier block m implies a block distribution of the threads with at
most m threads per memory. If m is omitted, its default value is: the
integer ceiling of the number of threads being distributed divided by the
number of memories being distributed across.
The qualifier cyclic n implies a cyclic distribution of the selected
threads across the selected memories. The threads are chosen in groups of
n and dealt out to the appropriate memory until all threads are used. If
n is omitted, its default value is one and the threads are dealt out to
the memories like a deck of cards.
If errors are encountered in the placement_file, the default procedure
for dplace is to print a diagnostic message to standard error specifying
where the error occurred in the placement_file and abort execution. If
errors are encountered in the libdplace.so library during the run-time
execution of program, then a diagnostic message is sent to standard
error, a default signal of SIGKILL is sent to all members of the process
group, and execution is aborted.
The mode signal expr statement allows a selection of a specific signal
number to be generated upon error. If the mode signal expr is specified,
the action taken when libdplace.so detects a run-time error is to send
the signal number derived from expr to the program invoked by dplace.
Under this condition the control is returned to the caller, which is the
program. The signal number can range from 1 to 32.
An example of how to set the signal number:
mode signal 16
Page 3
DPLACE(5) DPLACE(5)
Upon detecting an error in libdplace.so during run-time, signal 16
(defined as SIGUSR1) is sent to the calling process (in this case the
program and control is returned to the caller.
COMMON PLACEMENT FILE ERRORS [Toc] [Back] The most common placement file error is the use of either the run or
distribute directive without previously declaring both the number of
threads, and the number of memories using the memories and threads
directives.
In the above, a thread is an IRIX process or any of its descendants which
were created using sproc(2) or fork(2). Thread numbers are ordered (from
0 to the number of threads minus 1 ) in the same way as the pid's for
each process.
A memory is an instantiation of a physical memory.
A range is a virtual address range such as 0x100000 to 0x200000.
The placement policies are specified according to mmci(5). The current
valid policies are PlacementDefault, PlacementFixed, PlacementFirstTouch,
PlacementRoundRobin, PlacementThreadLocal, PlacementCacheColor.
dplace(1), dprof(1), dplace(3), numa(5), mmci(5), hwgraph(4)
PPPPaaaaggggeeee 4444 [ Back ]
|
