Skip to content

Z
ZIO
Commit 59d8d443 authored by Alessandro Rubini's avatar Alessandro Rubini
Browse files
Options

doc: manual review and update 

Signed-off-by: Alessandro Rubini's avatarAlessandro Rubini <rubini@gnudd.com>
Acked-by: 's avatarFederico Vaga <federico.vaga@gmail.com>
parent 9d7207a2
Hide whitespace changes
Inline Side-by-side
Showing with 568 additions and 369 deletions
doc/zio-manual.in
View file @ 59d8d443
......@@ -38,7 +38,7 @@
@finalout
@c #################################################################
@titlepage
@title ZIO User Manual
@title ZIO User Manual (version-1.0)
@subtitle @value{update-month}
@subtitle A kernel framework for laboratory I/O
......@@ -101,7 +101,7 @@ containing a single data sample are not expected to be common, as our
use case is concerned with devices that input or output several
thousand samples in a single shot, with a hardware-defined data rate
and a single time-stamp marking the beginning of the event. Blocks
containing zero data items are not forbidden, because they can be used
containing zero data items are allowed, because they can be used
to pass meta-information without associated data (e.g., TDC and DTC
devices, described later).
......@@ -120,25 +120,27 @@ The ZIO code base is designed around three main items:
@cindex trigger
@item Trigger
The trigger is code concerned with firing I/O, in response to
some event. ZIO supports both software triggers and hardware
triggers; in the latter case the software module is used to
configure to the device and retrieve status information.
The trigger is code concerned with arming the I/O event, in response to
some event. Peripheral devices with no internal timing will input
or output data when the trigger is armed; peripherals that are
self-timed are supported by means of the default trigger, without
the need to write custom trigger code for each of them.
See @ref{The Trigger}.
@cindex buffer
@item Buffer
A buffer stores blocks, either input blocks
generated by a trigger or output blocks generated by some
injecting code One end of the buffer is always connected to a
injecting code. One end of the buffer is always connected to a
trigger, and the other end is usually connected to a char
device (but ZIO buffers are not constrained to).
device, though this is not mandated by the framework.
See @ref{The Buffer}.
@end table
@cindex driver
The ZIO driver manages a @i{device}, which in turn may be either a
chip or a PCB (or something else). The driver is the @i{probe unit},
The ZIO driver manages a @i{device}, which in turn may be a
chip, a card or an external equipment connected through a field bus.
The driver is the @i{probe unit},
i.e. a batch of I/O peripherals that are plugged and unplugged as a
whole, usually based on some bus, like PCI, USB or SPI.
......@@ -155,7 +157,7 @@ A cset is a group of channels usually associated with a wire connected
to some backplane of the computer. All channels within a cset must be
alike: same sample size and same configuration options (i.e.,
configuration values may vary across channels in the same cset, but
all of them must feature the same set, like gain and offset).
all of them must feature the same set of parameters).
@cindex channel
The channel is the basic I/O entity, usually associated with a single
......@@ -163,7 +165,7 @@ electrical connection on some backplane. The layers are displayed
in @ref{fig:layers}
@float Figure,fig:layers
@image{img/zio-layers, 8cm, , the three layers of ZIO, gif}
@center @image{img/zio-layers, 8cm, , The three layers of ZIO, gif}
@end float
@sp 1
......@@ -172,6 +174,7 @@ The cset is the most important object for ZIO devices. Each cset is
associated with a trigger, and I/O events affects all channels in the
cset -- although you can disable individual channels as needed.
@cindex current_trigger
Each cset has a @i{current_trigger} attribute, which defines a trigger
type; for each cset using that trigger type, ZIO creates an instance of
the trigger. Each cset has a @i{current_buffer} attribute as well. ZIO creates
......@@ -183,8 +186,7 @@ refers to an the instances it is using. A cset has one trigger
instance overall and one buffer instance for each channel.
@float Figure,fig:cset
@image{img/zio-cset, 8cm, , A cset with trigger and buffers, gif}
@caption{Figure 2: The cset}
@center @image{img/zio-cset, 8cm, , A cset with trigger and buffers, gif}
@end float
@sp 1
......@@ -194,9 +196,10 @@ instance overall and one buffer instance for each channel.
@cindex devices, supported
This is a quick list of devices or device types that are supported in
the current version of ZIO. This is a cursory overlook, for some of
them more information is provided in specific sections of
@ref{Available Modules}.
the current version of ZIO. They are mainly examples, because the real
devices we are working with are developed as external packages.
Such external packages include a 100Ms ADC, a Fine-Delay module and a TDC
(both with 10ps resolution).
@table @r
......@@ -206,7 +209,8 @@ them more information is provided in specific sections of
The zero device is a software-driven input and output device,
it is used for demonstration and stress-testing. It behaves
like @file{/dev/zero}, @file{/dev/null} and similar devices.
like @file{/dev/zero}, @file{/dev/null} and similar devices,
but it inputs and outputs ZIO blocks.
@cindex zio-loop
@cindex loop device
......@@ -223,27 +227,35 @@ them more information is provided in specific sections of
@cindex mini device
@item mini device
This is the last software-only device we offer. It is designed
This is another software-only device we offer. It is designed
to be as simple as possible, to help new developers who are
learning to write ZIO drivers. It registers one cset with one
learning how to write ZIO drivers. It registers one cset with one
channel, by default, but module parameters allow to create several
csets and several instances of the device. With it you can also
stress-test the ZIO bus abstraction, and interaction with @i{udev}.
Each channel simply returns one sample in each block, which
represents a binary @t{struct timespec} that marks when the
input event happened.
@c FIXME: zio-irq-tdc
@c FIXME: zio-fake-dtc
@cindex gpio device
@cindex zio-gpio
@item gpio device
The gpio device acts on an array of input and output GPIO pins.
It relies on the Linux GPIO abstraction, so it can run in any
The gpio device acts on an array of input and output GPIO pins
(up to 8 pins for input, up to 8 for output). Each cset has
one channel only.
The driver relies on the Linux GPIO abstraction, so it can run in any
system that has registered GPIO pins. The configuration is
set with module parameters.
@cindex SPI
@item SPI devices
Some ADC devices connected to the SPI bus are supported, it should
be easy to add more.
Some ADC devices connected to the SPI bus are supported. The
driver supports AD7887 and AD7887 with a single source file;
it relies on the Linux SPI subsystem.
@cindex uart
@cindex line discipline
......@@ -251,15 +263,14 @@ them more information is provided in specific sections of
@c FIXME line discipline
We are working on a @i{ldisc} device driver, which however is
not included in this release yet. It is out test-bed for
interleaved support.
not included in this release yet. It is our test-bed for
interleaved support, that will be included in later versions.
@cindex drivers, external
@cindex external drivers
@item external drivers
There are a number of external device drivers that rely on
As said, there are a number of external device drivers that rely on
the ZIO framework. They are not part of this distribution because
they live in their own projects, and are quite specialized.
We have at least a @i{fine-delay} module, a @i{TDC} and a fast @i{ADC}.
......@@ -275,10 +286,11 @@ not listed here as they are not properly device drivers. The buffers
are generic RAM-based buffers (we offer @i{kmalloc} and @i{vmalloc},
the latter supports @i{mmap} as well), and the triggers are a set of
generic software-driven triggers: kernel timer (periodic),
high-resolution timers (both periodic and one-shot),
external-interrupt, and the transparent trigger (where I/O is driven
by either the user who tries to read or write, or by the internal
events in the device and it claims to ba self-triggered).
high-resolution timer (both periodic and one-shot),
external-interrupt, and the transparent trigger which is selected
by default. With the transparent trigger, I/O is driven
by either the user who initiates read and writes, or by the device
itself if it claims to be self-timed.
For more details see @ref{Available Buffers} and @ref{Available Triggers}.
......@@ -298,8 +310,8 @@ in @i{beta} stage, available from @i{git} branches in our @t{ohwr} repository.
@cindex input subsystem and ZIO
@item Supporting an @code{input_device} buffer type
A buffer type may inject input events to the Linux input
subsystem, and we feel it's an easy and interesting feature to offer.
A buffer that injects input events to the Linux input
subsystem, without requiring any change in other ZIO modules.
@cindex PF_ZIO, network protocol family
@cindex AF_ZIO, network address family
......@@ -308,9 +320,9 @@ in @i{beta} stage, available from @i{git} branches in our @t{ohwr} repository.
The protocol family @code{PF_ZIO} allows to see the whole
set of I/O peripheral devices as a network. Such network can
be internal to a single host, or it can span multiple hosts
by exchanging frames for type @t{ETH_P_ZIO}. With @t{PF_ZIO}
by exchanging frames of type @t{ETH_P_ZIO}. With @t{PF_ZIO}
each channel has a network address, with the @i{control} acting
as link-level protocol header. This will allow applications to
as link-level protocol header. This allows applications to
open only a single socket (or a few of them) instead of one
or two char devices for each channel being used.
This is already available, and is being stabilized. We
......@@ -332,16 +344,17 @@ in @i{beta} stage, available from @i{git} branches in our @t{ohwr} repository.
@cindex interface, to abstract file operations
@item Defining the ``zio_interface'' data structure.
The @i{interface} will abstract char device support out of
An @i{interface} layer may abstract char device support out of
the ZIO core, so users will be able to use different interfaces;
the PF_ZIO protocol will therefore be able to use
hardware-specific buffers when the device requires them.
with interfaces, the PF_ZIO protocol or a link with the input
subsystem can be a first-level alternative to char devices instead
of being a buffer type that doesn't prevent creation of char devices.
@cindex timestamping ZIO internals
@item Generalized timestamping.
@c FIXME: timestamping the pipeline: is it done?
We are going to add capabilities to timestamp the internal ZIO
We are soon going to add capabilities to timestamp the internal ZIO
pipeline. This allows to measure the overhead of the pipeline
(driver, trigger and buffer) independently of which objects are
being used -- so you can timestamp during production as well.
......@@ -367,15 +380,15 @@ converted to this color model, and we apologize for this.
The color model for all of our objects is shown in @ref{fig:color}.
@float Figure,fig:color
@center @image{img/zio-define-block, 5cm, , color model for blocks, gif} @image{img/zio-defines, 6cm, , color model for ZIO objects, gif}
@center @image{img/zio-define-block, 5cm, , Color model for blocks, gif} @image{img/zio-defines, 6cm, , Color model for ZIO objects, gif}
@end float
@sp 1
@cindex pipeline
@cindex input pipeline
@cindex output pipeline
The ZIO pipeline connects user space with the peripheral device,
and the main objects involved are the file operations in the
The ZIO pipeline connects user space with the peripheral device.
The main objects involved are the file operations in the
character device, the buffer where blocks are stored, the trigger that
sequences the actual data transfers and the peripheral driver that
is concerned with hardware operations when the trigger fires.
......@@ -385,7 +398,7 @@ where the path of the blue blocks is shown for both input and output.
The black arrows represent function calls.
@float Figure,fig:pipeline
@center @image{img/zio-pipeline, 14cm, , the input and output pipeline, gif}
@center @image{img/zio-pipeline, 14cm, , The input and output pipeline, gif}
@end float
@sp 1
......@@ -393,11 +406,11 @@ The black arrows represent function calls.
User space makes system calls, which are implemented by the
@code{file_operations} in the current buffer. Generic functions are
implemented in @code{chardev.c} and exported as
@code{zio_generic_fops} (currently all buffers included in the
@code{zio_generic_file_operations} (currently all buffers included in the
distribution are using these operations). This may slightly change in the
future if we introduce the @i{interface} concept, but the change
will not affect buffers which simply won't declare their file operations
any more). See @ref{Future Developments} about the @i{interface} idea.
will not affect buffers besides the removal of their @t{f_op} field.
See @ref{Future Developments} about the @i{interface} idea.
@findex store_block
@findex retr_block
......@@ -412,20 +425,23 @@ for the raw data transfer.
@section All the Steps in the Pipeline
This sections details all the steps a block takes from user space
to the device and back. You should get acquainted to the various
step when writing a ZIO module or debugging some module by your mates.
to the device and back. You should get acquainted with the various
step when writing a ZIO module or debugging it.
When you load the device driver, each of its cset wil receive an
instance of a trigger, and each channel in the cset an instance of a
@cindex preferred trigger
@cindex preferred buffer
When you load the device driver, each of its csets receives an
instance of a trigger, and each channel in the cset receives an instance of a
buffer; the preferred trigger type and the preferred buffer type are
selected by the device, or the default ones are used. When the user
selected by the cset or the device, otherwise default ones are used.
When the user
changes the trigger or buffer type for a cset, the operation is
atomic: it either succeeds or it fails. This design ensures that all
steps of the pipeline are always present, and we'll use @t{bi} as name
for the pointer to the current @i{buffer instance}, and @t{ti} as name
steps of the pipeline are always present; we always use @t{bi} as name
for the pointer to the current @i{buffer instance} and @t{ti} as name
for the pointer to the current @i{trigger instance}.
When the data is flowing, operation of the pipeline is quite
When data is flowing, operation of the pipeline is quite
straightforward, but there are always some corner cases to be dealt
with, so the description below is sometimes longer than expected, but
it covers all cases.
......@@ -434,6 +450,8 @@ Let's start with the input direction:
@itemize @bullet
@findex retr_block
@findex free_block
@item When a user program calls @i{read}, @i{poll} or @i{select} on
a ZIO character device, it means it is interested in data. The code in
the ZIO file operation calls @t{bi->retr_block}, which may succeed or
......@@ -441,136 +459,172 @@ fail. If it fails, the device is reported as not readable; the process
may be added to a wait queue or not, according to standard Unix
semantics. If @t{retr_block} succeeds, the block just retrieved is
managed by the @i{file operations} code until it is exhausted and
eventually freed calling @t{bi->free_block}.
eventually freed by calling @t{bi->free_block}.
@findex retr_block
@findex pull_block
@item The @t{retr_block} method lives in the buffer.
If the buffer is already hosting at least one block, it returns the
first block in its FIFO structures, and the loop closes. If not, it
first block in its FIFO structures, and the loop closes. If not, the function
checks @t{ti->pull_block}. If the pointer is NULL, the buffer can
only wait for the trigger to autonomously produce a block; otherwise
it calls @t{ti->pull_block}.
it calls the function to request an input event from the trigger.
@findex pull_block
@item Most trigger types do not implement @t{pull_block}: a time-driven
trigger or an event-driven trigger will fire by itself, whether or not
trigger or an event-driven trigger fire by themselves, whether or not
the user is reading. But if the trigger implements
the method, then it means it wants to know when the user is asking
for input data: such a trigger may return
for input data: such a trigger may thus return
an incomplete block, or just use @i{that} event to fire acquisition.
In any case, @t{pull_block} does not return anything, it just
informs the trigger about the request.
@cindex active block
@findex raw_io
@item The trigger is involved with driving actual input events, in any
way it wants. At some time, or at some event, it knows it wants input
to happen, so we say it is @i{armed}. When the trigger is armed, all
active channels in the cset have a valid @i{current block}, allocated
by the trigger. Then, the trigger notifies the event to the relevant
cset by calling @t{device->raw_io(cset)}. A @i{transparent} trigger
will arm the device at @t{pull_block} time, and each other trigger in
to happen, so we say the trigger is @i{armed}. When the trigger is armed, all
enabled channels in the cset should have a valid @i{active block}, allocated
by the trigger. (Actually, allocation may fail, so some channels
may have no active block when the trigger fires).
Then, the trigger notifies the event to the relevant
cset by calling @t{device->raw_io(cset)}. The @i{transparent} trigger
arms the device at @t{pull_block} time, and each other trigger in
its own specific way.
@findex raw_io
@item After the trigger is @i{armed}, the device is in charge. If
the device is self-timed (like a TDC or another data-driven device),
the trigger can remain armed for a while. If the device has no
internal timing, actual input begins immediately; it may complete
synchronously or asynchronously, for example reading GPIO pins is
synchronous, but if the device starts DMA an interrupt will signal its
completion at a later time. The @t{raw_io} device method returns 0 if
internal timing, actual input begins immediately. Input may complete
synchronously or asynchronously; for example reading GPIO pins is
synchronous, but if the device initiates DMA it needs to wait for
an interrupt that happens
at a later time. The @t{raw_io} device method returns 0 if
input completed synchronously, and it returns @t{-EAGAIN} to report
that data will be ready at a later time. In the former case the system
knows the trigger has fired, in the latter case ZIO knows the trigger
is still armed for this cset.
is still armed for this cset. Return values other than 0 and @t{-EAGAIN}
represent a real error, and ZIO knows the trigger is not armed.
@findex zio_trigger_data_done
@findex data_done
@findex store_block
@item At some time in the future, the device that reported @t{-EAGAIN}
is done with the input operation. It thus calls @t{zio_trigger_data_done()}
(that may rely on @i{ti->data_done}).
(that may rely on @i{ti->data_done} if the trigger defines it).
When this happens (or if @t{raw_io} completed synchronously), the
respective trigger changes its status to @i{idle}; the active block
respective trigger changes its status to un-armed; the active block
for each channel in the cset is stored to the respective buffer
instance, by calling @i{bi->store_block}. According to its own
semantics, the trigger can re-arm the trigger immediately, or it may
not.
@cindex alarms
@item At @i{data-done} time, ZIO raises an @i{alarm} bit for
each channel that is enabled by has no active block. The alarm is
persistent until cleared by writing to @i{sysfs}.
@item The buffer, in its @t{store_block} method, enqueues the block.
If there are already other blocks in the buffer, the loop is over.
If, instead, this is the first block, the buffer must awake the wait
queue, because there may be a user-space program waiting for that
If, instead, this is the first block, the buffer awakes its own wait
queue, because there may be a user-space program waiting for such
data.
@end itemize
The sequence of events for output is similar, but not in all
details:
The sequence of events for output is similar, but not in every
detail:
@itemize
@findex alloc_block
@findex store_block
@item When a user program calls @i{write} on
a ZIO character device, the data build up into a block, that the file
operation allocated by calling @t{bi->alloc_block}. The details of
a ZIO character device, the data builds up into a block, which has
been allocated by calling @t{bi->alloc_block}. Select for writing
is supported as well, for processes that need it: when allocation of
a block fails the char device is reported as not writable, and the
process may sleep on a wait queue, according to Unix semantics.
The details of
how the control and the data are filled are not described here (see
@ref{Details of Char Device Policies}). When the block is full, the
code calls @t{bi->store_block}. If the buffer reports failure, the
file is reported as not writable, and the process may be put to sleep,
according to Unix semantics. @i{poll} and @i{select} are supported in
the same way.
@item The buffer method @t{store_block} take hold of
the new block. If there already are blocks in its FIFO structure, the
new block is enqueued and nothing more is done. If it is the first
code calls @t{bi->store_block}, that never fails.
@findex store_block
@findex push_block
@item The buffer method @t{store_block} takes hold of
the new block. If there are already blocks in its FIFO structure, the
new block is enqueued and nothing more is done. If this is the first
block, the buffer calls @t{ti->push_block}. If the push succeeds, the
buffer remains empty (and will try @i{push} again next time).
If the push fails, the block is enqueued.
@item The trigger, in its @t{push_block}, may accept or
@cindex double buffering
@item As said, the trigger may accept or
refuse the push. In general, it should refuse the push if it
already has a block for the current channel, and it should accept
already is getting hold of a block for the current channel,
and it should accept
the push if no block is pending. A double-buffering trigger can
be implemented by properly writing @i{push}.
@cindex transparent trigger
@item After accepting the push, the trigger may, or may not, arm
the trigger, according to its own trigger policies -- which may
include cset-wide conditions or whatever else.
the trigger, according to its own trigger policies. For example, the
transparent trigger only arms when every enabled channel has an
active block -- because the output event happens for the whole
cset at once.
@findex raw_io
@item When the trigger module wants output to be performed, it
turns in to @i{armed} state, and calls @t{device->raw_io(cset)}.
turns into @i{armed} state, and calls @t{device->raw_io(cset)}.
Similarly to the input case described earlier, this method
can perform I/O synchronously and return 0 to mean ``success'',
or ti can perform I/O asynchronously and return @code{-EAGAIN}.
In the latter case the trigger remains in @i{armed} state.
or it can perform I/O asynchronously and return @code{-EAGAIN}.
In the latter case the trigger remains in @i{armed} state. Other
return values are considered errors, and the trigger is not considered
as being armed.
@findex zio_trigger_data_done
@findex data_done
@findex free_block
@findex retr_block
@item At some time in the future, the device that reported @t{-EAGAIN}
is done with hardware output, and calls @t{zio_trigger_data_done()}
is done with hardware output, so it calls @t{zio_trigger_data_done()}
(that may rely on @i{ti->data_done}).
When this happens (or if @t{raw_io} completed synchronously), the
trigger instance becomes @i{idle} again, and it frees the blocks
trigger instance becomes un-armed again, and it frees the blocks
for the cset by calling @t{bi->free_block}. In general,
the trigger will then call @t{bi->retr_block} in order to be
the trigger should call @t{bi->retr_block} in order to be
ready to re-arm the trigger.
@item If the buffer is not able to satisfy the @t{retr_block}
request, this means it is empty. As a consequence, the next time a
block is stored to the buffer, the buffer calls @t{ti->push} to
@findex retr_block
@item If the buffer is empty, it is not able to satisfy the @t{retr_block}
request. As a consequence, the next time this buffer receives a block
from user space it calls @t{ti->push} to
restart a loop that has been broken. If the buffer was completely
full, this @t{retr_block} makes some space available, and the buffer
awakes the associated wait queue.
@end itemize
We think the overall is pretty simple: we offer @i{pull} and @i{push}
We think the overall flow is pretty simple: we offer @i{pull} and @i{push}
to initiate a new data flow or restart one that stopped for lack of
data, but the normal flow involves just @i{store_block} and
@i{retr_block}, @i{raw_io} and @i{data_done}.
@i{retr_block} for the buffers, @i{raw_io} and @i{data_done} for the triggers.
The output design, in particular, resembles what happens with network
devices. A network card will accept pushes by the kernel (the method
is called @i{hard_start_xmit} until its internal buffers are full; at
thi point it calls @i{netif_stop_queue}, and when space is available
The output design, in particular, is similar to what happens with network
devices. A network card accepts pushes by the kernel (the method
is called @i{hard_start_xmit}) until its internal buffers are full; at
this point it calls @i{netif_stop_queue}, and when space is available
again it calls @i{netif_start_queue}.
ZIO output is similar to network transmission, but the ZIO core knows
that the trigger can accept only one or two blocks at most; so we
chose to not require the trigger to call @i{stop} and @i{start} in the
buffer, in favour for a clean failure mode for the push,
and reuse of the same @i{retr_block} that is used for the input flow.
buffer, providing instead a clean failure mode for the push,
so to reuse the same @i{retr_block} that is used for the input flow.
@c ==========================================================================
@node Lifetime of a Data Block
......@@ -614,7 +668,7 @@ system you thus need both items: the @i{driver} is in charge of any
@i{device} that appears in the system, while the @i{device} is a data
structure that describes the parameters of the specific hardware
instance. The two structures are bound by calling a @i{match}
function, that is at the core of the bus abstraction. If the device
function, which is at the core of the bus abstraction. If the device
and the driver match, the driver is asked to manage the new device
instance.
......@@ -624,7 +678,7 @@ software abstraction we are able to deal with several devices of the
same type. Our use case involves control systems where more than one
instance of the same PCI card is installed in each host. Actually, by
distributing our hostless I/O devices using a field-bus, we foresee
the need to register a few hundred ZIO devices, to be driven by the
the need to register up to a few hundred ZIO devices, to be driven by the
same driver.
@cindex modules, structure of
......@@ -633,10 +687,10 @@ I/O card. Since PCI is a @i{bus} in Linux, you'll need to register a
PCI @i{driver} to hook to any card present in the system (in this case
the associated @i{device} is created by the PCI controller, when it
scan the physical bus). The @i{probe} function of your PCI driver,
then, will register a ZIO @i{device}. In addition to registering a PCI
then, registers a ZIO @i{device}. In addition to registering a PCI
@i{driver}, your kernel module also needs to register a ZIO @i{driver}
for your hardware, so that Linux calls its @i{probe} function
for every ZIO @i{device} that matches.
for every ZIO @i{device} that matches such driver.
@cindex match function in ZIO bus
The ZIO bus is just a software abstraction, so its own @i{match}
......@@ -644,8 +698,8 @@ function is just comparing the two names: if a device and a driver
register the same name string, they match. This is exactly what
happens with the @i{platform bus}, a software-only bus implementation
used in every Linux system to take care of peripherals that are
directly connected to the computer system, without capabilities for
auto-discovery.
directly connected to the computer system, without relying on an enumerated
bus.
@cindex registering ZIO devices
@cindex registering ZIO drivers
......@@ -656,6 +710,7 @@ This is clearly visible in simple modules like @t{zio-zero} and
data structures. Nothing prevents another kernel module from
registering a new instance of the @i{zero} or @i{gpio} device (or both).
@cindex zio-mini
You can experiment with this concept using @t{zio-mini},
which actually registers more than one @i{device}
associated to its own @i{driver}.
......@@ -667,7 +722,7 @@ associated to its own @i{driver}.
@cindex data model
This chapter defines in detail the data model of ZIO. The
unconventional data model is the main idea behind the ZIO project,
together with the use of a real @i{bus} abstraction.
together with the use of a full-featured @i{bus} abstraction.
@c ==========================================================================
@node The Block
......@@ -675,17 +730,20 @@ together with the use of a real @i{bus} abstraction.
@cindex block
@cindex samples
Our typical use case handles several samples at the same time. We need
In the typical ZIO use case we handle several samples at the same time. We need
to manage thousands or even millions of samples at the same time,
because I/O is performed at hardware level (within an FPGA) while
because I/O and time-stamping is performed at hardware level
(within an FPGA) while
the Linux driver is only concerned with passing data around.
For this reason ZIO defines a structure called @i{block}. All data transfers
within ZIO happen in the context of a block, which is the atomic data
within ZIO happen in the context of a block, which is its own atomic data
item. A block includes both the actual data and the @i{meta-data} associated
with the input or output event. Inside ZIO, no data ever travels without
associated metadata. This design choice allows I/O data to be
transferred through unaware transport channels: only the endpoints need
associated metadata (even if the user is allowed to ignore meta-data
and only read or write data). This design choice allows I/O data to be
transferred through payload-agnostic transport mechanisms:
only the endpoints need
to agree about what the actual payload is. More on this later.
@cindex block
......@@ -729,12 +787,17 @@ The meaning of the fields is as follows:
The length of the memory area where @i{data} points. May
be zero for TDC or DTC devices.
@cindex user offset
@tindex uoff
@item uoff
@i{User offset}. This field is used to track partial data,
typically when user-space produces or consumes the block.
Both endpoints of the ZIO pipeline can use this field, as
the block never hosts partial data while within ZIO.
@i{User offset}. This field is used to track partial data, while
the block is produced or consumed. This happens, for example,
when the user accesses
a byte-oriented interface like a char device. The peripheral
driver can use the field to track partial data at its own
end of the pipeline, but for input blocks the field must
be zeroed before storing the block into a buffer.
@end table
......@@ -743,7 +806,7 @@ The flag bit hidden in @i{ctrl_data} is called @t{cdone}, and
is the counterpart of @i{uoff}: when a block is being produced or
consumed we need to track the status of the control. Unlike data,
the control is atomic: no partial controls ever exits, and one bit
is enough.
is enough to track its status within a block.
We chose to coalesce the pointer to control and the @code{cdone} bit
in a single filed in order to avoid wasting bytes and/or break alignment
......@@ -779,7 +842,8 @@ The @i{control} is the container of meta-information used to describe
a block of data. Its size is fixed to 512 bytes, with the optional
addition of TLV records at the end. Such @i{type-length-value} records
are not used in the current implementation, but they are designed
to introduce no incompatibilities on ZIO modules developed externally.
to introduce no incompatibilities on ZIO modules developed externally,
when we'll add them to the core.
More information about this is in @ref{TLV in the Control}.
The layout of the control is shown in @ref{fig:control}.
......@@ -800,17 +864,18 @@ the Linux kernel.
@cindex meta-information
The control is designed to offer the full meta-information needed to
describe a block. In includes the complete global identification
describe a block. In includes the unique global identification
of the channel, as well as the name and parameters for the
current trigger. Applications can thus pass around a
block without knowing what it is; thus, knowledge about device and
block without knowing what it is; knowledge about device and
trigger details can be concentrated in a single place, without the
need to spread information to all actors in the input/output pipeline.
@cindex offline data management
By using a control structure (i.e., zio blocks in their entirety)
ZIO users can perform
both offline data generation and offline data analysis. A program
both offline data generation for output channels and offline data analysis
for input channels. A program
may prepare an output waveform in advance and ask generic tools to
deliver it to the device; similarly, acquisition may be performed by
generic tools that concentrate data from a set if I/O computers to a
......@@ -822,7 +887,7 @@ blocks.
The control structure is defined in @code{zio-user.h} because it
must be accessed by both kernel and user space. The definition includes
a few sub-structures, to ease logical grouping of information.
This is the current definition (version 1.0 of the control block):
This is the current definition of the control block, version 1.0:
@tindex zio_control
@smallexample
......@@ -877,33 +942,39 @@ the fields are not yet being filled by the core.
The version is currently 1.0, as defined by @code{ZIO_MAJOR_VERSION}
and @code{ZIO_MINOR_VERSION} (in @code{zio.h}). The version is used
to ensure all actors agree on the meaning of the fields.
We ensure that all minor version changes will be compatible (for
example, 1.1 may have new flags that can be ignore by older code).
We warrant that all minor version changes will be compatible (for
example, 1.1 may have new flags that can be ignored by older code).
Any incompatible change will force an upgrade of the major number,
but we plan no such change in the foreseeable future -- we had
enough experience with the 0.x versions already, and we are confident
we covered all needs already.
we covered all needs that may arise in the future.
@cindex alarms
@tindex zio_alarms
@tindex drv_alarms
@item zio_alarms
@itemx drv_alarms
Two masks of active alarms. Alarms can be set by any module in the
pipeline, but they can only be cleared by writing to the
@i{sysfs} attribute of each channel. They are not currently being
used, though.
@c FIXME alarms
pipeline, and are persistent until cleared by means of a
@i{sysfs} attribute. ZIO alarms are defined
by the core, and we currently support ``lost block'' and ``lost
trigger''.
@cindex sequence numbers in control structures
@item seq_num
Block sequence number. It starts at 1 when a channel
is created, and it is incremented at each I/O event. The value 0 is
Block sequence number. It starts at 0 when a channel
is created, and it is incremented at each I/O event (so the
first block is number 1. The value 0 is
reserved as a signal that the sequence number is not used by the
entity that generated the block (e.g., user space, for output blocks).
@cindex size of data samples
@cindex sample size
@tindex nsamples
@tindex ssize
@tindex nbits
@item nsamples
@itemx ssize
@itemx nbits
......@@ -934,7 +1005,8 @@ the fields are not yet being filled by the core.
input, the trigger must fill it, and it may be software-generated or
hardware-generated. For output, some triggers use it and some
don't (for example, and external-irq trigger won't use pre-set
timestamps). The internals of this structure are defined later,
timestamps, but TDC devices use it to fire output pulses at specified
times). The internals of this structure are defined later,
in @ref{The Time Stamp}.
@cindex mmap support
......@@ -944,7 +1016,9 @@ the fields are not yet being filled by the core.
the data in the buffer's storage area. Please note that the
@i{data} pointer in the block structure is still valid: kernel
users should refer to @t{block->data}, while user space can
use @t{ctrl->mem_offset} while relying on @i{mmap}.
use @t{ctrl->mem_offset} while relying on @i{mmap}. We have no
clear ruses, yet, about how to use of @i{mmap} for output.
@c FIXME: mmap for output
@item flags
......@@ -952,6 +1026,7 @@ the fields are not yet being filled by the core.
for currently defined flags.
@cindex name of device and trigger
@cindex current_trigger
@item triggername
Name of the current trigger. For input, the name
......@@ -959,6 +1034,8 @@ the fields are not yet being filled by the core.
left blank). To change the trigger you should write to
@t{current_trigger} in @i{sysfs}.
@tindex attr_channel
@tindex attr_trigger
@item attr_channel
@itemx attr_trigger
......@@ -969,7 +1046,7 @@ the fields are not yet being filled by the core.
@item tlv
The TLV record is normally zeroed. Special csets may need more
information than what the control can host, and in this case
information than what the standard control can host, and in this case
this field is used. See @ref{TLV in the Control} for more information.
@end table
......@@ -980,7 +1057,9 @@ the inception of the ZIO project, and this is verified by a
compile-time check, to ensure nobody changes this in error while
working on a new version. Actually, the check already triggered during
development: a proposed layout failed because of different alignment
constraints on different architectures.
constraints on different architectures. In the future, we'll support
control structures bigger than 512 bytes with no incompatibilities
in already-written code. See @ref{TLV in the Control}.
We currently define the following flags, in @code{zio-user.h}:
......@@ -1016,17 +1095,21 @@ We currently define the following flags, in @code{zio-user.h}:
@subsection ZIO Address
@cindex address of a channel
The @code{zio_addr} structure is designed to uniquely identity
@tindex zio_addr
The @code{zio_addr} structure is designed to uniquely identify
a channel. Its main purpose is to allow generalized routing of the
blocks to the output channels, as well as traceability of the source
of any input data.
blocks to the output channels, as well as to trace the source
of any input data block.
@cindex PF_ZIO
@cindex AF_ZIO
@tindex sockaddr_zio
The chosen layout for @t{zio_addr} is matching @t{sockaddr_zio},
the addressing structure used for the @t{PF_ZIO} implementation,
using sockets to route I/O blocks.
a network protocol to route I/O blocks.
The address of each channel is also available as a binary file in
@i{sysfs}, called @t{address}.
This is the definition of @t{zio_addr}:
......@@ -1051,6 +1134,7 @@ The fields have the following meaning:
@item sa_family
Currently unused in the control, this is meant to be @t{AF_ZIO}
(in network byte order)
in all contexts where this structure is called @t{sockaddr_zio}.
By having the same layout in the control block we simplify conversions
between the I/O world and the networking world.
......@@ -1061,12 +1145,12 @@ The fields have the following meaning:
The first two bytes are used as @i{class} identifiers for
the following @t{hostid}. Currently we only support type 0,
which means @i{local delivery}; though ignored, all other
which means @i{local delivery}; all other
bytes are expected to be 0 as well. When @t{PF_ZIO} will route
frames over ethernet, we'll be able to use @t{hostid} as a MAC
frames over Ethernet, we'll be able to use @t{hostid} as a MAC
address. We chose to use one byte only for @t{host_type} to
avoid endianness problems; if needed, @t{filler} can be used as
a subclass of the host type.
a subclass for some of the future host types.
@cindex index of cset and channel
@cindex dev_id
......@@ -1103,7 +1187,7 @@ The fields have the following meaning:
@cindex timestamp
Time stamps in ZIO are represented by @code{struct zio_timestamp},
defined in @code{zio-user.h} because it is shared with user space.
It is is made of 3 64-bit fields:
It is is made of three 64-bit fields:
@tindex zio_timestamp
@example
......@@ -1125,10 +1209,10 @@ a consistent way:
The field should be used to host a TAI value, i.e.
the number of seconds since Jan 1, 1970. The field is 64-bits
wide to preserve alignment, to survive 2038 and
wide to preserve alignment, to survive 2038
and to allow for a different choice
of the epoch, if needed. When ZIO uses software timestamping
for input channels, the field hosts the @code{tv_sec} value
of the Epoch, if needed. When ZIO uses software timestamping,
the field hosts the @code{tv_sec} value
of a @code{struct timespec}.
@cindex nanoseconds in timestamps
......@@ -1146,19 +1230,18 @@ a consistent way:
This field should be used for any high-precision number as used
in the hardware. It may also be the only non-zero field, for
example if the hardware timestamp is taken from a custom counter
without a second or nanosecond component, if it fits in 64 bits.
unrelated to the standard Epoch.
@end table
@cindex White Rabbit
For example, @i{White Rabbit} devices use hardware timestamps made up
of three 32-bit values: seconds, nanoseconds with a granularity of 8ns
or 16ns and phase offsets, usually converted to
picoseconds within the 8ns or 16ns
interval. In this case, ZIO timestamps will store the seconds in
@code{secs}, the nanoseconds in @code{ticks}, though with a granularity
of 8 or 16 nanoseconds, and the phase in the @code{bins} field.
This setup allows generic ZIO users to still get most
and phase offsets as 12-bit fractional bit withing the 8ns tick.
In this case, ZIO timestamps will store the seconds in
@code{secs}, the nanoseconds in @code{ticks}, as a multiple of 8,
and the phase in the @code{bins} field.
This setup allows WR-unaware ZIO users to still get most
of the information, while our application can combine
@i{ticks} and @i{bins} to get the full resolution provided by
the hardware.
......@@ -1171,16 +1254,18 @@ the hardware.
@cindex extended control structure
The control structure already includes a TLV record at the end. This
is currently unused, but we'll need to use TLV records in future
versions. This section describes our plans and how the introduction
versions. This section describes our plans and how (actually, how little)
the introduction
of TLV will affect developers of external ZIO modules.
@cindex interleaved channels
The need for extending the control comes from interleaved
acquisition. Some input or output devices exchange buffers where
data is interleaved between the channels, but each channel has
its own attributes, like gain and offset. We may use a cset with
its own attributes, such as gain and offset. We may use a cset with
a single interleaved channel, but this has two problems: on one
side 32 device attributes may not be enough, and on the other
we need to describe how physical channels are mapped to the interleaved
side 32 device attributes may not be enough for all channels, and on the other
we need a way to describe how physical channels are mapped to the interleaved
data set.
The initial idea was to use several control structures attached to a
......@@ -1192,12 +1277,13 @@ it is now is not future-proof.
After serious discussion considering real use cases, we found that the
best approach is extending the control with a TLV structure. This
gives flexibility and generality at the same time: any user who
ignores some type can just use the length to skip over the data (but
ignores some type number can just use the length to skip over the data (but
still carry everything to others if the specific user is only one
step in a communication channel).
Thus, the last 16 bytes of the control, which were unused
are the first @i{lump} of a TLV chain.
@cindex lump, in TLV area
Thus, the last 16 bytes of the control, which were unused before version 1.0
of the data structure, are the first @i{lump} of a TLV chain.
A TLV record is defined like this, in @t{zio-user.h}:
......@@ -1211,85 +1297,90 @@ struct zio_tlv {
@end example
While 32 bits for the type and length may seem overkill, the choice is
meant to have 8-byte alignment in the payload to prevent
meant to ensure the payload is 8-byte aligned, to prevent
inconsistencies between 32-bit and 64-bit hosts.
The type is split into globally-assigned and locally-assigned numbers.
@cindex type, in TLV
The @i{type} is split into globally-assigned and locally-assigned numbers.
We reserve to define new types in the low half of the range in future
versions of ZIO, while our users are free to use any new TLV they need
in passing special meta-data throughout the ZIO pipeline.
Type 0 is the terminator. All TLV sequences must end with 16 bytes of
zeroes. The unextended control, with its trailing zeroes, is
@cindex terminator, in TLV
Type 0 is the terminator. All TLV sequences must end with a @i{lump}
of 16 zeroes. The unextended control, with its trailing zeroes, is
thus already TLV-compliant.
Type 1 is "read more". Its length is always 1 lump and its content
is a 32-bit count of how many bytes of TLV follow after this lump (the
remaining 4 bytes are unused). Thus, when a control is augmented
with trailing TLV data, it will feature this @i{read more} lump.
remaining 4 bytes are unused). Thus, when a control structure is augmented
with trailing TLV data, it must feature this @i{read more} lump.
The consumer of the control
will be able to read all the trailing data with a single system
is thus able to read all the trailing data with a single system
call.
@unnumberedsubsubsec Effect on device drivers
When we'll introduce TLV support in ZIO core, we'll simply add a
@i{controlsize} field to the cset structure. So your driver can state
@i{controlsize} field to the cset structure. So ZIO drivers can state
the overall size of the control and exchange extra metadata with
user space. If you are not using this feature your code will not
be affected and will work perfectly after recompilation under the new
ZIO version.
ZIO data structures.
Current drivers will work unchanged, because the core will turn an
unspecified @i{controlsize} to 512 at registration time.
Current drivers will need no modification, because the core knows
a zeroed @i{controlsize} field means 512 bytes.
Buffer modules won't be affected, as long as they use the provided ZIO
helpers to allocate the control structure. If buffer modules
need to allocate the control in a special way, they should call
@t{zio_get_control_size(cset)} to be future-proof. This for example
@t{zio_control_size(cset)} to be future-proof. This for example
applies to the @t{PF_ZIO} buffer, which needs to store the control
in the socket buffer it allocates through the network subsystem.
inside the socket buffer it allocates through the network subsystem.
Generic trigger drivers won't be affected because they only look
in trigger attributes in the base control.
at trigger attributes in the base control.
@unnumberedsubsubsec Effect on user space
User space code designed to work with a specific driver won't be
affected by the change, because the control size is not going to
affected by the change, because the control size for its own driver
is not going to
change. Current code will continue working as well as future code
that doesn't need to use TLV records. Needless to say, user-space
code designed to work with a device that uses TLV must be aware of
the TLV records.
such TLV records.
Generic user space tools, that use the ZIO abstraction to work with
any I/O peripherals, on the other hand, need to take care of TLV
On the other hand, generic user space tools
that use the ZIO abstraction to work with any I/O peripherals
need to take care of TLV
extensions using the suggested ZIO protocol. Such protocol already
works with the unextended control, and actually it is already used by
the example programs in the @i{tools} subdirectory. Clearly,
generic code that only manages data ignoring metadata is not affected.
works with the unextended control.
@c and actually it is already used
@c by the example programs in the @i{tools} subdirectory.
Clearly,
generic code that ignores metadata and only manages data, is not affected.
For the input direction, generic code that reads a ZIO control device
must work like this:
@itemize @bullet
@item Read 512 bytes (sizeof zio_control);
@item Read 512 bytes (@t{sizeof zio_control});
@item Check the first TLV lump, which is guaranteed to be type 0 or 1;
@item If it is 1 (@i{read-mode}), read the trailing part of the control.
@item If it is 1 (@i{read-more}), read the whole trailing part of the control.
@end itemize
For the output direction, generic code receives metadata from
elsewhere and sends it to the device. If metadata
is concatenated with data, because it is streamed through some other
elsewhere and sends it to the device. If metadata and data
are concatenated, because they both are streamed through some other
communication mechanism, generic code needs to follow the
same procedure described above.
This is how parser code should be implemented. But, again, please
remember that in most cases your control will be unextended and you
remember that most of the time your control will be unextended and you
can just ignore TLV extensions:
@example
......@@ -1308,7 +1399,7 @@ can just ignore TLV extensions:
@end example
The code above assumes native endianness, which always applies when
working on the local host. If you scan remote blocks, you should
working on the host where devices live. If you scan remote blocks, you should
convert all values using the endianness information from @t{ctrl->flags}.
......@@ -1317,12 +1408,13 @@ convert all values using the endianness information from @t{ctrl->flags}.
@section The Data
@cindex data block
The data pointed-to by a control structure is just opaque data for
ZIO. For output, only the device needs to make sense of it (or just
pass it hardware without knowing what it is); for input, the
The data pointed-to by a control structure is just opaque payload for
ZIO. For output, only the device driver
needs to make sense of it (or it may just
pass it hardware without ever knowing what it is); for input, the
final destination of the data will use it according to the
device/cset/channel it originated from, and the information about
sample size, bits and alignment found in the control structure.
device/cset/channel it originated from, as well as the information about
sample size, bits and alignment that is found in the control structure.
In practice, only the endpoints of the pipeline need to know what the
data is (usually relying on the metadata provided in the control
......@@ -1347,9 +1439,10 @@ master branch of the ZIO repository as soon as it is stabilized.
@node Details of Char Device Policies
@section Details of Char Device Policies
@cindex char devices
The default user-space interface of ZIO is based on character devices.
The framework registers two devices for each channel: one is used
to exchange the control, and the other to exchange the data. The name
to exchange the control, and the other is used to exchange the data. The name
of the device is automatically generated from the name of the driver,
its @i{device_id} field, the cset number and the channel number.
......@@ -1367,23 +1460,36 @@ they appear as follows:
@end smallexample
The exact name, unfortunately, depends on the version of @i{udev} you
are running (older versions did not create the directory @i{/dev/zio}).
are running (older versions did not make the directory @i{/dev/zio}
and just created the devices in @i{/dev}).
@cindex control device
@cindex data device
The role of the two devices is as represented in figure
@ref{fig:cdev}. The meta-data and data are strictly ordered in time,
but applications can choose to read either one or both. For input, if
you only read one device, the other is discarded; the framework makes
all devices readable when a block is available, and users can choose
what to read -- after you read some of the data, the next control read
will wait for the next block. For output, if you only write data the
default control is used; if you write the control you replace the
default control.
you only read one of the devices, the other items are discarded;
the framework marks both devices as
readable when a block is available, and users can choose
what to read -- after you read at least some of the data,
the next read from the control device discard trailing data and returns
the next control. For output, if you only write data the
default control is used; if you write the control you replace
the default control for the following data writes. (This is not
yet implemented in the current release, although we have a beta version).
@c FIXME: write control
@cindex DTC devices
If the channel is a zero-size device, user space must write only
control blocks. This is how the DTC devices work, and
@i{tools/test-dtc} shows how to do that.
@float Figure,fig:cdev
@image{img/zio-cdev, 15cm, , the two char devices, gif}
@end float
@sp 1
@cindex shell tools to access ZIO devices
This split allows simple programs to just ignore the metadata and
retrieve unadorned data in the usual way: @i{cat}, @i{dd}, @i{hexdump}
are all good tools for simple diagnostics of both input and output.
......@@ -1393,36 +1499,44 @@ ZIO data are described in @ref{User Space Utilities}.
You can think of this approach as what you normally do with industrial
food items. The envelope describes the specific content (including the
timestamp, name, quantity and weight of each item), and you can choose
to read the envelope or not; similarly you can choose to eat the food
timestamp, name, item size and item count of each package), and you can choose
to read the envelope or not; similarly, you can choose to eat the food
or not, and sometimes you make the choice only after reading the
envelope.
@tindex mem_offset
@cindex mmap for data access
@cindex vmalloc buffer type
Different buffer types can offer different functionalities; for example
the @i{vnalloc} buffers supports @i{mmap} by means of the @t{mem_offset}
field in the control. Applications can @i{mmap} the whole buffer, so
they can read the control and just access the data. This is represented
in @ref{fig:mmap}.
the @i{vmalloc} buffer type supports @i{mmap} by means of the @t{mem_offset}
field in the control. Applications can @i{mmap} the whole buffer; later
they can read the control and just access the data through a pointer.
This is represented in @ref{fig:mmap}.
@float Figure,fig:mmap
@image{img/zio-buffer-mmap, 15cm, , the vmalloc buffer and char devices, gif}
@end float
@sp 1
An attribute of the @i{vmalloc} buffer, can turn it into a
real circular buffer: individual buffer instances can merge
(sticks together)
several data blocks, in order to immediately release some control
@cindex vmalloc buffer type
An attribute of the @i{vmalloc} buffer, called @t{merge-data}
can turn it into a
real circular buffer: a buffer instance, if so configured, can merge
(stick together)
several data blocks, in order to immediately release some of the control
structures. The user may activate the attribute if the application knows
it won't need meta-data for every block: if you know you acquire at
1kHz, for example, time-stamping the first sample may be enough,
whereas further streaming is self-timed. The attribute doesn't break
the ZIO data model because whenever a new data block arrives it is
stuck to the previous data by releasing the new control and increasing
the block size of the previous block; when a block enters an empty
buffer, its own control is preserved (but its own @t{nsmaples} field may
be increased later if another data blob is merged to it). The @i{control +
data} abstraction will thus continue working towards user space. This
it won't need meta-data for every block: if you know you acquire a continuous
stream at
1kHz, for example, time-stamping the first sample may be enough, because
further data samples are self-timed.
By setting the attribute you don't break
the ZIO data model because whenever a new data block arrives, the buffer
sticks it to the previous data by releasing the new control and increasing
the block size of the previous block. When a block enters an empty
buffer, its own control is preserved, but its own @t{nsamples} field may
be later increased when another data block is merged to it. The @i{control +
data} abstraction thus still works towards user space. This
situation is shown in @ref{fig:circ}.
@float Figure,fig:circ
......@@ -1437,6 +1551,11 @@ situation is shown in @ref{fig:circ}.
The distribution includes a few device-independent tools in the
@i{tools} subdirectory.
@c --------------------------------------------------------------------------
@node zio-dump
@subsection zio-dump
@cindex zio-dump
The most important one is @t{zio-dump}: a program that reads pairs of
ZIO devices and prints to @i{stdout} both the meta-data and the data
it finds.
......@@ -1447,6 +1566,11 @@ of @i{zzero}. The first channel returns zeroes, the second is
random and the third is sequential; the current trigger is the
timer, with a 2-second period.
(Actually, the current version of @i{zio-dump} prints more
information from the control block, including alarms, but we had no
time to update this section).
@c FIXME: new zio-dump
@smallexample
spusa.root# ./tools/zio-dump /dev/zio/zzero-0000-0*
Ctrl: version 1.0, trigger timer, dev zzero-0000, cset 0, chan 0
......@@ -1503,18 +1627,27 @@ the device and the trigger appear as a named file in @i{sysfs}:
@smallexample
spusa.root# ls /sys/bus/zio/devices/zzero-0000/zero-input-8/trigger/
enable ms-period name post-samples power/ uevent
enable ms-period ms-phase name post-samples power/ uevent
@end smallexample
The mapping between names and attribute positions is available by using
The mapping between names and attribute positions in the control
data structure is available by using
the proper defines in source files. We'll offer a way to see the mapping
from @i{sysfs} in future versions.
@c --------------------------------------------------------------------------
@node zio-cat-file
@subsection zio-cat-file
@cindex zio-cat-file
@cindex mmap
An alternative to @t{zio-dump} is @t{zio-cat-file}: it receives a
single file name on the command line, and copies it to @i{stdout}
using @i{mmap} if so supported. The program works by
using @i{mmap} if so supported by the device (i.e., but the underlying
buffer type). The program works by
opening the ZIO control file associated with the named data file,
using the meta-information in there to time data accesses. If your
and by using the meta-information in there to time data accesses. If your
buffer is @i{vmalloc} or any other @i{mmap}-capable buffer, the
tool will memory-map the data device instead of reading it:
......@@ -1532,40 +1665,44 @@ spusa.root# ./tools/zio-cat-file /dev/zio/zzero-0000-0-2-data 3 | od -t x1z
By looking at the source code or using @i{strace} you can verify how
data is retrieved my memory mapping instead of reading.
@c ##########################################################################
@node Accessing ZIO from Kernel Space
@chapter Accessing ZIO from Kernel Space
@c --------------------------------------------------------------------------
@node test-dtc-file
@subsection test-dtc
Still to be written. Feel free to express your interest in this
chapter to the mailing list.
@cindex DTC devices
@cindex test-tdc
The simple @i{test-dtc} program writes control blocks to a control
file, specifying timestamps for a DTC device. By default the program
uses the name of the @i{zio-fake-dtc} device that uses an high-resolution
timer to create output events at specified times, according to the
control structures it receives.
@c FIXME: more info on test-dtc
@c ##########################################################################
@node Internals
@chapter Internals
This chapter details the design ideas and data structures that make up
the three main ZIO objects. It is what you need to know in order to
write your own ZIO modules, to be accompanied with references to the
real code part of the ZIO software distribution.
The section about devices includes the description of csets and
channels.
the three main ZIO objects, as well as the naming conventions we follow
throughout ZIO code.
@c ==========================================================================
@node Naming Conventions
@section Naming Conventions
@cindex naming conventions in ZIO code
To avoid confusion in the terminology used in ZIO code and
documentation, we list here the types and data structures used in
ZIO. For each item we list the structure name (without @t{struct}), the
name (english word) we consistently use in all documentation and the
name (English word) we consistently use in all documentation and the
preferred name for pointers to that data structure, used in the code.
@multitable @columnfractions .25 .2 .1 .45
@headitem Struct name or C type @tab Name @tab Var name @tab Description
@item zio_device
@tab full-device
@tab full device
@tab zdev
@tab The overall I/O device, either a board or a chip.
......@@ -1675,8 +1812,9 @@ all three levels of data structures (device, cset, channel), so this
chapter describes the important fields of those structures. It then
offers an overview of the registration steps and an example.
Unless you want to know all the details immediately, at a first read
we suggest to skip the first sections and jump to @ref{The Driver Structure}.
Unless you want to know all the details immediately, for a cursory
overview
you may want to skip the first sections and jump to @ref{The Driver Structure}.
@c --------------------------------------------------------------------------
@node The Device Structure
......@@ -1715,9 +1853,12 @@ are:
@item char *preferred_trigger
The device may specify a device-wide default trigger type
and/or buffer type. This allow a device with an hardware-internal
trigger to used that as soon as it is initialized, instead
of requiring the user to select it. If the fields are NULL
and/or buffer type. Csets can still make their specific choice,
which takes precedence, to easily support
kernel modules that register a device-specific trigger.
These ``preferred'' names are meant to be modified by module
parameters (see @t{zio-zero} and other examples).
If the fields are NULL
or the preferred type is not available in the running instance
of ZIO, the system-wide defaults for buffer and trigger type will
be used.
......@@ -1740,10 +1881,15 @@ As said, the cset is a homogeneous set of I/O channels belonging to a
single device. All channels in the set have the same physical
characteristics. This object is the most important in the ZIO device
hierarchy because all data transfers are cset-wide. Each cset includes
a pointer to the current trigger and buffer types, because the current
types are two cset-wide attributes. (We chose to set the preferred type
at device level because this fits most use cases, while avoiding
the need to repeat the default for each cset).
a pointer to the current trigger and buffer types, which are
cset-wide attributes.
@cindex trigger type for a cset
@cindex buffer type for a cset
The data structure includes two string fields (@t{zbuf} and @t{trig})
that name the requested trigger and buffer types for this cset.
If the fields are blank or the named object is unavailable, the
device-wide or zio-wide defaults are tried in this order.
@tindex zio_cset
The most important fields of @code{struct zio_cset} to be filled
......@@ -1826,9 +1972,9 @@ or used by the developer are:
@itemx void (*exit)(struct zio_cset *cset)
The function pointers, if not NULL, are called by ZIO at
cset registration and removal time, after allocating (before
removing) the channel array. They may useful to resp. setup
and release the @code{priv_d} field.
cset registration and removal time, after allocating and before
removing, resp., the channel array. They may useful to channel
sets that need to setup and release the @code{priv_d} field.
@end table
......@@ -1873,6 +2019,7 @@ The top-level structure that devices a device driver is similar to
what is used with PCI or USB. This is the definition of the @t{zio_driver}:
@tindex zio_driver
@tindex zio device table
@smallexample
struct zio_driver {
const struct zio_device_id *id_table;
......@@ -1896,6 +2043,7 @@ fields. See @ref{An Example Driver}.
@cindex identifier table
@cindex table of identifiers
@cindex device name
@cindex device table
@cindex match function
The table of identifiers, then, is defined like this in for ZIO devices:
......@@ -1908,8 +2056,9 @@ struct zio_device_id {
@end smallexample
The @t{name} field is a 12-byte string, the same string that is part
of the control block. The @t{template} is a a filled ZIO device,
the the core replicates for each and every device matching the name.
of the control block. The @t{template} is a ZIO device, with the
proper fields initialized. The ZIO core replicates such template
for each and every device matching the name.
The @t{id_table} field in @t{zio_driver} is a pointer to an array of
identifiers; the array is terminated with an empty entry, as customary
......@@ -1922,6 +2071,8 @@ in many Linux subsystems.
By virtue of how the Linux bus abstraction works, registering a ZIO
device is a two-step procedure, based on the above data structures.
@cindex probe function
@cindex remove function
The client driver must register a @t{struct zio_driver}, with the
associated table of identifiers; when the core finds a match, it
allocates a new device instance and calls the @i{probe} method.
......@@ -1929,6 +2080,7 @@ Please note that the @t{zio_device} passed to the @i{probe} method
is not the template, but a copy of it -- this is mandatory in order
to support several devices of the same type running on the same host.
@cindex private data in the device
If, for some reason, you need to save some private data in the device
structure, you must do that in the @i{probe} function for each new
device instance -- and undo any step in the associated @i{remove}
......@@ -1937,7 +2089,8 @@ function.
If your driver is able to drive two or more devices, that are similar
in nature but feature a different number of csets or differ in some
other detail, you should fill your table with more than one device
type, each of them with a different name.
type, each of them with a different name. This is exemplified by the
@t{ad788x} driver, part of the ZIO distribution.
@c --------------------------------------------------------------------------
@node Registering a ZIO Device
......@@ -2025,17 +2178,17 @@ static struct zio_driver zmini_zdrv = {
@end smallexample
With these structures in place, the @i{init} function of the module
will just need to call ``@t{zio_register_driver(&zmini_zdrv)}''.
just needs to call ``@t{zio_register_driver(&zmini_zdrv)}''.
Actually, the data structures above are more than half of the complete
driver: the only missing bit is the @t{raw_io} cset method, because
triggering, buffering and everything else is manage automatically by
triggering, buffering and everything else is managed automatically by
the ZIO framework.
This is a very simple case, where the table includes only one entry,
the device has only one cset and there's no need for @i{probe} and
@i{remove}. The seemingly-unneeded extra levels are there in order to
be able to support more complex cases, including the common situation
where more than one device is driven by a single host.
when more than one device is driven by a single host.
@c ==========================================================================
......@@ -2047,10 +2200,10 @@ Every data exchange, either input or output, is executed in response
to an event of some kind. ZIO offers a @i{trigger} abstraction
to describe all such events and configure their activation.
Each cset is using a trigger type, and a specific instance of that type.
Each cset is connected to a trigger type, and a specific instance of that type.
Different csets can use different trigger types, because the
@code{current_trigger} is an attribute of each cset. When the trigger
fires, it acts on all the non-disabled channels of the cset.
arms, it acts on all the non-disabled channels of the cset.
@cindex input and triggers
@cindex output and triggers
......@@ -2059,8 +2212,6 @@ For input data flows, the trigger receives blocks from the device and
stores them in the buffer. For output data flows the trigger retrieves
blocks from the buffer and sends them to the device.
@c FIXME: where does the control block come from?
@cindex trigger type, defining
When defining a new trigger type, the most important fields of
@code{struct zio_trigger_type} for the programmer are the following
......@@ -2072,11 +2223,11 @@ ones:
@item const struct zio_sysfs_operations *s_op
These are the operations used to read and write attributes.
It is the same structure used in the device.
It is the same set of operations used in the device data structure.
@item const struct zio_trigger_operations *t_op
The trigger operations, expanded below, are the ones that
The trigger operations, described below, are the ones that
implement the behavior of a trigger type.
@end table
......@@ -2103,7 +2254,7 @@ struct zio_trigger_operations {
void (*destroy)(struct zio_ti *ti);
void (*change_status)(struct zio_ti *ti,
unsigned int status);
void (*abort)(struct zio_cset *cset);
void (*abort)(struct zio_ti *ti);
};
@end smallexample
......@@ -2119,22 +2270,22 @@ The detailed meaning of the operations is as follows:
The operations are called when this trigger type is associated to
(resp. de-associated from) a new cset. @code{create} returns
a disabled trigger instance structure, which is usually part of a larger
structure that the instance itself will recover with the macro
@code{container_of}. Please look at existing triggers for details
(for example, @t{ti->cset} must be set at creation time.
structure, accessed by using
@code{container_of}. Please look at existing triggers for details.
@cindex push_block
@findex push_block
@item push_block
When a buffer has a complete block of data, it can send it to
the trigger using @code{push_block}. The trigger can either accept it
(returns 0) or not (returns @code{-EBUSY}). This because an
(returning 0) or not (returns @code{-EBUSY}). This happens because an
output trigger has only one pending data transfer. When the block is
consumed, the trigger will call @code{bi->retr_block} to get the next
one. Buffering is in the buffer, not in the trigger.
finally consumed,
the trigger must call @code{bi->retr_block} to get the next
one: buffering is in the buffer, not in the trigger.
@c FIXME: check push_block for double buffering
@cindex pull_block
@findex pull_block
@item pull_block
For input channels, a buffer may call @code{pull_block}. The trigger
......@@ -2144,17 +2295,19 @@ The detailed meaning of the operations is as follows:
a new block is available. In these cases the @code{pull_block}
method can be left @code{NULL}.
@cindex data_done
@findex data_done
@item data_done
This method, if defined, is called by the device by means of the helper
This method, if defined, is called by the core inside the helper
@t{zio_trigger_data_done()}. I/O in the device is
almost always asynchronous, so when asked to transfer a cset,
the device will prepare to do it, and will call @code{zio_trigger_data_done}
later. For output csets,
@code{zio_trigger_data_done} frees the blocks and prepares new blocks
for the next trigger event; for input, @code{zio_trigger_data_done} pushes
material to the buffers. This method, if present, is called
the device will prepare to do it, and will call
@code{zio_trigger_data_done} later. For output csets,
@code{zio_trigger_data_done} frees the blocks;
for input, @code{zio_trigger_data_done} pushes
material to the buffers. If the peripheral is self-timed,
the function also arms the trigger for the next event.
The method, if present, is called
while holding the @i{cset} spin lock, and the trigger still
in @t{ARMED} state (@t{zio_trigger_data_done} clears the flag only
when everything is over with the current trigger event).
......@@ -2167,6 +2320,7 @@ The detailed meaning of the operations is as follows:
structure, this callback allows the trigger to reconfigure itself.
@c FIXME: use the config trigger method
@cindex trigger abort
@item abort
Abort, if defined, is called when an already-armed trigger event
must be aborted. This happens, for example, because event
......@@ -2176,14 +2330,14 @@ The detailed meaning of the operations is as follows:
pointer to NULL. Please check how it is used in @t{helpers.c}.
The method
is called while holding the cset lock and cannot fail nor sleep.
A trigger implementing this method will usually call
the @i{stop_io} cset method. See the generic abort implementation
for reference.
See the generic abort implementation for reference.
@findex change_status
@cindex trigger enable and disable
@item change_status
The method, if defined, is called when the trigger is enabled or
disabled. On disable, ZIO calls @i{abort} beforehand.
The method is called while hoding the cset lock.
disabled. For disable, ZIO calls @i{abort} beforehand.
The method is called while holding the cset lock.
@end table
......@@ -2193,15 +2347,18 @@ The detailed meaning of the operations is as follows:
@cindex buffer
The buffer interface in ZIO allows to select between different
allocation techniques and memory access. By splitting the buffer
allocation techniques and memory access patterns. By splitting the buffer
to a separate ZIO object, the framework allows drivers with special
allocation needs to define their own hardware-specific buffer.
allocation needs to define their own hardware-specific buffer -- for
example, data samples may need to live in PCI memory.
@cindex csets and buffers
Each cset is using a buffer type, for which an instance exists for
each channel in the cset. Data transfers only happen if a channel
is enabled, so different buffers instances in the cset may at
Each cset is using a buffer type; an instance of that buffer type
exists for each channel in the cset. Actual I/O only happens if a channel
is enabled, so different buffer instances in the cset may at
times host a different number of blocks, but this is not the usual case.
When a buffer experiences overflow, a data block is lost, and this is marked
in the @t{zio_alarms} field for the specific channel.
The most important fields of @code{struct zio_buffer_type}, from
the point of view of the developer of anew buffer, are the various
......@@ -2214,16 +2371,17 @@ operations structures:
@item const struct zio_buffer_operations *b_op
The buffer operations, detailed later, are the function
The buffer operations, detailed later, are the functions
that define the actual behavior of a buffer.
@cindex file operations
@tindex zio_generic_fops
@item const struct file_operations *f_op
File operations are used to provide user-space access to
the buffer. ZIO exports a @code{zio_generic_fops} structure
that will work for most users. This pointer will be removed when
the @i{interface} idea is implemented.
that will work for most users. We may remove this pointer if
the @i{interface} idea is implemented in the ZIO core.
@cindex virtual memory operations
@cindex mmap support
......@@ -2232,14 +2390,15 @@ operations structures:
Buffer types supporting @i{mmap} must implement the
virtual memory operations. They allow to keep track of active
uses of the buffer instance and handle page faults for
program accessing the buffer. The operations are associated
the processes that are accessing the buffer.
The operations refer
to each @code{vma} mapped to the char device associated to
a buffer instance of this type. Use is exactly like what
you do in normal char drivers, with the only difference the
the @code{open} method is called when @i{mmap} happens.
you do in normal char drivers, with the only difference that
the core calls @t{v_op->open} every time @i{mmap} happens
(because the driver doesn't run its won @i{mmap} system call back-end.
Please refer to the @code{zio-buf-vmalloc} implementation for
details.
@c FIXME: vmalloc reference to in-manual vmalloc desc
@end table
......@@ -2274,39 +2433,40 @@ This is the specific role of each method in the structure:
When ZIO associates a buffer with a new channel, it calls the
@code{create} operation. The returned @code{zio_bi} structure
will usually be part of a bigger structure used internally
by the buffer implementation, using the @code{container_of}
macro to access it from the @code{zio_bi} pointer. If creation
is usually be part of a bigger structure, accessible using
@code{container_of}. If creation
fails, the method must return an @t{ERR_PTR} value, not NULL.
@c FIXME: check ERR_PTR use
@cindex alloc_block
@cindex free_block
@findex alloc_block
@findex free_block
@findex zio_alloc_control
@item alloc_block
@itemx free_block
The buffer is concerned with memory management, so whenever
The buffer is concerned with data storage, so whenever
the trigger or the @i{write} system call need a new block,
they ask it to the buffer type. Similarly, the buffer type
is asked to release blocks. The function must also allocate
the control, likely with @t{zio_alloc_control}. Such control
will be filled with the current values for the channel in due time.
(For input this copy happens late). On error it return NULL.
is asked to release blocks. @t{alloc_block} must also allocate
the control, through the helper @t{zio_alloc_control}. Such control
is filled with the current values for the channel in due time.
(For input this copy happens late).
On error allocation must return NULL.
@cindex store_block
@cindex retr_block
@findex store_block
@findex retr_block
@item store_block
@itemx retr_block
The functions simply add a block to an existing buffer instance
or ask to retrieve a block out of it. In addition to managing
storage according to its own will, the buffer is requested to
or ask to extract a block out of it. In addition to managing
storage according to its own needs, the buffer is requested to
make two special actions. When @code{store_block} inserts
the first block in an empty output buffer, the method must
call the @code{push_block} method of the associated trigger.
When @code{retr_block} asks for a block from an empty input buffer,
the method must call @code{pull_block} in the associated trigger,
if not NULL. Please refer to existing implementations
for details.
call the @code{ti->push_block}.
When @code{retr_block} is called on an empty input buffer,
the method must call @code{ti->pull_block}, if the function exists.
Please refer to existing implementations for details.
@end table
......@@ -2314,55 +2474,114 @@ This is the specific role of each method in the structure:
@node The Attributes
@section The Attributes
Still to be written. Feel free to express your interest in this
section to the mailing list.
This section is till to be written. Feel free to express your interest
in this section to the mailing list. Meanwhile, please refer to
current code.
@c ##########################################################################
@node Available Modules
@chapter Available Modules
@c FIXME: include the list of types and abbreviations (from the wiki)
The current ZIO repository includes a number of modules for devices
triggers and buffers. They are meant to act as test cases, examples
and tools to stress-test the code. Some of them are useful in the
real world, despite their simple and straightforward design.
The first hardware parts for the real use cases are going to be
available during February 2012, so this list will soon increase.
@c ==========================================================================
@node Available Devices
@section Available Devices
Still to be written. Feel free to express your interest in this
section to the mailing list.
Drivers are listed in @ref{Supported Devices}, so they are not
repeated here.
@c ==========================================================================
@node Available Triggers
@section Triggers
Still to be written. Feel free to express your interest in this
section to the mailing list.
@cindex triggers in the distribution
This release of ZIO includes the following trigger types:
@table @t
@cindex transparent trigger
@cindex user trigger
@item user
This is the default trigger, linked in @t{zio.ko}. It is
a transparent trigger: then user space reads an input channel,
it arms the input event, and when user space writes to output
channels, the event is armed when all channels have been provided
their output block. If the device is self-timed (e.g., TDC and DTC
devices, or an ADC that streams data at a certain rate) the
trigger is armed as soon as possible, to allow the device itself
to set its own pace
@cindex timer trigger
@item timer
The timer trigger uses a kernel timer and is periodic. While
you cannot set it to an absolute time, it includes support to
set the @i{phase}, to be able to schedule in time I/O on several
csets. How exactly phase support works is documented in the
source code and the commit messages.
@cindex high-resolution-timer trigger
@item hrt
High-resolution timer. This trigger can work as both periodic
and one-shot. It is configured for absolute times or for some
delay in the future (to ease testing with shell scripts). It accepts
both scalar nanoseconds (as low-half and high-half values) and
seconds + nanoseconds. Another attribute specifies the allowed
slack to be used in programming the kernel resource.
@cindex irq trigger
@cindex gpio as a trigger source
@item irq
External-interrupt. The module receives a @t{irq=} parameter,
or a @t{gpio=} parameter (the latter only available for systems
with @i{libgpio} (and a working @t{gpio_to_irq} function).
You can have several instances of this trigger type, but all
of them are bound to the same interrupt. This is mainly used
for demonstration purposes.
@end table
@c ==========================================================================
@node Available Buffers
@section Buffers
Still to be written. Feel free to express your interest in this
section to the mailing list.
@cindex buffers in the distribution
This release of ZIO includes the following buffer types.
@c ##########################################################################
@node Writing ZIO Modules
@chapter Writing ZIO Modules
@table @t
This chapter explains how to write ZIO modules, using existing modules
as a working example. This chapter shows how to use the data structures
described earlier in this manual.
@cindex kmalloc buffer
@item kmalloc
@c ==========================================================================
This is the default buffer. It allocates blocks by calling
@i{kmalloc}. The buffer size is expressed in number of blocks,
and it defaults to 16. You can change it in @i{sysfs} for
each instance.
@cindex vmalloc buffer
@item vmalloc
This buffer allocates memory using @i{vmalloc}, and it supports
@i{mmap}. It also supports the @t{merge-data} attribute, as
described in @ref{Details of Char Device Policies}. Its size
is expressed in kilobytes, and it default to 128. Currently,
it cannot be changed, as it still doesn't count the number of
active @i{mmap} users.
@c FIXME: mmap users of vmalloc buffer
@end table
There is currently no way to change the buffer size at module load time,
but there's nothing preventing it, besides our own time supply.
@c ##########################################################################
@node Locking Policies
@section Locking Policies
@chapter Locking Policies
@cindex locking policies
@cindex spin locks in ZIO
......@@ -2404,7 +2623,7 @@ The following locks are defined used in ZIO core and data structures:
serialize access the I/O operations, as well as trigger-related
events (i.e. the @code{ti->flags} bits like
@code{ZIO_TI_ARMED}). For this reason, the field @t{ti->cset}
must be immediateky assigned when the trigger instance is created.
must be immediately assigned when the trigger instance is created.
I/O is serialized by trigger code in the ZIO core:
only one trigger event
can be pending for each cset, using the ARMED flag. The cset
......@@ -2473,27 +2692,6 @@ the utility functions @code{zio_generic_data_done} and
Device modules will need to serialize some of their non-atomic
hardware access primitives, in this case by declaring their own locks.
@c ==========================================================================
@node Writing a Device
@section Writing a Device
Still to be written. Feel free to express your interest in this
section to the mailing list.
@c ==========================================================================
@node Writing a Trigger
@section Writing a Trigger
Still to be written. Feel free to express your interest in this
section to the mailing list.
@c ==========================================================================
@node Writing a Buffer
@section Writing a Buffer
Still to be written. Feel free to express your interest in this
section to the mailing list.
@page
@c ##########################################################################
@node Index
......@@ -2506,4 +2704,5 @@ section to the mailing list.
@c LocalWords: gnudd titlepage iftex texinfo CERN documentlanguage settitle
@c LocalWords: documentencoding setfilename afourpaper paragraphindent sysfs
@c LocalWords: setchapternewpage finalout timestamping timestamp csets gpio
@c LocalWords: ohwr cset cindex itemx inutuitive
@c LocalWords: ohwr cset cindex itemx inutuitive timespec ldisc vmalloc mmap
@c LocalWords: uart ctrl zzero smallexample nsamples
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment