spec-sw.in

\input texinfo    @c -*-texinfo-*-
%
% spec-sw.in - main file for the documentation
%
%%%%

%------------------------------------------------------------------------------
%
%                         NOTE FOR THE UNAWARE USER
%                         =========================
%
%    This file is a texinfo source. It isn't the binary file of some strange
%    editor of mine. If you want ASCII, you should "make spec-sw.txt".
%
%------------------------------------------------------------------------------

%
% This is not a conventional info file...
% I use three extra features:
%   - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
%   - leading blanks are allowed (this is something I can't live without)
%   - braces are automatically escaped when they appear in example blocks
%

@comment %**start of header
@documentlanguage en
@setfilename spec-sw.info
@settitle spec-sw
@iftex
@afourpaper
@end iftex
@comment %**end of header

@setchapternewpage off

@set update-month September 2012

@finalout

@titlepage
@title SPEC Software Support
@subtitle (@value{update-month})
@subtitle A driver for the SPEC card and its FMC modules
@author Alessandro Rubini for CERN (BE-CO-HT)
@end titlepage
@headings single

@c ##########################################################################
@iftex
@contents
@end iftex

@paragraphindent 3mm

@c ##########################################################################
@node Top
@top Introduction

This is the manual for the SPEC device driver. SPEC is the @i{Simple
PCI-Express Carrier} for FMC cards, developed at
@url{http://www.ohwr.org/projects/spec}. This manual is part of the
associated software project, hosted at
@url{http://www.ohwr.org/projects/spec-sw}, whose @i{git} repository
hosts the latest version.

@c ##########################################################################
@node History and Overview
@chapter History and Overview

This driver is pretty different from the initial implementation, such as
the one used by @i{fine-delay-sw-v1.1}.  If you use such version, please
compile the manual you find in your source code repository.

The package currently includes both the @i{spec} driver and the
@i{fmc} bus driver (documented separately).
Moreover, it includes some drivers for fmc cards.

@c ##########################################################################
@node Compiling the Drivers
@chapter Compiling the Drivers

The kernel modules part of this package live in the @i{kernel}
subdirectory. To compile them, you need to 
set the following variables in your environment:

@table @code

@item LINUX

	The top directory of the kernel sources for the version you
        are going to run the driver under. I'm testing mostly with 3.4,
	but this version compiles against Linux-2.6.37 and later ones
        (2.6.36 had a different interface for hardware timestamping,
        so @i{fmc} and @i{spec} compile fine, but @i{wr-nic} does not
        because of its support for timestamping Ethernet frames.

@item CROSS_COMPILE

	If you are cross-compiling, you need to set this variable.
	It is not usually needed for the PC, but if you are using
        the @i{Powec} board, you'll most likely need this. It is not
        needed if you compile for a different-sized PC (see below).

@item ARCH

	If you are cross-compiling, set this variable. Use @code{powerpc}
        for the @i{Powec}, @code{x86-64} if you compile on a 32-bit PC
        to run on a 64-bit PC and @code{i386} if you compile on a 64-bit
        PC to run on a 32-bit PC.

@end table

To compile run ``@code{make}'' with the previous variables set.  To
install run ``@code{make install} to install under
@code{/lib/modules/3.2.0} (or other version-based directory).  You can
set @code{INSTALL_MOD_PATH} to set a prefix before @code{/lib/modules}.
For example, if your target computer's filesystem is mounted under
@code{/mnt/target} you can run

@example
   make install INSTALL_MOD_PATH=/mnt/target
@end example

The modules are installed under the subdirectory @code{extra}. In
the previous case your driver will end up being installed
(together with the other modules) as

@example
   /mnt/target/lib/modules/3.2.0/extra/spec.ko
@end example

@c ##########################################################################
@node Role of spec.ko
@chapter Role of spec.ko

The @code{spec.ko} driver depends on @code{fmc.ko}, that must
be loaded first (if you don't rely on automatic dependencies).

It registers itself as a PCI driver,
using both the ``old'' vendor and device ID (the Gennum identifiers)
and the new ones (CERN vendor and SPEC device).

@c ==========================================================================
@node SPEC Initialization
@section SPEC Initialization

For each new SPEC device found on the system, the driver performs the
following steps:

@itemize @bullet
@item It enables MSI interrupts, the only ones supported in this package.
@item It loads the @code{fmc/spec-init.bin} ``golden'' gateware file.
@item It checks that the content of the binary is as expected (using
      a minimal @i{sdb}-based verification).
@item It reads the whole I2C EEPROM found on the mezzanine.
@item It allocates an @i{fmc_device} structure and registers as
      a new device in the @i{fmc} bus.
@end itemize

Failure of any of the above steps is fatal.

The suggested @code{spec-init.bin} gateware binary is always available
from the @i{files} area of the @i{spec-sw} project on @code{ohwr.org}.
As I write this, the direct link is
@url{http://www.ohwr.org/attachments/download/1454/spec-init.bin-2012-07-24}.

@b{Note:} currently the SPEC driver does not re-write the golden
binary file when the sub-driver releases control of the card. This
allows a further driver to make use of an existing binary, and may be
useful during development. The EEPROM contents are still being
available to the new sub-driver (even if it cannot write to it, because
the carrier refused to act on registers after the golden gateware is replaced
by an new binary unknown to the carrier itself).

@c ==========================================================================
@node SPEC Module Parameters
@section SPEC Module Parameters

The module can receive the following parameters to customize its operation:

@table @code

@item test_irq

	If not zero, this parameter requests to self-test interrupt
        generation, using the Gennum registers. This usually does not
        work on my host, for yet unknown reasons (and that's why it is
        disabled by default).

@item i2c_dump

	If not zero, this parameter requests to @i{printk} the content
        of the FMC eeprom, for diagnostic purposes.

@item fw_name

	This string parameter can be used to override the default name
        (@code{fmc/spec-init.bin}) for the initialization binary file.

@end table

Any mezzanine-specific action must be performed by the driver for the
specific FMC card, including reprograming the FPGA with the final
gateware file.  Similarly, the @i{spec} driver is not concerned with
programming the LM32 image, when it makes sense to. This is different
from the role splitting in previous versions of the driver.

@b{Note:} the gateware binary is looked-for in @i{/lib/firmware/fmc},
which is where all fmc-related external files are expected to live.
That's because our own installations share firmware for COTS peripherals
but mount a host-specific NFS subdirectory.

Please refer to the @i{fmc-bus} document for details about the overall
design of the interactions of carriers and mezzanines.

@b{Warning:} currently the @i{match} function of the bus always
returns success: the mezzanines I currently have for testing have no
ID records written in their internal EEPROM, so I'm not able to setup
the associated data structures and code.  For this reason there is no
@i{module_alias} support nor autoprobe of drivers: any @i{fmc} driver
you load will drive all SPEC cards found on the system unless it
limits itself through parameters (see below)

@c ==========================================================================
@node Sub-Module Parameters
@section Sub-Module Parameters

Since most of the FMC drivers, also called sub-modules, need the same
set of kernel parameters, this package includes support
to implement common parameters, by means of fields
in the @code{fmc_driver} structure and simple macro definitions.

The parameters are carrier-specific, in that they rely on the @i{busid}
concept, that varies among carriers (here it is a PCI bus-devfn number).
Drivers for other carriers will most likely offer something similar
but not identical; some code duplication is unavoidable.

This is the list of parameters, to see how they are used in sub-modules,
please look at @i{spec-trivial.c}.

These parameters are offered:

@table @code

@item busid=

	This is an array of integers, specifying the PCI bus and PCI devfn,
        each of the fields being 8 bits (and the latter is most likely 0).
        For example: @code{0x0400} (bus 4,
        slot 0).  If any such ID is specified, the sub-driver will only
        accept to drive cards that appear in the list (even if the
        FMC ID matches).

@item gateware=

	The argument is an array of strings. If no @i{busid=} is
        specified, the first string of @i{gateware=} is used for
        all cards; otherwise the identifiers and gateware names are
        pared one by one, in the order specified.

@end table

For example, if you are using the trivial driver to load two different
gateware files to two different cards, you can use the following
parameters to load different binaries to the cards, after looking up
the PCI identifiers:

@smallexample
   insmod fmc-trivial.ko \
                         busid=0x0200,0x0400 \
                         gateware=fmc/fine-delay.bin,fmc/simple-dio.bin
@end smallexample

@c ##########################################################################
@node fmc-trivial.ko
@chapter fmc-trivial.ko

The simple module @i{fmc-trivial} is just a simple client that
registers an interrupt handler. I use it to verify the basic mechanism
of the FMC bus and how interrupts work.

The module is a user of generic SPEC parameters, so it can program a
different gateware file in each card. The whole list of parameters it
accepts are:

@table @code

@item busid=
@itemx gateware=
      Generic parameters. See @ref{Sub-Module Parameters}.

@c no docbook is generated, so the following is like "#if 0"
@ifdocbook
@item sdb=

	This is an array of integers, specifying the @i{sdb}
        starting address after programming the gateware. See
        @ref{Sub-Module Parameters} about how this is used in multi-board
        environments.
       
@item lm32=

	This is a array of strings like @code{gateware=},
        listing the name or names
        to be used to reprogram the internal LM32. The same rules as for
        @code{gateware=} above are used for matching binaries and cards.
        The carrier will copy the @i{lm32} executable to the first
        SDB record that is mapped as ``@code{WB4-BlockRAM}'' (thus,
        it needs the @code{sdb=} values to be specified.
@end ifdocbook
@end table

This driver is worth reading, but it is not worth describing here.

@c ##########################################################################
@node fmc-write-eeprom.ko
@chapter fmc-write-eeprom.ko

This module is designed to load a binary file from @i{/lib/firmware}
and to write it to the internal EEPROM of the mezzanine card. This
driver uses the @code{busid} generic parameter, but doesn't use the
other ones (even if @i{modinfo} reports them).

Overwriting the EEPROM is not something you should do daily, and is
expected to only happen during manufacturing. For this reason, the
module makes it unlikely for the random user to change a working eeprom.

The module takes the following measures:

@itemize @bullet

@item It accepts a @code{file=} argument (within @i{/lib/firmware})
and if no such argument is received, it doesn't write anything
(i.e. there is no default file name)

@item If the file name ends with @code{.bin} it is written verbatim
starting at offset 0.

@item If the file name ends with @code{.tlv} it is interpreted as
type-length-value (i.e., it allows @i{writev(2)}-like operation).

@item If the file name doesn't match any of the patterns above, it is
ignored and no write is performed.

@item Only cards listed with @code{busid=} are written to. If no
@i{busid} is specified, no programming is done (and the probe function
of the driver will fail).

@end itemize

Each TLV tuple is formatted in this way: the header is 5 bytes,
followed by data. The first byte is @code{w} for @i{write}, the
next two bytes represent the address, in little-endian byte order, and
the next two represent the data length, in little-endian order. The length does
not include the header (it is the actual number of bytes to be
written).

This is a real example: that writes 5 bytes at position 0x110:

@smallexample
   spusa.root# od -t x1 -Ax /lib/firmware/try.tlv
   000000 77 10 01 05 00 30 31 32 33 34
   00000a
   spusa.root# insmod /tmp/fmc-write-eeprom.ko busid=0x0200 file=try.tlv
   [19983.391498] spec 0000:03:00.0: write 5 bytes at 0x0110
   [19983.414615] spec 0000:03:00.0: write_eeprom: success
@end smallexample

Please note that you'll most likely want to use SDBFS to build your
EEPROM image, at least if your mezzanines are being used in the White Rabbit
environment. For this reason the TLV format is not expected to be used much
and is not expected to be developed further.

@c ##########################################################################
@node The WR-NIC
@chapter The WR-NIC

With the current code base, the @code{wr-nic.ko} driver is designed
to run with the simple 5-Channel DIO board, because it includes both
code to access the network card and code to act on the DIO channels,
using time tags that are provided by the White Rabbit mechanism.
Similarly, both incoming and outgoing frames can be time-stamped by
White Rabbit.

This driver is the most important driver in this package: it is
a generic implementation of the @i{spec-sw} framework which can be
useful by itself as a White Rabbit starter kit.  Moreover, it is a
complete driver that can serve as a model for other developments.

Within @i{White Rabbit} we have other full-featured drivers for
specialized FMC mezzanines hosted on the SPEC carrier. They are not
part of this package because of their specialized nature; all of them
are nonetheless hosted on @url{www.ohwr.org}, usually as a
@i{software} subproject of the related gateware project.

@c ==========================================================================
@node Code Layout
@section Code Layout

This section is mainly for the developers who look in the code, and for
me to make order in my own mind. SPEC users are expected to skip to
the next section.

The @code{wr-nic.ko} is built using a number of headers and source files,
spread over several directories:

@table @file

@item wbgen-regs/

	The directory hosts the register definitions for the various
        core that are included in the FPGA binary. The headers are
        generated by running @i{wbgen2} over the @code{.wb} files that
        are part of the VHDL source repositories; unfortunately some
        minor editing is needed on the @i{wbgen2} output, so there
        is a @code{Makefile} that takes care of this.  This package
        includes both the input file and the output header; the log
        messages details the upstream origin of each @code{.wb} file.
        The directory started out as a direct copy of the directory
        with the same name found in @code{wr-switch-sw}, release 3.0.

@item wr_nic/

	The directory started out as an unchanged copy of the driver
        used in the @code{wr-switch-sw} package, release 3.0. The directory
        name is the same in both projects. All later
        commits take care of differences in the SPEC with regard to the
        switch, but we plan to clean up those later changes and reach
        a unified code base between the White Rabbit switch and the White
        Rabbit node. The NIC driver itself is a @i{platform driver},
        instantiated by @i{platform devices} defined externally.

@end table

The @i{wr-nic} driver refers to several headers, in addition to the
register definitions. This is the role of each of them:

@table @file

@item include/linux/fmc.h
@itemx include/linux/sdb.h
@itemx include/linux/fmc-sdb.h

	These three headers are used to define the interface to the
        FMC bus abstraction and the SDB self-description of the
	internal FPGA bus. They are used by other @i{spec-sw} files as well.
        We include them as @code{<linux/fmc.h>} because we plan to
        have them upstreamed to the official kernel, and we don't want to
        introduce incompatibilities in the related source files.

@item wr_nic/nic-mem.h
@itemx wr_nic/nic-hardware.h
@itemx wr_nic/wr-nic.h

	These headers come from @i{wr-switch-sw/wr_nic} with minor spec-
        related modifications. @i{nic-mem.h} defines the memory map and
        is now almost obsoleted by SDB; @i{nic-hardware.h} is a collection
        of inline functions used by the driver; @i{wr-nic.h} defines
        all the important data structures and @i{ioctl} commands. Because
        of @i{ioctl} commands, it has a rather generic name and
        is meant to be included by user space as well as kernel space.

@item spec.h

	Definitions related to the SPEC carrier (Gennum registers and
        other SPEC-internal stuff).  It is currently used by
        @i{wr-nic-eth.c}, which is not completely carrier-independent.

@item spec-nic.h

	The header defines the SDB vendor and device values used in
        @i{wr-nic} as well as the data structures and prototypes used
        internally by the driver.

@item wr-dio.h

	The header hosts the user interface to access the DIO channels.
        It is included by @i{wr-nic-dio.c} as well as the user-space
        tools that want to configure DIO operation.

@end table

The source code of the driver itself is split in several files, in addition
to the NIC platform driver hosted in @file{wr_nic/}:

@table @file

@item wr-nic-core.c

	The file is the @i{fmc} driver: it implements the @i{probe}
        and @i{remove} methods and deals with loading the firmware
        file and the LM32 program binary (called @i{wrc}: White Rabbit
        Core).

@item wr-nic-eth.c

	This is concerned with creating the platform device for the
        network interface card. It maps the needed device memory, allocates
        the platform data and sets up the internal interrupt controller
        to route interrupts to the platform driver.

@item wr-nic-dio.c

	This is the mezzanine-specific driver. It implements the
        @i{ioctl} commands that allow user space to talk with
        the mezzanine. It only implements the @i{ioctl} method
        and support functions for it (e.g., interrupt management).
        If you want to port  @i{wr-nic} to a different mezzanine, this
        is the file you need to replace.

@end table

@c ==========================================================================
@node Overview of the Driver
@section Overview of the Driver

The @i{wr-nic} driver is basically an Ethernet driver with support for
hardware time stamping.  The DIO mezzanine card can be used by means
of @i{ioctl} commands. Such commands are designed to be portable, so
user-space programs should able to identify which mezzanine is
connected to the SPEC network card and act accordingly.

The driver loads two binaries, using the @i{firmware loader} mechanism
offered by the Linux kernel: one is the gateware file, that is requested
through the @i{reprogram} carrier method; the other is the LM32 program
binary, which is only loaded on user request.

The default file names are as follows:

@table @file

@item fmc/wr_nic_dio.bin

	This is the @i{gateware} file. The default name can be changed
        using the @code{file=} module parameter.

@item fmc/wr_nic_dio-wrc.bin

	This is the LM32 program file, or @i{White Rabbit Core}, WRC.
        The file is not loaded automatically, because we expect to deliver
        a gateware file that already includes the correct LM32 program
        (but the binary currently suggested does not include it).
        To request loading the file you should pass @code{wrc=1}. To
        request loading a different WRC file name, you should pass
        the actual file name. For example ``@code{wrc=recompiled-wrc.bin}''.

@end table

The binaries suggested for this software release are available from
the @i{files} tab of the Open Hardware Repository. The direct
links are:

@example
http://www.ohwr.org/attachments/download/1596/wr_nic_dio.bin-2012-09-20
http://www.ohwr.org/attachments/download/1597/wr_nic_dio-wrc.bin-2012-09-20
@end example

The date is included in the binary name so we won't need to remove the
binaries: @i{spec-sw} releases are expected to continue working in the
future.  You can copy the following command sequence to your shell
in order to fill your @file{/lib/firmware/fmc} with everything that's
needed to run @i{wr-nic}:

@smallexample
cd /tmp
wget -O wr_nic_dio.bin \
   http://www.ohwr.org/attachments/download/1596/wr_nic_dio.bin-2012-09-20
wget -O wr_nic_dio-wrc.bin \
   http://www.ohwr.org/attachments/download/1597/wr_nic_dio-wrc.bin-2012-09-20
sudo mv wr_nic_dio wr_nic_dio-wrc /lib/firmware/fmc
@end smallexample

@c ==========================================================================
@node Controlling the White Rabbit Core
@section Controlling the White Rabbit Core

In this release the driver is not controlling the White Rabbit Core
and the default mode of operation is @i{free-running master}.  You can
use the serial port and interact with the WRC shell to change the
operation mode and do other supported interaction with the PTP daemon.

The complete reference of the shell commands is included in the
@i{wrpc-sw} manual in the @i{files} tab of the project.  The direct
link is
@url{http://www.ohwr.org/attachments/download/1586/wrpc-v2.0.pdf}.

The most useful ones are repeated here for your convenience

@table @code

@item mode grandmaster
@itemx mode master
@itemx mode slave

	The commands change the current PTP mode.

@item time raw

	Prints the internal device time as seconds and nanoseconds.

@item mac get

	Reports the MAC address used by WRPC (it should match what is
        reporte by @code{ifconfig} in Linux.

@end table

Please note that you may also need to configure the SFP module
you are using, with the @code{sfp} WRC command, as described in the
@code{wrc-v2.0.pdf} manual referenced above.

@c ==========================================================================
@node Transferring Data
@section Transferring Data

The @i{wr-nic} driver registers a Linux network interface card for
each SPEC device it drives.  The cards are called @code{wr%d} (i.e.,
@i{wr0}, @i{wr1}, ...).

The MAC address of the device is retrieved from the internal White
Rabbit registers, because at the time when Linux configures the
interface the WRC code has already configured the Ethernet port and
generated a MAC address using the serial number of the internal
thermometer.

The user is thus only expected to assign an Internet address to the
Ethernet port and just use it to transfer data. If you need to change
the MAC address, the command you need is something like the following:

@example
   ifconfig wr0 hw ether 12:34:56:78:9a:bc
@end example

The port supports hardware timestamping for user frames through the
standard Linux mechanisms, but at this point no sample code is provided
in this package.

@b{Warning}: time stamping is not working, but we are on the problem.

@c ==========================================================================
@node Accessing the DIO Channels
@section Accessing the DIO Channels

In order to access the DIO channels, user-space programs are expected
to issue device-specific @i{ioctl} commands.  The driver supports two
commands, allocated at the end of the range of command numbers reserved
for device-specific use:

@table @code

@item PRIV_MEZZANINE_ID

	The command is used to identify the features of the
        specific NIC device.  It tells user space which mezzanine is
        currently plugged and also which type of carrier is this.
        The command exchanges a data structure with user space in order
        to be able to extend its functionality over time, with a sub-command
        field. (For example, we may return EEPROM contents to user space
        on request).

        @b{Warning}: the command is not implemented because we still have
        no mezzanine identification in place

@item PRIV_MEZZANINE_CMD

	The command is based on the exchange of a data structure: by
        means of sub-commands included in such structure user space
        programs can request different services to the mezzanine driver.
        In the case of the DIO mezzanine this includes generating pulses
        and timestamping input events; other mezzanine drivers will
        be able to use the command in a different way. The application
        expected to first run @code{PRIV_MEZZANINE_ID} to ensure the
        NIC device is connected to the right mezzanine.

@end table

In the specific case of this @i{wr-nic} driver, the data structure
is defined in @code{wr-dio.h} and is not repeated here.

The structure includes a few integer fields and an array of
@code{struct timespec}.  Such structures define time stamps with
nanosecond precision, and the simple DIO mezzanine and its gateware
are able to time-stamp input events and generate output events with a
resolution of 8ns.

When the device is asked to timestamp input events, the array is used
to return such events to user space.  When the device is asked to
generate output pulses at specific points in time, the array is used
to pass three values: the beginning of the pulse, the duration of the
pulse and the period of the pulse train.  It is possible to configure
several outputs with a single command: in this case the structures are
used in groups of 5: structures 0..4 represent the starting time,
structures 5..9 represent the durations and structures 10..14
represent the periods. By passing a channel mask the application
selects which channels to act on; @code{timespec} items for
non-selected channels are ignored by the driver.  To stop output
generation, the application must request a starting time of 0.0
seconds for the channels it wants to stop.

@b{Warning}: currently the @i{pulse train} mode is not supported
by software.
      
Specifics about the use of individual fields is shown in the header
(in a big comment block), in the driver itself and in the user-space
programs that call @i{ioctl}.

In lab environments you may be concerned about the duration of the
@i{ioctl} implementation, because it sometimes seems to do more work
than needed. To verify whether we have an over-engineering problem in
kernel space, I provided a simple measurement of how much time is
spent in the @i{Ioctl} itself. The @i{make} variable
@code{WR_NIC_CFLAGS} can be used to pass extra flags to the compiler,
and the macro @code{DIO_STAT} enables the time measurement.
Compiling with the following command thus enable such measurement
and associated @i{printk} -- the time is usually 5 microseconds for me:

@example
   make WR_NIC_FLAGS=-DDIO_STAT
@end example

@c ==========================================================================
@node WR-NIC Command Tool
@section WR-NIC Command Tool

In the @file{tools/} subdirectory of this project, you find the
@file{wr-dio-cmd} program, which is a command-line interface to the
@i{ioctl} command to simplify initial access to the DIO device. Other
@code{wr-dio-} tools are provided (and described below) but this is
the most generic one.

Please note that neither timestamping nor pulse generation work
if the WR core is not running or has an invalid time: it must either be
a master or a synchronized slave.

Moreover, please note that this tool is just a demonstration to quickly
test the I/O features of the device (and for me to verify the kernel
part is actually working): for serious use you should call
@i{ioctl} by yourself with proper arguments, and avoid all the parsing
of ASCII and repeated invocation of this program.

This is the general syntax of the command:

@example
   wr-dio-cmd <ifname> <cmd> [<arg> ...]
@end example

The arguments have the following meaning

@table @code
@item ifname

	The name of the network interface, most likely @code{wr0}
        (if you have more than one SPEC card, the other interfaces are
        called @code{wr1}, @code{wr2} and so on).

@item cmd

	The specific command. Supported commands are listed below.
        Each command takes zero or more of arguments. If you pass
        a wrong number of arguments you'll get help, and
        if one argument is wrong (e.g., not a number) the
	error message is meant to be directly helpful.

@end table

The current version of the tool supports the following commands:

@table @code

@item stamp [<channel>] [wait]
@itemx stampm [<channel-mask>]

	The commands are used to retrieve timestamps from the card.
        If no arguments are passed, the tool reports to @i{stdout} all
        timestamps for all channels (they are ordered by channel, not
        by time). If one integer argument is passed, it can be a channel
        number in the range 0 to 4 (@code{stamp} command) or a mask
        in the range 0 to 0x1f (@code{stampm} command). If getting stamps
        for an individual channel, you can add the @code{wait} option
        to have the tool wait for (and report) new timestamps until killed.

@item pulse <channel> <duration> <when> [<period> <count>]

	Channel is an integer in the range 0 to 4. The duration must
        be specified as a fraction of a second (decimal number, less than
        one second), the @code{when} argument can be the string @code{now},
        an absolute time (@code{<seconds>.<fraction>}) or a relative
        time (@code{+<seconds>.<fraction}). In the last case, the
        current second is added to @code{<seconds>} while the fraction
        is not modified. The @code{+} form is useful for simple checks with
        visual inspection.  @code{period}, if specified, requests for
        a pulse train, with that separation from the raising edges;
        @code{count} is the number of instances to run (0 means forever).

@item mode <channel> <mode> [<channel> <mode> ...]
@itemx mode <mode0><mode1><mode2><mode3><mode4>

	Configure one or more channel for the specified mode. Each mode
        is one character, so the second form receives a 5-byte
        string argument. Available modes are ``@code{I}'' (input),
        ``@code{0}'' and ``@code{1}'' (output, steady state),
        ``@code{D}'' (DIO core output) or ``@code{-}'' (unchanged).
        A channel managed by the DIO core is normally low and can
        pulse high on request (see @code{pulse}
        command. Uppercase @code{D} or @code{I} select the termination
        resistor, while lowercase @code{d} or @code{i} disable the termination
        (output is always without termination, but I may add @code{L} and
        @code{H} if needed: the driver supports all combinations).
        Please note that the @code{pulse} command turns the affected channel in
        @i{DIO} mode anyways (without changing the termination).

@end table

Please note that generation of a pulse train is performed in kernel space,
because the @i{simple DIO} card and gateware can only emit one pulse at
a requested @i{White Rabbit} time. For this reason you'll have a lower
limit for the pulse duration, according to how powerful your computer
is -- and how much you loaded with other tasks.

There is no command to flush the timestamp FIFOs, but you can
always ``@code{wr-dio-cmd stamp > /dev/null}'' if needed.

Example uses of the tool follow:

@example
   # Pulse channel 4 for 0.1 seconds now
   wr-dio-cmd wr0 pulse 4 .1 now

   # Pulse for 10 microseconds in the middle of the next second
   wr-dio-cmd wr0 pulse 4 .00001 +1.5

   # Pulse for 1ms at 17:00 today
   wr-dio-cmd wr0 pulse 4 .001 $(date +%s --date 17:00)

   # Get timestamps for the output events above
   wr-dio-cmd wr0 stamp 4

   # Make a train of 5 pulses, 0.5ms wide, every ms at next second
   wr-dio-cmd wr0 stamp 4 0.0005 +1 .001 5

   # Configure channel 0 as input with termination, 1 as input, 4 as low
   wr-dio-cmd wr0 mode Ii--0
@end example

@c ==========================================================================
@node WR-NIC Pulse per Second
@section WR-NIC Pulse per Second

To better show how to write your own application with the SPEC driver
and the @i{simple DIO} mezzanine card, this package includes
@file{wr-dio-pps}, which features a very small and readable source
file.

The program just fires a 1ms-long @i{pps} pulse on one of the output
channels. The NIC defaults to @i{wr0} but can specify a different one;
the channel number is a mandatory argument.

@example
   # run pps on channel 2 of the default SPEC card
   ./wr-nic-pps 2

   # run pps on channel 0 of the "second" card
   ./wr-nic-pps wr1 0
@end example

The following figure shows two such @i{pulse-per-second} signals
retrieved from two different @i{simple-DIO} cards, connected by a 10km
roll of fiber, after syncing the two @i{White Rabbit} on-board systems
(one as master and the other as slave).  Increasing the horizontal
scale of the scope, found the leading edge of the PPS signal differed
by exactly 8ns, because of different delay figures in the two pairs of
boards (the SPEC and the @i{simple-DIO}).  A more complete
experimental setup would include calibration of the internal delays of
the boards, like the mechanism in place for the @i{fine-delay}
mezzanine card (see
@url{http://www.ohwr.org/projects/fmc-delay-1ns-8cha} and
@url{http://www.ohwr.org/projects/fine-delay-sw}).

@sp 1
@center @image{two-pps, 10cm}
@sp 1


@c ==========================================================================
@node Distributing Output Pulses
@section Distributing Output Pulses

A typical application for @i{White Rabbit} (or any time
synchronization system) is being able to generate output signals at the
same time in different output boards; another typical application is
time stamping input events.

By using the Ethernet interface included in the SPEC,
an application can exchange data with other @i{White
Rabbit} devices; thus, it can easily request output event to
other output peripherals, or collect remote input events.
The tool-set offered by the driver
is made up of the the @code{PRIV_MEZZANINE_CMD} @i{ioctl} command,
amd the usual Posix API for network communication.

The @i{DIO-specific} @i{ioctl} command is the one used by the
@code{wr-dio-cmd} tool described above, while network communication
should be known to most users of this package.  In order to ease new
@i{White Rabbit} users, though, this package includes sample code to
implement a simple dual-headed system with concurrent output.  The
examples are also meant to show the basic code that uses the provided
@i{ioctl} command, without all the boring parameter parsing that is
required in more generic tools like @code{wr-dio-cmd}. For this reason,
the pulse width is hardwired to 1ms.

The example is made up of two programs: @code{wr-dio-agent} and
@code{wr-dio-ruler} (the former is a dumb actor, while the latter
states the rules). To keep things simple the two programs
assume that the SPEC is connected point-to-point to another SPEC
and both carry the @i{simple-DIO} mezzanine.

Under this simplified assumption, the @i{ruler} transmits RAW Ethernet
frames to the broadcast address, while the @i{agent} receives almost
everything that appears the cable.  This also allows plugging two SPEC
cards in the same computer and running the example, whereas adding IP
addresses and using UDP would prevent a setup with a single PC to work
through the fiber.  The simplification above, however, most likely
prevents this pair of demo programs from working within a more complex
network topology.  I expect real @i{White Rabbit} users to add proper
network addressing in the real applications.

If you have a single SPEC card, you can still use the @i{ruler} by
itself to mirror an input channel to an output channel of the same
card, with a specified delay.

@sp 1

The @i{agent} program silently listens to the network interface and
receives a data structure ready to be passed to @i{ioctl}.  Its only
command line argument is the name of the @i{White Rabbit} interface to
use (for most users it is @code{wr0}):

@example
   wr-dio-agent wr0
@end example

The @i{ruler} command, on the other hand, waits for timestamps
to appear on its own input channel; when a positive-going edge is detected it
replicates the edge on one or more outputs. Each output can be
local or remote, and can have a different delay.

If you lack an input signal, you can make an output pulse with
@code{wr-dio-cmd} or other means and use it as a trigger. Please note
that the @i{ruler} does not configure the channel mode, so you might
want to use the @code{mode} command of @code{wr-dio-cmd} in advance.

The following command waits for input channel 0 on the card connected
to @i{wr1}, and replicates the evebt with a delay of 1ms on channel 3 of
both the local and the remote card; it also replicates with a 2ms
delay to local channel 4.

@example
   wr-dio-ruler wr1 IN0 L3+0.001 R3+0.001 L4+0.002
@end example

There is no sample code that generates trains of pulses at this time, nor
other options than generating 1ms-long pulses, but the code is
thoroughly commented in order to serve as a starting point for
more complex lab environments.

As a final remark, please note that all pulse generation is driven by
host software, after an hardware interrupt reports the input event.
For this reason, you'll not be able to reliably replicate pulses with
delays smaller than a few hundred microseconds, depending on the
processing power of your computer and the load introduced by other
processes.  For remote connections, you must also count the overhead
of network communication as well as transmission delays over
the fiber (a 10km fiber introduces a
delay of 50 microseconds). 

The following example shows use of the @i{ruler} and @i{agent} on
two hosts, called @code{spusa} and @code{tornado}. The input events
on @code{spusa} are replicated to one local channel and two remote channels,
with a delay of 1ms. The input events in this case are from a @i{pulse-per-second} signal:

@smallexample
   tornado.root# /tmp/wr-dio-agent wr0 &

   spusa.root# wr-dio-ruler wr1 IN4 L3+.001 R4+.001 R2+.001
   wr-dio-ruler: configured for  local channel 3, delay 0.001000000
   wr-dio-ruler: configured for remote channel 4, delay 0.001000000
   wr-dio-ruler: configured for remote channel 2, delay 0.001000000

   [... wait a few seconds ...]

   spusa.root# wr-dio-cmd wr1 stamp 3
   ch 3,       385.001000000
   ch 3,       386.001000000
   ch 3,       387.001000000
   ch 3,       388.001000000
   tornado.root# wr-dio-cmd wr0 stamp 2
   ch 2,       385.001000000
   ch 2,       386.001000000
   ch 2,       387.001000000
   ch 2,       388.001000000