Skip to content
Snippets Groups Projects
Commit ea1e3c65 authored by Alessandro Rubini's avatar Alessandro Rubini
Browse files

doc: spec-sw now documents wr-nic 


Signed-off-by: default avatarAlessandro Rubini <rubini@gnudd.com>
parent 1d7052f7
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 463 additions and 16 deletions
doc/spec-sw.in
+ 463
− 16
View file @ ea1e3c65
......@@ -34,7 +34,7 @@
@setchapternewpage off
@set update-month August 2012
@set update-month September 2012
@finalout
......@@ -164,7 +164,9 @@ As I write this, the direct link is
binary file when the sub-driver releases control of the card. This
allows a further driver to make use of an existing binary, and may be
useful during development. The EEPROM contents are still being
available to the new sub-driver (even if it cannot change it any more).
available to the new sub-driver (even if it cannot write to it, because
the carrier refused to act on registers after the golden gateware is replaced
by an new binary unknown to the carrier itself).
@c ==========================================================================
@node SPEC Module Parameters
......@@ -256,7 +258,7 @@ These parameters are offered:
For example, if you are using the trivial driver to load two different
gateware files to two different cards, you can use the following
parameters to load differnt binaries to the cards, after looking up
parameters to load different binaries to the cards, after looking up
the PCI identifiers:
@smallexample
......@@ -345,7 +347,7 @@ of the driver will fail).
Each TLV tuple is formatted in this way: the header is 5 bytes,
followed by data. The first byte is @code{w} for @i{write}, the
next two bytes represent the address, in little-endian byte order, and
the next two the datalength, in lttle-endian format. The length does
the next two represent the data length, in little-endian order. The length does
not include the header (it is the actual number of bytes to be
written).
......@@ -360,26 +362,436 @@ This is a real example: that writes 5 bytes at position 0x110:
[19983.414615] spec 0000:03:00.0: write_eeprom: success
@end smallexample
Please note that you'll most likely want to use SDBFS to build your
EEPROM image, at least if your mezzanines are being used in the White Rabbit
environment. For this reason the TLV format is not expected to be used much
and is not expected to be developed further.
@c ##########################################################################
@node The WR-NIC
@chapter The WR-NIC
@c FIXME
The @i{wr-nic} driver is being worked on. It should be a model of how
FMC drivers should behave, so it is hosted in this package.
With the current code base, the @code{wr-nic.ko} driver is designed
to run with the simple 5-Channel DIO board, because it includes both
code to access the network card and code to act on the DIO channels,
using time tags that are provided by the White Rabbit mechanism.
Similarly, both incoming and outgoing frames can be time-stamped by
White Rabbit.
This driver is the most important driver in this package: it is
a generic implementation of the @i{spec-sw} framework which can be
useful by itself as a White Rabbit starter kit. Moreover, it is a
complete driver that can serve as a model for other developments.
Within @i{White Rabbit} we have other full-featured drivers for
specialized FMC mezzanines hosted on the SPEC carrier. They are not
part of this package because of their specialized nature; all of them
are nonetheless hosted on @url{www.ohwr.org}, usually as a
@i{software} subproject of the related gateware project.
@c ==========================================================================
@node Code Layout
@section Code Layout
This section is mainly for the developers who look in the code, and for
me to make order in my own mind. SPEC users are expected to skip to
the next section.
The @code{wr-nic.ko} is built using a number of headers and source files,
spread over several directories:
@table @file
@item wbgen-regs/
The directory hosts the register definitions for the various
core that are included in the FPGA binary. The headers are
generated by running @i{wbgen2} over the @code{.wb} files that
are part of the VHDL source repositories; unfortunately some
minor editing is needed on the @i{wbgen2} output, so there
is a @code{Makefile} that takes care of this. This package
includes both the input file and the output header; the log
messages details the upstream origin of each @code{.wb} file.
The directory started out as a direct copy of the directory
with the same name found in @code{wr-switch-sw}, release 3.0.
@item wr_nic/
The directory started out as an unchanged copy of the driver
used in the @code{wr-switch-sw} package, release 3.0. The directory
name is the same in both projects. All later
commits take care of differences in the SPEC with regard to the
switch, but we plan to clean up those later changes and reach
a unified code base between the White Rabbit switch and the White
Rabbit node. The NIC driver itself is a @i{platform driver},
instantiated by @i{platform devices} defined externally.
@end table
The @i{wr-nic} driver refers to several headers, in addition to the
register definitions. This is the role of each of them:
@table @file
@item include/linux/fmc.h
@itemx include/linux/sdb.h
@itemx include/linux/fmc-sdb.h
These three headers are used to define the interface to the
FMC bus abstraction and the SDB self-description of the
internal FPGA bus. They are used by other @i{spec-sw} files as well.
We include them as @code{<linux/fmc.h>} because we plan to
have them upstreamed to the official kernel, and we don't want to
introduce incompatibilities in the related source files.
@item wr_nic/nic-mem.h
@itemx wr_nic/nic-hardware.h
@itemx wr_nic/wr-nic.h
These headers come from @i{wr-switch-sw/wr_nic} with minor spec-
related modifications. @i{nic-mem.h} defines the memory map and
is now almost obsoleted by SDB; @i{nic-hardware.h} is a collection
of inline functions used by the driver; @i{wr-nic.h} defines
all the important data structures and @i{ioctl} commands. Because
of @i{ioctl} commands, it has a rather generic name and
is meant to be included by user space as well as kernel space.
@item spec.h
Definitions related to the SPEC carrier (Gennum registers and
other SPEC-internal stuff). It is currently used by
@i{wr-nic-eth.c}, which is not completely carrier-independent.
@item spec-nic.h
The header defines the SDB vendor and device values used in
@i{wr-nic} as well as the data structures and prototypes used
internally by the driver.
@item wr-dio.h
The header hosts the user interface to access the DIO channels.
It is included by @i{wr-nic-dio.c} as well as the user-space
tools that want to configure DIO operation.
@end table
The source code of the driver itself is split in several files, in addition
to the NIC platform driver hosted in @file{wr_nic/}:
@table @file
@item wr-nic-core.c
The file is the @i{fmc} driver: it implements the @i{probe}
and @i{remove} methods and deals with loading the firmware
file and the LM32 program binary (called @i{wrc}: White Rabbit
Core).
@item wr-nic-eth.c
This is concerned with creating the platform device for the
network interface card. It maps the needed device memory, allocates
the platform data and sets up the internal interrupt controller
to route interrupts to the platform driver.
@item wr-nic-dio.c
This is the mezzanine-specific driver. It implements the
@i{ioctl} commands that allow user space to talk with
the mezzanine. It only implements the @i{ioctl} method
and support functions for it (e.g., interrupt management).
If you want to port @i{wr-nic} to a different mezzanine, this
is the file you need to replace.
@end table
@c ==========================================================================
@node Overview of the Driver
@section Overview of the Driver
The @i{wr-nic} driver is basically an Ethernet driver with support for
hardware time stamping. The DIO mezzanine card can be used by means
of @i{ioctl} commands. Such commands are designed to be portable, so
user-space programs should able to identify which mezzanine is
connected to the SPEC network card and act accordingly.
The driver loads two binaries, using the @i{firmware loader} mechanism
offered by the Linux kernel: one is the gateware file, that is requested
through the @i{reprogram} carrier method; the other is the LM32 program
binary, which is only loaded on user request.
The default file names are as follows:
@table @file
@item fmc/wr_nic_dio.bin
This is the @i{gateware} file. The default name can be changed
using the @code{file=} module parameter.
@item fmc/wr_nic_dio-wrc.bin
This is the LM32 program file, or @i{White Rabbit Core}, WRC.
The file is not loaded automatically, because we expect to deliver
a gateware file that already includes the correct LM32 program
(but the binary currently suggested does not include it).
To request loading the file you should pass @code{wrc=1}. To
request loading a different WRC file name, you should pass
the actual file name. For example ``@code{wrc=recompiled-wrc.bin}''.
@end table
The binaries suggested for this software release are available from
the @i{files} tab of the Open Hardware Repository. The direct
links are:
@example
http://www.ohwr.org/attachments/download/1584/wr_nic_dio.bin-2012-09-13
http://www.ohwr.org/attachments/download/1585/wr_nic_dio-wrc.bin-2012-09-13
@end example
The date is included in the binary name so we won't need to remove the
binaries: @i{spec-sw} releases are expected to continue working in the
future. You can copy the following command sequence to your shell
in order to fill your @file{/lib/firmware/fmc} with everything that's
needed to run @i{wr-nic}:
@smallexample
cd /tmp
wget -O wr_nic_dio.bin \
http://www.ohwr.org/attachments/download/1584/wr_nic_dio.bin-2012-09-13
wget -O wr_nic_dio-wrc.bin \
http://www.ohwr.org/attachments/download/1585/wr_nic_dio-wrc.bin-2012-09-13
sudo mv wr_nic_dio wr_nic_dio-wrc /lib/firmware/fmc
@end smallexample
@c ==========================================================================
@node Controlling the White Rabbit Core
@section Controlling the White Rabbit Core
In this release the driver is not controlling the White Rabbit Core
and the default mode of operation is @i{free-running master}. You can
use the serial port and interact with the WRC shell to change the
operation mode and do other supported interaction with the PTP daemon.
The complete reference of the shell commands is included in the
@i{wrpc-sw} manual in the @i{files} tab of the project. The direct
link is
@url{http://www.ohwr.org/attachments/download/1586/wrpc-v2.0.pdf}.
The most useful ones are repeated here for your convenience
@table @code
@item mode grandmaster
@itemx mode master
@itemx mode slave
The commands change the current PTP mode.
@item time raw
Prints the internal device time as seconds and nanoseconds.
@item mac get
Reports the MAC address used by WRPC (it should match what is
reporte by @code{ifconfig} in Linux.
@end table
Please note that you may also need to configure the SFP module
you are using, with the @code{sfp} WRC command, as described in the
@code{wrc-v2.0.pdf} manual referenced above.
@c ==========================================================================
@node Transferring Data
@section Transferring Data
The @i{wr-nic} driver registers a Linux network interface card for
each SPEC device it drives. The cards are called @code{wr%d} (i.e.,
@i{wr0}, @i{wr1}, ...).
The MAC address of the device is retrieved from the internal White
Rabbit registers, because at the time when Linux configures the
interface the WRC code has already configured the Ethernet port and
generated a MAC address using the serial number of the internal
thermometer.
The user is thus only expected to assign an Internet address to the
Ethernet port and just use it to transfer data.
The port supports hardware timestamping for user frames through the
standard Linux mechanisms, but at this point no sample code is provided
in this package.
@b{Warning:} While frame transmission is perfectly working, there is
still a problem in the RX data path and no data can be received with
this version of the software and gateware.
@c ==========================================================================
@node Accessing the DIO Channels
@section Accessing the DIO Channels
In order to access the DIO channels, user-space programs are expected
to issue device-specific @i{ioctl} commands. The driver supports two
commands, allocated at the end of the range of command numbers reserved
for device-specific use:
@table @code
@item PRIV_MEZZANINE_ID
The command is used to identify the features of the
specific NIC device. It tells user space which mezzanine is
currently plugged and also which type of carrier is this.
The command exchanges a data structure with user space in order
to be able to extend its functionality over time, with a sub-command
field. (For example, we may return EEPROM contents to user space
on request).
@b{Warning}: the command is not implemented because we still have
no mezzanine identification in place
@item PRIV_MEZZANINE_CMD
The command is based on the exchange of a data structure: by
means of sub-commands included in such structure user space
programs can request different services to the mezzanine driver.
In the case of the DIO mezzanine this includes generating pulses
and timestamping input events; other mezzanine drivers will
be able to use the command in a different way. The application
expected to first run @code{PRIV_MEZZANINE_ID} to ensure the
NIC device is connected to the right mezzanine.
@end table
In the specific case of this @i{wr-nic} driver, the data structure
is defined in @code{wr-dio.h} and is not repeated here.
The structure includes a few integer fields and an array of
@code{struct timespec}. Such structures define time stamps with
nanosecond precision, and the simple DIO mezzanine and its gateware
are able to time-stamp input events and generate output events with a
resolution of 8ns.
When the device is asked to timestamp input events, the array is used
to return such events to user space. When the device is asked to
generate output pulses at specific points in time, the array is used
to pass three values: the beginning of the pulse, the duration of the
pulse and the period of the pulse train. It is possible to configure
several outputs with a single command: in this case the structures are
used in groups of 5: structures 0..4 represent the starting time,
structures 5..9 represent the durations and structures 10..14
represent the periods. By passing a channel mask the application
selects which channels to act on; @code{timespec} items for
non-selected channels are ignored by the driver. To stop output
generation, the application must request a starting time of 0.0
seconds for the channels it wants to stop.
@b{Warning}: there is a problem with pulse output at a specified time,
only immediate pulse is working. The bug is in the FPGA binary and is
being worked on.
@b{Warning}: currently the @i{pulse train} mode is not supported
by software.
Specifics about the use of individual fields is shown in the driver itself
and in the user-space programs that call the command.
@c ==========================================================================
@node WR-NIC User Tool
@section WR-NIC User Tool
In the @file{tools/} subdirectory of this project, you find the
@file{wr-dio-cmd} program, which is a command-line interface to the
@i{ioctl} command to simplify initial access to the DIO device.
Please note that neither timestamping nor pulse generation works
if the WR core is running and has valid times: it must either be
a master or a slave in synchronized state.
This is the general syntax of the command:
@example
wr-dio-cmd <ifname> <cmd> [<args> ...]
@end example
@c FIXME: wr-dio-cmd <ifname> <cmd> [<args> ...] [<cmd> [<args> ...]]
The arguments have the following meaning
@table @code
@item ifname
The name of the network interface, most likely @code{wr0}
(if you have more than one SPEC card, the other interfaces are
called @code{wr1}, @code{wr2} and so on).
The driver will register an Network Interface Card on the host computer,
to allow exchanging data on the same fiber channel where White-Rabbit
synchronization is taking place. The driver relies on the 5-channel
digital I/O mezzanine to time-stamp incoming pulses and generate
precisely timed output signals.
@item cmd
The specific command. Currently the tool supports @code{stamp}
and @code{pulse}.
@end table
@c FIXME: argument to stamp
The @code{stamp} command currently takes no arguments. It reports
to stdout all the available timestamps, scanning the FIFO for all
channels in order. Later I'll add a <i>channel</i> argument to restrict
the report to only one channel (the driver already supports this).
The @code{pulse} command receives three arguments:
@example
wr-dio-cmd wr0 pulse <channel> <fraction> <start>
@end example
The @code{channel} argument is a number, in the range 0-4. The
@code{fraction} argument is a decimal number, for example @code{.1}
representing the length in second of the pulse; the length is
truncated to a multiple of 8ns. The @code{start} argument is
either the string @code{now}, or an absolute number of seconds, or
a relative number of seconds. A relative time is expressed as
@code{+@i{nr}} (for example, @code{+3}) and requests the pulse
at the start of a second, @i{nr} seconds in the future.
@c FIXME: support fractional starting times
@b{Warning}: due to problems in the current gateware, only immediate
pulses are actually generated by hardware.
@c ==========================================================================
@node The Future of WR-NIC
@section The Future of WR-NIC
The @i{grand plan} for this driver is to detach the NIC functionality
from the mezzanine driver.
Some future version of this package will thus feature a different lay
out of code, and the NIC will be a mezzanine-independent feature that
may be activated on request of the mezzanine driver -- it's the
mezzanine driver that requests its own gateware to be loaded, and only
that driver can know whether or not NIC functionality is part of its
feature set.
In any case, no change to user-space access is expected, because the
current way to handle mezzanine-specific @i{ioctl} commands is already
portable to the new code arrangement.
@c ##########################################################################
@node User-Space Tools
@chapter User-Space Tools
The @i{tools} subdirectory of this package includes a few host-side
programs that may be useful while working with the SPEC device.
programs that may be useful while working with the SPEC device. This
section does not describe the @i{wr-nic} specific tool; for that see
@ref{WR-NIC User Tool}.
@b{Warning:} the tools need some more audit and cleanup. For example,
the tools may fail with a segmentation fault if invoked as a
non-privileged user.
They are all base on the same @i{speclib}, part of the same directory,
so all of them accept some parameters in common, in order to identify
one specific SPEC card if you have more than one:
......@@ -432,7 +844,7 @@ The tools currently available are:
@item spec-fwloader
This is a user-space loader for the gateware file. It simply
receives a file name argument (after the optinal bus number for
receives a file name argument (after the optional bus number for
the device).
@item spec-vuart
......@@ -443,10 +855,45 @@ The tools currently available are:
@end table
@c ##########################################################################
@node Bugs and Missing Features
@chapter Bugs and Missing Features
@itemize @bullet
@item Identification of the mezzanine is completely missing; every @i{fmc}
driver at this point takes hold of every device.
@item Both spec and wr-nic should have GPIO support with @i{gpiolib};
there is skeletal support but no real code for actual I/O.
@item The NIC data transfer only works in the transmit direction. This looks
like a bug in the gateware and we are investigating on both sides.
@item The NIC driver should directly support setting the White Rabbit
mode for each card (grandmaster, free-running master or slave).
@item DIO support in @i{wr-nic} is missing some of the features listed
in @file{wr-dio.h}.
@item DIO support should use interrupts to allow pulse trains to be generated
and timestamps to be collected without polling the FIFO.
@item The @i{wr-nic} functionality should be completely detached from
the specific mezzanine.
@item The package should be verified with a wide range of kernel versions.
Currently I compiled and tested only under Linux-3.4.
@item A number of temporary @i{printk} should be removed.
@end itemize
@c ##########################################################################
@bye
@c LocalWords: gnudd titlepage iftex texinfo CERN documentlanguage settitle
@c LocalWords: documentencoding setfilename afourpaper paragraphindent
@c LocalWords: setchapternewpage finalout
@c LocalWords: documentencoding setfilename afourpaper paragraphindent EEPROM
@c LocalWords: setchapternewpage finalout eeprom gateware devmem devfn busid
@c LocalWords: speclib Gennum
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment