\input texinfo    @c -*-texinfo-*-
%
% gnurabbit.in - main file for the documentation
%
% Copyright (C) 2010 CERN (www.cern.ch)
% Author: Alessandro Rubini <rubini@gnudd.com>
%
% Released according to the GNU GPL, version 2 or any later version.
%
% This work is part of the White Rabbit project, a research effort led
% by CERN, the European Institute for Nuclear Research.
%%%%

%------------------------------------------------------------------------------
%
%                         NOTE FOR THE UNAWARE USER
%                         =========================
%
%    This file is a texinfo source. It isn't the binary file of some strange
%    editor of mine. If you want ASCII, you should "make gnurabbit.txt".
%
%------------------------------------------------------------------------------

%
% This is not a conventional info file...
% I use three extra features:
%   - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
%   - leading blanks are allowed (this is something I can't live without)
%   - braces are automatically escaped when they appear in example blocks
%

@comment %**start of header
@documentlanguage en
@documentencoding ISO-8859-1
@setfilename gnurabbit.info
@settitle Gnu Rabbit
@iftex
@afourpaper
@end iftex
@paragraphindent none
@comment %**end of header

@setchapternewpage off

@set update-month January 2011

@finalout

@titlepage
@title Gnu Rabbit
@subtitle Code for GENNUM and White Rabbit
@subtitle @value{update-month}

@author Alessandro Rubini (@code{rubini@@gnudd.com})
@author Work sponsored by CERN (@code{www.cern.ch})
@end titlepage
@setchapternewpage off
@headings single

@c ##########################################################################
@node Top, Raw PCI I/O, (dir), (dir)
@top Introduction

This package is basically a
module for raw PCI I/O, used at CERN as development tool to help
prototyping White Rabbit hardware and software.

The main target of this code is the GN4124 PCI-E evaluation board, but
there is little stuff here that is specific to the gennum chip -- mainly
it's the firmware loader.

All code and documentation is released according to the GNU GPL, version
2 or (at your option) any later version.

@menu
* Raw PCI I/O::                 
* Firmware Loader::             
@end menu

@c ##########################################################################
@node Raw PCI I/O, Firmware Loader, Top, Top
@chapter Raw PCI I/O

The kernel module for raw I/O is called @i{rawrabbit}. After
running @i{make} you'll find a file called @code{rawrabbit.ko}.
I'm sorry for this mismatch in naming, but initially the package was
expected to host another module in addition to @i{rawrabbit}, something
Gennum-specific like @i{gnuuser} for ``GN user''.

To compile you may optionally set the following three variables in
your environment:

@table @code
@item CROSS_COMPILE
	The variable defaults to the empty string, used for native compilation
@item ARCH
	The variable defaults to the build architecture
@item LINUX
	This is the location of the kernel source against which you
        are compiling. It defaults to the place where the currently running
        kernel has been compiled (assuming it was compiled on this same
        system).
@end table

The code has been run-tested on version 2.6.34 and 2.6.35.
It has also been compile-tested on
2.6.24.7-rt27, 2.6.29.4-rt15, 2.6.31.6-rt19, 2.6.31.12-rt21.

The module creates a @i{misc} char device driver, with major number 10
and minor number 42.  If you are running @i{udev} the special file
@code{/dev/rawrabbit} will be created automatically.

@b{Warning:} future releases of this package may change the device
number or switch to several devices, I'm yet undecided on this choice.

@menu
* General features of rawrabbit::  
* Interrupt management::        
* Bugs and misfeatures::        
* The DMA buffer::              
* System calls implemented::    
* Ioctl commands::              
* User space demo programs::    
* User space benchmarks::       
@end menu

@c ==========================================================================
@node General features of rawrabbit, Interrupt management, Raw PCI I/O, Raw PCI I/O
@section General features of rawrabbit

The driver is designed to act as a misc device (i.e. a char device)
towards user programs and as a PCI driver towards hardware, declaring
the pair @i{vendor}/@i{device} it is able to drive.

The pair of identifiers is predefined at compile time but can be changed
at run time. The defaults (set forth) in @code{rawrabbit.h} refer to
the GN4124 evaluation board.  The values can also be changed at module
load time by setting the @code{vendor} and @code{device} arguments.
For example the following command
sets @i{rawrabbit} to look for a 3com Ethernet device:

@example
    insmod rawrabbit.ko vendor=0x10b7 device=0x9055
@end example

When the driver is loaded it registers as a PCI driver for the
preselected vendor/device pair, but loading succeeds even if no
matching peripheral exists on the system, as user space programs can
request a different vendor/device pair at runtime.  Since a single bus
might host several instances of the same peripheral, user space
programs can also specify the @i{bus} and @i{devfn} values in order to
select a specific instance of the hardware device. Similarly, the pair
@i{subvendor}/@i{subdevice} may be specified.

@c mmap
User programs can use @i{read} and @i{write}, @i{mmap} and @i{ioctl}
as described later (@b{Note:} mmap is not currently
supported).  Each and every command refers to the device
currently selected by means of the vendor/device pair as well as
bus/devfn and/or subvendor/subdevice if specified.

The driver allows access to the PCI memory regions for generic I/O
operations, as well as some limited interrupt management in user space.
Programs can also access a DMA buffer, for which they can know the
physical address on a page-by-page basis.

In the source file, each global function or variable declared in the
file itself or in the associated header
has @code{rr_} as prefix in the name, even if its scope is
static. Local variables have simple names with no prefix, like @code{i}
or @code{dev}.  This convention is followed so when reading
random parts of the source you can immediately know whether the symbol
is defined in the same file (like @code{rr_dev}) or is an external Linux
resource (like @code{pci_driver}).

@c ==========================================================================
@node Interrupt management, Bugs and misfeatures, General features of rawrabbit, Raw PCI I/O
@section Interrupt management

The driver is able to handle the interrupt for the active device. User
space is allowed to wait for an interrupt and acknowledge it later at
will.  To allow this latency, the driver disables the interrupt as soon as
it is reported, so user-space can do the board-specific I/O before asking
to re-enable the interrupt.

The interrupt handler is registered as a shared handler, as most PCI
cards must share the interrupt request line with other peripherals. In
particular, on my development motherboard both the PCI-E and the PCI
slot share the interrupt with other core peripherals and I couldn't test
stuff if I didn't enable sharing.

Unfortunately, the @i{rawrabbit} interrupt handler
can't know if the interrupt source is its
own board or another peripherals, so all it can do is saying it
handled to the interrupt (by returning @code{IRQ_HANDLED}) and disable
it.  If you are running other peripherals on the same interrupt line,
you'll need to acknowledge the interrupt pretty often, to avoid a
system lock or data loss in your storage or network device.

@c ==========================================================================
@node Bugs and misfeatures, The DMA buffer, Interrupt management, Raw PCI I/O
@section Bugs and misfeatures

@c FIXME: single open
This version of @i{rawrabbit} creates a single device and can act on a
single PCI peripheral at a time. This limitation will be removed in
later versions, as time permits.

The @i{read} and @i{write} implementations don't enforce
general data-size constraints: reading or writing 1, 2, 4, 8 bytes at a
time forces 8, 16, 32, 64 bit accesses respectively, while bigger
transfers use unpredictable access patterns to I/O memory, as
the driver uses @i{copy_from_user} and @i{copy_to_user}.

@c FIXME: interrupt is always requested
The interrupt line is always requested and handled (by disabling it).
This means that if
the line is shared with other devices, you can't avoid it being disabled
thus breaking the other devices.  A specific ioctl to request/release the
handler is needed.

@c FIXME: odd bars
The driver assumes to work with PCI-E so odd BAR areas are not supported.
This limitation may be lifted in future versions if needed.

@c FIXME: cache effects
@b{Important:} please note that there may be bugs related to cache
memory.  When using DMA you may encounter incorrect data due to
missing flush or invalidate instructions.  If the problem is real
please report the bug to me, with as much information as possible
about the inconsistency, and I'll do my best to find the proper
solution.  One solution might be adding two @i{ioctl} commands: one to
flush the buffer after it has been written and one to invalidate it
before reading; however better solutions, with no API changes, may be
viable.  Or the problem may just not appear as things are already
correct, I can't tell for sure.

@c ==========================================================================
@node The DMA buffer, System calls implemented, Bugs and misfeatures, Raw PCI I/O
@section The DMA buffer

At module load time, a 1MB buffer is allocated. The actual size
can be changed by means of a module parameter, but it currently
can't be bigger than 4MB. 

The buffer is allocated with @i{vmalloc}, so it is contiguous in
virtual space but not in physical space.  User space can read and
write the buffer like it was BAR 12 (0xc) of the device, using
contiguous offsets from 0 up to the buffer size.

In order to DMA data to/from the buffer, the peripheral device must be
told the physical address to use.  Since allocation is page-grained,
you need a different physical address for each 4kB page of data.  The
driver can thus return the list of @i{page frame numbers} that make up
the @i{vmalloc} buffer.  A PFN is a 32-bit number that identifies the
position of the page in physical memory. With 4kB pages, you can shift
by 12 bits to have the physical address, and a 32-bit PFN can span up
to 44 bits of physical address space.

The details about how PFNs are returned to user space are described later
where the @i{ioctl} commands are discussed.  A working example is in the
@i{rrcmd} user space tool.

Unlikely what happens with I/O memory, reading and writing the DMA
buffer uses the @i{copy_*_user} functions for all accesses, so the
pattern of actual access to memory can't be controlled, but this is
not a problem for RAM (as opposed to registers).

@c ==========================================================================
@node System calls implemented, Ioctl commands, The DMA buffer, Raw PCI I/O
@section System calls implemented

The following system calls are implemented in @i{rawrabbit}:

@table @i

@item open
@itemx close
	These system calls are used to keep a reference count of device use.
        If the device has been opened more than once, it will refuse
        to change the active device, to prevent possible confusion in
        another process using @i{rawrabbit} at the same time. Please note
        that after @i{fork} the device is still opened twice but the
        driver can't know about it, so in this case changing the active
        device is allowed, but it can be confusing nonetheless.

@item llseek
	The @i{seek} family of system calls is implemented using the
        default kernel implementation. A process may seek the device to
        access specific registers in specific BAR areas, or the DMA
        buffer. The offset
        being used selects the BAR and the offset within the BAR
        at the same time. Each BAR is limited to an extension of 512MB:
        so BAR0 starts at 0, BAR 2 starts at 0x2000.0000 and BAR 4 starts
        at offset 0x4000.0000; if you prefer symbolic names,
        @code{RR_BAR_0}, @code{RR_BAR_2} and @code{RR_BAR_4}
        are defined in @code{rawrabbit.h}. The DMA buffer is accessed
        like it was BAR 12 (@code{RR_BAR_BUF}), so @code{0xc} or @code{c}
        can be used in @i{rrcmd} (see @ref{rrcmd}).

@item read
@itemx write
	By reading and writing the device, a process can access on-board
        I/O space.  The file position (set through @i{llseek} or by
        sequential access of file data) is used to specify both the BAR
        and the offset within the BAR as described above. Access to
        an inexistent BAR returns @code{EINVAL}, access outside the BAR
        size returns @code{EIO}.
        If the hardware device offers I/O ports (instead of I/O memory), the
        system calls are not supported and you must use @i{ioctl} -- @i{read}
        and @i{write} will return @code{EINVAL} like the BAR was not
        existent.

	As a special case, reading past the DMA buffer size returns 0 (EOF),
        and writing returns @code{ENOSPC}, since the DMA buffer is a memory
        region and a file-like interface is best suited for command-line tools
        like @code{dd}.

@item mmap
	@b{Warning:} mmap is not yet implemented in this version.

	The @i{mmap} system call allows direct user-space access to the
        I/O memory. The device offset has the same meaning as for @i{read},
        but accesses to undefined pages cause a @code{SIGBUS} to be sent.
        If the device offers I/O ports (instead of I/O memory), the
        @i{mmap} method can't be used on such BAR areas.

@item ioctl
	A number of @i{ioctl} commands are supported, they are listed
        in the next section. Note that the commands to read and write
        can act both on memory and ``I/O ports'' areas.

@end table        

@c ==========================================================================
@node Ioctl commands, User space demo programs, System calls implemented, Raw PCI I/O
@section Ioctl commands

The following @i{ioctl} commands are currently implemented. The type
of the third argument is shown in parentheses after each command:

@table @code

@item RR_DEVSEL (struct rr_devsel *)

	The command copies device selection information to kernel space.
        If the device has been opened more than once the command fails
        with @code{EBUSY}; otherwise the pci driver is unregistered and
        re-registered with a new @code{pci_id} item. If no device matches
        the new selection @code{ENODEV} is returned after a timeout of
        100ms.

@item RR_DEVGET (struct rr_devsel *)

	The command returns to user space device information: vendor/device,
        subvendor/subdevice and bus/devfn. If no device is currently
        managed by the driver, @code{ENODEV} is returned.

@item RR_READ (struct rr_iocmd *)
@itemx RR_WRITE (struct rr_iocmd *)

	The commands can read or write one register from an even BAR
        area (BAR 0, 2, 4) of within the DMA buffer (BAR 12, 0xc).
        The @code{address} field of the structure
        specifies both the BAR and the offset (see @code{rawrabbit.h} or
        the description of @i{llseek} above for the details).  Access outside
        the size of the area returns @code{ENOMEDIUM}. 
        The @code{datasize} field of @code{rr_iocmd}
        can be 1, 2, 4 or 8 and is a byte count.
        The other fields, @code{data8} through @code{data64} are used to
        host the register value; these fields are collapsed together in an
        unnamed union (see the @i{gcc} documentation about unnamed unions),
        so the same code works with little-endian and big-endian systems.

@item RR_IRQWAIT (no third argument)

	The command waits for an interrupt to happen on the device. If an
        interrupt did already happen, @code{EAGAIN} is returned, otherwise
        an interrupt is waited for and 0 is returned. After the interrupt
        fired, the interrupt line is disabled by the kernel handler.
         Please note that this may
        be a serious problem if the line is shared with other peripherals,
        like your hard drive or ethernet card.

@item RR_IRQENA (no third argument)

	The command re-enables the interrupt. The user is assumed to have
        acknowledged the interrupt in the board itself, or another interrupt
        will immediately fire. If the interrupt did not happen, @code{EAGAIN}
        is returned, otherwise the command returns the number of nanoseconds
        that elapsed since the interrupt occurred. If more than one
        second elapsed, the command returns 1000000000 (one billion), to
        avoid overflowing the signed integer return value of @i{ioctl}.

@item RR_GETDMASIZE (no third argument)

	The command simply returns the size, in bytes, of the DMA buffer,
        Currently such size can only be changed at module load time and is
        fixed for the lifetime of the module.

@item RR_GETPLIST (array of 1024 32-bit values)

	The command returns the PFNs for the current DMA buffer. The initial
        part of the page passed as third argument is filled with 32-bit
        values.  The array must be a complete 1024-entry array, even if
        only part of it is used.  Each value written represents a @i{page
        frame number} that can be shifted by 12 bits to obtain the physical
        address for the associated page.  The @i{rawrabbit} module can only
        work with 4kB pages, and a compile-time check is built into the code
        to prevent compilation with a different page size; at least not
        before a serious audit of the code.

@end table

@c ==========================================================================
@node User space demo programs, User space benchmarks, Ioctl commands, Raw PCI I/O
@section User space demo programs

The subdirectory @code{user/} of this package includes the user-space
sample tools.  The helper for @i{rawrabbit} (@i{rr}) is called @i{rrcmd}.

@menu
* rrcmd::                       
@end menu

@c --------------------------------------------------------------------------
@node rrcmd,  , User space demo programs, User space demo programs
@subsection rrcmd

The @i{rrcmd} program can do raw I/O and change the active binding of
the device.

Every command line can change the binding and issue a command. Since
binding is persistent, you can issue commands without specifying a new
binding.  The initial binding is defined by module parameters, or by
default as a GN4124 device.

To specify a new binding, the syntax is
``@code{@i{vendor}:@i{device}/@i{subvendor}:@i{subdevice}@@@i{bus}:@i{devfn}}''
where the first pair is mandatory and the following ones are optional.

The following is an example session with @i{rrcmd}, from the
compilation directory, note that in this case I'm using the GN4124
device and an ethernet port without active driver.

@example
   tornado% sudo insmod kernel/rawrabbit.ko
   tornado% ./user/rrcmd info
   /dev/rawrabbit: bound to 1a39:0004/1a39:0004@0001:0000
   tornado% ./user/rrcmd 10b7:9055
   tornado% ./user/rrcmd info
   /dev/rawrabbit: bound to 10b7:9055/10b7:9055@0004:0000
   tornado% ./user/rrcmd 1a39:0004 info
   /dev/rawrabbit: bound to 1a39:0004/1a39:0004@0001:0000
   tornado% ./user/rrcmd 10b7:9055@01:0
   ./user/rrcmd: /dev/rawrabbit: ioctl(DEVSEL): No such device
   tornado% ./user/rrcmd info
   /dev/rawrabbit: not bound
@end example

The ``no such device'' error above depends on the chosen @i{bus:devfn}
parameter.  Please note that trying to bind to a device already driven
by a kernel driver returns @code{ENODEV} in the same way, as the probe
function of the PCI driver registered by @i{rawrabbit} will not be
called.

To read and write data with @i{rrcmd} you can use the @code{r} and @code{w}
commands. The syntax of the commands is as follows:

@example
    r[<sz>] <bar>:<addr>
    w[<sz>] <bar>:<addr> <val>
    <sz> = 1, 2, 4, 8 (default = 4)
    <bar> = 0, 2, 4
@end example

Actually, since an interactive user often reads and writes the same
register, the @code{r} and @code{w} commands are the same, and a read
or write is selected according to the number of arguments. You can think
of @code{r} as ``register'' and @code{w} as ``word'' if you prefer.

In this example two Gennum leds are turned off, and the value is read back.
Address 0xa08 in BAR 4 is the ``output drive enable'' register for the
GPIO signals from the GN4124 chip, and enabling the drive without any
other change from default settings is enough to turn the leds off.

@example
    tornado% ./user/rrcmd r 4:a08
    0x00000000
    tornado% ./user/rrcmd r 4:a08 0x3000
    tornado% ./user/rrcmd r 4:a08
    0x00003000
@end example

Note, in the example above, that ``@code{r}'' is used for writing
as well as reading. If you forget the @code{r} or @code{w}
command name, however, the program will understand the argument
as a @i{vendor}:@i{device} pair, and will unbind the driver.
This can be construed as a design bug and you can blame me at will.

Reading data with a different-from-default size returns the right number
of hex digits, to make clear what data size that has been read:

@example
   tornado% ./user/rrcmd r1 4:a08
   0x00
   tornado% ./user/rrcmd r2 4:a08
   0x3000
   tornado% ./user/rrcmd r4 4:a08
   0x00003000
   tornado% ./user/rrcmd r8 4:a08
   0x0000000000003000
@end example

Interrupt management with @i{rrcmd} can be performed using two
commands: @code{irqwait} and @code{irqena}. The former is used to wait
for an interrupt to happen; the latter re-enables the interrupt in the
controller. You should probably acknowledge the interrupt in the device
between these two operations.  The @code{irqwait} command returns
@code{EAGAIN} if the interrupt has already happened; the @code{irqena}
command returns @code{EAGAIN} if the interrupt has not happened
yet.

For example, this script waits for an interrupt in a BT878 frame
grabber and acknowledges it for 100 times:

@example
   # select device and enable vsync interrupt (bit 1, value 0x2)
   ./user/rrcmd 109e:036e w 0:104 2
   # now wait for irq, acknowledging bit 1 for vsync
   for n in $(seq 1 100); do
      ./user/rrcmd irqwait
      ./user/rrcmd w 0:100 2
      ./user/rrcmd irqena
   done
   # finally, disable the interrupt in the device, ack and enable
   ./user/rrcmd w 0:104 0
   ./user/rrcmd w 0:100 2
   ./user/rrcmd irqena
@end example

The other commands are @i{getdmasize} and @i{getplist}, that work
as follows:

@example
    tornado% ./user/rrcmd getdmasize
    dmasize: 1048576 (0x100000 -- 1 MB)
    tornado% ./user/rrcmd getplist | head
    buf 0x00000000: pfn 0x00029a3c, addr 0x000029a3c000
    buf 0x00001000: pfn 0x0002dbb1, addr 0x00002dbb1000
    buf 0x00002000: pfn 0x00029a34, addr 0x000029a34000
    buf 0x00003000: pfn 0x00029839, addr 0x000029839000
    buf 0x00004000: pfn 0x00029838, addr 0x000029838000
    buf 0x00005000: pfn 0x000298ed, addr 0x0000298ed000
    buf 0x00006000: pfn 0x000298ec, addr 0x0000298ec000
    buf 0x00007000: pfn 0x00029843, addr 0x000029843000
    buf 0x00008000: pfn 0x00029842, addr 0x000029842000
    buf 0x00009000: pfn 0x0002dbab, addr 0x00002dbab000
@end example

@c ==========================================================================
@node User space benchmarks,  , User space demo programs, Raw PCI I/O
@section User space benchmarks

The package includes a few trivial programs used to benchmark performance
of the various I/O primitives.

@menu
* bench/ioctl::                 
* bench/irq878::                
* Benchmarking read and write::  
@end menu

@c --------------------------------------------------------------------------
@node bench/ioctl, bench/irq878, User space benchmarks, User space benchmarks
@subsection bench/ioctl

The program tests how many ioctl output operations can be performed
per second. It issues a number of register writes assuming the driver
is currently accessing the Gennum evaluation board.

The data written makes the 4 GPIO leds blink with different duty
cycles, so you should see them lit at different light levels.

On my system, the program reports more than 3 million operations per
second:

@example
    tornado% ./bench/ioctl 1000000
    1000000 ioctls in 303611 usecs
    3293688 ioctls per second
    tornado% ./bench/ioctl 10000000
    10000000 ioctls in 3068384 usecs
    3259044 ioctls per second
@end example

@c --------------------------------------------------------------------------
@node bench/irq878, Benchmarking read and write, bench/ioctl, User space benchmarks
@subsection bench/irq878


The program does the same kind of
operation as the script shown earlier: it handles BT878 interrupts in
user space, and prints the delays from actual interrupt to
end-of-acknowledge.  While the script shown earlier reports times in
the order of 10ms, since several processes are executed between
the interrupt and the final @code{irqena}, this shows the system call
overhead which is just a few microseconds:

@example
     tornado% ./bench/irq878 100
     got 100 interrupts, average delay 6389ns
@end example

@c --------------------------------------------------------------------------
@node Benchmarking read and write,  , bench/irq878, User space benchmarks
@subsection Benchmarking read and write

No specific program is provided to check access to the DMA buffer, as
@i{dd} is enough to verify read and write speed.  A script like the
following will work:

@smallexample
IF="if=/dev/rawrabbit"
OF="of=/dev/rawrabbit"
# test dmabuf read
for BS in 1 2 4 8 16 32 64 128 256 512 1024 2048 4096; do
   dd bs=$BS skip=$(expr $(printf %i 0xc0000000) / $BS) $IF of=/dev/null \
	2>&1 | grep MB/s
done
# test dmabuf write
for BS in 1 2 4 8 16 32 64 128 256 512 1024 2048 4096; do
   dd bs=$BS seek=$(expr $(printf %i 0xc0000000) / $BS) $OF if=/dev/null \
	2>&1 | grep MB/s
done
@end smallexample

To benchmark access to I/O memory, the @i{rdwr} utility is offered.
It repeatedly accesses the GPIO register (bar 4, offset 0xa08 of the
GN4124 board) as a 32bit register and measures the time it takes:

@example
   tornado% ./bench/rdwr 1000000
   1000000 writes in 361487 usecs
   2766351 writes per second
   1000000 reads in 1041681 usecs
   959986 reads per second
@end example

It's interesting to note that reads are slower than writes. Even more
interesting is that direct
writes are slower than writing through @i{ioctls} (compare with
@code{bench/ioctl}).  The difference is probably due use of
@i{lseek} between one @i{read} or @i{write} and the next one, so for an
@i{ioctl}-based I/O operation you need one system call, but to
achieve the same using @i{read} or @i{write} you need two system
calls.


@c ##########################################################################
@node Firmware Loader,  , Raw PCI I/O, Top
@chapter The Firmware Loader

The kernel driver supports loading a binary file from user space.
Such a loading is Gennum-specific, so currently the code performs
the operation only when the bound device has @code{1a39:0004} as
@i{vendor:device} pair.

@menu
* Loading at Probe Time::       
* The Firmware File Name::      
* The Firmware File Format::    
* Working Without Udev::        
@end menu

@c ==========================================================================
@node Loading at Probe Time, The Firmware File Name, Firmware Loader, Firmware Loader
@section Loading at Probe Time

Whenever a device is probed for, the driver requests a firmware file. FIXME

Such @i{probe} actions happen when the driver is matched to a device.
This happens at load time, if the PCI bus is already hosting a device
with the proper vendor end product numbers, but also when @i{rrcmd}
or other user-space tools ask the driver to bind to a different device.

In practice, you can load a new firmware binary simply by re-binding
the driver to the same device. For example this command re-loads the
firmware:

@example
   rrcmd 1a39:0004
@end example

@c ==========================================================================
@node The Firmware File Name, The Firmware File Format, Loading at Probe Time, Firmware Loader
@section The Firmware File Name

The name of the binary file being requested is generated from the various
identifiers of the board: vendor and product, subvendor and subproduct,
bus number and devfn.  For example, in my system the file being
requested is: @code{rrabbit-1a39:0004-1a39:0004@@0001:0000} .

Since no message is reported to system logs when the file is not found,
the module uses @i{printk} to report the generated name. The file
can be placed in @i{/lib/firmware} or other places, according to how
your @i{hotplug} or @i{udevd} is configured.  The firmware-loader
mechanism of the kernel arranges for the file to be made available to
@i{rawrabbit}, whatever the file name.

The current version is still too verbose in its messages, but this is
work in progress so you'll forgive me. This is for example what I get
then loading the module with a correct firmware file:

@c FIXME
@smallexample
[ 3048.954185] rr_ask_firmware: called with preempt_cont == 0x00000001
[ 3048.960525] rr_ask_firmware: current = 3061: insmod
[ 3048.965486] rr_load_firmware: rrabbit-1a39:0004-1a39:0004@0001:0000
[ 3048.971834] rr_load_firmware: called with preempt_cont == 0x00000000
[ 3048.978342] rr_load_firmware: current = 13: events/2
[ 3048.983441] request firmware returned 0
[ 3049.048928] rr_loader_complete: called with preempt_cont == 0x00000000
[ 3049.055572] rr_loader_complete: current = 3066: firmware/rrabbi
[ 3049.061527] eda3a3c0: size 594412 (0x911ec), data f8b02000
[ 3049.067045] programming with bar4 @ fe7ff000, vaddr f8494000
[ 3050.574029] __rr_gennum_load: 220: done after 148603
@end smallexample

In the message above, you can see both the file name (always starts
with @i{rrabbit-1a39:0004}) and the context of execution: @i{current}
is the current process, and as you see a new kernel thread is created
to run the completion function of the loading mechanism (here
the @i{pid} is 3066 and the name is @i{firmware/rrabbi} -- only the
leading 15 bytes of the name are printed.  The final @i{done after 148603}
reports that the @i{done} bit in Gennum registers became true after
writing that number of 32-bit words. Hopefully, this matches the file
size divided by 4.

This version of the package allows changing the file name for the
firmware at will (but it doesn't allow picking it from a different
directory from the ones looked-for by the @i{hotplug} or @i{udev}
daemons).  The default firmware name is set forth in a module parameter,
@code{fwname}; the predefined value of the string is
``@code{rrabbit-%P-%p@@%b}''; such a string is expanded at run time,
using the following markers:

@itemize @bullet
@item @code{%P} expands to the PCI vendor and device ID
@item @code{%p} expands to the subsystem vendor and device ID
@item @code{%b} expands to the bus and @i{devfn} values
@item @code{%%} expands to a single percent
@end itemize

Each such marker is expanded as @code{%04x:%04x}, using the pairs
vendor:device or bus:devfn. Trailing spaces and newlines are allowed,
as the module removed them automatically; other @code{%} escape on the
other hand are currently forbidden, and result in an error.

The version string can be changed at load time as a module parameter,
or later by writing in @code{/sys/module/rawrabbit/fwname}. When changing
the content of the file, a new firmware-load action is triggered, so
you don't need to rebind the device (may I say I dislike this feature?
you may just @code{rrcmd 1a39:0004} or rebind in @i{sysfs}).

The driver tries to check the @i{fwname} string: if it's too long or
uses a wrong format, writing it is refused and the previous name
remains.  I fear, though, that there may be some buglet about
allocation/deallocation, so you'd better not write wrong names, to be
safe.

This is an example session:
@smallexample
   tornado.root# insmod kernel/rawrabbit.ko fwname="fw-%P"
   [22194.161358] rr_load_firmware: fw-1a39:0004
   tornado.root# cat /sys/module/rawrabbit/parameters/fwname
   fw-%P
   tornado.root# echo -n "rabbit-%P-%p-bus-%b" \
           > /sys/module/rawrabbit/parameters/f
   [22898.924501] rr_load_firmware: rabbit-1a39:0004-1a39:0004-bus-0001:0000
@end smallexample

@c ==========================================================================
@node The Firmware File Format, Working Without Udev, The Firmware File Name, Firmware Loader
@section The Firmware File Format

Testing has been performed with the binary firmware provided by Gennum
as an example. The file is called @file{lambo.gfw} and is in an
yet-unknown format. Binaries generated with the Xilinx tools are in
a different format and don't work (although they load successfully).

The Gennum binary must be preprocessed before it can be programmed in
the hardware: the header must be removed and the bits in each byte
must be reversed.  This work is expected to be performed in user
space, and the file in @file{/lib/firmware} is expected to be the
raw binary.  For bit-reversal, you can use @file{user/flip} which is
part of this package.

The Gennum header is simple: after 4 bytes of magic number there is
a teo-byte little-endian count of header size. In the @file{lambo.gfw}
this is 34 bytes (0x22). The following script is an example of how
to process this file, assuming @code{GFW} is the input file name and
@code{RRABBIT} is the top directory of this package:

@example
   # print as decimal-2bytes, no address, 2bytes per line,
   # then get line 3, and delete any space in the string. We get "34".
   hsize=$(od -t d2 -An -w2 $GFW | sed -n 3p | tr -d ' ')
   # skip 1 block, of size 34, and bit-reverse the resulting file.
   dd bs=$hsize skip=1 if=$GFW | $RRABBIT/user/flip > $OUTNAME
@end example

With the original @file{lambo.gfw} (size 594446, md5sum:
7d9d289b8014ff2be0550816496463f9) you'll get a file with md5sum
2aec561e9aa289944d87b2f3ec988b66.

You can check that this works by writing and reading at offset 0x4000 of
BAR0, where FPGA RAM is exposed:

@example
    tornado% sudo ./user/rrcmd w4 0:4000 c0ffee
    tornado% sudo ./user/rrcmd r4 0:4000
    0x00c0ffee
@end example

@c ==========================================================================
@node Working Without Udev,  , The Firmware File Format, Firmware Loader
@section Working Without Udev

If needed, you can work withoPlease note that with some kernel versions, you won't be able to
remove the module while it is bound to a device. You must first force
it to unbind from the device, either by running @i{rrcmd} with
an unexistent @i{vendor:device} pair, or by using the @i{unbind}
method in @i{sysfs}:

@example
   tornado.root# rmmod rawrabbit
   ERROR: Module rawrabbit is in use
   tornado.root# DEV=$(basename /sys/bus/pci/drivers/rawrabbit/*:*)
   tornado.root# echo $DEV > /sys/bus/pci/drivers/rawrabbit/unbind
   tornado.root# rmmod rawrabbit
   tornado.root#
@end example
ut @i{udev} or @i{hotplug}, which is good
if your system is meant to be a small embedded-like, fast-boot device.

Without @i{udev}, the binary being called by the userspace-helper of
the kernel is the path appearing in @file{/proc/sys/kernel/hotplug}.
So, for example, you can run this script as @file{hotplug},
to show what's happening at firmware-load time:

@example
   #!/bin/sh
   date > /tmp/hp.$$
   echo Called as: $0 "$@" >> /tmp/hp.$$
   echo Environment: >> /tmp/hp.$$
   env >> /tmp/hp.$$
@end example

To activate it, just make it executable as
@i{/usr/local/bin/hotplug-logger} and run this command (without
@i{udev} or other interfering stuff running):

@example
   echo /usr/local/bin/hotplug-logger > /sys/kernel/uevent_helper
@end example

The following file appeared on my system as @file{/tmp/hp.3190} when
firmware loading happened (3190 is the @i{pid} of the script):

@smallexample
    Wed Jan 12 09:26:16 CET 2011
    Called as: /usr/local/bin/hotplug-logger firmware
    Environment:
    SUBSYSTEM=firmware
    ASYNC=1
    DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/firmware/0000:01:00.0
    FIRMWARE=rrabbit-1a39:0004-1a39:0004@0001:0000
    PATH=/sbin:/bin:/usr/sbin:/usr/bin
    ACTION=add
    PWD=/
    TIMEOUT=60
    SHLVL=1
    HOME=/
    SEQNUM=1351
    _=/usr/bin/env
@end smallexample

Thus, with those environment variables, you can use this script as
user-space helper for firmware loading. Again, just place it as
executable file in some place and write its pathname to
@file{/sys/kernel/uevent_helper}.

@smallexample
    #!/bin/sh
    export HOTPLUG_FW_DIR=/lib/firmware/
    # FIRMWARE and DEVPATH are provided in the event
    if [ -z "$FIRMWARE" ]; then exit 0; fi
    echo "$0: loading $FIRMWARE to $DEVPATH" > /dev/console
    echo 1 > /sys/$DEVPATH/loading
    cat $HOTPLUG_FW_DIR/$FIRMWARE > /sys/$DEVPATH/data
    echo 0 > /sys/$DEVPATH/loading
@end smallexample

Note that the userspace helper is also called at device creation and
removal, so you'll get several invocations of either script. This
leads to several log files in the former case.  Also, please note that
is no @i{udev} is running, you'll need to @i{mknod} manually, with 10
as major number and 42 as minor number.

As a final note, please remember that when running the simple
@i{hotplug-logger} shown above, no firmware is actually loaded, so the
module remains busy until the timeout (1 minute) expires, and you
won't be able to unload it. You may want to change such timeout with
a command like:

@example
   echo 3 > /sys/class/firmware/timeout
@end example

@c ##########################################################################
@iftex
@contents
@end iftex

@bye
