Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
\input texinfo @c -*-texinfo-*-
%
% fmc-bus.in - main file for the documentation
%
%%%%
%------------------------------------------------------------------------------
%
% NOTE FOR THE UNAWARE USER
% =========================
%
% This file is a texinfo source. It isn't the binary file of some strange
% editor of mine. If you want ASCII, you should "make fmc-bus.txt".
%
%------------------------------------------------------------------------------
%
% This is not a conventional info file...
% I use three extra features:
% - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
% - leading blanks are allowed (this is something I can't live without)
% - braces are automatically escaped when they appear in example blocks
%
@comment %**start of header
@documentlanguage en
@setfilename fmc-bus.info
@settitle fmc-bus
@iftex
@afourpaper
@end iftex
@paragraphindent none
@comment %**end of header
@setchapternewpage off
@set update-month July 2012
@finalout
@titlepage
@title FMC Bus Abstraction for Linux
@subtitle @value{update-month}
@subtitle Implementing a bus abstraction for fmc mezzanines
@author Alessandro Rubini for CERN (BE-CO-HT)
@end titlepage
@headings single
@c ##########################################################################
@iftex
@contents
@end iftex
@c ##########################################################################
@node Top
@top Introduction
This document describes the implementation of the @i{fmc} bus for
Linux. FMC (FPGA Mezzanine Carrier) is the standard we use for our
I/O devices, in the context of White Rabbit and related hardware.
In our I/O environments we need to write drivers for each mezzanine
card, and such drivers must work independent of the carrier being used.
To achieve this, we abstract the FMC interface
We have a carrier for PCI-E called @i{SPEC} and one for VME called
@i{SVEC}, but more are planned. Also, we support stand-alone devices
(usually plugged on a SPEC card), controlled through Etherbone,
developed by GSI.
Currently, code and documentation for the FMC bus is part of the
@i{spec-sw} project at @code{ohwr.org}.
@menu
* What is a Linux Bus::
* FMC Device::
* FMC Driver::
* The API Offered by Carriers::
@end menu
@c ##########################################################################
@node What is a Linux Bus
@chapter What is a Linux Bus
Within the Linux kernel, a @i{bus} is a data structure with a few
methods. It's main role is registering a list of devices and a list
of drivers, offering a @i{match} function that compares the respective
identifiers (in a bus-specific way) to assign drivers to devices.
Activation and deactivation of devices happens through the @i{probe}
and @i{remove} functions of the respective driver; an advanced user
can also use @i{sysfs} to change the binding of drivers to devices
(for example, if more than one driver can drive the same device you
may want to force the choice).
@c ##########################################################################
@node FMC Device
@chapter FMC Device
Within this framework, the FMC device is created and registered by the
carrier driver. For example, the PCI driver for the SPEC card fills a
data structure for each SPEC that it drives, and registers an
associated FMC device. The SVEC driver can do exactly the same for
the VME carrier (actually, it should do it twice, because the SVEC
carries two FMC mezzanines. Similarly, an Etherbone driver will be
able to register its own FMC devices, offering communication primitives
through frame exchange.
The contents of the EEPROM within the FMC will be used for
identification purposes, i.e. for matching the device with its own
driver. For this reason the device structure includes a complete copy
of the EEPROM (actually, the carrier driver may choose to only return
the leading part of it).
This is the current structure defining a device. Please note that all
the machinery is in place but some details may still change in the future.
For this reason, there is a version field at the beginning of the structure.
As usual, the minor number will change for compatible changes (like a new
flag) and the minor number will increase when an incompatible change
happens (for example, a change in layout of some @i{fmc} data structures).
Device writers should just set it to the
value @t{FMC_VERSION}, and be ready to get back @t{-EINVAL} at
registration time.
@smallexample
struct fmc_device {
unsigned long version; /* to be set to FMC_VERSION */
struct fmc_device_id id; /* for the match function */
struct fmc_operations *op; /* carrier-provided */
int irq; /* according to host bus. 0 == none */
int eeprom_len; /* Usually 8kB, may be less */
uint8_t *eeprom; /* Full contents or leading part */
char *carrier_name; /* "SPEC" or similar, for special use */
void *carrier_data; /* "struct spec *" or equivalent */
__iomem void *base; /* May be NULL (Etherbone) */
struct device dev; /* For Linux use */
struct device *hwdev; /* The underlying hardware device */
struct sdb_array *sdb;
void *mezzanine_data;
@c ##########################################################################
@node FMC Driver
@chapter FMC Driver
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
Within this framework the FMC driver is expected to be independent of
the carrier being used. The matching between device and driver is
only based on the content of the EEPROM (as mandated by the FMC
standard) and the driver will perform I/O accesses only by means of
carrier-provided functions.
In some special cases it is possible for a driver to directly access
FPGA registers, by means of the @code{base} field of the device
structure. This is needed for high-bandwidth peripherals like fast ADC
cards. The @code{base} pointer is NULL for remote devices, and the driver
will refuse to work with them if it needs direct access.
In even more special cases, the driver may access carrier-specific
functionality: the @code{carrier_name} string allows the driver to
check which is the current carrier and make use of the
@code{carrier_data} pointer. We chose to use carrier names rather
than numeric identifiers for greater flexibility, but also to avoid a
central registry within the @code{fmc.h} file -- we hope other users
will exploit our framework with their own carriers.
@c ##########################################################################
@node Functions Exported by the Core
@chapter Functions Exported by the Core
The FMC core exports the usual 4 functions that are needed for a bus
to work:
@smallexample
int fmc_driver_register(struct fmc_driver *drv);
void fmc_driver_unregister(struct fmc_driver *drv);
int fmc_device_register(struct fmc_device *tdev);
void fmc_device_unregister(struct fmc_device *tdev);
@end smallexample
They should be self-explicative, so nothing is added here.
@c ##########################################################################
@node The API Offered by Carriers
@chapter The API Offered by Carriers
The carrier provides a number of methods by means of the
@code{fmc_operations} structure, which currently is defined like this
(again, it is a moving target, please refer to the header rather than
this document):
@smallexample
struct fmc_operations {
uint32_t (*readl)(struct fmc_device *fmc, int offset);
void (*writel)(struct fmc_device *fmc, uint32_t value, int offset);
int (*reprogram)(struct fmc_device *f, struct fmc_driver *d, char *gw);
int (*validate)(struct fmc_device *fmc, struct fmc_driver *drv);
int (*irq_request)(struct fmc_device *fmc, irq_handler_t h,
void (*irq_ack)(struct fmc_device *fmc);
int (*irq_free)(struct fmc_device *fmc);
int (*gpio_config)(struct fmc_device *fmc, struct fmc_gpio *gpio,
int ngpio);
int (*read_ee)(struct fmc_device *fmc, int pos, void *d, int l);
int (*write_ee)(struct fmc_device *fmc, int pos, const void *d, int l);
The individual methods perform the following tasks:
@table @code
@item readl
@itemx writel
These functions access FPGA registers by whatever means the
carrier offers. They are not expected to fail, as most of the time
they will just make a memory access to the host bus. If the
carrier provided a @i{base} pointer, the driver may use direct
access through it instead. For this reason the header offers
the inline functions @i{fmc_readl} and @i{fmc_writel} that
access @i{base} if respective method is NULL.
For Etherbone, or other non-local carriers,
error-management is still to be defined.
@item validate
Module parameters are used to manage different applications for
the @i{bus_id} module parameter (if provided) and returns the
matching index in the array. If no match is found, @code{-ENOENT}
is returned; if the argument has not been specified, all
devices match the driver and 0 is returned. The value returned
by the validate method can be used as index into other parameters
(for example, some drivers use the @code{lm32=} parameter in this
way). Such ``generic parameters'' are currently documented in the
@i{spec-sw} manual; this @i{validate} method is on show in
@code{spec-fmc.c} and it is used by @code{fmc-trivial.c}.
@item reprogram
The carrier enumerates FMC devices by loading a standard (or
@i{golden} FPGA binary that allows EEPROM access. Each driver, then,
will need to reprogram the FPGA by calling this
function. If the name argument is NULL,
the carrier will reprogram the golden binary. If the gateware name
has been overridden through module parameters (in a carrier-specific
way) the file loaded will match the parameters.
@item irq_request
@itemx irq_ack
@itemx irq_free
Interrupt management is carrier-specific, so it is abstracted
as operations. The interrupt number is listed in the device
structure, but it's only informative for the mezzanine driver.
The handler will receive the @i{fmc} pointer as @i{dev_id}; the
@i{flags} argument is still to be defined.
@item gpio_config
The method allows to configure a GPIO pin in the carrier, and
read its current value if it is configured as input. See
@ref{The GPIO Abstraction} for details.
@item read_ee
@itemx write_ee
Read or write the EEPROM. The functions are expected to be only
called before reprogramming and the carrier will refuse them
with @code{ENODEV} after reprogramming. The offset is limited
to 8kB but addresses up to 1MB are reserved to fit bigger I2C
devices in the future. Carriers may offer
access to other internal flash memories using these same methods:
for example the SPEC driver may define that its own I2C memory
is seen at offset 1M and the internal SPI flash is seen at offset
16M. This is carrier-specific and should only be used by the
driver after checking the @code{carrier_name} field.
@end table
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
@c ##########################################################################
@node The GPIO Abstraction
@chapter The GPIO Abstraction
Support for GPIO pins in the @i{fmc-bus} environment is a little
heavy, and deserves special discussion.
While the general idea of a carrier-independent driver seems to fly,
configuration of specific signals within the carrier needs at least
some knowledge of the carrier itself. For this reason, the specific
driver can request to configure carrier-specific GPIO pins, numbered
from 0 to at most 4095. Configuration is performed by passing
a pointer to an array of @t{struct fmc_gpio} items, as well as
the number of those items:
@example
struct fmc_gpio {
char *carrier_name;
int gpio;
int _gpio; /* internal use by the carrier */
int mode; /* GPIOF_DIR_OUT etc, from <linux/gpio.h> */
int irqmode; /* IRQF_TRIGGER_LOW and so on */
};
@end example
By specifying a @i{carrier_name} for each pin, the driver may access
different pins in different carriers. The @i{gpio_config} method
returns the number of pins successfully configured, and each carrier
just ignores requests for other carriers. So, for example, a driver
that has been developed and tested on both the SPEC and the SVEC may
request configuration of two different GPIO pins, and expect one such
configuration to succeed -- if none succeeds it most likely means that
the current carrier is a still-unknown one. (FIXME: the return value
is not actually used this way in current code).
If, however, your GPIO pin has a specific known role, you can
pass a special number in the @t{gpio} field. The header defines
the following macros:
@example
#define FMC_GPIO_RAW(x) (x) /* 4096 of them */
#define FMC_GPIO_IRQ(x) ((x) + 0x1000) /* 256 of them */
#define FMC_GPIO_LED(x) ((x) + 0x1100) /* 256 of them */
#define FMC_GPIO_KEY(x) ((x) + 0x1200) /* 256 of them */
#define FMC_GPIO_TP(x) ((x) + 0x1300) /* 256 of them */
#define FMC_GPIO_USER(x) ((x) + 0x1400) /* 256 of them */
@end example
Use of virtual GPIO numbers (anything but @t{FMC_GPIO_RAW}) is allowed
provided the @i{carrier_name} field is left unspecified (NULL). Each
carrier is responsible for providing a mapping between virtual and
physical GPIO numbers (and possibly cache the raw number in the
@t{_gpio} field). All carriers must map their I/O lines
to the sets above starting from zero. The SPEC, for example, maps
interrupt 0 and 1, and test points 0 through 3.
If, for example, a driver requires a free led and a test point (for a
scope probe to be plugged at some point during development) it may ask
for @t{FMC_GPIO_LED(0)} and @t{FMC_GPIO_TP(0)}. Each carrier will
provide suitable GPIO pins. Clearly, the person running the drivers
will know the order used by the specific carrier driver in assigning
leds and testpoints, so to make a carrier-dependent use of the diagnostic tools.
In theory, some form of autodetection should be possible: a driver
like the @i{wr-nic} (which uses IRQ(1) on the SPEC card) should
configure IRQ(0), make a test with software-generated interrupts and
configure IRQ(1) if the test fails -- the @i{wr-nic} gateware is
known to use IRQ1 on the SPEC, but the driver should be
carrier-independent if possible and thus use IRQ(0) as a first bet.
If a pin is configured as input, the @i{gpio_config} method returns 0
or 1, to report its current value. Invalid GPIO numbers will cause
@code{-ENODEV} to be returned for physical numbers and @code{-ENOENT}
for virtual mappings.
@bye
@c LocalWords: gnudd titlepage iftex texinfo CERN documentlanguage settitle
@c LocalWords: documentencoding setfilename afourpaper paragraphindent SVEC
@c LocalWords: setchapternewpage finalout Etherbone EEPROM