fmc-dio.in 34.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
\input texinfo    @c -*-texinfo-*-
% fmc-dio.in - main file for the documentation
%
%
%%%%

%------------------------------------------------------------------------------
%
%                         NOTE FOR THE UNAWARE USER
%                         =========================
%
%    This file is a texinfo source. It isn't the binary file of some strange
%    editor of mine. If you want ASCII, you should "make spec-sw.txt".
%
%------------------------------------------------------------------------------

%
% This is not a conventional info file...
% I use three extra features:
%   - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
%   - leading blanks are allowed (this is something I can't live without)
%   - braces are automatically escaped when they appear in example blocks
%

@comment %**start of header
@documentlanguage en
@setfilename fmc-dio.info
@settitle fmc-dio
@iftex
@afourpaper
@end iftex
@comment %**end of header

@setchapternewpage off

@set update-month July 2020
@c the release name below is substituted at build time
@set release __RELEASE_GIT_ID__

@finalout

@titlepage
@title FMC DIO Software Support
@subtitle @value{update-month} (@value{release})
@subtitle A driver for the FMC DIO module
Jorge Machado's avatar
Jorge Machado committed
46 47
@author Alessandro Rubini for CERN (BE-CO-HT)
@author Jorge Machado Cano for Seven Solutions
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
@end titlepage
@headings single

@c ##########################################################################
@iftex
@contents
@end iftex

@paragraphindent 1

@c ##########################################################################
@node Top
@top Introduction

This is the manual for the fmc-dio device driver. 

@c ##########################################################################
@node History and Overview
@chapter History and Overview

68 69 70 71 72 73 74 75 76 77 78
This version of the driver adds support for generating a train pulses in HW and
a new virtual channel that generates interrupts and is fully configurable by the user.
The HW train pulse generator improves the precision of the pulses which are
not any more generated in software as it was the case in the previous versions.
In this implementation, the software only takes care of the number of pulses to
disable them when all the pulses have been generated.

This and previous versions of the driver provide support for immediate pulse
generation, programmed pulse generation synchronized with White Rabbit clock,
and input pulse timestamping. The driver provides an ioctl interface to configure
the hdl core.
79 80 81 82 83

@c ##########################################################################
@node Compiling the Drivers
@chapter Compiling the Drivers

Jorge Machado's avatar
Jorge Machado committed
84
The kernel modules that are part of this package live in the @i{sw/kernel}
85 86 87 88 89 90 91 92
subdirectory. To compile them, you need to 
set the following variables in your environment:

@table @code

@item LINUX

	The top directory of the kernel sources for the version you
Jorge Machado's avatar
Jorge Machado committed
93 94
        are going to run the driver under. I'm testing mostly with 4.15,
	but this version compiles against Linux-4.15-4.18 and later ones.
95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113

@item CROSS_COMPILE

	If you are cross-compiling, you need to set this variable.
	It is not usually needed for the PC, but if you are using
        the @i{Powec} board, you'll most likely need this. It is not
        needed if you compile for a different-sized PC (see below).

@item ARCH

	If you are cross-compiling, set this variable. Use @code{powerpc}
        for the @i{Powec}, @code{x86-64} if you compile on a 32-bit PC
        to run on a 64-bit PC and @code{i386} if you compile on a 64-bit
        PC to run on a 32-bit PC.

@end table

To compile run ``@code{make}'' with the previous variables set.  To
install run ``@code{make install} to install under
Jorge Machado's avatar
Jorge Machado committed
114
@code{/lib/modules/4.15} (or other version-based directory).  You can
115 116 117 118 119 120 121 122 123 124 125 126 127 128
set @code{INSTALL_MOD_PATH} to force and installation directory
(as a prefix followed by @code{/lib/modules/...}).
For example, if your target computer's filesystem is mounted under
@code{/mnt/target} you can run

@example
   make install INSTALL_MOD_PATH=/mnt/target
@end example

The modules are installed under the subdirectory @code{extra}. In
the case shown above your driver will end up being installed
(together with the other modules) as

@example
Jorge Machado's avatar
Jorge Machado committed
129
   /mnt/target/lib/modules/4.15.0/extra/fmc-dio.ko
130 131
@end example

132
Please note that by default the package compiles and installs the
133 134 135 136 137 138 139 140 141 142 143 144
@i{fmc-bus} modules, too (the project is a @i{git} submodule).

@c ##########################################################################
@node Role of fmc-dio.ko
@chapter Role of fmc-dio.ko

The FMC-DIO driver consists of a set of functions accesible through an IOCTL 
call to write on the registers of each channel of the FMC-DIO board to generate 
totally configurable pulses. It also recovers timestamps of external and generated
pulses.

@c ==========================================================================
Jorge Machado's avatar
Jorge Machado committed
145 146
@node Initialization of fmc-dio 
@section Initialization of fmc-dio 
147 148 149 150 151 152 153

There is no need to explicitly initialize the FMC-DIO driver. 

@c ==========================================================================
@node Interrupts in fmc-dio.ko
@section Interrupts in fmc-dio.ko

Jorge Machado's avatar
Jorge Machado committed
154
There are two types of interrupts: 
155

Jorge Machado's avatar
Jorge Machado committed
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173
@table @file

        @item I/O channel interrupt

                This interrupt is generated when one channel generates an 
                output pulse or detects an input pulse. The timestamp of the 
                pulse is stored into the timestamp FIFO queue of the channel.
        
        @item Dedicated interrupt channel

                This interrupt is generated only when the interrupt channel
                is triggered. The timestamp of the interrupt is stored into 
                the interrupt timestamp FIFO.

@end table

Both interrupt types comes from an unique HW interrupt. The software interrupt 
handler manages both types.
174 175 176 177 178 179 180 181 182 183 184 185 186
@c ==========================================================================
@node Code Layout
@section Code Layout

This section is mainly for the developers who look in the code, and for
me to make order in my own mind. FMC-DIO users are expected to skip to
the next section.

The @code{fmc-dio.ko} is built using a number of headers and source files,
spread over several directories:

@table @file

Jorge Machado's avatar
Jorge Machado committed
187
@item sw/fmc-bus/
188 189 190

        FMC-BUS kernel driver support (see specific doc inside)

Jorge Machado's avatar
Jorge Machado committed
191
@item sw/kernel/
192 193 194 195

        This directory contains the low level functions to configure each
        physical channel and the interrupt management.

Jorge Machado's avatar
Jorge Machado committed
196
@item sw/tools/
197 198 199 200 201 202 203 204 205 206 207

        This directory hosts the userspace tools that uses the low level 
        functions.

@end table

The @i{fmc-dio} driver refers to several headers, in addition to the
register definitions. This is the role of each of them:

@table @file

Jorge Machado's avatar
Jorge Machado committed
208 209
@item sw/kernel/fmc-dio.c
@item sw/kernel/fmc-dio.h
210 211 212 213 214

        Code and headers of the driver initialization. It also detects
        the version of the FPGA binary to able coherent memory accesses
        through all the code.

Jorge Machado's avatar
Jorge Machado committed
215
@item sw/kernel/fmc-dio-gpio.c
216

Jorge Machado's avatar
Jorge Machado committed
217
        DIO GPIO support skeleton to be developed using gpiolib Linux feature.
218

Jorge Machado's avatar
Jorge Machado committed
219
@item sw/kernel/fmc-dio-internal.c
220

Jorge Machado's avatar
Jorge Machado committed
221 222
        Code that implements the low-level functions to handle FMC DIO devices
        through the IOCTL and interrupt handler.
223

Jorge Machado's avatar
Jorge Machado committed
224
@item sw/kernel/fmc-dio-mdev.c
225 226 227

        This code created the device and connects the IOCTL.

Jorge Machado's avatar
Jorge Machado committed
228 229 230 231 232 233 234
@item sw/kernel/hw/ppsg-regs.h
@item sw/kernel/hw/wr-dio-regs.c
@item sw/kernel/hw/wr-dio-regs.h
@item sw/kernel/hw/wr-dio-regs_v1.c
@item sw/kernel/hw/wr-dio-regs_v1.h
@item sw/kernel/hw/wr-dio-regs_v2.c
@item sw/kernel/hw/wr-dio-regs_v2.h
235

Jorge Machado's avatar
Jorge Machado committed
236
        Code and headers with the registers map of the fmc-dio. It also
237 238 239 240
        implements some functions to get the offset of each register.

@end table

Jorge Machado's avatar
Jorge Machado committed
241
The @i{sw/tools} directory contains all the userspace tools the manage the 
242 243 244 245
@i{fmc-dio} driver. This is the role of each of them:

@table @file

Jorge Machado's avatar
Jorge Machado committed
246 247
@item sw/tools/net_tstamp.h
@item sw/tools/stamp-frame.c
248 249 250

        Code and headers of timestampig related functions.

Jorge Machado's avatar
Jorge Machado committed
251
@item sw/tools/wr-dio-agent.c
252

Jorge Machado's avatar
Jorge Machado committed
253 254 255
        Code that implements a very basic server to receive contol packets from 
        ruler.  Once received them, it executes some operation to the FMC DIO device 
        (mode change, pulse generation, etc).
256

Jorge Machado's avatar
Jorge Machado committed
257
@item sw/tools/wr-dio-cmd.c
258 259 260 261

        Code that implements the userpace commands to manage the fmc-dio
        channels.

Jorge Machado's avatar
Jorge Machado committed
262
@item sw/tools/wr-dio-pps.c
263 264 265

        Example of a PPS generation.

Jorge Machado's avatar
Jorge Machado committed
266
@item sw/tools/wr-dio-ruler.c
267

Jorge Machado's avatar
Jorge Machado committed
268 269
        Code that implements a basic client to send DIO command to the agent. 
        It also is able to run local FMC DIO commands.
270 271 272 273 274 275 276

@end table

@c ==========================================================================
@node Overview of the Driver
@section Overview of the Driver

277 278
This fmc-dio driver provides support for both versions of the fmc-dio hdl core,
i.e. the current version with extended functionality and a previous version.
Jorge Machado's avatar
Jorge Machado committed
279 280 281
The fmc-dio hdl core provides immediate pulse generation, immediate pulse 
train generator, programmed pulse/pulse train generation synchronized with 
White Rabbit clock, programmed/immediate interrupt generation and input 
282
pulse timestamping.
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305

@c ==========================================================================
@node Accessing the DIO Channels
@section Accessing the DIO Channels

In order to access the DIO channels, user-space programs are expected
to issue device-specific @i{ioctl} commands.  The driver supports two
commands, allocated at the end of the range of command numbers reserved
for device-specific use:

@table @code

@item PRIV_MEZZANINE_ID

	The command is used to identify the features of the
        specific NIC device.  It tells user space which mezzanine is
        currently plugged and also which type of carrier you are talking to.
        The command exchanges a data structure with user space in order
        to be able to extend its functionality over time, and such data
        structure includes  a sub-command
        field. (For example, we may return EEPROM contents to user space
        on request).

306
        @b{Warning}: The command is not implemented because we still have
307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349
        no mezzanine identification in place. The error being returned
        is @code{EAGAIN}; user code can rely on that error to know it
        is actually talking with a SPEC device, even if no identification
        is currently possible.

@item PRIV_MEZZANINE_CMD

	The command is based on the exchange of a data structure: by
        means of sub-commands included in such structure user space
        programs can request different services to the mezzanine driver.
        In the case of the DIO mezzanine this includes generating pulses
        and timestamping input events; other mezzanine drivers will
        be able to use the command in a different way. The application
        is expected to first run @code{PRIV_MEZZANINE_ID} to ensure the
        NIC device is connected to the right mezzanine.

@end table

The data structure is defined and explained in @code{wr-dio.h} and 
is not repeated here.

The structure includes a few integer fields and an array of
@code{struct timespec}.  Such structures define time stamps with
nanosecond precision, but the @i{simple-DIO} mezzanine and its gateware
are able to time-stamp input events and generate output events with a
resolution of 8ns.

When the device is asked to timestamp input events, the array of
@code{struct timespec} is used
to return such events to user space.  When the device is asked to
generate output pulses at specific points in time, the array is used
to pass three values: the beginning of the pulse, the duration of the
pulse and (optionally) the period of the pulse train.

Specifics about the use of individual fields are shown in the header
(in a big comment block), in the driver itself and in the user-space
programs that call @i{ioctl}.

In lab environments you may be concerned about the duration of the
@i{ioctl} implementation, because it sometimes seems to do more work
than needed. To verify whether we have an over-engineering problem in
kernel space, I provided a simple measurement of how much time is
spent in the @i{Ioctl} itself. The @i{make} variable
Jorge Machado's avatar
Jorge Machado committed
350
@code{WR_DIO_FLAGS} can be used to pass extra flags to the compiler,
351 352
and the macro @code{DIO_STAT} enables the time measurement.
Compiling with the following command thus enable such measurement
Jorge Machado's avatar
Jorge Machado committed
353
and associated @i{printk}
354 355

@example
Jorge Machado's avatar
Jorge Machado committed
356
   make WR_DIO_FLAGS=-DDIO_STAT
357 358 359 360 361 362 363 364 365 366 367 368
@end example

Previous versions of this manual described how to command pulses on
several channels with a single @i{ioctl} command, but that feature has
never been implemented (one of the reasons is that @i{ioctl} revealed
fast, so calling it several times is acceptable).


@c ==========================================================================
@node FMC-DIO Command Tool
@section FMC-DIO Command Tool

Jorge Machado's avatar
Jorge Machado committed
369
In the @file{sw/tools/} subdirectory of this project, you find the
370 371 372 373 374
@file{wr-dio-cmd} program, which is a command-line interface to the
@i{ioctl} command that acts on the @i{simple-DIO} mezzanine card.  Other
@code{wr-dio-} tools are provided (and described below) but this is
the most generic one.

Jorge Machado's avatar
Jorge Machado committed
375
@b{Please note that neither timestamping nor pulse generation work
376
if the WR core is not running or has an invalid time: it must either be
Jorge Machado's avatar
Jorge Machado committed
377
a master or a synchronized slave.}
378 379 380 381 382 383 384 385 386 387

Moreover, please note that this tool is just a demonstration to quickly
test the I/O features of the device (and for me to verify the kernel
part is actually working): for serious use you should call
@i{ioctl} by yourself with proper arguments, and avoid all the parsing
of ASCII and repeated invocation of this program.

This is the general syntax of the command:

@example
Jorge Machado's avatar
Jorge Machado committed
388
   wr-dio-cmd <devname> <cmd> [<arg> ...]
389 390 391 392 393
@end example

The arguments have the following meaning

@table @code
Jorge Machado's avatar
Jorge Machado committed
394
@item devname
395

Jorge Machado's avatar
Jorge Machado committed
396
	The name of the device, most likely @code{/dev/fmc-dio-1:0}
397
        (if you have more than one SPEC card, the other interfaces are
Jorge Machado's avatar
Jorge Machado committed
398
        called @code{/dev/fmc-dio-1:1}, @code{/dev/fmc-dio-1:2} and so on).
399 400 401 402 403 404 405 406 407 408 409 410 411

@item cmd

	The specific command. Supported commands are listed below.
        Each command takes zero or more of arguments. If you pass
        a wrong number of arguments you'll get help, and
        if one argument is wrong (e.g., not a number) the
	error message is meant to be directly helpful.

@end table

@b{Note}: This command, like everything else in this package, numbers
channels from 0 through 5, whereas the back panel of the mezzanine
412
numbers them 1 through 5. The last channel does not represent a physical channel,
413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440
it is a virtual channel to generate interrupts. 

The current version of the tool supports the following commands:

@table @code

@item stamp [<channel>] [wait]
@itemx stampm [<channel-mask>]

	The commands are used to retrieve timestamps from the card.
        If no arguments are passed, the tool reports to @i{stdout} all
        timestamps for all channels (they are ordered by channel, not
        by time). If one integer argument is passed, it can be a channel
        number in the range 0 to 5 (@code{stamp} command) or a mask
        in the range 0 to 0x3f (@code{stampm} command). If getting stamps
        for an individual channel, you can add the @code{wait} option
        to have the tool wait for (and report) new timestamps until killed.
        @b{Warning}: use of @code{wait} is dangerous because it has
        been implemented against the rules. You must
        terminate any waiting process before you unload the driver, or
        your PC will explode and will destroy your academic career.

@item pulse <channel> <duration> <when> [<period> <count>]

	Channel is an integer in the range 0 to 4. The duration must
        be specified as a fraction of a second (decimal number, less than
        one second), the @code{when} argument can be the string @code{now},
        an absolute time (@code{<seconds>.<fraction>}) or a relative
Jorge Machado's avatar
Jorge Machado committed
441
        time (@code{+<seconds>.<fraction>}). In the last case, the
442 443
        current second is added to @code{<seconds>} while the fraction
        is used unchanged. The @code{+} form is useful for simple checks with
Jorge Machado's avatar
Jorge Machado committed
444
        visual inspection. The @code{period} parameter, if specified, requests for
445
        a pulse train, with the specified time period between raising edges;
Jorge Machado's avatar
Jorge Machado committed
446
        the @code{count} parameter is the number of times to run pulse generation (-1 means forever,
447
        0 means ``stop ongoing pulse generation'').
448 449 450 451

@item irq <when> [<period> <count>]

        This command only generates interrupts and does not generate any 
452
        physical pulse. The @code{when} argument can be the string @code{now}, 
Jorge Machado's avatar
Jorge Machado committed
453
        an absolute time (@code{<seconds>.<fraction>}) or a relative time 
Jorge Machado's avatar
Jorge Machado committed
454
        (@code{+<seconds>.<fraction>}). In the last case, the current second 
455 456
        is added to @code{<seconds>} while the fraction is used unchanged. 
        The @code{+} form is useful for simple checks with visual inspection.  
Jorge Machado's avatar
Jorge Machado committed
457 458 459
        The @code{period} parameter, if specified, requests for a interrupt train, 
        with the specified time period between interrupts. It must be specified in 
        a range between 1ms and 2s; the @code{count} parameter is the number 
460
        of times to run interrupt generation (-1 means forever, 0 means ``stop ongoing interrupt generation'').
461 462 463 464 465 466 467 468 469

@item mode <channel> <mode> [<channel> <mode> ...]
@itemx mode <m0><m1><m2><m3><m4>

	Configure one or more channels for the specified mode. Each mode
        is represented by one character; the latter form above thus
        configures all 5 channels by means of a 5-bytes-long string,
        each characters specifies a mode according to the next table.
        The irq channel can not be configured with this command, it is
470
        should be configured with the @code{irq} command.
471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493

This is the list of supported modes for channels:

@table @code

@item I i

	Input mode, with termination or without termination. Uppercase
        forces the termination on (the rationale is you need a stronger
        signal to drive the input). Termination is 50 Ohm. Pulses on
        an input channel are timestamped.

@item 0 1

	Digital output mode, from the GPIO logic core, resp. low and high.
	Both modes disable the termination resistor.

@item D d

	DIO core output (not the GPIO core). With this mode the channel
        is driven by the WR-aware digital I/O, and can thus generate pulses
        and so on.  Uppercase enables the termination.
        A channel managed by the DIO core is normally low and can
Jorge Machado's avatar
Jorge Machado committed
494
        pulse high on request (see @code{pulse})
495 496 497 498 499 500
        command. The termination resistor doesn't make much sense for
        output, but the code is provided for consistency with input modes.

@item P p (Channel 0)

	Pulse per second output from the White Rabbit core that can be used
Jorge Machado's avatar
Jorge Machado committed
501
	only for the first channel (ch0). This @i{PPS} is sharper in its absolute 
502 503 504 505 506 507 508 509
	time than the one that can be
        generated by software using DIO pulses. Again, upper case
        selects the termination, for symmetry with input modes.

@item C c (Channel 4)

	``Clock'' input.  This mode (with or without termination) is
        used to feed a clock to the White Rabbit core (currently the
Jorge Machado's avatar
Jorge Machado committed
510
        WR core supports a 10MHz input on channel 4 (the last one)).
511 512 513 514 515 516
        For other channel, the mode is just like @t{I} or @t{i} but
        without timestamping (and thus without a software interrupt).

@end table


Jorge Machado's avatar
Jorge Machado committed
517 518 519 520 521 522 523 524 525
@item update_width <channel> <new_width>

        This command updates the width of the last pulse train programmed.
        The duration must be specified as a fraction of a second (decimal number, 
        less than one second). The command reads the previous period and checks if
        the new width is compatible.

@item mask_irq <channel> <Y/y>
        
526 527 528 529 530 531 532
        This command disables interrupt coming from the selected channel if the second
        argument is 'y' and enables it if the argument is 'Y'. In normal operation
        (if not disabled), an interrupt is generated on a channel (0-4) when a pulse
        is received or transmitted on this channel. For example, if channel 0 is 
        configured to generate 1 Pulse Per Second, an interrupt is generated every 
        time 1 PPS is generated on this channel. This command allows to disable
        such generation of interrupts.
Jorge Machado's avatar
Jorge Machado committed
533 534 535 536 537 538



@end table


539 540 541 542 543 544
@b{Note}: The first channel (channel 0) has been modified and now support only the
@t{P}/@t{p} as output mode. You will not be able to use @t{D},@t{d},@t{1},@t{0} modes.

@b{Note}: At startup, the DIO channels are configured by default as:
@code{wr-dio-cmd mode p00ic}. 

545 546

The new hardware pulse generation feature allows to configure a pulse train with a higher 
Jorge Machado's avatar
Jorge Machado committed
547
frequency than the previous version. However, there is still a software limitation due to the 
Jorge Machado's avatar
Jorge Machado committed
548 549 550 551 552 553
interrupts handling. If the interrupt of the channel is disabled, the pulse train generation can 
be as fast as the reference clock period allows, but there will be no timestamps in the FIFO and 
the pulse train will be indefinite because the pulse counting mechanism remains excecuted in the 
sofware interrupt handler. This frequency should be controlled in a proper way just to avoid PC 
freezing. Because of this, the maximum frequency depends on your current machine configuration 
and can be easily characterized with the existing tools for fmc-dio project.
554 555 556 557 558 559 560 561

There is no command to flush the timestamp FIFOs, but you can
always ``@code{wr-dio-cmd stamp > /dev/null}'' if needed.

Example uses of the tool follow:

@example
   # Pulse channel 4 for 0.1 seconds now
562
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .1 now
563 564

   # Pulse for 10 microseconds in the middle of the next second
565
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .00001 +1.5
566 567

   # Pulse for 1ms at 17:00 today
568
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .001 $(date +%s --date 17:00)
569 570

   # Get timestamps for the output events above
571
   sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
572 573

   # Make a train of 5 pulses, 0.5ms wide, every ms at next second
574
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 0.0005 +1 .001 5
575 576

   # Configure channel 0 as input with termination, 1 as input, 4 as low
577
   sudo wr-dio-cmd /dev/fmc-dio-1:0 mode Ii--0
578 579

   # Generate interrupt now
580
   sudo wr-dio-cmd /dev/fmc-dio-1:0 irq now
581

Jorge Machado's avatar
Jorge Machado committed
582
   # Generate a train of 100 interrupts every 10 ms starting at the next second
583
   sudo wr-dio-cmd /dev/fmc-dio-1:0 irq +1 .01 100
Jorge Machado's avatar
Jorge Machado committed
584 585

   # Make a train of indefinite pulses, 1ms wide every 5ms
586
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 1 0.001 now 0.005 -1
Jorge Machado's avatar
Jorge Machado committed
587 588

   # Stop the previous pulse train
589
   sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 1 0.001 now 0.005 0
Jorge Machado's avatar
Jorge Machado committed
590 591

   # Update pulse width in channel 1 to 10ms
592
   sudo wr-dio-cmd /dev/fmc-dio-1:0 update_width 1 0.01 
Jorge Machado's avatar
Jorge Machado committed
593 594

   # Disable interrupts in channel 0
595
   sudo wr-dio-cmd /dev/fmc-dio-1:0 mask_irq 0 y
Jorge Machado's avatar
Jorge Machado committed
596 597

   # Enable interrupts in channel 2
598
   sudo wr-dio-cmd /dev/fmc-dio-1:0 mask_irq 2 Y
599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614
@end example


@c ==========================================================================
@node Timestamping Fast Input Signals
@section Timestamping Fast Input Signals

When timestamping pulses in the @i{simple-DIO} mezzanine board, an
interrupt is generated to notify the driver that a new timestamp is
pending.  On recent computers this works reliably up to more than
100kHz, but clearly there is a point where the system locks up,
because it spends all of its time in interrupt handling.

This problem is transient: as soon as you remove the offending cable
the system recovers. However, you need a 10MHz input signal if you
want to run your SPEC device to be a White Rabbit @i{grandmaster}. In
615
order to support that, the driver automatically disables DIO interrupts when the
616 617 618 619 620 621 622 623 624 625 626 627 628 629
time spent in interrupt management exceeds 80% of the total time,
averaged over one thousand interrupt events.  Ethernet interrupts
are not affected.  The fact is reported by a kernel message, using
the PCI address of the card that triggered the problem.

@example
  spec 0000:04:00.0: DIO irq takes > 80% CPU time: disabling
@end example

This choice allows stamping your pulse trains up to a few dozen
kilohertz and still be able to feed higher frequencies without manual
intervention.  However, after DIO interrupts are disabled, the only
way to re-enable them is removing and reloading the device driver.

630 631 632 633 634 635 636 637
On the other hand, you can manually disable interrupts on a channel where
you expect high frequency input signals by using the mask_irq command, see
previous section.

@b{Note}: If you run two SPEC cards, and one is fed with high frequency
pulses without disabling the interrupts manually, it may happen that interrupts
are automatically disabled on both boards. The safeguard is currently not 
very refined, as it was implemented in a hurry.
638 639 640 641 642 643 644 645 646 647 648 649

@c ==========================================================================
@node WR-DIO Pulse per Second
@section WR-DIO Pulse per Second

To better show how to write your own application with the SPEC driver
and the @i{simple-DIO} mezzanine card, this package includes
@file{wr-dio-pps}, which features a very small and readable source
file.

The program is meant as a source code example, more than a real PPS
signal. If you need a real WR-driven pulse-per-second, you should
Jorge Machado's avatar
Jorge Machado committed
650
use the channel0 which is "hard-wired" to the PTP core and can be configured by 
651 652
executing: ``@t{wr-dio-cmd mode 0 p}''.

Jorge Machado's avatar
Jorge Machado committed
653 654
The program just fires a 1ms-long @i{PPS} pulse on one of the output
channels. The device name defaults to @i{wri0} but can specify a different one;
655 656 657 658 659 660 661 662 663 664 665 666
the channel number is a mandatory argument.

@b{Note}: This command, like everything else in this package, numbers
channels from 0 through 4, whereas the back panel of the mezzanine
numbers them 1 through 5.  My devices have no panel, so I just made
the wrong guess.

@example
   # run pps on channel 2 of the default SPEC card
   ./wr-dio-pps 2

   # run pps on channel 0 of the "second" card
Jorge Machado's avatar
Jorge Machado committed
667
   ./wr-dio-pps wri1 0
668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720
@end example

The following two figures show two such @i{pulse-per-second} signals
retrieved from two different @i{simple-DIO} cards, connected by a 10km
roll of fiber, after syncing with @i{White Rabbit}.  In some cases, it
may happen that the leading edges differ by almost exactly 8ns; this
happens because in the @i{simple-DIO} all times are quantized by
8ns-long clock ticks. The differences in internal delays (which depend
on the carrier, the mezzanine and the FPGA binary), are not
self-measured and calibrated in this simple design and may appear in
the output after quantization.  A more complete experimental setup
would include calibration of the internal delays of the boards, like
the mechanism in place for the @i{fine-delay} mezzanine card (see
@url{http://www.ohwr.org/projects/fmc-delay-1ns-8cha} and
@url{http://www.ohwr.org/projects/fine-delay-sw}).

@sp 1
@center @image{two-pps-whole, 10cm}
@sp 1
@center @image{two-pps-detail, 10cm}
@sp 1

@c ==========================================================================
@node Distributing Output Pulses
@section Distributing Output Pulses

A typical application for @i{White Rabbit} (or any time
synchronization system) is being able to generate output signals at the
same time in different output boards; another typical application is
time stamping input events.

By using the Ethernet interface included in the SPEC,
an application can exchange data with other @i{White
Rabbit} devices; thus, it can easily request output event to
other output peripherals, or collect remote input events.
The tool-set offered by the driver
is made up of the the @code{PRIV_MEZZANINE_CMD} @i{ioctl} command,
amd the usual Posix API for network communication.

The @i{DIO-specific} @i{ioctl} command is the one used by the
@code{wr-dio-cmd} tool described above, while network communication
should be known to most users of this package.  In order to ease new
@i{White Rabbit} users, though, this package includes sample code to
implement a simple dual-headed system with concurrent output.  The
examples are also meant to show the basic code that uses the provided
@i{ioctl} command, without all the boring parameter parsing that is
required in more generic tools like @code{wr-dio-cmd}. Because of this
simplification of parameter passing, the pulse width is hardwired to 1ms.

The example is made up of two programs: @code{wr-dio-agent} and
@code{wr-dio-ruler} (the former is a dumb actor, while the latter
states the rules). To keep things simple the two programs
assume that the SPEC is connected point-to-point to another SPEC
Jorge Machado's avatar
Jorge Machado committed
721 722
and both carry the @i{simple-DIO} mezzanine. @b{The wr-nic driver should
be installed to run this example @url{https://ohwr.org/project/wr-nic}}
723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746

Under this simplified assumptions, the @i{ruler} transmits raw Ethernet
frames to the broadcast address, while the @i{agent} receives almost
everything that appears on the cable.  This choice allows plugging two SPEC
cards in the same computer and run the example; if the example
used an IP-based protocol (like UDP) it couldn't be used
with two cards on a single PC -- and a fiber through them.
The simplification above, however, most likely
prevents the programs from working within a more complex
network topology.  I expect real @i{White Rabbit} users to add proper
network addressing in their applications.

If you have a single SPEC card, you can still use the @i{ruler} by
itself to mirror an input channel to an output channel of the same
card, with a specified delay.

@sp 1

The @i{agent} program silently listens to the network interface and
receives a data structure ready to be passed to @i{ioctl}.  Its only
command line argument is the name of the @i{White Rabbit} interface to
use (for most users it is @code{wr0}):

@example
Jorge Machado's avatar
Jorge Machado committed
747
   wr-dio-agent [-V] <wr-if> <dio-dev>
748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770
@end example

The @i{ruler} command, on the other hand, waits for timestamps
to appear on the specified input channel;
when notified about a positive-going edge, ot
replicates the edge on one or more outputs. Each output can be
local or remote, and can use a different delay from the input pulse.

If you lack an input signal, you can make an output pulse with
@code{wr-dio-pps} or other means and use it as a trigger. Please note
that the @i{ruler} does not configure the channel mode, so you might
want to use the @code{mode} command of @code{wr-dio-cmd} in advance.

The following command waits for events on channel 0 of the card connected
to @i{wr1}, and replicates the event with a delay of 1ms on channel 3 of
both the local and the remote card; it also replicates with a 2ms
delay to local channel 4. Please note that the delays should be no
more than the interval between input pulses, because the tools
reprograms all output triggers at each input event.

@b{Note}: As usual, channels are numbered 0 through 4.

@example
Jorge Machado's avatar
Jorge Machado committed
771
   wr-dio-ruler wr1 /dev/fmc-dio-1:0 IN0 L3+0.001 R3+0.001 L4+0.002
772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797
@end example

There is no sample code that generates trains of pulses as a response
to events, nor support for other than 1ms-long output pulses; anyways,
the code is
thoroughly commented in order to serve as a starting point for
more complex lab environments.

As a final remark, please note that all pulse generation is driven by
host hardware, after an hardware interrupt reports the input event, 
but the software controls the number of number of pulses that has been
generated.
For this reason, you'll not be able to reliably replicate pulses with
delays smaller than a few hundred microseconds, depending on the
processing power of your computer and the load introduced by other
processes.  For remote connections, you must also count the overhead
of network communication as well as transmission delays over
the fiber (a 10km fiber introduces a
delay of 50 microseconds). 

The following example shows use of the @i{ruler} and @i{agent} on
two hosts, called @code{spusa} and @code{tornado}. The input events
on @code{spusa} are replicated to one local channel and two remote channels,
with a delay of 1ms. The input events in this case are from a @i{pulse-per-second} signal:

@smallexample
Jorge Machado's avatar
Jorge Machado committed
798
   tornado.root# /tmp/wr-dio-agent wr0 /dev/fmc-dio-1:0 &
799

800
   spusa.root# sudo wr-dio-ruler wr1 /dev/fmc-dio-1:0 IN4 L3+.001 R4+.001 R2+.001
801 802 803 804 805 806
   wr-dio-ruler: configured for  local channel 3, delay 0.001000000
   wr-dio-ruler: configured for remote channel 4, delay 0.001000000
   wr-dio-ruler: configured for remote channel 2, delay 0.001000000

   [... wait a few seconds ...]

807
   spusa.root# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
808 809 810 811
   ch 3,       385.001000000
   ch 3,       386.001000000
   ch 3,       387.001000000
   ch 3,       388.001000000
812
   tornado.root# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 2
813 814 815 816
   ch 2,       385.001000000
   ch 2,       386.001000000
   ch 2,       387.001000000
   ch 2,       388.001000000
817
   tornado.root# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
818 819 820 821 822 823
   ch 4,       385.001000000
   ch 4,       386.001000000
   ch 4,       387.001000000
   ch 4,       388.001000000
@end smallexample

Jorge Machado's avatar
Jorge Machado committed
824 825 826 827
@c ##########################################################################
@node Interrupt Demo
@chapter Interrupt Demo

Jorge Machado's avatar
Jorge Machado committed
828
The interrupt demo software can be found in the @code{sw/irq-demo} directory.
Jorge Machado's avatar
Jorge Machado committed
829 830 831 832 833 834 835 836 837 838 839 840
This demo sofware disables the interrupts of all channel except the new one, only 
dedicated to interrupts. 
By default, it setups the interrupt with 100 milliseconds between them.

The tool provides the total number of interrupts, the interrupts per second and 
some statistic information calculated with the difference between OS time and 
WR time.

This is the general syntax of the command:

@table @code

841
@item irq-demo -f <dio-device> [-p <period in ns> -c <MONOTONIC/REALTIME>]
Jorge Machado's avatar
Jorge Machado committed
842 843 844 845 846 847 848

@end table

Example uses of the tool follow:

@example
   # Generate one interrupt each 100 ms
849
   sudo irq-demo -f /dev/fmc-dio-1:0
Jorge Machado's avatar
Jorge Machado committed
850 851

   # Generate one interrupt each 10 ms
852
   sudo irq-demo -f /dev/fmc-dio-1:0 -p 10000000
Jorge Machado's avatar
Jorge Machado committed
853 854
@end example

855 856 857 858 859 860
@c ##########################################################################
@node Bugs and Missing Features
@chapter Bugs and Missing Features

@itemize @bullet

Jorge Machado's avatar
Jorge Machado committed
861 862 863 864 865 866
@item Identification of the mezzanine is completely missing; every @i{fmc}
driver at this point takes hold of every device. We are working on this,
and the next version of fmc-dio will support identification, with a flag
to run without identification for users whose EEPROM has not been programmed.


867 868 869 870 871
@item The NIC driver should directly support setting the White Rabbit
mode for each card (grandmaster, free-running master or slave). This
will be supported at module load time, not at runtime (for that please
use the UART).

Jorge Machado's avatar
Jorge Machado committed
872
@item DIO support is missing some of the features listed
873 874 875 876 877 878 879 880 881 882 883 884 885 886
in @file{wr-dio.h} (i.e. DAC control)>

@item Locking in kernel code should be verified with a serious audit
effort. There are no known issues at this point, but some code may
be made safer.

@end itemize

@c ##########################################################################
@node Portability
@chapter Portability

This package should be portable. However I didn't test it on a wide
variety of systems.  Currently most of my use is
Jorge Machado's avatar
Jorge Machado committed
887
on a 64-bit x86 host, running version between 4.15-4.18 of the kernel.
888 889 890 891 892 893 894 895 896 897

@c ##########################################################################
@bye


@c  LocalWords:  gnudd titlepage iftex texinfo CERN documentlanguage settitle
@c  LocalWords:  documentencoding setfilename afourpaper paragraphindent EEPROM
@c  LocalWords:  setchapternewpage finalout eeprom gateware devmem devfn busid
@c  LocalWords:  speclib Gennum timestamps stampm ifname timespec timestamp
@c  LocalWords:  timestamping linux FPGA ohwr http