\input texinfo    @c -*-texinfo-*-
%
% fine-delay.in - main file for the documentation
%
%%%%

%------------------------------------------------------------------------------
%
%                         NOTE FOR THE UNAWARE USER
%                         =========================
%
%    This file is a texinfo source. It isn't the binary file of some strange
%    editor of mine. If you want ASCII, you should "make fine-delay.txt".
%
%------------------------------------------------------------------------------

%
% This is not a conventional info file...
% I use three extra features:
%   - The '%' as a comment marker, if at beginning of line ("\%" -> "%")
%   - leading blanks are allowed (this is something I can't live without)
%   - braces are automatically escaped when they appear in example blocks
%

@comment %**start of header
@documentlanguage en
@documentencoding UTF-8
@setfilename fine-delay.info
@settitle fine-delay
@iftex
@afourpaper
@end iftex
@paragraphindent 3
@comment %**end of header

@setchapternewpage on

@set update-month December 2012

@finalout

@titlepage
@title Fine Delay Design Notes
@subtitle June 2013
@author CERN BE-CO-HT / Tomasz Włostowski

@end titlepage
@headings single


@iftex
@contents
@end iftex

@c ##########################################################################

@unnumbered Revision history
@multitable  @columnfractions .10 .15 .25 .50
@headitem Revision @tab Date @tab Author @tab Changes
@item 1 @tab 03.07.2013 @tab Tomasz Włostowski @tab Initial version
@end multitable


@page

@node Top
@chapter Introduction

This document contains some detailed information on the hardware design of the
Fine Delay Mezzanine (also called FmcDelay1ns4cha and further abbeviated as the FD) and its VHDL core.
It is not very useful for the FD's users and it is certainly not formal. 
Its target are driver developers, carrier/hardware integrators,
people interested in building similar devices and looking for hints and inspiration
or the folks curious why something was done in that and not another way. It also explains
things that are not obvious in the VHDL/test program code, such as the calibration mechanisms
and Acam's TDC quirks.

The hardware/HDL description contains very frequent references to the card's schematics [1], PCB design and VHDL sources [3]. It is a good idea to print or open them before continuing reading.

Note that this description refers to the latest version of the hardware, that is EDA-02267-V5-2. It can be considered almost accurate for the versions since V3-0. Future and past improvements to the hardware can be found in the @i{Issues} section of the project's Wiki [3].

@page
@chapter The Hardware

@section Overview

The FD is a VITA-57 LPC FPGA Mezzanine card, whose basic function is taking TTL pulses and reproducing them on one or more of four TTL
outputs after a given time. Delay can be programmed to any value between 600 ns and 12 seconds with 10 ps resolution. 
It is also possible to control width, spacing and repetition rate of the output pulses. 
The exact, detailed specifications can be found in the User's Manual.

@float Figure,fig:fd_block
@center @image{drawings/block_diagram, 15cm,,,.pdf}
@caption{Block diagram of a FD card.}
@end float

@ref{fig:fd_block} depicts a detailed hardware block diagram of the FD. The major design blocks are:
@itemize
@item The TDC, built with the Acam's TDC-GPX chip.
@item Output stages, based on LVPECL '195 programmable delay chips.
@item Clock distribution circuit, encompassing a multi-output PLL synthesizer (AD9516).
@item Power supplies, SPI general-purpose IO, sensors and ID EEPROM.
@end itemize

@float Figure,fig:fd_simple
@center @image{drawings/simple_diagram, 15cm,,,.pdf}
@caption{Simplified principle of FD operation.}
@end float

The FD's principle of operation is explained in @ref{fig:fd_simple}. The card time tags an input pulse (using a Time-to-Digital converter),
adds the desired delay to the time tag and produces a pulse on the output when the internal time base counter hits the computed sum. The fine part (that is, less than a single clock cycle) is adjusted by an external programmable delay line.



@section Clock distribution

Relevant files: @code{clock_generator.SchDoc}

The FD requires a number of different clock signals to synchronize the TDC,
the output stages and the FPGA core altogether. All clocks are generated by the Analog Devices'
AD9516-4 integrated PLL/clock fanout (IC4). This particular chip was chosen due to its wide 
configuration capabilities (frequency settings, fine per-output phase adjustment), support 
for multiple I/O standards (PECL, single-ended, etc) and low inter-output skew. 
The PLL outputs are programmed as follows:
@itemize
@item OUT9: 125 MHz FPGA reference clock (LVDS). The choice of 125 MHz reference is forced 
by compliance with White Rabbit and Gigabit Ethernet for distributed, 
sub-nanosecond synchronization of multiple cards. All other clocks used in the design 
are derived from 125@tie{}MHz.
@item OUT0..3: 250 MHz clocks that drive the output flip flops. The frequency value 
comes from the functional requirement for generation of 10 MHz, WR-aligned clock - 250 MHz is the 
smallest common multiple of both 10 MHz and 125 MHz WR clock. Note that outputs 0, 1 and 2 are inverted to
simplify PCB routing (this is compensated by AD9516's programmable output polarity control).
@item OUT7: 31.25 Mhz TDC reference clock (LVCMOS). Must be lower than the TDC maximum 
reference frequency (40 MHz) and an integer fraction of 125 MHz. 31.25 MHz is the largest
possible value. Low value would significantly reduce TDC throughput and increase timestamping latency, as it clocks the Acam's internal timestamp processing pipeline.
@item OUT4, OUT5: 7.8125 MHz TDC start signals (LVPECL to the TDC, LVCMOS to the FPGA). 
Rising edges of these clocks are the reference points for time interval measurement in the TDC. The TDC start signal
is further divided by two (by toggling TDC's @code{StartDis} pin) to avoid exceeding Acam's maximum start frequency of 7 MHz. Unfortunately it
is not possible to achieve higher division ratios directly in the PLL chip.
@end itemize

The AD9516's PLL bandwidth is set to approx. 10 kHz by the loop filter components (R41, C33 and neighbours), whose values were calculated using Analog Devices @i{AdiSimClk} software [11] with default performance settings (the jitter of the TDC and output stages is an order of magnitude larger than worst-case jitter of the AD9516 PLL). The PLL is referenced to a 25 MHz VCTCXO OSC5, a Mercury Crystal VM53S3-series oscillator. The TCXO can be digitally
tuned within (±10 ppm) range by the 16-bit  DAC IC14 (AD5662). Low-cost shunt regulator IC10 
(LM336) provides the reference voltage for the DAC (it needs not be extremely stable because the 
whole circuit usually works in a feedback loop, see @ref{fig:wr_pll}). The combination 
of the used DAC and oscillator meets the requirements of Synchronous Ethernet and White Rabbit 
for synchronization: 1kHz PLL bandwidth and tuning sensitivity of < 1 ppb and range of > ±2.5 ppm. 
Careful readers should have noticed at this point that it is not possible to directly feed an 
external reference clock to the card. This limitation is caused by the lack of carrier to 
mezzanine clock signals in the low pin version of an FMC connector and is solved by locking 
the cards' clock to the the external frequency with a PLL implemented in the carrier FPGA. 

The PLL is powered from 3.3V filtered by an LC circuit (L3 and its surroundings), following 
manufacturer's guidelines. Programming is done via the SPI bus, directly connected 
to the FPGA. A multipurpose STATUS signal is also routed to the FPGA, mainly for fast 
loss of lock detection.

@section Input stage

Relevant files: @code{input_stage.SchDoc}

The role of the input stage is to adapt the incoming trigger pulses so that they can be digested 
by the TDC and the FPGA. Following the signal path starting from the input connector:
@itemize
@item fuse F5 protects the input stage against a serious overvoltage/overcurrent 
(e.g. connecting the input to a 12 V DC power supply).
@item resistors R110, R116, R117 along with the MOSFET T2 constitute a programmable 
50 Ohm termination. 3 resistors connected in parallel were used to give more freedom
for the PCB designer (the board is packed quite tightly) and ease power dissipation. 
R76 ensures calibration mode is off by default.
@item PIN diodes D6 (BAR66) and resistor R57 form a fast overvoltage clamping circuit. 
R57 reduces the D6's clamping current (~200 mA) for small overloads, clamping currents 
above 200 mA will anyway blow the fuse F5. R108 pulls down the input, lowering its impedance 
and preventing an unconnected input from taking glitches/EMI as legitimate trigger pulses.
@item FET switch IC18 (TS3USB221) selects the signal that drives the TDC input 
between the trigger input and a calibration line driven by the FPGA (the calibration process
will be discussed later). @code{TRIG_SEL} line selects the active input (by default, it's the LEMO trigger connector).
@item The output of the switch IC18 feeds three other components: an LVTTL to LVPECL buffer (IC5, 100EPT29), 
which drives the TDC's Stop input, an LVCMOS buffer (IC23, LVC125) that feeds the trigger signal
to the FPGA and a single-gate D flip flop which takes part in the DDMTD calibration. The physical length of the signal path between these components 
is very short (5mm on the PCB) to avoid stubs and reflections.
@end itemize
 
@section The TDC

Relevant files: @code{acam_tdc.SchDoc}

The TDC, just as its name says, converts an incoming pulse into a digital value, denoting the 
moment in time at which the pulse's rising (or falling) edge occured... Well, that is not entirely 
true. In The Real World, a TDC must be provided with a notion of time, so its output is the time difference 
between two inputs, called Start and Stop. The start input is used to provide the time base 
(i.e. pulses occuring at well defined moments in time) and the stop input takes the
pulses to be timestamped.

The FD's TDC is a single-chip solution, called TDC-GPX, made by Acam (IC8) [5]. It can simultaneously 
timestamp from 1 to 8 inputs, with accuracy and repetition rate depending on the mode of 
operation (R, I, M, G-modes, more details in the TDC-GPX datasheet). In the FD, the Acam serves two purposes:
@itemize
@item The obvious one: @b{time tagging trigger pulses}. The TDC in configured in the G-mode, with a single
stop input, providing 7 Mpulses/second throughput, measurement range of 40 us and resolution (one-sigma) 
of 42 ps. In this mode, LVPECL inputs are used, the Start being connected to the 7.125 MHz PLL 
clock (divided internally by 2), and the Stop feeding in trigger pulses. As a result, we get the time difference between 
the rising edge of the most recent start pulse and the stop pulse - for a (7.125 / 2) MHz start event
frequency, it gives a measurement range of 0 to 256 ns. The coarse part of the timestamp is 
produced with a counter inside the FPGA and the two values are merged together to obtain the 
final, accurate value (@xref{Timestamp postprocessing}).

@item @b{Calibration} of the output delay lines, done during initialization 
of the card. Its goal is to determine the setpoint for each delay line that results with a delay of exactly 8 ns 
(single reference clock cycle), and thus compensates the @i{Process-Voltage-Temperature} (PVT) effects. The TDC is working in the I-mode 
(81 ps resolution), with one LVTTL start connected to an FPGA output generating arbitrary pulses and 
the 4 LVTTL stop inputs wired to the outputs of the delay chips (@xref{Output stage calibration}). 
@end itemize

The interesting part about the TDC is how it measures time differences so accurately - obviously, 
it does not have a counter running at 1/82 ps (approx. 12 GHz). Instead it employs a tapped delay line (see @ref{fig:tdc_principle}), 
fabricated in silicon as a series of identical buffers or inverters. The input of the line is connected 
to the Start signal, while the output of each buffer drives a latch disabled by the Stop signal. 
The later the Stop signal comes after Start, the more ones will be latched in, and the higher 
will be the output value, at least in @i{The Ideal World}. 

@float Figure,fig:tdc_principle
@center @image{drawings/tdc_principle, 8cm,,,.pdf}
@caption{Simplified schematic of a tapped delay line TDC.}
@end float

Unfortunately, in @i{The Real World}, PVT 
effects come into play, causing the delay introduced by each buffer to vary with temperature, 
voltage and between different chips. The TDC by Acam employs a clever trick to compensate for this 
delay spread. It has another delay line, with more or less identical silicon layout but with 
positive feedback, turning it into a ring oscillator. The frequency of this oscillator 
is continuously measured and compared against a reference value corresponding to the desired 
bin size. The resulting error signal drives a servo that controls the voltage powering both the 
oscillator and the measurement delay line(s) in such way that bin size stays constant (assuming 
that delays introduced by each of the buffers scale with very similar factors). This explains 
why the power circuitry for the TDC is so complex (3 LM1117 LDO regulators - IC6, IC89 and IC21). 
The servo output signal @code{PHASE} is PWM-modulated and after filtering in R17-C22-R16 and R20-C42-R22 
circuits, set bias voltage on the ADJ pins of the regulators, which directly determine
their output voltages (Vadj = Vout - 1.25@tie{}V). The values of the voltage divider components and 
large number of decoupling capacitors come from the TDC-GPX reference design provided by Acam. 

The TDC communicates with the FPGA using a simple asynchronous address/data bus, with 4 address 
bits and 28 data bits. Aside from standard bus lines (@code{CS}, @code{RD}, @code{WR}), the FPGA drives the TDC FIFO 
purge signal (@code{TDC_ALUTRIGGER}) and receives the Timestamp FIFO Empty and Error flags (@code{TDC_EF} and @code{TDC_ERR}).
Series resistors on TDC data lines are provided to match the impedance of the PCB traces, 
improving signal integrity and EMC performance. 

Note that the address inputs of the TDC are not driven by the FPGA, but by the SPI GPIO controller (MC23S17, IC19).
This is due to lack of free pins in the FMC connector. Fortunately, the address value needs to be modified
only when setting up the TDC and it stays constant during data readout, having no impact on performance.

@section Output stages

Relevant files: @code{delay_channel.SchDoc}, @code{output_driver.SchDoc}, @code{drv_single_channel.SchDoc}.

The role of the FD's output stages is to:
@itemize
@item Reduce the jitter and de-skew coarse output pulses produced by the FPGA.
@item Adjust the fine part of the delay.
@item Properly drive a 50 Ohm load.
@end itemize

The first task is done by the discrete LVPECL flip-flops (IC3). The D inputs of the FFs are
connected to FPGA pins (output counter comparators), while the CLK inputs supply four low-skew
250 MHz clocks, synchronous with the FPGA's reference 125 MHz clock. This way, poor quality
pulses produced by the FPGA are retimed and deskewed, resulting with an output-to-output skew
of less than 100 ps and jitter level comparable to noise of the PLL chip. 

Retimed output signals are fed to the delay lines (IC2, IC7). A SY89295 chip from
Micrel was used, mainly because of availabilty in QFN packages (PCB space constraints). 
The delay lines are configured by outputting the number of delay taps on @code{D0..9} and latching it in by asserting @code{LEN} input low. Value @code{0} corresponds to 2.2 ns and @code{1023} to 12.5 ns, giving
quite a lot of headroom (we need a 4 ns range). Delays are updated immediately after laching in a new value. All delay chips share
same data bus, which is arbitrated by the FPGA (again, due to lack of pins in the FMC connector). The price is higher minimum possible delay setpoint. Resistors R11 and R53 select the signalling
level for the control inputs (LVTTL). 

The differential signal is converted back to LVTTL by IC1 (SY100EPT23) and fed to a single ended driver IC25,
built using a fast high current operational amplifier AD8009. Resistors R69 and R71 set the gain to get 6 V peak
level on a high impedance load. R75 ensures proper biasing of the bipolar output stage of IC1. The opamp is powered from
a dedicated switched mode power supply (+8 V/-2 V). Amplified signal passes through the SSR switch SW1, serving the purpose 
of enabling/disabling the output in a rock-solid, glitch-proof way. D4 / F1 circuit provides ESD/overcurrent/overvoltage protection. R72, together with
the opamp's output and SSR's series impedances form a 50 Ohm source termination. R65 forces low state on the output when the card is disabled. 
The lowpass circuit R42/C49 makes sure the SSR is switched without glitching. IC26 buffers the output signal for driving feedback TDC input (calibration).
The D flip flop IC27 is a part of DDMTD-based calibration circuit. R66 and R34 constitute a voltage divider, bringing down the 6 V output of the opamp to a level
that is acceptable by LVC/AUP logic. Yet again, due to lack of pins, calibration flip flop outputs are ANDed together, making a trivial multiplexer (one output is calibrated at a time,
while the rest is driven to 1). The output stage meets the edge rising time requirement of 2 V/ns (thanks to the very high output voltage rise speed of AD8009 of 5.5 kV/us) for both 50 Ohm and high impedance loads and is capable of producing neat, clean pulses of 3 V level on 50 Ohms, suitable for directly driving TTL inputs (see @ref{fig:output_shape}).


@page
@float Figure,fig:output_shape
@center @image{drawings/output_shape, 10cm,,,.png}
@caption{Shape of an output pulse rising edge on a 50 Ohm load.}
@end float

@section Power supply

The FD takes power from the following lines in the FMC connector:
@itemize
@item P3V3, that is used for all LVPECL/LVTTL logic on the board, including the AD9516 and TDC I/O power. Approximate current consumption is 1.5 A.
@item P12V, used for powering the output stages and Acam's PLL. This rail draws approximately 200 mA.
@item P3V3_AUX, powering only the configuration EEPROM (less than 5 mA).
@end itemize

Aside from the standard FMC power lines, the FD contains two dedicated switching mode power converters:
@itemize
@item The buck converter IC11 for +8 V / 600 mA supply, powering the output stage opamps and Acam's PLL linear regulators. The TVS diode D7 provides additional overvoltage protection for the outputs (when a high current DC voltage is connected, D7 clips it to a safe level, because the P8V PSU is unable to sink current).
@item The inverting converter IC30, producing -2 V @ 600 mA, used solely for powering the output driver.
@end itemize

The P8V, P3V3 and VADJ supplies are monitored by a reset/voltage supervisor IC8 (TPS3307-33), which ensures that the card is un-reset only when all supply voltages have stabilized. The VADJ rail, even though it is not used to power any part of the FD, is monitored as it powers the FPGA LVTTL/LVDS drivers in the carrier and its failure would also render the mezzanine useless. The RC circuit (R78, D1, C57) determines the power-on-reset time and deglitches the reset signal coming from the FMC connector.

The average power dissipation of the FD is approximately 7 - 7.5 W, being large enough to heat the PCB to 60 - 70 @textdegree C in poorly cooled environments such as Kontron KISS PCs. Forced airflow is required. In order to increase the card reliability in high temperatures, all non-ceramic capacitors (single type in the project: 10 uF/16 V) are Sanyo Poscap TQC series, rated 200000 hours at 65 @textdegree C.
@page
@section Everything else

Relevant files: @code{FMC_Delay_1ns_4cha.SchDoc}.

Below are listed all components that didn't qualify to the previous sections:
@itemize
@item SPI GPIO expander (IC19 - MCP23S17): drives seldom changing digital signals, saving some pins in the FMC
plug for more important purposes. These signals are: TDC address lines, output stages SSR enable (4x),
calibration mode select and termination enable. See table below for GPIO pin mapping:
@multitable  @columnfractions .20 .80
@headitem GPIO pin @tab Signal
@item @code{A0} @tab Input termination enable (active high)
@item @code{A2} @tab Output 4 enable (active high)
@item @code{A3} @tab Output 3 enable (active high)
@item @code{A4} @tab Output 2 enable (active high)
@item @code{A5} @tab Output 1 enable (active high)
@item @code{A6} @tab Trigger select (0 = external, 1 = FPGA)
@item @code{B0} @tab Acam address bit @code{0}
@item @code{B1} @tab Acam address bit @code{1} 
@item @code{B2} @tab Acam address bit @code{2}
@item @code{B3} @tab Acam address bit @code{3}
@end multitable
@item Buffers (IC12, IC28, IC29 - LVC1G125): ensure correct operation of all SPI peripherals by 
adapting 2.5 V LVCMOS levels from the FPGA to 3.3 V LVTTL.
@item Buffers (IC32, IC31 - LVC1G125): boost output current & voltage levels of the FPGA for driving calibration
inputs (DDMTD clock and calibration TDC pulses).
@item 1-Wire temperature sensor (IC13, DS18B20U+): measures the temperature of the output delay lines 
(used by on-the-fly delay drift compensation) and gives each board a unique ID number.
@item Two LEDs and a configiuration EEPROM (IC22) - standard components of every FMC card. The EEPROM is also used for storage of calibration parameters.
@end itemize


@page
@chapter The VHDL

This chapter provides a brief description of the VHDL design of the FD core. For more detailed explanations, you may need to refer to the comments in the source code [3]. 

@section Core interface

The table below lists all I/O ports of the VHDL FD core and the corresponding nets on the mezzanine schematic, if applicable. Note that the exact pin locations are not provided here - in order to save your time and minimize chances of making an error, a script that generates an UCF file for different carriers is provided (in @code{scripts/ucfgen.py} directory).

@multitable  @columnfractions .20 .12 .55 .20
@headitem Port @tab Direction @tab Purpose @tab SCH net 
@item @code{clk_ref_0_i} @tab input @tab Reference clock (from AD9516), in-phase
@item @code{clk_ref_180_i} @tab input @tab Reference clock (from AD9516), inverted via FPGA's DCM/PLL. Used for DDR output drivers
@item @code{clk_sys_i} @tab input @tab System clock (Wishbone). Must be slower than @code{clk_ref_0_i}
@item @code{clk_dmtd_i} @tab input @tab DMTD clock from the WR Core (optional)
@item @code{rst_n_i} @tab input @tab Global core reset, active low
@item @code{dcm_reset_o} @tab output @tab Reset signal for the DCM/PLL produdcing DDR @code{clk_ref} clocks
@item @code{dcm_locked_i} @tab output @tab Lock detection signal from the DCM/PLL produdcing DDR @code{clk_ref} clocks
@item @code{trig_a_i} @tab input @tab FPGA trigger input @tab  @code{TRIG_TO_FPGA}
@item @code{tdc_cal_pulse_o} @tab output @tab FPGA calibration trigger output @tab @code{CAL_PULSE}
@item @code{tdc_start_i} @tab input @tab 7.8125 MHz TDC start signal @tab @code{FPGA_TDC_START}
@item @code{dmtd_fb_in_i} @tab input @tab DDMTD input calibration feedback @tab @code{DMTD_FB_IN}
@item @code{dmtd_fb_out_i} @tab input @tab DDMTD output calibration feedback @tab @code{DMTD_FB_OUT}
@item @code{dmtd_samp_o} @tab output @tab DDMTD calibration flip-flop clock @tab @code{DMTD_CLK}
@item @code{led_trig_o} @tab output @tab Trigger LED @tab @code{LED_TRIG}
@item @code{ext_rst_n_o} @tab output @tab FMC hardware reset line @tab @code{EXT_RESET_N}
@item @code{pll_status_i} @tab input @tab AD9516 STATUS pin @tab @code{PLL_STATUS}
@item @code{acam_d_o}, @code{acam_d_i}, @code{acam_d_oen_o} @tab tristate @tab Acam data bus. Tristate enable signal is active LOW @tab @code{TDC_D}
@item @code{acam_emptyf_i} @tab input @tab Acam empty flag @tab @code{TDC_EF}
@item @code{acam_alutrigger_o} @tab output @tab Acam ALU Trigger line (used as FIFO purge signal) @tab @code{TDC_ALUTRIGGER}
@item @code{acam_wr_n_o} @tab output @tab Acam write enable @tab @code{TDC_WRN}
@item @code{acam_rd_n_o} @tab output @tab Acam read enable @tab @code{TDC_RDN}
@item @code{acam_start_dis_o} @tab input @tab Acam Start Disable pin @tab @code{TDC_START_DIS}
@item @code{acam_stop_dis_o} @tab input @tab Acam Stop Disable pin @tab @code{TDC_STOP_DIS}
@item @code{spi_cs_dac_n_o} @tab output @tab AD5662 DAC SPI chip select @tab @code{SPI_DAC_CSN}
@item @code{spi_cs_pll_n_o} @tab output @tab AD9516 PLL SPI chip select @tab @code{SPI_PLL_CSN}
@item @code{spi_cs_gpio_n_o} @tab output @tab MCP23S17 GPIO SPI chip select @tab @code{SPI_IO_CSN}
@item @code{spi_sclk_o} @tab output @tab SPI clock @tab @code{SPI_SCK}
@item @code{spi_mosi_o} @tab output @tab SPI master data output @tab @code{SPI_MOSI}
@item @code{spi_miso_i} @tab input @tab SPI master data input @tab @code{SPI_MISO}
@item @code{delay_len_o} @tab output @tab SY89295 Latch Enable pins (0 = channel 1) @tab @code{DELAY_LEN}
@item @code{delay_pulse_o} @tab output @tab Coarse pulse outputs @tab @code{DIN}
@item @code{delay_val_o} @tab output @tab Number of fine delay taps @tab @code{DELAY_D}
@item @code{owr_en_o}, @code{owr_i} @tab open-drain @tab One-wire bus for the temperature sensor/ID chip. When @code{owr_en_o} is high, 1-wire bus is shorted to GND @tab @code{ONEWIRE}
@item @code{i2c_scl_o}, @code{i2c_scl_oen_o}, @code{i2c_scl_i} @tab tristate @tab I2C SCL line @tab @code{SCL}
@item @code{i2c_sda_o}, @code{i2c_sda_oen_o}, @code{i2c_sda_i} @tab tristate @tab I2C SDA line @tab @code{SDA}
@item @code{fmc_present_n_i} @tab input @tab FMC presence pin @tab @code{PRSNT_M2C_L}


@item @code{tm_link_up_i} @tab input @tab WR link state indication
@item @code{tm_time_valid_i} @tab input @tab WR timing validity indication
@item @code{tm_cycles_i} @tab input @tab WR 8 ns cycles counter
@item @code{tm_tai_i} @tab input @tab WR TAI seconds counter
@item @code{tm_clk_aux_lock_en_o} @tab output @tab Enables disciplining of @code{clk_ref_i} to WR clock
@item @code{tm_clk_aux_locked_i} @tab input @tab Indicates if @code{clk_ref_i} is locked to WR clock
@item @code{tm_clk_dmtd_locked_i} @tab input @tab Indicates lock status of @code{clk_dmtd_i}
@item @code{tm_dac_value_i} @tab input @tab WR servo tuning word for AD5662 DAC
@item @code{tm_dac_wr_i} @tab input @tab Valid indication for WR servo tuning word

@item @code{wb_XXX_i}, @code{wb_XXX_o} @tab in/out @tab Wishbone bus for control registers access

@item @code{tdc_seconds_o}, @code{tdc_cycles_o}, @code{tdc_frac_o} @tab output @tab 
Direct TDC timestamp output, for host-less environments. 
@item @code{tdc_valid_o} @tab output @tab 
Validity indication for direct timestamp output.

@item @code{outx_seconds_o}, @code{outx_cycles_i}, @code{outx_frac_i} @tab input @tab 
Direct pulse start input, for host-less environments.
@item @code{tdc_valid_o} @tab output @tab 
@code{1} latches in a trigger timestamp for a given output channel.

@end multitable

The core also requires a few generic parameters to be set up in order to work:

@multitable @columnfractions .30 .70
@headitem Parameter @tab Description
@item @code{g_with_wr_core} @tab Enables/disables White Rabbit support
@item @code{g_simulation} @tab Reduces internal timeouts and delays to speed up simulations. Synthesizing with @code{g_simulation} enabled will very likely produce a non-functional bitstream
@item @code{g_with_direct_timestamp_io} @tab Enables/disables direct timestamp I/O ports (@code{tdc_XXX_o} and @code{outx_XXX_i}). @b{When enabled, delay mode is not available!}
@item @code{g_interface_mode}, @code{g_address_granularity} @tab Wishbone slave mode (classic/pipelined) and address granularity (32-bit word or byte addressing).
@end multitable


@section Top Level
Relevant files: @code{fine_delay_core.vhd}, @code{fine_delay_pkg.vhd}.

@float Figure,fig:hdl_top
@center @image{drawings/fpga_diagram, 15cm,,,.pdf}
@caption{Block diagram of the FD VHDL core.}
@end float

The top level diagram of the FD core is shown in @ref{fig:hdl_top}. Its major components are:
@itemize
@item Acam timestamper unit, producing time tags for input pulses.
@item 4 FD channel drivers, which take these time tags, add the desired delay and produce a number pulses of given width and spacing.
@item Platform-specfic dual-edge (DDR) output flip-flops.
@item Ring buffer, providing timestamps of input/output pulses for the host system.
@item Time base and reset generators, calibration logic and peripheral I/O cores (e.g. Onewire).
@end itemize
All of these are accessible from the host via a Wishbone bus. There are six Wishbone slaves in the design: 4 output register banks (one per each channel driver),
the main register bank (shared between all other sub-cores) and a 3rd-party OneWire core. Custom register banks were generated using the @code{wbgen2} tool [12]. 
An SDB descriptor is provided for plug&play integration on the carrier with other cores [9]. 

Aside from the Wishbone registers, the FD core provides direct timestamp I/O ports, which can be used to easily collect timestamps and trigger output pulses from other cores in your design. Note that in order to use the direct I/O it is still necessary to program the core and the TDC via Wishbone. 

@subsection Clocks and time base
Relevant files: @code{fine_delay_core.vhd}, @code{fd_csync_generator.vhd}.

The FD core requires a number of clock signals to operate:
@itemize
@item @code{clk_ref_0_i} and @code{clk_ref_180_i}: Reference clock, coming from the AD9516 PLL. 
Since the FD's PLL provides only the @code{clk_ref_0_i} signal, the designer of the
carrier firmware must generate the shifted version with a platform-specific PLL/DCM component outside the FD core. This clock
references the entire pulse processing path (TDC core, output drivers, calibration logic & time base generator).
@item @code{clk_sys_i}: System clock, used by Wishbone busses, the direct timestamp I/O port and peripheral logic. Must not be faster than
@code{clk_ref_0_i}. Having a separate system clock domain simplifies integration with other IP cores on the same Wishbone bus.
@item @code{clk_dmtd_i}: DDMTD offset clock (close to 62.5 MHz) from the associated WR core. Used exclusively by the DDMTD calibration logic, not necessary for normal operation of the card.
@end itemize

The time base for the FD core is provided by the @code{fd_csync_generator} unit. By ``time base'', we mean the the signals representing the core's internal notion of time, which are synchronous to the reference clock @code{clk_ref_0}:
@itemize
@item @code{csync_utc}: TAI seconds.
@item @code{csync_coarse}: number of @code{clk_ref_0} cycles since beginning of the current second.
@item @code{csync_pps}: Pulse-per-second signal, generated 3 cycles in advance, to accommodate for pipeline delays in the TDC and output drivers.
@end itemize

Timing-wise, the FD can work in two modes:
@itemize 
@item @b{Local time base} mode, where @code{clk_ref_0} oscillator is free running and the time counters 
are coarsely initialized by the host through @code{TM_SECH}, @code{TM_SECL}, @code{TM_COARSE} and @code{TCR} registers. 
In this mode, the TDC input/output events cannot be very accurately related to other cores/devices, 
and the delay accuracy is as good as of the local oscillator (±2.5@tie{}ppm), 
which means worst case error of 2.5 ns for a delay setting of 1 ms.
@item @b{White Rabbit time base} mode, in which the reference clock is phase-locked to the WR master clock (by means of the SoftPLL
inside the WR Core) and the time base signals are following the second/cycles counters provided by the WR Core 
(all @code{tm_} prefixed signals in the top level). WR provides inter-card synchronization better than 1 ns. In this mode the accuracy of the delays and timestamps is determined by that of the WR Master clock.
@ref{fig:wr_pll} shows interactions between the hardware and the cores while in White Rabbit mode.
@end itemize

@float Figure,fig:wr_pll
@center @image{drawings/wr_pll, 15cm,,,.pdf}
@caption{Operation of the FD core with White Rabbit as a time source.}
@end float

Mode selection is controlled by the @code{TCR} register and the @code{p_whiterabbit_fsm} state machine, 
which also informs driver about the status of WR/local operation and can generate interrupts whenever the state 
of the synchronization source changes. When the WR link goes down, the FSM automatically
switches the card to local time base mode, retaining the previous value of time base counters 
(so the time will slowly drift away from the WR time scale, but not ``jump''). In case of the WR link recovery after a failure, the resynchronization procedure must be triggered by the host (no seamless, automatic switchover yer - see issue @i{769} in [3]). Loss or acquisition of WR synchronization is signalled to the host via @code{TCR} status bits or an interrupt.

@section The TDC block
Relevant files: @code{fd_acam_timestamper.vhd}, @code{fd_acam_timestamp_postprocessor.vhd}, @code{fd_timestamper_stat_unit.vhd}.

The TDC controller interfaces with the Acam TDC-GPX chip and does whatever is neccessary to output timestamps aligned with the FD core's time base, as fast (in terms of delay) as possible. That is:
@itemize
@item Detection of input pulses, checking their width and generation of coarse (256 ns granularity) timestamps by taking a snapshot of an internal counter.
@item Reading out the fine part from the Acam.
@item Merging these values together, aligning the result to the local timebase and outputting everything in a format digestible by the pulse generators.
@end itemize

@subsection Timestamp format

The FD core uses standard White Rabbit timestamp format (@ref{fig:ts_format}), where each timestamp consists of 3 fields:
@itemize
@item 40 bit @code{seconds}: number of TAI seconds since 01.01.1970 (Unix epoch).
@item 28-bit @code{coarse}: number of reference clock cycles since the beginning of current second. In case of the FD, reference clock is 125 MHz, so @code{coarse} range is 0 to 124999999.
@item 12-bit @code{frac}: fraction of 8 ns, scaled to span full 12 bit range. @code{frac = 4095} @code{8 ns * 4095/4096}.
@end itemize
@float Figure,fig:ts_format
@center @image{drawings/ts_format, 13cm,,,.pdf}
@caption{WR timestamp format used by the FD.}
@end float

For example, a hardware timestamp of @code{12:50000:1000} is 12 seconds + (50000 * 8 ns) + (1000 / 4096 * 8 ns) = 12.000400001953125 s.

@subsection TDC timing

Before we time tag any pulses, we need to make sure the TDC is referenced to something and the time shift between 
this something and the Acam's internal time base is known (or even better, constant). Inside the TDC core, the time base consists of:
@itemize
@item @code{utc_count}: seconds counter.
@item @code{coarse_count}: coarse start cycle counter incremented after each TDC start event. In our case, the TDC start period of 256 ns is achieved by driving the Start input with a 7.8125 MHz clock coming from the PLL and gating out every second cycle via Acam's StartDis input to effectively divide it by 2.
@item @code{start_count}: start subcycle counter (0..31), reset at start event.
@item @code{timebase_offset}: offset between the Acam's and core's internal timescales.
@end itemize

A snapshot of these counters is sampled for every input event. The values are later merged with the fine part read from the Acam TDC to obtain the final, White Rabbit-formatted timestamp.

The interesting thing is how the TDC timebase is related to the external time scale, as TDC start events occur at fixed multiplies of 256 ns, but a PPS pulse (@code{csync_p1_i}) from the counter sync 
unit can come anytime (with 8 ns granularity). The most straightforward way would be to align the TDC start
pulse with the PPS pulse by resetting the output dividers in the AD9516. This solution wouldn't be very safe though, as the PLL clock before division is 750 MHz. Ensuring correct setup/hold times between the FPGA output pin and the counter synchronization input in the PLL at such frequency is very difficult.
The TDC core employs a simple trick here: when a resync pulse comes, it stores the difference between the lowest 
bits of the @code{csync_coarse} counter and the start subcycle counter (@code{timebase_offset} signal in @code{p_start_subcycle_counter} process). This difference is added to the timestamps at the postprocessing stage to compensate for time base shift. See @ref{fig:tdc_timebase} for a graphical explanation.

@float Figure,fig:tdc_timebase
@center @image{drawings/tdc_timebase, 15.5cm,,,.pdf}
@caption{TDC core time base signals.}
@end float

@subsection Input stage and TDC control

This part of the core takes care of the coarse input pulse and TDC start/stop enable signals. It is responsible for:

@itemize
@item @b{Sampling coarse pulse input.}
This is done in the @code{p_sync_trigger} process which implements 
a simple rising edge detector with a 2-stage synchronizer and input enable/disable logic.

@item @b{Safely enabling and disabling the TDC input}.
There are three processes involved here: @code{p_safety_counter}, which disables the input after a rising edge
of and re-enables it a certain time (@code{c_FALLING_REENABLE_TIMEOUT}) after the falling 
edge of the currently processed pulse. This prevents the core from generating incorrect timestamps when the 
input pulses come too close to each other or in case of poor quality or noise on the falling edge of the signal, which could be misinterpreted as a spurious input event. Even if the signal quality is fine, such spurious pulses may occur while plugging the trigger input to a live signal. The remaining processes: @code{p_gen_acam_start_dis} 
and @code{p_gen_acam_stop} drive the TDC's START and STOP disable signals. The former ensures that the start 
input is not enabled in the middle of a rising edge of the 7.125 MHz start clock and gates the StartDis TDC input to effectively divide the start clock by 2. The latter enables the stop input when the TDC has received at least one correct start pulse.

@item @b{Rejecting pulses that do not meet the requirements}. In our case - shorter than 24 ns or containing glitches. Width detection is done in the main state machine (@code{p_main_fsm}) by shifting in subsequent samples into a register (@code{width_check_sreg}) and checking if the register contains only ones. Glitch detection exploits sensitivity of Acam's Stop input: since we disable the trigger input right after we have detected a rising edge, the Acam shall normally produce only one timestamp. Therefore, if the FIFO does not become empty immediately after read, a spurious pulse must have been tagged. Such situation may occur when the board is fed with a train of densely spaced pulses (where the shift register doesn't notice the gaps between them, but the Acam does). If any glitch or incorrect pulse is detected, the timestamp is ignored, and the Acam is reset by asserting @code{AluTrigger} line.

@item @b{Reading out the Acam FIFO}, done by the main state machine. After detecting a rising edge on the coarse pulse input, the FSM waits for the timestamp to appear in Acam's FIFO and reads it out from register @code{8}. Several wait states are introduced by the FSM to ensure the read sequence does not cause setup/hold violations or signal integrity (SI) problems. The fine value is passed along with the coarse counters and offsets to the postprocessing unit.

@end itemize

@subsection Timestamp postprocessing

@node{Timestamp postprocessing}

The postprocessor combines the fine value read from the Acam with the coarse value captured using a counter into a WR-compatible timestamp and aligns it to selected time base. Postprocessing is done in the @code{p_postprocess_tags} process. It consists of 4 pipelined stages:
@itemize

@item Subtract the start offset value from the fine part. Acam's ALU can't handle negative numbers, therefore each timestamp is internally adjusted by a value defined in the @code{StartOff[1,2]} Acam's control registers. This step subtracts this value (programmable via @code{ASOR} register, so that a pulse that occured in phase with a start pulse gets a fine value near to 0. Some timestamps may have negative fine values after start offsest subtraction. This results from the internal Acam's delays - sometimes it may reference a stop event to a start event 
 that occured afterwards. The range of fine values is therefore wider than the start period - an event occuring 250 ns after a start pulse can be either timestamped as 250 or -6 ns. 

@item Rescale the fine value from Acam (expressed as a number of 41 ps bins) to WR time format (where 1 ps = 8 ns / 4096). This is done by simply multiplying by a constant scalefactor (programmable via the @code{ADSFR} register).
@item Check consistency between the coarse counter @code{coarse_count} and the fine part. In ideal case, the final timestamp should be a simple sum of @code{coarse_count} * 256 ns and the fine part. In @i{The Real World}, transitions of the fine value (i.e. 256 ns to 0 ns) in the Acam are not consistent with transitions of the coarse counter in the FPGA. Acam's internal start is shifted forward with respect to the FPGA's start signal. In certain cases, the fine part may have already flipped the 256 ns boundary, while the coarse counter has not counted up yet, producing a timestamp with an error of 256 ns (see @ref{fig:tdc_merge}). Also, big fine values at the the end of the range may be interleaved with negative ones, depending on Acam's mood.

The postprocessor mitigates this problem by using the @code{start_count} counter. If @code{start_count} value is low (indicating that we are close to the beginning of an FPGA start cycle), while the fine part is high (meaning that the TDC has not yet noticed the ``fresh'' start pulse), the timestamp's @code{coarse_count} need to be adjusted by subtracting one start period. Thresholds for the comparisons are programmable via @code{ATMCR} register, their values were obtained experimentally.

@item Align the timestamp to our local/WR timebase, by simply adding the @code{timebase_offset} value obtained during time base counter resynchronization.

@end itemize

@float Figure,fig:tdc_merge
@center @image{drawings/tdc_merge, 15.5cm,,,.pdf}
@caption{Relations between coarse and fine timestamp parts.}
@end float

As a result, we get a WR-formatted timestamp, which may have a constant offset with respect to the WR/local timescale. This offset results from PCB trace lengths, component properties and the choice of the postprocessing register values and is compensated in software by adding a correction to all timestamps read from the card and all programmed delay values. The correction value is determined individually for each mezzanine during factory or DDMTD calibration and written in the calibration EEPROM. Typical value is 127.5 ns.

Note that the postprocessor can be disabled for debugging purposes by setting @code{TSBCR.RAW} bit. In such case, raw counter and fine values are written to the buffer instead of the final timestamp. This feature is used in production tests.

@subsection Statistics unit

The Acam core includes a statistics unit. Its purpose is to collect data that may be helpful in debugging and performance tuning of the core:
@itemize
@item Count all detected input pulses.
@item Count pulses that have been correctly tagged.
@item Measure worst-case input-to-timestamp latency (which is important for delay applications as it defines minimum safe delay value). In case of the Acam configured in G-Mode, the latency is 360 ns. 
@end itemize

@subsection Acam host interface

The main TDC state machine lets the host directly access Acam's control registers. Host (a.k.a. bypass) mode is active when @code{GCR.BYPASS} bit is set. The Acam must be programmed for G-mode operation prior to enabling the hardware readout. In order to write a single Acam register follow the procedure below:
@itemize
@item Set the address of the Acam register by programming the GPIO expander.
@item Write the desired word to @code{TDR} register.
@item Write 1 to @code{TDCSR.WRITE} bit.
@item Wait at least 1 microsecond before commencing another write.
@end itemize
Read procedure is quite similar:
@itemize
@item Set the address of the Acam register by programming the GPIO expander.
@item Write 1 to @code{TDCSR.READ} bit. 
@item Wait at least 1 microsecond.
@item Read the value returned by Acam from @code{TDR} register.
@end itemize
Note that once the TDC is programmed, its address (via the SPI expander) must be set to @code{8} (Acam FIFO1 register), so that the FSM can read correct data from the right FIFO register.

@subsection Host timestamp readout
Relevant files: @code{fd_ts_buffer.vhd}.

The FD provides the values of all input and output timestamps through a 1024-entry ring buffer. Each timestamp is associated with a sequence number and the source channel identifier. Timestamp readout can be enabled anytime and for any mode of operation (delay/TDC/pulse generator). The readout procedure goes as follows:
@itemize
@item Set channels we are interested in reading from in @code{TSBCR.CHAN_MASK}. Enable readout by setting @code{TSBCR.ENABLE}.
@item Poll the buffer by reading @code{TSBCR.EMPTY} bit or by handling the TS buffer interrupt. Do not attempt both ways simultaneously.
@item Read the timestamp from @code{TSBR} registers. Order doesn't matter.
@item Release the timestamp from the buffer and proceed to the next one by writing anything to the @code{TSBR_ADVANCE} register.
@end itemize

In case of an overflow, the oldest timestamps in the buffer are subsequently replaced by the most recent ones. Loss of timestamps due to overflow can be detected by comparing the sequence numbers. If the buffer is handled through interrupts, coalescing mechanism is provided to reduce CPU load for larger amounts of timestamps. See @code{TSBIR} register description for details.

@section Output stages

Relevant files: @code{fd_delay_channel_driver.vhd}, @code{fd_delay_line_arbiter.vhd}.

An output stage produces one or more pulses of given width and spacing starting at a TDC timestamp adjusted with the programmed delay value (delay mode) or at an arbitrary time (pulse generator mode). It handles a single output channel, programmable through a separate Wishbone register block. 

@float Figure,fig:vhdl_output_stage
@center @image{drawings/vhdl_output_stage, 13cm,,,.pdf}
@caption{Output stage VHDL overview.}
@end float

The structure of the output stage VHDL is shown in @ref{fig:vhdl_output_stage}. The datapath consists of two accumulateing timestamp adders that calculate start- and end-of-pulse timestamps. The adders' outputs are compared with the time base counter, resulting in pulse start/end strobe signals. The timing of output pulses is defined by 3 sets of registers:
@itemize
@item @code{start}: delay between TDC timestamp and the rising edge of the output pulse (delay mode) or absolute time of the rising edge of the output pulse (pulse generator mode).
@item @code{end}: same for the falling edge.
@item @code{delta}: delay between subsequent output pulses.
@end itemize
Multiplexers are used to configure the data path for a given output mode. Comparators and adders drives a simple, sequential state machine, which:
@enumerate
@item Waits for a TDC timestamp (delay) or a write to @code{DCR.PG_ARM} (pulse generator),
@item Takes the fractional part of the rising edge output timestamp, multiplies it by the calibration factor @code{FRR} and sends it to the delay line for a given channel,
@item Waits until the arbiter updates the delay line with the new fractional value,
@item Waits for the start comparator hit and asserts coarse output high,
@item Repeats points 2...4 for the falling edge,
@item Checks if we want more than one pulse - if true, it adds @code{delta} value to the start/end timestamp and goes to point 2. If not, it goes idle.
@end enumerate

Access to the delay lines is multiplexed by a round-robin arbiter (@code{fd_delay_line_arbiter}). Worst-case update latency is 4 * 32 ns = 128 ns, imposing a width/spacing limit of 200 ns. Shorter/denser pulses (up to 50 ns) can be still produced by setting @code{DCR.NO_FINE} bit, with width/spacing values being restricted to multiplies of 4 ns. Given the TDC latency of 360 ns and a few clock cycles taken by pipelining, the minimum safe delay setting is therefore 600 ns.

@page
@subsection Programming the output stage
@itemize
@item Set the mode in @code{DCR.MODE} bit.
@item Set the absolute start time or delay and pulse spacing in @code{DCR.x_START}, @code{DCR.x_END} and @code{DCR.x_DELTA} registers.
@item Acknowledge the changes by writing @code{DCR.UPDATE} bit.
@item If the output is not already enabled, write @code{DCR.ENABLE} bit and enable the corresponding SSR switch through the SPI GPIO.
@end itemize

@subsection Other important things
@itemize
@item The design is highly pipelined to meet timing for a 125 MHz clock with a rather wide datapath (40 + 28 + 12 = 80 bit add/compare operations).
@item Start/end/delta and mode selection registers are shadowed to ensure atomic updates of output pulse timings. By shadowing we mean that there are each of these registers has an internal copy that is used by the output logic. All copies are updated simultaneously when @code{DCR.UPDATE} bit is written.
@item Offset resulting from pipelining and data path delays is systematic and must be compensated by adjusting the start/end values in the software.
@item Checking if the output has triggered can be done by polling @code{DCR.PG_TRIG} bit.
@item @code{FRR} register must be initialized with correct calibration coefficient (see @ref{Output stage calibration}).
@end itemize

@section Other logic

@subsection OneWire

The FD core incorporates a dedicated Dallas's 1-Wire bus master core for accessing the temperature sensor/ID chip from a non-deterministic host (1-Wire requires tight timing to operate correctly). The core's documentation is available at the Opencores project page [6].

@subsection I2C

There is also a simple bit-banged I2C master for talking with the I2C EEPROM, accessible through the @code{I2CR} register. The driver uses it for retreiving calibration data and identification of the mezzanine.

@subsection SPI Master

The SPI Master is a specialized core (not a general-purpose Wishbone SPI master), accessible via the @code{SCR} register and interfacing with all SPI peripherals (VCXO DAC, GPIO and AD9516). Special features include:
@itemize
@item Two concurrent, arbitrated write ports: software via @code{SCR} register and hardware via @code{tm_dac_value} port.
@item The hardware port takes priority over software access and lets the SoftPLL update the VCXO DAC in a deterministic way, regardless of driver's accesses to the GPIO and PLL chips.
@item Atomic read/write access thanks to a single control/status/data register.
@end itemize

@page
@subsection Testing logic

There are two extra cores used during production testing and characterization of the card:
@itemize
@item A PWM driver, accessible via the @code{TDER2} register. Used in lab tests for driving a Peltier module in order to characterize the temperature effects on the mezzanine.
@item A frequency meter (@code{TDER1}), measuring the mezzanine's VCXO frequency against the carrier's system clock. Used to characterize the tuning range of the oscillator during production test.
@end itemize

@section Initializing the card

Since initialization of the FD mezzanine is not simple and straightforward, a brief description of the procedure is provided below:
@enumerate
@item Check for presence of the FD core by verifying @code{IDR} register.
@item Check mezzanine presence through @code{GCR.FMC_PRESENT} bit.
@item Read the calibration EEPROM via the I2C master. Parse and verify its contents.
@item Reset the mezzanine via @code{RSTR} register. Hold the FD core (except SPI & I2C) in reset.
@item Program the AD9516 PLL. Use register values included with the driver/test program code.
@item Initialize 1-wire temperature sensor. Read card serial number and temperature.
@item Initialize MCP23S17 GPIO. Disable termination, select internal trigger, disable all outputs.
@item Clocks are initialized, un-reset the core through @code{RSTR} register.
@item Enable TDC host mode by setting @code{GCR.BYPASS} bit.
@item Program the TDC in I-Mode. Run output stage calibration. Store calibrated values.
@item Program the TDC in G-Mode. Disable bypass and trigger input.
@item Load timestamp postprocessor configuration (@code{ADSFR}, @code{ASOR}, @code{ATMCR} values). Use values
provided in the test program.
@item Set board time to 0 via @code{TCR} register.
@item Purge timestamp readout buffer (@code{TSBCR.PURGE} bit).
@item Enable trigger input via @code{GCR} register.
@end enumerate


Now the card should be ready for timestamp readout and output programming. For code examples please look at @code{fdelay_lib.c} file in the test program [3].

@page
@section Carrier implementation example

Relevant files: @code{svec_top.vhd}.

An example FD VHDL core implementation on a SVEC FMC carrier [8] is shown in @ref{fig:svec_top_block}.

@float Figure,fig:svec_top_block
@center @image{drawings/svec_top_block, 16cm,,,.pdf}
@caption{Implementation of two FD cores on a SVEC carrier.}
@end float

The top level is only a passive interconnect between the FD cores, White Rabbit and FPGA-specific blocks, such as clock/IO buffers and PLLs. The components of our example design are:
@itemize
@item 2 FD cores,
@item White Rabbit core, providing synchronization and Ethernet connectivity,
@item Additional White Rabbit components: GTP serdes wrapper and SPI DAC driver,
@item Wishbone interconnect,
@item Etherbone core, allowing for Ethernet access to entire memory space of the card,
@item Vectored Interrupt Controller (VIC), multiplexing interrupts from all sources (in our case, the two FD cores). VIC's vector table is preinitialized with base addresses of the cores sourcing interrupts to enable automatic IRQ line detection,
@item VME64x core, bridging between VME64x bus and Wishbone (with all necessary VME buffers/tristates),
@item Clock distribution and PLLs (producing the DDMTD and system clocks and DDR clocks for the output stages of the FD cores),
@item Utility cores - power up reset and SVEC front panel LED driver for status indication.
@end itemize

For more details, refer to the source files and comments inside.

@page
@chapter Calibration procedures

@section Output stage calibration

@node{Output stage calibration}

The role of this calibration mechanism is to make sure the SY89295 fine delay lines introduce delays consistent with the programmed settings. In @i{The Ideal World}, the tap size of a SY89295 is 10 ps, so programming the chip to 800 shall result with a delay of 8 ns. In reality, the tap size depends on PVT effects - we observed that some chips, when set to 800 taps produce 7.5 ns or 8.5 ns instead of the requested 8 ns.  
As mentioned earlier, the FD output stage works by producing an 8 ns resolution coarse pulse with a counter and adjusting it precisely in an SY99295 chip, according to the equation:

@code{t_measured = floor(t_out / 8ns) + FRR * (t_out mod 8ns)}


where @code{t_measured} is the measured timestamp of the output pulse, and @code{t_out} is the one the output stage was requested to produce. The @code{FRR} parameter relates the fractional (modulo) part of the timestamp with the number of fine delay line taps required to accurately reproduce it. In @i{The Ideal World}, it should be 100 taps/ns. If the alpha value is wrong, output pulses will be imprecise. To make things worse, the error will not be proportional to the requested delay, but only to its modulo part. Therefore, pulses whose timestamps have small fractional value (for example, 1000 ns mod 8 ns = 0) will have no error at all, while other ones (e.g. 1007.9 ns) will have an error of as much as 1 ns. 

Since there is absolutely no corellation between the pulses coming to the TDC and the card's reference clock, fractional parts of the input timestamps and the values written to the SY89295 for each output pulse look purely random. Therefore, one pulse may have an error of 0, while the next one might be off by almost a nanosecond - an effect that in technical terms is called huge, non-gaussian jitter, exceeding by far the 100 ps specification requirement.

@float Figure,fig:calib_why
@center @image{drawings/calib_why, 15.5cm,,,.pdf}
@caption{Effects of uncalibrated output delay line.}
@end float

Fortunately, this effect is mitigated by calibrating the @i{FRR} value for each SY89295 delay line. This is done every time the card starts up.

The output stage calibration mechanism is depicted in @ref{fig:calib_output}. It works by feeding the output stage with 
calibration pulses and measuring the in-out delay of SY89295 delays for different tap settings in order to find a point at which they
delay the signal by exactly 8 ns more than at tap setting of 0. The TDC, reconfigured in the I-mode (single ended start input and
4 single ended stop inputs, one per output) is reused as a calibrator (thanks to its' voltage adjusting PLL, we know that its definition of 8 ns
is not worse than of the reference oscillator). Precision better than 10 ps rms (single tap) is achieved by averaging multiple measurements.
This calibration is performed by the device driver every time the card is initialized. In order to speed up calibration, the software assumes monotonicity of the delay line
(guaranteed by design) and instead of sweeping all possible tap counts, employs a simple divide-and-conquer algorithm.

@float Figure,fig:calib_output
@center @image{drawings/output_calibration, 12cm,,,.pdf}
@caption{Output stage jitter calibration mechanism.}
@end float

Note that while the calibration in progress, the output switch is disabled to stop our calibration pulses from reaching devices driven by the card. This unfortunately prevents
executing the calibration during runtime without risk of losing pulses. We provided an alternative mechanism to overcome this limitation, which
exploits the fact that since process and supply voltage remain constant during operation, only temperature has significant impact on the output stage delay.
A function of 8 ns tap delay error vs temperature was measured in the lab by cooling/heating up the card to temperatures between 30 and 90 degrees C with a Peltier cell
 and is used to relate the temperature and 8ns tap setting measured at the card startup with its' current temperature. Simple 2nd order polynomial fitting allows for updating
the output stage scale factor without disturbing the outputs with extra calibration pulses.

The same method (with full range sweeping instead of divide-and-conquer) is used to measure linearity (INL/DNL) of the delay lines during production test.

@page
@section DDMTD I/O delay calibration

Careful readers may have noticed that the previous calibration process only minimizes jitter. The purpose of DDMTD calibration is to measure 
the end-to-end delay of an (almost) entire mezzanine. @ref{fig:ddmtd_calibration} shows the calibration components:
@itemize
@item The input of the TDC is fed with a square waveform of @code{clk_ref_0} / 144, simulating real input pulses with the fastest allowed frequency (to speed up measurements and increase resolution) and some safety margin.
@item The delay path is programmed to a minimum insertion delay of 600 ns. 
@item Input and output pulses are sampled by another clock with two identical flip flops. Frequency of the samling clock is slightly offset with respect to 
@code{clk_ref_0} / 144 (in our case the offset is 1/16384). The flip flop outputs are hence downconverted versions of the in/out pulses and keep their timing relations, but
scaled down by a factor of 16384, so a delay of 10 picoseconds is seen as 16.384 ns. This is very easy to measure using a simple counter.
@end itemize

@float Figure,fig:ddmtd_calibration
@center @image{drawings/ddmtd_calibration, 10cm,,,.pdf}
@caption{End-to-end delay calibration using DDMTDs.}
@end float

Since the offset clock is produced by the PLL in the White Rabbit core, DDMTD calibration
is possible only with WR-enabled carriers. It is not done by default in the driver, but
can be run using @code{ddmtd_calibration} tool from the @code{software/tests/} subdirectory in the repo [3]. 

Note that this method is still not ideal - it is prone to PVT differences between the calibration flip flops and it does not take into account the delays introduced by
the output cutoff and input selection switches. Therefore, production tests involve calibration with an external time interval meter. Tests performed on a batch of 80 cards
have shown that the error between DDMTD calibration mechanism and the external time interval meter did not exceed 800 ps.

More information on DDMTD phase/time measurement techniques is available in Tom's MSc thesis [2].


@page
@chapter Registers description

@section Memory layout

@multitable  @columnfractions .20 .80
@headitem Base Address @tab Core
@item @code{0x000} @tab Main registers. @xref{Main registers}
@item @code{0x100} @tab Channel 1 registers. @xref{Output stage registers}
@item @code{0x200} @tab Channel 2 registers. @xref{Output stage registers}
@item @code{0x300} @tab Channel 3 registers. @xref{Output stage registers}
@item @code{0x400} @tab Channel 4 registers. @xref{Output stage registers}
@item @code{0x500} @tab 1-wire master registers
@end multitable

@section Main registers
@node{Main registers}
The main register block controls all subsystems of the FD core excluding the OneWire thermometer and output channels.

@macro regsection{name}
@subsection \name\
@end macro

	@include fd_main_regs.in

@section Output stage registers
@node{Output stage registers}

The output stage register block controls a single FD output stage.

    @include fd_channel_regs.in

@page

@chapter References
@node{References}

@enumerate
@item Official schematics and PCB design (CERN EDMS)

@url{https://edms.cern.ch/nav/EDA-02267-V5-2}
@item Tom's MSc thesis (a bible of White Rabbit timing)

@url{http://www.ohwr.org/documents/80}
@item Hardware homepage & Wiki

@url{http://www.ohwr.org/projects/fmc-delay-1ns-8cha}
@item Official user's manual

@url{http://www.ohwr.org/documents/179}

@item Acam TDC-GPX datasheet

@url{http://www.acam.de/fileadmin/Download/pdf/English/DB_GPX_e.pdf‎}

@item Sockit 1-Wire master project page

@url{http://opencores.org/}

@item Git repository with VHDL & test program sources

@url{http://ohwr.org/projects/fmc-delay-1ns-8cha/repository}
@item SVEC FMC Carrier project

@url{http://ohwr.org/projects/svec}

@item Software Described Bus (SDB) project

@url{http://www.ohwr.org/projects/fpga-config-space}

@item Fine Delay Production Test System

@url{http://www.ohwr.org/projects/pts}

@item Analog Devices PLL design software (@i{AdiSimClk})

@url{http://www.analog.com/en/rf-tools/adisimclk/topic.html}

@item @code{wbgen2} - a Wishbone slave generator

@url{http://www.ohwr.org/projects/wishbone-gen}

@end enumerate

@bye
