WR2RF timing calibration methodology
To commission a WR2RF card for operational performance each RF channel and its two trigger units must be observed to be operating well away from metastable sampling points.
Metastability can enter the system because the White Rabbit clocked nco_reset signal is sampled in the RF clock domain. Then for each trigger unit there is an external flip-flop that resamples the trigger unit output from the FPGA. These trigger unit signals must be delayed to avoid the setup and hold window the external flip-flop.
The process and methodology is described in the following document.
Extracted document:
Configuring timing delays for WR2RF
Before commencing to calibrate timing delays for a WR2RF card it is important that the card has reached thermal equilibrium in its installation. The card has an oven controlled oscillator and this takes approximately 5 minutes to reach operating conditions. It's recommended to start calibration after 15 minutes.
Note: calibration may result in slightly different values from board to board (although not expected), or from a different system (like ertm). So the reset might be visible on the RF clock output at slightly different time. This is not a real problem as it corresponds at the end to a difference of cable length.
Configuring timing delays for the DDS
The picture below depicts the connections between the DDS and FPGA that are important to consider for timing calibration.
The signal IOUpdate resets the phase accumulators on the DDS AD9910. Internally, this signal is sampled by a clock called SYNC_CLK which is divided by 4 version of the SYS_CLK frequency. The SYS_CLK is derived from the incoming 1 GHz reference clock from the PLL. We have to ensure that IOUpdate is cleanly sampled by the SYNC_CLK. The FPGA provides a fine delay circuit to adjust the output timing of the IOUpdate signal, implemented using OSERDES and ODELAY cells.
To achieve this we must provide a stream of NCO resets which in turn will trigger an IOUpdate. We then increase the delay of IOUpdate until we observe the DDS output jump onto the next SYNC_CLK sampling edge. The SYNC_CLK has a 4 ns period, so the DDS output will phase will be reset 4 ns later. We have to observe this phase reset, and observe it slip by two SYNC_CLK periods and then place the delay for IOUpdate between these values.
To observe and make these measurement, configure an oscilloscope to trigger off the nco_reset
signal and observe the RF out signal (configured to be the DDS signal). The place of measurement
is usually around 120 – 130 ns after nco_reset
.
With wr2rf_init.sh
, nco_reset
is visible on tmgio3
.
The easiest way to check calibration is to use the command:
wr2rf -s SLOT calib-dds-ioupdate
which generates an NCO reset every 200ms, unmask iopudate and outputs DDS on RF1.
The DDS ioupdate delay can be modified using the command:
wr2rf -s SLOT api-dds-ioupdate-delay FDELAY ODELAY
ODELAY controls an odelaye2
cell, each step is about 70ps. FDELAY controls an oserdes
cell, each step is 2ns.
The default values (FDELAY=2, ODELAY=10) are known to be ok.
A single NCO reset can be generated using the command:
wr2rf -s SLOT nco-reset
Configuring timing delays for the trigger units in an RF channel on WR2RF
For each RF channel on a WR2RF card, there are three places where metastability can be introduced into the system and observed on the two trigger unit outputs, located on the front panel. This may be due to incorrect timings and delays having been set, or are the default when initialising the card.
The aim of this section is to define a procedure that allows calibration of these delays so they may be avoided. The following functional overview highlights the different clock domains and their interfaces. Blue, denotes a WR clock domain, red denotes RF clocks and signals and orange/yellow indicate the VTU clock, a divided (by 8) version of the RF clock.
Where can metastability be introduced into the system?
-
The phase of the RFNCO is reset within the WR clock domain. This signal must be synchronised into the RF clock domain via the first trigger unit. However, there is a fixed phase relationship for any given RF frequency.
-
Trigger unit 1 drives its output off the FPGA and into an external jitter cleaning flip-flop. This T1 signal must avoid the set-up and hold window of this flip-flop.
-
Trigger unit 2 drives its output off the FPGA and into an external jitter cleaning flip-flop. This T2 signal must avoid the set-up and hold window of this flip-flop. The path between T1 and T2 is fully timed by the FPGA.
Stable RF
There are two stages to the process:
The first is to consider that the RF signal produced by the RFNCO, nonIQMod and WR2RF PCB is not stable until 950 ns after nco_reset has occurred. This is a minimum value.
For a fuller description of this issue please refer to this page: https://ohwr.org/project/wr2rf-vme/wikis/wr2rf-operational-timings
To avoid this issue, there is a coarse delay (cdelay) block, provided within the WR clock domain to delay sync signal into the first trigger unit. It provides up to 2 us of delay. It is followed by a fine delay circuit that can delay the signal by an additional 16 ns, in 1 ns steps. In experiments with proton injection frequencies, 75 clock cycles or 1.2 us has provided stable results.
Method – Requirements
To follow steps 1 to 4 it is necessary that the card is receiving an RF train that provides regular NCO resets.
Method – Step 1 – coarse delay
To be sure we are clear of any RF instabilities, inspect the RF output signal whilst triggering on a delayed version of the NCO reset. This signal (cdelay applied only) can be observed via a front panel lemo:
./wr2rf -s $slot lemo-dbg-sel tmgio3 rf1_nco_reset_cdelayed
Try setting the coarse delay value for rf1 to 75 cycles via this command:
./wr2rf -s $slot nco-reset-delay 1 75 0
(1 for rf-1, 75 for the number of cycles, 0 for the 2ns steps)
If the delayed version of NCO reset is clear of the RF stabilities we can proceed to configure the fine delay.
As the RF-NCO is reset only with frames, it is not possible to use wr2rf nco-reset
.
But it is possible to generate rffame every .5sec using the command:
./wr2rf -s $slot calib-vtu 1
Methods – Steps 2, 3 and 4 - overview
We plan to increase the programmable delays in each of the three places where metastability can occur, such that our signal is being sampled on the following clock edge. At this point, our output will appear 1 clock cycle later. Measuring the skew between the nco_reset and either T1 or T2 will allow us to determine if the output has been sampled on a later clock edge. The delay at which the sampling change occurs, is the worst choice. We need to be half a clock period away from this delay.
For steps 2-4, route the signal nco_reset to the front panel:
./wr2rf -s $slot lemo-dbg-sel tmgio3 nco_reset
(Note: TBC but the rf1_nco_reset_cdelayed should be ok)
Method – Step 2 – fine delay
Observe the skew between nco_reset and the T1 trigger unit output. Sweep across the full range of fine delay values 0 through to 15. This should be sufficient to observe two or three changes in the RF clocks sampling points. Choose a fine delay value mid-way between these sampling changes. Tune with the odelay value from 0 to 31.
./wr2rf -s $slot nco-reset-delay 1 75 [0-15] [0-31]
Method – Step 3 – T1 flip-flop
Observe the skew between nco_reset and the T1 trigger unit output. Changing the programmable delay on the trigger unit output from the FPGA has two elements:
-
An ODELAY cell that provides changes in delay of 78 ps, up to 31 x 78 ps
-
A half-cycle delay (RF clock cycle)
Combined together, the half-cycle delay and ODELAY should provide close to 5 ns of total delay, if the incoming RF clock has a 200 MHz frequency.
Binary chop through this delay space until the RF sampling point changes. A sequence may look something like this for trigger unit 1 on RF channel 1:
./wr2rf -s $slot vtu-odelay 1.1 half_cycle_delay odelay_value
./wr2rf -s $slot vtu-odelay 1.1 0 0x0
./wr2rf -s $slot vtu-odelay 1.1 1 0x1f
./wr2rf -s $slot vtu-odelay 1.1 1 0x0
./wr2rf -s $slot vtu-odelay 1.1 1 0x10
and so on, depending on where the RF sampling point changes. The value to be set is just the inverted half-cycle and same odelay.
There is a small chance that the metastability region cannot be found. In this event, set the delay to midpoint of the search, e.g.
./wr2rf -s $slot vtu-odelay 1.1 0 0x1f
Method – Step 4 – T2 flip-flop
As per Step 3 and the T1 flip-flop, but now we should observe the skew between nco_reset and the T2 output.