procedure.tex 14.7 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
\subsection{Initial state}
Initially none of the transmission/reception latencies and asymmetry of White 
Rabbit devices or fiber parameters are known. Therefore {\bf the values of 
appropriate parameters should be set to 0 for all nodes and switches which will
be calibrated}. In the White Rabbit PTP Core this can be done by modifying the SFP
database and setting $\Delta_{TX}$, $\Delta_{RX}$ and $\alpha$ values to 0 for all
supported transceivers\footnote{Please refer to the White Rabbit PTP Core manual
\url{}}. For the White Rabbit Switch 
please modify the \emph{/wr/etc/sfp\_database.conf} and
\emph{/wr/etc/wrsw\_hal.conf} configuration files.

\subsection{Reference fiber latency}

In this very first step of the calibration procedure the total propagation delay
($\delta = \delta_{MS} + \delta_{SM}$) of a fiber cable is measured. It requires
two different fiber cables with LC connectors (i.e. fibers that have
different length). For further considerations let's mark them $f_1$ and $f_2$.
Fiber $f_1$ can be called helper and is just a few meters long. It is used to
measure the parameters of $f_2$ and in hardware calibration procedures described
later. Fiber $f_2$ is of the same type that will be used in the WR installation.
It is a few kilometers long and its parameters will be measured.

27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102
		\caption{Measuring total fiber propagation latency}

	\item Take two White Rabbit devices, configure one of them as WR Master 
		and the other as WR Slave. 
	\item Connect them with fiber $f_1$ and wait until they become 
		synchronized (fig.\ref{fig:refiber:latency}, step 1). You can 
		use the available monitoring software \footnote{\emph{gui} WRPC 
		Shell command for the WR PTP Core, or \emph{wr\_mon} program for the WR 
		Switch} on the Slave device to check when it happens (indicated by 
		the WR servo in the \emph{TRACK\_PHASE} state and an offset below 1 ns).
	\item Write down the round trip delay ($delay_{MM1}$) reported by
		the monitoring software and the bitslide values for both
		Master($\epsilon_{M1}$) and Slave($\epsilon_{S1}$). Since the fixed delays
    are initially set to 0, the bitslide values are reported by the monitoring
		software simply as the reception delays.
	\item Connect the same two WR Devices with fiber $f_2$
		(fig.\ref{fig:refiber:latency}, step 2), wait again until the WR Slave
		becomes synchronized and write down the values of
		$delay_{MM2}$, $\epsilon_{M2}$ and $\epsilon_{S2}$.
	\item Connect fibers $f_1$ and $f_2$ together using an LC-LC Adapter and use
		this link to connect again the same WR Devices together(fig.
		\ref{fig:refiber:latency}, step 3). Wait until the WR Slave becomes 
		synchronized and write down the values of $delay_{MM3}$, 
		$\epsilon_{M3}$, $\epsilon_{S3}$.
	\item Subtract the bitslide values to obtain an approximation of the round
		trip delay, the sum of Master and Slave latencies ($\Delta_{TXM}$,
		$\Delta_{RXM}$, $\Delta_{TXS}$, $\Delta_{RXS}$) and the fiber round-trip
			delay_{MM1}' = delay_{MM1} - \epsilon_{M1} -
			\epsilon_{S1} \\
			delay_{MM2}' = delay_{MM2} - \epsilon_{M2} -
			\epsilon_{S2} \\
			delay_{MM3}' = delay_{MM3} - \epsilon_{M3} -
	\item Calculate the latency of the fiber $f_1$ ($\delta_1$) and $f_2$
			\delta_1 = delay_{MM3}' - delay_{MM2}' \\
			\delta_2 = delay_{MM3}' - delay_{MM1}'
		{\bf Note:} The LC-LC Adapter also has its own unknown latency that
		introduces an error in the $\delta_1$ and $\delta_2$ calculations.
		However, this error is a few picoseconds and can be disregarded.
Further mathematical explanation for this calibration step can be found in
appendix \ref{subsec:app_filat}. Fiber cable $f_1$ with a round-trip latency
$\delta_1$ will be used further to pre-calibrate the selected WR Calibrator.

\subsection{Fiber asymmetry}

The calibration of a fiber cable is needed to find out its asymmetry coefficient
$\alpha$. In the previous step we managed to calculate the total propagation
latency of fibers $f_1$ and $f_2$. However, to calibrate any device, the fiber
asymmetry has to be known.

The $\alpha$ coefficient used in the White Rabbit protocol to express fiber 
asymmetry is defined as:
	\alpha = \frac{\delta_{MS} - \delta_{SM}}{\delta_{SM}}

To get its value, you have to make two connections with WR Devices and fibers
$f_1$, $f_2$ as presented in figure \ref{fig:fiasym}.

104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180
		\caption{Measuring fiber $f_2$ asymmetry}

	\item Connect any two WR Devices with a few meters long fiber $f_1$, set
		the $\alpha$ value in their configuration to 0, configure one of them as WR
		Master, and the other as WR Slave. Connect their 1-PPS outputs to an
		It is important to use oscilloscope cables of the same length
		and type or cables with known delays so that no additional,
		unknown latency affects the measurements.

	\item Use the monitoring software to check when the Slave becomes
		synchronized to the Master (offset reported by WR PTP daemon below 
		1 ns and WR servo in the \emph{TRACK\_PHASE} state).

	\item Measure 1-PPS skew between the Slave and Master in picoseconds by
		comparing the 1-PPS signals with an oscilloscope ($skew_{PPS1} 
		= t_{PPS_S} - t_{PPS_M}$)

	\item Repeat the same procedure using a few kilometers long fiber $f_2$ to
		obtain the PPS skew for the second connection ($skew_{PPS2} =
		t_{PPS_S}' - t_{PPS_M}'$)

	\item Calculate the $\alpha$ coefficient using the following equation:
			\alpha = \frac{2(skew_{PPS2} -
			skew_{PPS1})}{\frac{1}{2}\delta_2 -

  \item The White Rabbit Switch uses the $\alpha$ value calculated from the equation
    above without any conversion. However, the White Rabbit PTP Core requires
    converting $\alpha$ into a natural number using the formula:
      \alpha_N = 2^{40}(\frac{\alpha+1}{\alpha+2} - 0.5)

  \item $\alpha$ or $\alpha_N$ (depending on the WR Device) should be stored
		into the configuration parameters of a WR Calibrator but also into
		each Slave device that will use this fiber.

Further mathematical explanation for this calibration step can be found in
appendix \ref{subsec:app_fiasym}

{\bf Note:} Both WR Devices used at CERN: the WR Switch and the WR PTP Core can
be used to calculate the $\alpha$ value. However, the 1-PPS signal generated by
the WR PTP Core running on the SPEC card with a DIO mezzanine is more jittery
than the 1-PPS generated by the WR Switch (check measurement uncertainty
considerations in appendix \ref{subsec:errors:fiasym}). For most applications
the inaccuracy of $\alpha$ measured using two SPEC cards will be negligible. If
that isn't the case, the most precise value of the $\alpha$ parameter can be
obtained from the measurements described above done with two WR Switches.
Alternatively, measurements done with two SPEC cards (or any other hardware
based on the SPEC design) can be repeated multiple times to produce an averaged
value of 1-PPS skew and a more accurate $\alpha$.

\subsection{Calibrator pre-calibration}

Having the parameters of fiber $f_1$ measured, a WR Calibrator has to be
selected among the set of available WR-compatible devices. The only
constraint for the selection is having two instances of the same device
for the pre-calibration procedure described here. They must have the
same PCB layout and the same FPGA bitstream.

To determine what are the transmission and reception latencies ($\Delta_{TX}$,
$\Delta_{RX}$) of the WR Calibrator, the connection shown in figure
\ref{fig:calibrator} has to be established.

182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245
		\caption{Measuring Calibrator latencies}

	\item Since both Master and Slave are exactly the same devices (the same FPGA
		bitstreams, PCB layout, complementary SFPs from the same producer) the sum
		of their TX and RX latencies is equal:
			\Delta_{TXM} + \Delta_{RXM} = \Delta_{TXS} + \Delta_{RXS} 
		That means, the total transmission delay caused by each of those devices can
		be determined by simply dividing by 2 the round trip delay without the RX
		bitslide of each device and the fiber round-trip latency:
			\Delta_{TX} + \Delta_{RX} = (delay_{MM1}' - \delta_1) /
	\item Naturally, our Calibrator has some internal asymmetry, which means that
		$\Delta_{TX} \neq \Delta_{RX}$. There are two ways to determine the exact
		values of $\Delta_{TX}$ and $\Delta_{RX}$. It can be done either by
		establishing a WR link with a WR Device having a well-known internal
		asymmetry - which we don't have since all other devices will be calibrated
		from the WR Calibrator; or taking apart the Calibrator and measuring the
		latency of PCB traces, electronic components, SFP transceivers - that is
		possible, not trivial at all, and raises many other problems like measuring
		the latency between the SFP electrical input and its optical output, or
		measuring the latency inside the FPGA chip. Therefore at this point we set
		up a convention, that the WR Calibrator has no asymmetry:
			\Delta_{TX} = \Delta_{RX}
		That means, the parameters describing the transmission and reception delays
		of the WR Calibrator should be set in the device configuration to:
			\Delta_{TX} = \Delta_{RX} = (delay_{MM1}' - \delta_1) /

		{\bf Note:} Don't worry, assuming that the WR Calibrator has no asymmetry
    does not affect the calibration quality. It is taken into account later
    during the actual calibration of each unknown WR Device. It also does not
    affect the WR PTP protocol when a link is established between two WR
    Calibrators, since the daemon only uses the precise value of the one-way,
    Master to Slave link delay (where the TX latency of the Master and RX
    latency of the Slave are summed together).
	\item Configure your WR Calibrator with the calculated $\Delta_{TX}$ and
		$\Delta_{RX}$ values - modify the SFP database for the WR PTP Core, or the
		\emph{/wr/etc/wrsw\_hal.conf} file for the WR Switch.

\subsection{WR Device calibration} 

Having the WR Calibrator selected among the available WR Devices, and the fiber
cable used in the previous steps (with known $\alpha$ coefficient and total
propagation delay $\delta$), you can start calibrating all other devices that
will later create a White Rabbit Network (fig.\ref{fig:devices}). 

247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311
		\caption{WR Device calibration with WR Calibrator and known
		fiber $f_1$}

	\item Connect your Calibrator to an unknown WR Device using the fiber with the
		known $\alpha$ measured earlier (\ref{subsec:fiasym}). Remember to use an
		appropriate SFP transceiver on each side depending on whether your device
		under calibration is supposed to run as a WR Master or a WR
		Slave\footnote{There is a wiki page describing which wavelength should be
		used on each side: \url{}}.
		If you want to have the flexibility of selecting Master/Slave mode later
		through device configuration, the calibration procedure described below
		should be performed twice - with the SFP that will be used in Master, and
		the SFP that will be used in Slave mode. In further steps of this procedure
		the assumption is made that the WR Device being calibrated runs in WR Slave
	\item Run the monitoring software on the Slave node and write down the
		round-trip delay ($delay_{MM}$) and fixed delays of both Master and Slave
		($\Delta_{TXM}$, $\Delta_{RXM}$, $\Delta_{TXS}$, $\Delta_{RXS}$). The
		transmission delay for Slave is reported to be 0, while the reception delay
		may be non-zero. It is the current RX bitslide value $\epsilon_S$.
	\item Calculate the average (coarse) transmission and reception delays for the
		Slave device:
			\Delta_{TXS}' = \Delta_{RXS}' = \frac{1}{2} \Delta_S = 
			\frac{1}{2}(delay_{MM} - \Delta_{TXM} - \Delta_{RXM} - 
			\epsilon_S - \delta_1)
	\item Write the $\Delta_{TXS}$ and $\Delta_{RXS}$ delays to the configuration
		of your device, and restart its PTP daemon so that it synchronizes again
		using the new values.
	\item At this point your WR Device knows the coarse values of the transmission
		and reception delays. You can connect the 1-PPS signals generated by the
		calibrator and the device to an oscilloscope to observe that there is still
		some offset of (probably) a few nanoseconds between the Slave and Master
		even if the PTP daemon reports it is close to 0. That is because the
		transmission and reception delays are not equal (as the coarse values
		above). This asymmetry has to be measured and used to correct the coarse
		values of $\Delta_{TXS}$ and $\Delta_{RXS}$.
	\item Measure the Slave to Master offset by comparing the 1-PPS skew with the
		oscilloscope. This is the correction value that has to be subtracted from
		the coarse $\Delta_{TX}$ and added to $\Delta_{RX}$ to compensate the
		hardware asymmetry:
			\Delta_{TXS} = \frac{1}{2}\Delta_S - skew_{PPS} \\
			\Delta_{RXS} = \frac{1}{2}\Delta_S + skew_{PPS}

		{\bf Note:} The asymmetry measured in this stage of calibration is in fact
		the sum of WR Device and WR Calibrator asymmetry. However, since both
		transmission and reception delays are modified with this value, the
		component for WR Calibrator asymmetry will cancel when connecting two WR
		Devices calibrated to the same Calibrator, see \ref{subsec:apx:devices} for
		a mathematical proof.
	\item After putting the new delay values in the configuration of the WR
		Device, the PTP daemon can be restarted and this time the device will
		synchronize to the Calibrator with an offset below 1ns.
	\item This procedure has to be repeated for all other WR Devices (WR Masters
		and WR Slaves). When you want to calibrate a WR Switch, this calibration
		procedure has to be performed for each WR port of the device.