\section{Mathematical proofs} \subsection{Reference fiber latency} \label{subsec:app_filat} The same pair of WR Devices is used for all three connections in this procedure. That is why, when considering round-trip delay after subtracting the bitslide values, the transmission and reception delays of both devices are summed together and remain constant for fiber $f_1$, $f_2$, $f_1 + f_2$: $$\Delta = \Delta_{TXM} + \Delta_{RXM} + \Delta_{TXS} + \Delta_{RXS}$$ When two fibers $f_1$, $f_2$ are joined together the fiber latency for this connection will be the sum of $\delta_1$ and $\delta_2$. After eliminating the bitslide value, the remaining part of a round trip delay consists of the following elements: $$delay_{MM1}' = \Delta + \delta_1\\$$ $$delay_{MM2}' = \Delta + \delta_2\\$$ $$delay_{MM3}' = \Delta + \delta_1 + \delta_2$$ This equation system has three unknowns and after solving gives the formulas for the round-trip fiber latencies: \begin{align} \delta_1 = delay_{MM3}' - delay_{MM2}'\\ \delta_2 = delay_{MM3}' - delay_{MM1}' \end{align} \subsection{Fiber asymmetry} \label{subsec:app_fiasym} In this step of the calibration procedure two WR connections with the same pair of devices are established. For each of them the offset between the WR Slave and WR Master is calculated by the WR PTP software as: $$offset_{MS} = t_1 - t_2 + delay_{MS}$$ \noindent where $delay_{MS}$ is an estimated one-way link delay. When $\alpha$ is initially equal to 0, $delay_{MS}$ is estimated as half of the round-trip delay, which results in a distorted offset between the two devices: $$offset_{MS}' = t_1 - t_2 + \frac{1}{2} delay_{MM}$$ \noindent Then $skew_{PPS}$ measured with an oscilloscope is equal to the uncompensated link asymmetry (the sum of the fiber asymmetry and hardware asymmetry): \begin{align} \label{equ:app_fiasym:skew1} skew_{PPS1} &= offset_{MS1} - offset'_{MS1} = delay_{MS1} - \frac{1}{2} delay_{MM1}\\ \label{equ:app_fiasym:skew2} skew_{PPS2} &= offset_{MS2} - offset'_{MS2} = delay_{MS2} - \frac{1}{2} delay_{MM2} \end{align} \noindent We also know what factors build up the round trip delays and one-way delays for both connections. Please notice since the same pair of the devices is used in both cases, fixed hardware delays stay the same: \begin{align} delay_{MM1} &= \Delta + \delta_1\\ delay_{MM2} &= \Delta + \delta_2\\ delay_{MS1} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS1}\\ delay_{MS2} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS2}\\ \delta_1 &= \delta_{MS1} + \delta_{SM1}\\ \delta_2 &= \delta_{MS2} + \delta_{SM2} \end{align} \noindent Using the formulas above, equations \ref{equ:app_fiasym:skew1} and \ref{equ:app_fiasym:skew2} can be expanded: \begin{align} skew_{PPS1} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS1} - \frac{1}{2}\Delta - \frac{1}{2} \delta_1\\ skew_{PPS2} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS2} - \frac{1}{2}\Delta - \frac{1}{2} \delta_2 \end{align} \noindent Subtracting the two skew measurements eliminates any asymmetry due to fixed hardware delays: \begin{align} \label{equ:app_fiasym:skew_pps} skew_{PPS} &= skew_{PPS2} - skew_{PPS1}\\ &= \Delta_{TXM} + \Delta_{RXS} - \Delta_{TXM} - \Delta_{RXS} + \delta_{MS2} - \delta_{MS1} - \frac{1}{2}\Delta + \frac{1}{2}\Delta - \frac{1}{2} \delta_2 + \frac{1}{2} \delta_1 \nonumber\\ &= \delta_{MS2} - \delta_{MS1} - \frac{1}{2}\delta_2 + \frac{1}{2}\delta_1 \end{align} \noindent However, if fiber $f_1$ is just a few meters long, then its asymmetry is negligible. That means its one-way Master-to-Slave latency equals half of the total fiber latency: $$\delta_{MS1} = \frac{1}{2} \delta_1$$ \noindent This results in a simplified formula describing $skew_{PPS}$: \begin{align} skew_{PPS} &= \delta_{MS2} - \frac{1}{2}\delta_2 = \delta_{MS2} - \frac{1}{2}\delta_{MS2} - \frac{1}{2}\delta_{SM2} \nonumber\\ \label{equ:app_fiasym:final_skew} &= \frac{1}{2}(\delta_{MS2} - \delta_{SM2}) \end{align} \noindent Having in mind that $\alpha = \frac{\delta_{MS} - \delta_{SM}}{\delta_{SM}}$, using the already known value of the $f_2$ round-trip latency $\delta_2$ and equations \ref{equ:app_fiasym:skew_pps}, \ref{equ:app_fiasym:final_skew} we get the expression for $\alpha$ used in the calibration procedure: $$\alpha = \frac{2(skew_{PPS2} - skew_{PPS1})}{\frac{1}{2}\delta_2 - (skew_{PPS2} - skew_{PPS1})}$$ \subsection{WR Device calibration} \label{subsec:apx:devices} After the WR PTP daemon on a Slave device is synchronized to Master, the $skew_{PPS}$ observed on an oscilloscope can be treated as an error of a clock correction on the Slave side: $$\label{equ:devices:corrs} corr = corr_{ideal} - skew_{PPS}$$ The correction value that should be applied to the Slave clock by the daemon ($corr_{ideal}$) is calculated based on timestamps and a $delay_{MS}$ estimation: $$corr_{ideal} = t_1 - t_2 + delay_{MS_{ideal}}$$ The one-way delay is the sum of the fiber latency, Master transmission delay and Slave reception delay: $$\label{equ:devices:ideal_delay} delay_{MS_{ideal}} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \Delta_{RXS}$$ However, the Slave reception delay used by the daemon is the result of the first 4 steps of the procedure in \ref{subsec:devices} ($\frac{1}{2}\Delta_S$). That means, it has to be corrected by an asymmetry coefficient $\beta$ to get the right value that produces $corr_{ideal}$ above: $$\label{equ:devices:delta_rxs} \Delta_{RXS} = \frac{1}{2}\Delta_S + \beta$$ The round-trip delay value and the sum of hardware delays are fixed, which means the same asymmetry factor has to be subtracted from the Slave transmission delay to preserve those sums: $$\label{equ:devices:delta_txs} \Delta_{TXS} = \frac{1}{2}\Delta_S - \beta$$ \noindent Taking it back to equation \ref{equ:devices:ideal_delay} we get: $$delay_{MS_{ideal}} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \frac{1}{2}\Delta_S + \beta$$ However, the Master to Slave delay calculated by the daemon using the values without the asymmetry taken into account is: $$delay_{MS} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \frac{1}{2}\Delta_S$$ So the correction value for the reception asymmetry is also the difference between the $delay_{MS}$ estimations: $$delay_{MS_{ideal}} = delay_{MS} + \beta$$ \noindent Putting this back into the equation for $corr_{ideal}$: $$corr_{ideal} = t_1 - t_2 + delay_{MS} + \beta$$ \noindent Please remember though, $t_1 - t_2 + delay_{MS}$ is in fact the correction value ($corr$) derived from the coarse (without asymmetry) Slave delays: $$corr_{ideal} = corr + \beta$$ Comparing the equation above with \ref{equ:devices:corrs}: $$\beta = skew_{PPS}$$ That means, the difference between 1-PPS signals observed on the oscilloscope has to be used as the correction factor for the coarse delays of the Slave device.\\ The asymmetry of each calibrated Tx/Rx delay is set to compensate also the asymmetry of the WR Calibrator. Equations \ref{equ:devices:delta_rxs} and \ref{equ:devices:delta_txs} can be expanded to show the components of asymmetry $\beta$ of two WR Devices calibrated to the same WR Calibrator (where $\beta_C$ is the calibrator asymmetry and $\beta_1$, $\beta_2$ are the internal asymmetries of each device): \begin{align} \Delta_{TX1} = \frac{1}{2}\Delta_1 - \beta_{C1} = \frac{1}{2}\Delta_1 - \beta_1 + \beta_C \\ \Delta_{RX1} = \frac{1}{2}\Delta_1 + \beta_{C1} = \frac{1}{2}\Delta_1 + \beta_1 - \beta_C \\ \Delta_{TX2} = \frac{1}{2}\Delta_2 - \beta_{C2} = \frac{1}{2}\Delta_2 - \beta_2 + \beta_C \\ \Delta_{RX2} = \frac{1}{2}\Delta_2 + \beta_{C2} = \frac{1}{2}\Delta_2 + \beta_2 - \beta_C \end{align} After connecting those two WR Devices together, the transmission circuits of each one communicate with the reception circuits of the other, resulting in a one-way link delay (without fiber propagation latency): \begin{align} \Delta_{1-2} = \Delta_{TX1} + \Delta_{RX2} = \frac{1}{2}\Delta_1 - \beta_1 + \beta_C + \frac{1}{2} \Delta_2 + \beta_2 - \beta_C = (\frac{1}{2}\Delta_1 - \beta_1) + (\frac{1}{2}\Delta_2 + \beta_2) \\ \Delta_{2-1} = \Delta_{TX2} + \Delta_{RX1} = \frac{1}{2}\Delta_2 - \beta_2 + \beta_C + \frac{1}{2} \Delta_1 + \beta_1 - \beta_C = (\frac{1}{2}\Delta_2 - \beta_2) + (\frac{1}{2}\Delta_1 + \beta_1) \end{align} This proves that devices which have been calibrated using the same WR Calibrator can use the asymmetries found during the calibration process to synchronize one another. \subsection{Measurement with a loop-back fiber} For both measurements the same loop-back fiber, optical transmitter and optical receiver are used. There is also a requirement in the measurement procedure (section \ref{subsec:loopback}) saying that both transmitter and receiver should have a constant delay that doesn't vary for each connection. That means, for both steps, the loop-back link has some unknown latency $\delta_{L}$.\\ In the first case, the 1-PPS skew measured on the WR Master side can be represented with the formula: $$\label{equ:loopback:skew1} skew_{PPS1} = t_{PPSM1} - (t_{PPSS1} + \delta_{L})$$ where $t_{PPSM1}$ is a WR Master absolute time of 1-PPS generation, $t_{PPSS1}$ is a WR Slave absolute time of 1-PPS generation. The latency of the loop-back fiber $\delta_{L}$ is added to $t_{PPSS1}$, because in the first step the Slave 1-PPS signal observed on the WR Master side is delayed by $\delta_{L}$ picoseconds.\\ In the second step, the situation is reversed. The measurement is made on the WR Slave side, which means the 1-PPS generated from the WR Master is observed $\delta_{L}$ picoseconds later: $$\label{equ:loopback:skew2} skew_{PPS2} = (t_{PPSM2} + \delta_{L}) - t_{PPSS2}$$ The actual $skew_{PPS}$ that we want to measure within this procedure is the difference between the absolute time of the 1-PPS generation on Master and Slave: $$\label{equ:lookback:offset} skew_{PPS} = t_{PPSM1} - t_{PPSS1} = t_{PPSM2} - t_{PPSS2}$$ Of course we can make those subtractions equal only because the measurement in both cases is done when WR Master and WR Slave are synchronized. Now, putting together equations \ref{equ:loopback:skew1}, \ref{equ:loopback:skew2} and \ref{equ:lookback:offset} the following system of equations with two unknowns is produced: \begin{align} skew_{PPS1} = skew_{PPS} - \delta_{L}\\ skew_{PPS2} = skew_{PPS} + \delta_{L} \end{align} Solving it creates the final formula to calculate the 1-PPS skew between the WR Master and the WR Slave: $$skew_{PPS} = \frac{1}{2} (skew_{PPS1} + skew_{PPS2})$$ \subsection{Recovering the calibrator} The new WR Calibrator has unknown transmission and reception delays as any other, uncalibrated WR Device. We represent them using the mean (coarse) delay ($\Delta_{C2}$) and the asymmetry factor ($\beta_{C2}$): \begin{align} \Delta_{TXC2} = \frac{1}{2}\Delta_{C2} - \beta_{C2}\\ \Delta_{RXC2} = \frac{1}{2}\Delta_{C2} + \beta_{C2} \end{align} We already know from the previous sections that a WR Device (D1) calibrated to the primary calibrator (C1) compensates its own asymmetry but also the asymmetry of the WR Calibrator: \begin{align} \Delta_{TXD1} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \beta_{C1} \\ \Delta_{RXD1} = \frac{1}{2}\Delta_{D1} + \beta_{D1} - \beta_{C1} \end{align} In an ideal case, when each WR Device knows its delays, the Master-to-Slave (one-way) delay without the fiber propagation latency would be: $$\label{equ:recc:delaymsideal} \Delta_{D1-C2_{ideal}} = \Delta_{TXD1_{ideal}} + \Delta_{RXC2_{ideal}} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \frac{1}{2}\Delta_{C2} + \beta_{C2}$$ On the other hand, since the WR Device \emph{D1} compensates also the asymmetry of the primary calibrator \emph{C1} and initially $\beta_{C2}$ is unknown (set to 0), the actual fixed delay for \emph{D1}-\emph{C2} connection is: $$\label{equ:recc:delayms} \Delta_{D1-C2} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \beta_{C1} + \frac{1}{2}\Delta_{C2}$$ Comparing equations \ref{equ:recc:delayms} and \ref{equ:recc:delaymsideal} it can be noticed that the factor $\beta_{C1}$ partially compensates the asymmetry of the new calibrator \emph{C2}. The uncompensated part: $$\beta'_{C2} = \beta_{C2} - \beta_{C1}$$ produces an additional skew of the 1-PPS signals in the same way as the uncompensated asymmetry of the WR Device in section \ref{subsec:apx:devices}: $$skew_{PPS} = \beta_{C2} - \beta_{C1}$$ This remaining asymmetry of the \emph{D1}-\emph{C2} connection is compensated in the calibration procedure by using the 1-PPS skew as the correction factor. Then, the transmission and reception delays of the new calibrator \emph{C2} are presented in the equations: \begin{align} \Delta_{TXC2} = \frac{1}{2}\Delta_{C2} - skew_{PPS} = \frac{1}{2}\Delta_{C2} - \beta_{C2} + \beta_{C1}\\ \Delta_{RXC2} = \frac{1}{2}\Delta_{C2} + skew_{PPS} = \frac{1}{2}\Delta_{C2} + \beta_{C2} - \beta_{C1} \end{align} Each of them has the asymmetry factor $\beta_{C2}$ reduced by $\beta_{C1}$ so that the actual hardware asymmetry is reduced only partially. The remaining, uncompensated part equals the asymmetry of the primary calibrator \emph{C1}, so that the new calibrator \emph{C2} behaves for all practical purposes as the old calibrator \emph{C1}. \subsection{1-PPS skew measurement} Reading proposed calibration procedure one can start wondering what is the influence of 1-PPS propagation time - from the inside of FPGA to the physical connector - on the 1-PPS skew measurement. Let's consider one more time two WR Devices (\emph{D1}, \emph{D2}) being calibrated to the calibrator \emph{C}. This time we take into account the 1-PPS propagation delay from the inside of FPGA to the physical connector where we take it for skew measurement (figure \ref{fig:ppsdel:calibration}). Those delays are marked $\tau_C$, $\tau_1$, $\tau_2$ for the calibrator, device under calibration 1 and device under calibration 2. \begin{figure}[ht] \begin{center} \includegraphics[width=\textwidth]{calibration/calibration_pps_delay.pdf} \caption{Calibration with 1-PPS delays taken into account} \label{fig:ppsdel:calibration} \end{center} \end{figure} Therefore e.g. 1-PPS signal generated inside the FPGA of the WR Calibrator at time $t_{CPPS}$ is observed on the oscilloscope at time $t_{CPPS} + \tau_C$. Taking this into account, our skew measured in section \ref{subsec:devices} can be expanded as: \begin{align} skew'_{1C} &= (t_{1PPS} + \tau_1) - (t_{CPPS} + \tau_C) \nonumber\\ &= (t_{1PPS} - t_{CPPS}) + (\tau_1 - \tau_C) \nonumber\\ &= skew_{1C} + (\tau_1 - \tau_C) \end{align} According to the calibration procedure we apply the measured skew (in our case $skew'_{1C}$) as an asymmetry factor ($\beta_{C1}$ in section \ref{subsec:apx:devices}) to calculate fixed hardware delays $\Delta_{TX}$, $\Delta_{RX}$ for the device under calibration. Thus, our asymmetry factor also contains the difference in 1-PPS propagation times: $$\beta'_{C1} = \beta_{C1} + (\tau_1 - \tau_C)$$ As a consequence, fixed transmission and reception delays for device $D_1$ calculated from the coarse delay and asymmetry factor $\beta'_{C1}$ will also contain 1-PPS propagation times: \begin{align} \Delta'_{TX1} &= \frac{1}{2}\Delta_1 - \beta'_{C1} = \frac{1}{2}\Delta_1 - \beta_{C1} - (\tau_1 - \tau_C) = \Delta_{TX1} - (\tau_1 - \tau_C)\\ \Delta'_{RX1} &= \frac{1}{2}\Delta_1 + \beta'_{C1} = \frac{1}{2}\Delta_1 + \beta_{C1} + (\tau_1 - \tau_C) = \Delta_{RX1} + (\tau_1 - \tau_C) \end{align} By analogy we get the same result for device $D_2$ calibration: \begin{align} \Delta'_{TX2} &= \frac{1}{2}\Delta_2 - \beta'_{C2} = \frac{1}{2}\Delta_2 - \beta_{C2} - (\tau_2 - \tau_C) = \Delta_{TX2} - (\tau_2 - \tau_C)\\ \Delta'_{RX2} &= \frac{1}{2}\Delta_2 + \beta'_{C2} = \frac{1}{2}\Delta_2 + \beta_{C2} + (\tau_2 - \tau_C) = \Delta_{RX2} + (\tau_2 - \tau_C) \end{align} We can see that after performing the calibration procedure, both of these devices have their fixed hardware delays distorted by the difference in 1-PPS propagation delay of the device and the calibrator. \begin{figure}[ht] \begin{center} \includegraphics[width=.5\textwidth]{calibration/wr_devices_pps_delay.pdf} \caption{Synchronization of WR Devices calibrated to the same calibrator} \label{fig:ppsdel:sync} \end{center} \end{figure} When we connect two devices together and let them synchronize (figure \ref{fig:ppsdel:sync}), the propagation delay of calibrator's PPS signal gets canceled in the one-way delay calculation: \begin{align} delay'_{MS} &= \delta_{MS} + \Delta'_{TX1} + \Delta'_{RX2} = \delta_{MS} + (\Delta_{TX1} - \tau_1 + \tau_C) + (\Delta_{RX2} + \tau_2 - \tau_C)\\ delay'_{MS} &= delay_{MS} + (\tau_2 - \tau_1) \end{align} Now, the one-way delay is distorted only by the difference between $D_2$ and $D_1$ 1-PPS propagation delays. Having in mind the formula for a correction factor applied on the Slave side ($corr_{ideal}$ in section \ref{subsec:apx:devices}) we can see that in our case the distortion of $delay_{MS}$ directly affects $corr_{ideal}$ as well: $$corr = corr_{ideal} + (\tau_2 - \tau_1)$$ The change in the $corr$ value shifts the timescale of the slave device ahead by $(\tau_2 - \tau_1)$ so for this connection every 1-PPS pulse from $D_2$ is generated earlier than it should be in the ideal case: $$t'_{2PPS} = t_{2PPS} - (\tau_2 - \tau_1)$$ $skew'_{12}$ between $D_1$ and $D_2$ measured with the oscilloscope is: \begin{align} skew'_{21} &= (t'_{2PPS} + \tau_2) - (t_{1PPS} + \tau_1) = t_{2PPS} - \tau_2 + \tau_1 + \tau_2 - t_{1PPS} - \tau_1\\ skew'_{21} &= t_{2PPS} - t_{1PPS} \end{align} This shows that the difference between 1-PPS propagation delays causes Slave device to have its internal time shifted to compensate for this difference when two devices (calibrated earlier to the same calibrator) are connected together. Therefore when these devices are synchronized, their 1-PPS signals will be aligned and the difference in their propagation times is properly compensated. 1-PPS socket is our calibration reference plane. One can move the reference plane to any other point, also to the inside of the FPGA. However, this requires the knowledge of precise 1-PPS delay value to be taken into account in the PPS skew measurements.