appendix.tex 18.8 KB
 Grzegorz Daniluk committed Sep 10, 2014 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 \section{Mathematical proofs} \subsection{Reference fiber latency} \label{subsec:app_filat} The same pair of WR Devices is used for all three connections in this procedure. That is why, when considering round-trip delay after subtracting the bitslide values, the transmission and reception delays of both devices are summed together and remain constant for fiber $f_1$, $f_2$, $f_1 + f_2$: \Delta = \Delta_{TXM} + \Delta_{RXM} + \Delta_{TXS} + \Delta_{RXS} When two fibers $f_1$, $f_2$ are joined together the fiber latency for this connection will be the sum of $\delta_1$ and $\delta_2$. After eliminating the bitslide value, the remaining part of a round trip delay consists of the following elements: delay_{MM1}' = \Delta + \delta_1\\ delay_{MM2}' = \Delta + \delta_2\\ delay_{MM3}' = \Delta + \delta_1 + \delta_2 This equation system has three unknowns and after solving gives the formulas for the round-trip fiber latencies: \begin{align} \delta_1 = delay_{MM3}' - delay_{MM2}'\\ \delta_2 = delay_{MM3}' - delay_{MM1}' \end{align} \subsection{Fiber asymmetry} \label{subsec:app_fiasym} In this step of the calibration procedure two WR connections with the same pair of devices are established. For each of them the offset between the WR Slave and WR Master is calculated by the WR PTP software as: offset_{MS} = t_1 - t_2 + delay_{MS} \noindent where $delay_{MS}$ is an estimated one-way link delay. When $\alpha$ is initially equal to 0, $delay_{MS}$ is estimated as half of the round-trip delay, which results in a distorted offset between the two devices: offset_{MS}' = t_1 - t_2 + \frac{1}{2} delay_{MM} \noindent Then $skew_{PPS}$ measured with an oscilloscope is equal to the uncompensated link asymmetry (the sum of the fiber asymmetry and hardware asymmetry): \begin{align} \label{equ:app_fiasym:skew1} skew_{PPS1} &= offset_{MS1} - offset'_{MS1} = delay_{MS1} - \frac{1}{2} delay_{MM1}\\ \label{equ:app_fiasym:skew2} skew_{PPS2} &= offset_{MS2} - offset'_{MS2} = delay_{MS2} - \frac{1}{2} delay_{MM2} \end{align} \noindent We also know what factors build up the round trip delays and one-way delays for both connections. Please notice since the same pair of the devices is used in both cases, fixed hardware delays stay the same: \begin{align} delay_{MM1} &= \Delta + \delta_1\\ delay_{MM2} &= \Delta + \delta_2\\ delay_{MS1} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS1}\\ delay_{MS2} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS2}\\ \delta_1 &= \delta_{MS1} + \delta_{SM1}\\ \delta_2 &= \delta_{MS2} + \delta_{SM2} \end{align} \noindent Using the formulas above, equations \ref{equ:app_fiasym:skew1} and \ref{equ:app_fiasym:skew2} can be expanded: \begin{align} skew_{PPS1} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS1} - \frac{1}{2}\Delta - \frac{1}{2} \delta_1\\ skew_{PPS2} &= \Delta_{TXM} + \Delta_{RXS} + \delta_{MS2} - \frac{1}{2}\Delta - \frac{1}{2} \delta_2 \end{align} \noindent Subtracting the two skew measurements eliminates any asymmetry due to fixed hardware delays: \begin{align} \label{equ:app_fiasym:skew_pps} skew_{PPS} &= skew_{PPS2} - skew_{PPS1}\\ &= \Delta_{TXM} + \Delta_{RXS} - \Delta_{TXM} - \Delta_{RXS} + \delta_{MS2} - \delta_{MS1} - \frac{1}{2}\Delta + \frac{1}{2}\Delta - \frac{1}{2} \delta_2 + \frac{1}{2} \delta_1 \nonumber\\ &= \delta_{MS2} - \delta_{MS1} - \frac{1}{2}\delta_2 + \frac{1}{2}\delta_1 \end{align} \noindent However, if fiber $f_1$ is just a few meters long, then its asymmetry is negligible. That means its one-way Master-to-Slave latency equals half of the total fiber latency: \delta_{MS1} = \frac{1}{2} \delta_1 \noindent This results in a simplified formula describing $skew_{PPS}$: \begin{align} skew_{PPS} &= \delta_{MS2} - \frac{1}{2}\delta_2 = \delta_{MS2} - \frac{1}{2}\delta_{MS2} - \frac{1}{2}\delta_{SM2} \nonumber\\ \label{equ:app_fiasym:final_skew} &= \frac{1}{2}(\delta_{MS2} - \delta_{SM2}) \end{align} \noindent Having in mind that $\alpha = \frac{\delta_{MS} - \delta_{SM}}{\delta_{SM}}$, using the already known value of the $f_2$ round-trip latency $\delta_2$ and equations \ref{equ:app_fiasym:skew_pps}, \ref{equ:app_fiasym:final_skew} we get the expression for $\alpha$ used in the calibration procedure: \alpha = \frac{2(skew_{PPS2} - skew_{PPS1})}{\frac{1}{2}\delta_2 - (skew_{PPS2} - skew_{PPS1})} \subsection{WR Device calibration} \label{subsec:apx:devices} After the WR PTP daemon on a Slave device is synchronized to Master, the $skew_{PPS}$ observed on an oscilloscope can be treated as an error of a clock correction on the Slave side: \label{equ:devices:corrs} corr = corr_{ideal} - skew_{PPS} The correction value that should be applied to the Slave clock by the daemon ($corr_{ideal}$) is calculated based on timestamps and a $delay_{MS}$ estimation: corr_{ideal} = t_1 - t_2 + delay_{MS_{ideal}} The one-way delay is the sum of the fiber latency, Master transmission delay and Slave reception delay: \label{equ:devices:ideal_delay} delay_{MS_{ideal}} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \Delta_{RXS} However, the Slave reception delay used by the daemon is the result of the first 4 steps of the procedure in \ref{subsec:devices} ($\frac{1}{2}\Delta_S$). That means, it has to be corrected by an asymmetry coefficient $\beta$ to get the right value that produces $corr_{ideal}$ above: \label{equ:devices:delta_rxs} \Delta_{RXS} = \frac{1}{2}\Delta_S + \beta The round-trip delay value and the sum of hardware delays are fixed, which means the same asymmetry factor has to be subtracted from the Slave transmission delay to preserve those sums: \label{equ:devices:delta_txs} \Delta_{TXS} = \frac{1}{2}\Delta_S - \beta \noindent Taking it back to equation \ref{equ:devices:ideal_delay} we get: delay_{MS_{ideal}} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \frac{1}{2}\Delta_S + \beta However, the Master to Slave delay calculated by the daemon using the values without the asymmetry taken into account is: delay_{MS} = \frac{1+\alpha}{2+\alpha}(delay_{MM} - \Delta) + \Delta_{TXM} + \frac{1}{2}\Delta_S So the correction value for the reception asymmetry is also the difference between the $delay_{MS}$ estimations: delay_{MS_{ideal}} = delay_{MS} + \beta \noindent Putting this back into the equation for $corr_{ideal}$: corr_{ideal} = t_1 - t_2 + delay_{MS} + \beta \noindent Please remember though, $t_1 - t_2 + delay_{MS}$ is in fact the correction value ($corr$) derived from the coarse (without asymmetry) Slave delays: corr_{ideal} = corr + \beta Comparing the equation above with \ref{equ:devices:corrs}: \beta = skew_{PPS} That means, the difference between 1-PPS signals observed on the oscilloscope has to be used as the correction factor for the coarse delays of the Slave device.\\ The asymmetry of each calibrated Tx/Rx delay is set to compensate also the asymmetry of the WR Calibrator. Equations \ref{equ:devices:delta_rxs} and \ref{equ:devices:delta_txs} can be expanded to show the components of asymmetry $\beta$ of two WR Devices calibrated to the same WR Calibrator (where $\beta_C$ is the calibrator asymmetry and $\beta_1$, $\beta_2$ are the internal asymmetries of each device): \begin{align} \Delta_{TX1} = \frac{1}{2}\Delta_1 - \beta_{C1} = \frac{1}{2}\Delta_1 - \beta_1 + \beta_C \\ \Delta_{RX1} = \frac{1}{2}\Delta_1 + \beta_{C1} = \frac{1}{2}\Delta_1 + \beta_1 - \beta_C \\ \Delta_{TX2} = \frac{1}{2}\Delta_2 - \beta_{C2} = \frac{1}{2}\Delta_2 - \beta_2 + \beta_C \\ \Delta_{RX2} = \frac{1}{2}\Delta_2 + \beta_{C2} = \frac{1}{2}\Delta_2 + \beta_2 - \beta_C \end{align} After connecting those two WR Devices together, the transmission circuits of each one communicate with the reception circuits of the other, resulting in a one-way link delay (without fiber propagation latency): \begin{align} \Delta_{1-2} = \Delta_{TX1} + \Delta_{RX2} = \frac{1}{2}\Delta_1 - \beta_1 + \beta_C + \frac{1}{2} \Delta_2 + \beta_2 - \beta_C = (\frac{1}{2}\Delta_1 - \beta_1) + (\frac{1}{2}\Delta_2 + \beta_2) \\ \Delta_{2-1} = \Delta_{TX2} + \Delta_{RX1} = \frac{1}{2}\Delta_2 - \beta_2 + \beta_C + \frac{1}{2} \Delta_1 + \beta_1 - \beta_C = (\frac{1}{2}\Delta_2 - \beta_2) + (\frac{1}{2}\Delta_1 + \beta_1) \end{align} This proves that devices which have been calibrated using the same WR Calibrator can use the asymmetries found during the calibration process to synchronize one another. \subsection{Measurement with a loop-back fiber} For both measurements the same loop-back fiber, optical transmitter and optical receiver are used. There is also a requirement in the measurement procedure (section \ref{subsec:loopback}) saying that both transmitter and receiver should have a constant delay that doesn't vary for each connection. That means, for both steps, the loop-back link has some unknown latency $\delta_{L}$.\\ In the first case, the 1-PPS skew measured on the WR Master side can be represented with the formula: \label{equ:loopback:skew1} skew_{PPS1} = t_{PPSM1} - (t_{PPSS1} + \delta_{L}) where $t_{PPSM1}$ is a WR Master absolute time of 1-PPS generation, $t_{PPSS1}$ is a WR Slave absolute time of 1-PPS generation. The latency of the loop-back fiber $\delta_{L}$ is added to $t_{PPSS1}$, because in the first step the Slave 1-PPS signal observed on the WR Master side is delayed by $\delta_{L}$ picoseconds.\\ In the second step, the situation is reversed. The measurement is made on the WR Slave side, which means the 1-PPS generated from the WR Master is observed $\delta_{L}$ picoseconds later: \label{equ:loopback:skew2} skew_{PPS2} = (t_{PPSM2} + \delta_{L}) - t_{PPSS2} The actual $skew_{PPS}$ that we want to measure within this procedure is the difference between the absolute time of the 1-PPS generation on Master and Slave: \label{equ:lookback:offset} skew_{PPS} = t_{PPSM1} - t_{PPSS1} = t_{PPSM2} - t_{PPSS2} Of course we can make those subtractions equal only because the measurement in both cases is done when WR Master and WR Slave are synchronized. Now, putting together equations \ref{equ:loopback:skew1}, \ref{equ:loopback:skew2} and \ref{equ:lookback:offset} the following system of equations with two unknowns is produced: \begin{align} skew_{PPS1} = skew_{PPS} - \delta_{L}\\ skew_{PPS2} = skew_{PPS} + \delta_{L} \end{align} Solving it creates the final formula to calculate the 1-PPS skew between the WR Master and the WR Slave: skew_{PPS} = \frac{1}{2} (skew_{PPS1} + skew_{PPS2}) \subsection{Recovering the calibrator} The new WR Calibrator has unknown transmission and reception delays as any other, uncalibrated WR Device. We represent them using the mean (coarse) delay ($\Delta_{C2}$) and the asymmetry factor ($\beta_{C2}$): \begin{align} \Delta_{TXC2} = \frac{1}{2}\Delta_{C2} - \beta_{C2}\\ \Delta_{RXC2} = \frac{1}{2}\Delta_{C2} + \beta_{C2} \end{align} We already know from the previous sections that a WR Device (D1) calibrated to the primary calibrator (C1) compensates its own asymmetry but also the asymmetry of the WR Calibrator: \begin{align} \Delta_{TXD1} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \beta_{C1} \\ \Delta_{RXD1} = \frac{1}{2}\Delta_{D1} + \beta_{D1} - \beta_{C1} \end{align} In an ideal case, when each WR Device knows its delays, the Master-to-Slave (one-way) delay without the fiber propagation latency would be: \label{equ:recc:delaymsideal} \Delta_{D1-C2_{ideal}} = \Delta_{TXD1_{ideal}} + \Delta_{RXC2_{ideal}} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \frac{1}{2}\Delta_{C2} + \beta_{C2} On the other hand, since the WR Device \emph{D1} compensates also the asymmetry of the primary calibrator \emph{C1} and initially $\beta_{C2}$ is unknown (set to 0), the actual fixed delay for \emph{D1}-\emph{C2} connection is: \label{equ:recc:delayms} \Delta_{D1-C2} = \frac{1}{2}\Delta_{D1} - \beta_{D1} + \beta_{C1} + \frac{1}{2}\Delta_{C2} Comparing equations \ref{equ:recc:delayms} and \ref{equ:recc:delaymsideal} it can be noticed that the factor $\beta_{C1}$ partially compensates the asymmetry of the new calibrator \emph{C2}. The uncompensated part: \beta'_{C2} = \beta_{C2} - \beta_{C1} produces an additional skew of the 1-PPS signals in the same way as the uncompensated asymmetry of the WR Device in section \ref{subsec:apx:devices}: skew_{PPS} = \beta_{C2} - \beta_{C1} This remaining asymmetry of the \emph{D1}-\emph{C2} connection is compensated in the calibration procedure by using the 1-PPS skew as the correction factor. Then, the transmission and reception delays of the new calibrator \emph{C2} are presented in the equations: \begin{align} \Delta_{TXC2} = \frac{1}{2}\Delta_{C2} - skew_{PPS} = \frac{1}{2}\Delta_{C2} - \beta_{C2} + \beta_{C1}\\ \Delta_{RXC2} = \frac{1}{2}\Delta_{C2} + skew_{PPS} = \frac{1}{2}\Delta_{C2} + \beta_{C2} - \beta_{C1} \end{align} Each of them has the asymmetry factor $\beta_{C2}$ reduced by $\beta_{C1}$ so that the actual hardware asymmetry is reduced only partially. The remaining, uncompensated part equals the asymmetry of the primary calibrator \emph{C1}, so that the new calibrator \emph{C2} behaves for all practical purposes as the old calibrator \emph{C1}.  Grzegorz Daniluk committed May 23, 2015 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437  \subsection{1-PPS skew measurement} Reading proposed calibration procedure one can start wondering what is the influence of 1-PPS propagation time - from the inside of FPGA to the physical connector - on the 1-PPS skew measurement. Let's consider one more time two WR Devices (\emph{D1}, \emph{D2}) being calibrated to the calibrator \emph{C}. This time we take into account the 1-PPS propagation delay from the inside of FPGA to the physical connector where we take it for skew measurement (figure \ref{fig:ppsdel:calibration}). Those delays are marked $\tau_C$, $\tau_1$, $\tau_2$ for the calibrator, device under calibration 1 and device under calibration 2. \begin{figure}[ht] \begin{center} \includegraphics[width=\textwidth]{calibration/calibration_pps_delay.pdf} \caption{Calibration with 1-PPS delays taken into account} \label{fig:ppsdel:calibration} \end{center} \end{figure} Therefore e.g. 1-PPS signal generated inside the FPGA of the WR Calibrator at time $t_{CPPS}$ is observed on the oscilloscope at time $t_{CPPS} + \tau_C$. Taking this into account, our skew measured in section \ref{subsec:devices} can be expanded as: \begin{align} skew'_{1C} &= (t_{1PPS} + \tau_1) - (t_{CPPS} + \tau_C) \nonumber\\ &= (t_{1PPS} - t_{CPPS}) + (\tau_1 - \tau_C) \nonumber\\ &= skew_{1C} + (\tau_1 - \tau_C) \end{align} According to the calibration procedure we apply the measured skew (in our case $skew'_{1C}$) as an asymmetry factor ($\beta_{C1}$ in section \ref{subsec:apx:devices}) to calculate fixed hardware delays $\Delta_{TX}$, $\Delta_{RX}$ for the device under calibration. Thus, our asymmetry factor also contains the difference in 1-PPS propagation times: \beta'_{C1} = \beta_{C1} + (\tau_1 - \tau_C) As a consequence, fixed transmission and reception delays for device $D_1$ calculated from the coarse delay and asymmetry factor $\beta'_{C1}$ will also contain 1-PPS propagation times: \begin{align} \Delta'_{TX1} &= \frac{1}{2}\Delta_1 - \beta'_{C1} = \frac{1}{2}\Delta_1 - \beta_{C1} - (\tau_1 - \tau_C) = \Delta_{TX1} - (\tau_1 - \tau_C)\\ \Delta'_{RX1} &= \frac{1}{2}\Delta_1 + \beta'_{C1} = \frac{1}{2}\Delta_1 + \beta_{C1} + (\tau_1 - \tau_C) = \Delta_{RX1} + (\tau_1 - \tau_C) \end{align} By analogy we get the same result for device $D_2$ calibration: \begin{align} \Delta'_{TX2} &= \frac{1}{2}\Delta_2 - \beta'_{C2} = \frac{1}{2}\Delta_2 - \beta_{C2} - (\tau_2 - \tau_C) = \Delta_{TX2} - (\tau_2 - \tau_C)\\ \Delta'_{RX2} &= \frac{1}{2}\Delta_2 + \beta'_{C2} = \frac{1}{2}\Delta_2 + \beta_{C2} + (\tau_2 - \tau_C) = \Delta_{RX2} + (\tau_2 - \tau_C) \end{align} We can see that after performing the calibration procedure, both of these devices have their fixed hardware delays distorted by the difference in 1-PPS propagation delay of the device and the calibrator. \begin{figure}[ht] \begin{center} \includegraphics[width=.5\textwidth]{calibration/wr_devices_pps_delay.pdf} \caption{Synchronization of WR Devices calibrated to the same calibrator} \label{fig:ppsdel:sync} \end{center} \end{figure} When we connect two devices together and let them synchronize (figure \ref{fig:ppsdel:sync}), the propagation delay of calibrator's PPS signal gets canceled in the one-way delay calculation: \begin{align} delay'_{MS} &= \delta_{MS} + \Delta'_{TX1} + \Delta'_{RX2} = \delta_{MS} + (\Delta_{TX1} - \tau_1 + \tau_C) + (\Delta_{RX2} + \tau_2 - \tau_C)\\ delay'_{MS} &= delay_{MS} + (\tau_2 - \tau_1) \end{align} Now, the one-way delay is distorted only by the difference between $D_2$ and $D_1$ 1-PPS propagation delays. Having in mind the formula for a correction factor applied on the Slave side ($corr_{ideal}$ in section \ref{subsec:apx:devices}) we can see that in our case the distortion of $delay_{MS}$ directly affects $corr_{ideal}$ as well: corr = corr_{ideal} + (\tau_2 - \tau_1) The change in the $corr$ value shifts the timescale of the slave device ahead by $(\tau_2 - \tau_1)$ so for this connection every 1-PPS pulse from $D_2$ is generated earlier than it should be in the ideal case: t'_{2PPS} = t_{2PPS} - (\tau_2 - \tau_1) $skew'_{12}$ between $D_1$ and $D_2$ measured with the oscilloscope is: \begin{align} skew'_{21} &= (t'_{2PPS} + \tau_2) - (t_{1PPS} + \tau_1) = t_{2PPS} - \tau_2 + \tau_1 + \tau_2 - t_{1PPS} - \tau_1\\ skew'_{21} &= t_{2PPS} - t_{1PPS} \end{align} This shows that the difference between 1-PPS propagation delays causes Slave device to have its internal time shifted to compensate for this difference when two devices (calibrated earlier to the same calibrator) are connected together. Therefore when these devices are synchronized, their 1-PPS signals will be aligned and the difference in their propagation times is properly compensated. 1-PPS socket is our calibration reference plane. One can move the reference plane to any other point, also to the inside of the FPGA. However, this requires the knowledge of precise 1-PPS delay value to be taken into account in the PPS skew measurements.