Frequently Asked Questions

about White Rabbit.

Frequently Asked Questions

White Rabbit general questions

Q: What's the easiest way to start playing with White Rabbit?

A: For starting to use WR technology a WR starting kit is available. The manual you can find there shows clearly what is possible with just two PCIe plug-in cards. Later on you can expand your kit with a White Rabbit switch.

Q: I am considering using White Rabbit in my next project. Where can I get support?

A: White Rabbit (WR) is open source and commercial. That means that you can get the best of both worlds. You can avoid vendor lock-in because the technology is open source, so you can switch vendors if you are unhappy with the service a particular vendor provides. Because it is open source, provided your team has adequate know-how, you can even decide to not rely on commercial companies for support. But if you want the convenience of paying somebody for guaranteed support, you can also have that. In general, each WR user has a particular problem to solve, and it would be hard to design a one-size-fits-all WR solution. WR starter kits illustrate the basic capabilities of WR and provide a good basis to build upon. There is often a need for local design customization, and that is why we recommend that WR users have either the backing of a local design team or a commercial company. There is also the [white-rabbit-dev](https://forums.ohwr.org/c/white-rabbit-dev/344 discourse forum, where you can ask questions. But for obvious reasons, experts in white-rabbit-dev cannot guarantee they will answer all your questions in a timely manner, or even at all. That would not be scalable. Answers on white-rabbit-dev are therefore provided on a good-will basis. The same is true for email you send privately to individual core WR developers. If you need guaranteed support, a local team or a company is the way to go.

Q: How can I test White Rabbit switch equipment? Or can I buy some test equipments from your company?

A: CERN is not a company, but a publicly funded intergovernemental organization. We develop the technology but don't produce the equipment, we buy WR switches from commercial vendors. Anyone can produce a WR switch as all the sources are on our pages (just have a careful look at the wiki, sub-projects, ets... )
For example, there is wiki page about the WR switch on which you can find the companies that are involved in switch development and the companies from which you can buy a WR switch (Commercial producers, at the bottom). On the wiki page about SPEC card (an example of WR Node) you will find companies producing SPEC cards.

Q: If I don’t participate in WR workshop meetings, is the presentation material available?

A: All we do is public. All information from the workshops is directly accessible.

Q: How can I trust White Rabbit?

A: Most White Rabbit equipment is Open Hardware and Open Software. You have full access to all design data and can inspect and recompile the source code yourself to verify anything you like.
Many tests have been done by independent institutes to verify the timing synchronisation and data transmission performance. Some institutes indeed had found issues that have been solved since then. Other issues have been found and are being worked on.

Q: How important is White Rabbit?

A1: Have a look at the many institutes and companies who are using White Rabbit at the list of Users of White Rabbit Technology.
A2: That's a hard question. It is important enough to have been specifically mentioned in many job offers at different institutes and companies:

Q: Who to contact if I cannot find the right information about White Rabbit?

A: You may contact Erik van der Bij - CERN or you may ask your question in the [white-rabbit-dev](https://forums.ohwr.org/c/white-rabbit-dev/344 discourse forum.

White Rabbit use cases

Q: I have many GPS receivers on our company site. Any advantage to use White Rabbit here?

A: The cost of the GPS receiver, although even those cost over 10 K (well above the price of a simple WR installation), may not be your problem. It's the antenna installation and big cable for it that make it all a hassle to manage GPS installations. Everyone can nowadays install fibres and likely they are already installed in buildings, so within a day you may add WR timing capabilities to other buildings or offices. GPS receivers give you only the time at one single point.

Likely you can also manage the whole thing better, including backup GPS receivers on completely other places in the network as WR can do automatic switchover between timing sources. Redundancy is taken into account in the design, much in the same way as implemented by the Ethernet standards that it follows. Also for data transfer we make sure that high priority packets 'never' get lost.

Q: Can I use copper SFPs in WR network for connecting nodes that don't require WR timing?

A: Yes. You can plug copper SFP into the Switch and use it to connect your node. You will be able to transmit Ethernet traffic, but you won't get WR timing for that node through copper SFP. However, you will be able to receive PTP time from WR Switch and synchronize to it. Of course it won't be sub-ns accuracy any more. See also Copper SFP modules.

Actually it is really not worth bothering using copper SFPs as fibre-optic SFPs are cheap (a pair is less than 100 Euro, cables are not expensive either), you are using standard equipment that most WR adapts are using and the delay parameters of the ones shown in [2] are already known by the White Rabbit switch and other equipment so that you have an out-of-the-box good performance not requiring a calibration.

Q: Are WR devices compatible with 100 Mbit/s copper SFP?

A: No. WR switches/nodes support only 1GbE (copper/fiber). If you want to connect a non-1GbE device to a WR device, you need to do it through a switch that can speak both (most of 1GbE switches should do 10/100M)

Q: Will synchronization between WR devices in a WR network be equally accurate with and without a Grandmaster connected to a good reference (10MHz, 1PPS) ?

A: You can have your grandmaster switch free running completely, or connected only to 1PPS+10MHz but not the source of UTC. The synchronization accuracy in a WR network is guaranteed between the grandmaster and any WR device, regardless of the external source. However, having a good external frequency source is generally a good idea.

Q: Is it possible/foreseen to allow more than one Grandmaster in a WR Network?

A: It will be possible in the future using the seamless redundancy tools that are currently in a proof-of-concept state, see slide 7-11 of Redundancy presentation from WR Workshop. These mechanisms allow to connect as many Grandmasters as you want to a single switch downstream, provided the Grandmasters are synchronized at picoseconds (or tens of ps) level.

The logic to detect failure is very simple. If there is only one backup (e.g. 2 GMs), the switchover is triggered by: (1) physical failure, (2) notification from the master (i.e. if it looses the source of time). If there are two or more backups (e.g. 3 and more GMs), there is a third trigger of switchover: a kind-of majority voting based on comparison of all sources.

Byzantine faults and other more difficult problems are not handled. The main problem with that faults is that it takes considerably more time to detect them. So, even if you do detect them reliably, you are already off by much more than 1ns, so your timing does not meet the specification (sub-ns synchronization), so why bother. The first step to any attempt of solving this problem is much more stable local oscillator on the switch, so you have more time to decide and you can actually use the oscillator as a third source of time, and fall back to the majority voting...

Q: White Rabbit will give very low time differences between the devices connected to the network, in the low nano seconds. But what about UTC?

A: The time in WR network is transferred like in PTP (TAI plus information about leap seconds). The usual scenario is that you connect the grandmaster WR switch to 1PPS+10MHz from a GPS or Cesium clock and you get the UTC information from NTP.
Actually the WR system uses the PTP standard. And this standard uses TAI and not UTC. Using UTC in WR (and also in PTP) is a dangerous strategy as the behavior during a leap second jump is uncertain.

White Rabbit technical questions

Q: What is the WR performance? Accuracy and precision.

We understand accuracy as mean offset, and precision as standard deviation of that offset between PPS output signals. These definitions are quite simplistic for high-end time/frequency transfer.

For "standard" WR switch (current release v.5.0.1), the old numbers are quite accurate for accuracy/precision. A newer and more complete figure can be found in Maciej's thesis, page 50, Figure 4.5, it is also in the WR repo. In general, for this case, in constant temperature, the accuracy/precision for a single link are <100ps/10ps. Accuracy depends on calibration, it should be <100ps with proper calibration and this is what we typically see. It cannot be guaranteed it to be better than that because of bitslide problem. The source of inaccuracy and imprecision in WR and WRSv5.0.1 are summarized in Maciej's presentation from EFTS 2019, slides 22-30 (see White Rabbit lecture at European Frequency and Time Seminar, including the bitslide problem and relevant references. See also this article. Regarding the WR Switch, two improvements are addressed by:
- Low Jitter Daughtboard - improves precision
- Low Phase Drift Calibration (LPCD) - improves accuracy
WR switch with Low Jitter Daughtboard (LJD)
- See https://www.ohwr.org/project/wrs-low-jitter
- This is an additional feature, one must buy a WRS with LJD
- Related article that described the improvements and in general limits of WR switch time/frequency transfer : "White Rabbit Clock Synchronization: Ultimate Limits on Close-In Phase Noise and Short-Term Stability Due to FPGA Implementation".
- Regarding performance of LJD, the improvement in precision is not characterized with standard deviation (to simplistic metric) but rather jitter RMS and Modified Allan Deviation, which improve as follows
  - GM: 9ps to 1ps RMS 1Hz-100kHz
    2E-12 to < 5E-13 τ =1s ENBW 50Hz
  - BC: 11ps to <2ps RMS 1Hz-100kHz
    4E-12 to <7E-13 τ =1s ENBW 50H
WR with Low Phase Drift Calibration (LPCD)
- as of next WR switch release (V6.0), this feature will work by default on ports 1-12 of the WR switch (we do not have sufficient number of clocking resources in the FPGA to make it available for all ports)
- It mitigates imprecision of bitslide measurement (~+/-35ps for GTX) which limited the accuracy to 100ps on a single link.
- what we know so far is that it allows to keep the offset between PPS of the master and a slave to within 10ps after replugging the link and restarting switches.
- the expectation is that it will improve the accuracy of synchronization on a single link to <10ps (so tenfolds)
- We need proper tests to confirm this.

Note that all the above is for "standard" WR links (<10km, single mode bidirectional fibers). (ML, March 2020)

Answer from May 2016: A: A WR Switch can easily achieve a jitter (measured from 1Hz to 100kHz) less than 20ps with two levels of hierarchy. Most of the jitter power (99+%) is confined in the 1Hz-2kHz bandwidth. If you are using a WR-enabled SPEC with a digital I/O card, the jitter performance may deteriorate but still will be better than 50ps RMS. The accuracy is specified better than 1ns, but this specific offset will be very stable having only the mentioned jitter on top of it. Temperature effects will give a very slow drift.
The White Rabbit low jitter project may show more detailed results and shows ways of how the jitter (or phase-noise) may be improved (May 2016)

Q: Can I get links longer than 10 km?

A: If you want to build a larger distance network, you may want to know that there are several meteorological institutes working on extending the 10 km range. One may replace the standard SFP's by longer distance ones (see Non-compliant SFP types), and there may be ways of multiplexing specific wavelengths on a dark channel in a Telecom network. Also light amplifiers may be used. See for example the data of Vrije Universiteit Amsterdam and VSL at Users of WR technology. ~~At the moment of writing (April 2013) no links longer than 16 km have been made though.~~ The Finnish Metrology Institute MIKES has set up a White Rabbit link between two PCs that are connected with a 1000 (one thousand!) kilometer link.

The NEAT-FT Workshop on Optical Networks for Accurate Time and Frequency Transfer held in 2012 may give you more ideas and you possibly can find partner institutes that can help you with such a system. In 2016, other links, spanning well over 100 km have been presented at the Ninth White Rabbit Workshop.

Q: How much of the 1 Gbps bandwidth can I use?

A: We tested synchronization with a load of +95% bandwidth. In principle, PTP/WR requires much less than 1% of bandwidth (i.e. each second: 4 x min size frame + 1 frame of less than 100 bytes). If your network design is such that you plan to use over 99% of the bandwidth, it seems a rather bad design. If you leave a reasonable 10-20% of bandwidth margin, this is more than enough for WR/PTP to work. What is more, WR synchronization relies heavily on L1 (physical) syntonization which is independent from traffic (the frequency is transferred in the carrier with and without traffic), an incidental loss of PTP frames due to temporary peak of traffic load is acceptable by the PTP and will not break WR sub-ns synchronization which is maintained constantly by L1 syntonization.

Q: Where can I put additional delays coming e.g. from optical amplifiers to be able to compensate them?

A: For compensating additional (constant) assymetry you can modify deltaTx and deltaRx parameters in the SFP database stored in EEPROM (for WR PTP Core) or in a config file (for Wr Switch). Please take a look on our calibration procedure draft (https://www.ohwr.org/project/white-rabbit/wikis/Documents/White-Rabbit-calibration-procedure) and the WR PTP Core User's Manual (https://www.ohwr.org/documents/192). Basically what you have to do is to use "sfp add" command to specify Tx and Rx delays for particular SFP transceiver. In fact our delay model says that those Tx/Rx delays are the sum of constant delays of transmission/reception path so you can add your correction value there.

Those values are loaded by the PTP daemon only at the initialization time, so if one wants to update them, restarting the daemon is required.

Q: What limits the timing performance for the long-distance WR links?

A: Basically, the timing performance of a WR network can be limited by any link asymmetry not taken into account in the White Rabbit link delay model:

If, instead of WDM in a single fiber, two separate fibers are used for the transmission in both directions the latency of both links has to be measured and compensated (asymmetry caused by not equal fibers' length). Moreover, if those two fibers are not part of the same cable, their operation temperature will differ causing additional link asymmetry not foreseen in the WR link delay model.
If optical amplifiers are used, their delays have to be measured and taken into account. Asymmetry caused by different operation temperature of each amplifier are not taken into account in WR link delay model.
Constant delays (e.g. delays from optical amplifiers) can be taken into account during the calibration and summed together with Tx and Rx hardware delays (check White Rabbit Calibration).
Large values of delays or asymmetry are not a problem. The PTP software uses 64-bit variables as well as 64-bit mathematical operations.

Other effects are:

Temperature effects on the fibre itself. Over a large temperature range and 10km, this can vary up to 200ps.
Restarts of links may give an internal difference of up to 200ps. (see also another FAQ question)

So at the end of the day, just count on it being better than 1ns for links up to 10km. Only with much effort one can reach much better than that.

Q: May we use optical patch panels in the network or do you need direct connections?

A: There is no need to have direct connections between WR nodes. You can use patch panels in a WR network without any problem. Just make sure you have enough optical budget as each patch attenuates a bit of the light. This usually should not be a problem, but it is good to calculate this anyway.

Q: Any restriction on which fibre type should be used?

A: In 2010 we recommended to use a G652 type, that is optimized for dispersion. I am not sure if this requirement is still needed. (EB, 19/1/16)

Q: We have existing fibres already available which are meant for 10 Gb/s networks. Can I use these?

A: White Rabbit uses a 1 Gb/s transmission over a single mode fibre. So as long as the fibre is for single-mode transmission, any fibre is OK. The speed of data that you transfer over a fibre is not important.
So you will get the full White Rabbit performance with the fibre you have installed.

If you look at the datasheet of the fibre you have, it will say somewhere 9 um (mode field diameter). This specifies that the light can only follow a straight line through it, i.e. single mode.
Multi-mode fibres have a field diameter of 50 um or 62.5 um, meaning that the light can follow many different routes through it, including ones that reflect via the sides. This makes that a short light pulse will be spread out during the transmission. That’s why you cannot run long distances with multi-mode fibres.

Q: WR uses a single single-mode fibre, but would a short (~5m) multi-mode fibre link work too?

A: Multi-mode fibre links work fine. However you should redo timing calibration for your particular setup. Alpha (i.e. fiber asymmetry) is now caused by the difference in fibre length, not by chromatic dispersion. So for precise timing you need to calibrate each individual link.

Q: What is the accuracy and resolution of the round-trip time that is calculated?

A: The accuracy of the round-trip time measurement will be mostly determined by the accuracy of the frequency source you feed to the Grandmaster (the switch that is assigned to distribute the time, in many cases connected to a GPS or other clock). The resolution is approximately (1/62.5MHz)*(1/2^14) = 1 ps. However, also the implementation of the calculation can make a big difference: the implementation in the WRS is better than 1 ps, while the calculation in the end nodes may introduce an error in the order of 100 ps for a 10 km link.

The measurement of round-trip time is done by counting ticks of a 62.5MHz clock (which is ultimately derived from whatever clock signal you feed to the GM) and enhancing that result (integer number of ticks) with a fractional part using a DDMTD phase detector. The DDMTD gives you a zooming effect of 2^N, with N=14 for the current switch gateware.
More details can be found in the document White Rabbit clock synchronization: ultimate limits ...
Note that the synchronization is achieved by calculating master-to-slave delay based on the round-trip measurement and calculation. This is quite a tricky calculation because it involves large and small values (i.e. round-trip and alpha parameter). The implementation of this calculation can influence accuracy of synchronization on very long links (e.g. quantization errors and limited number of bits when doing the calculations in fixed-point arithmetic). The current implementations of this calculations for the WR switch and node are explained in the a short, unpublished, write-up (these might change for the PTP-HA High Accuracy implementation in future, contact Maciej Lipinski if you need more information).
The calculation on the WRS and on a node (using WRPC) is done differently. The calculation on the WRS is more precise that the calculation on WRPC. This is because on a WRS the ARM processor can use floating point calculations that cannot be used in the WRPC that has more limited resources (notably RAM for storing the program). The synchronization error due to limited calculation precision on WRPC (e.g. quantization error) may be in the order of 100 ps for a 10 km link (to be further characterized). While White Rabbit switches will remain very precisely synchronized, an error (in for example the PPS and 10 MHz outputs) on an end node can be caused by this calculation. This is work requires more study and may be improved in the PTP-HA implementation. (CERN, 22/11/18)

Q: What about start-up variations in shown cRTT values?

A: Every time a link is restarted, the round trip time is somewhat different. On a SPEC measurements show that it can be around 100 ps (A.Wallin), while a restart of the WR Switch may show variations of 10 ps (A.Wallin) to 30 ps (J.Diaz).

This start-up variations are caused by some internal functionality in the gigabit serialisers and is being worked on (Nov.2018). Actually a part of it may be caused by the precision of the calculations (see previous Q&A).

Update (March 2020):

WR with Low Phase Drift Calibration (LPCD)

as of next WR switch release (V6.0), this feature will work by default on ports 1-12 of the WR switch (we do not have sufficient number of clocking resources in the FPGA to make it available for all ports)
It mitigates imprecision of bitslide measurement (~+/-35ps for GTX) which limited the accuracy to 100ps on a single link.
What we know so far is that it allows to keep the offset between PPS of the master and a slave to within 10ps after replugging the link and restarting switches.
The expectation is that it will improve the accuracy of synchronization on a single link to <10ps.

Q: How could one create a performant wire tap on a port of a WR Switch ? / How to create a stable mirroring of a given port of a WR Switch to its another port?

A: As of v6.0.0, WR Switch supports mirroring of ports (separately tx and rx can be set), see options of the command rtu_stat:

ctdwa-774-cbt#rtu_stat -h
usage: rtu_stat <command> <values>
   help:              Show this message
   list:              List the routing table (same as empty command)
  [..cut..]
   mirror ingress <src_port> <dst_port>: Enable mirroring of ingress traffic
                                         from src_port to dst_port
   mirror egress  <src_port> <dst_port>: Enable mirroring of egress traffic
                                         from src_port to dst_port
   mirror off:      Turn off port mirroring

Faqwr