Frequently Asked Questions
about White Rabbit.
White Rabbit general questions
Q: What's the easiest way to start playing with White Rabbit?
A: For starting to use WR technology a WR starting kit is available. The manual you can find there shows clearly what is possible with just two PCIe plug-in cards. Later on you can expand your kit with a White Rabbit switch.
Q: I am considering using White Rabbit in my next project. Where can I get support?
A: White Rabbit (WR) is open source and commercial. That means that you can get the best of both worlds. You can avoid vendor lock-in because the technology is open source, so you can switch vendors if you are unhappy with the service a particular vendor provides. Because it is open source, provided your team has adequate know-how, you can even decide to not rely on commercial companies for support. But if you want the convenience of paying somebody for guaranteed support, you can also have that. In general, each WR user has a particular problem to solve, and it would be hard to design a one-size-fits-all WR solution. WR starter kits illustrate the basic capabilities of WR and provide a good basis to build upon. There is often a need for local design customization, and that is why we recommend that WR users have either the backing of a local design team or a commercial company. There is also the white-rabbit-dev mailing list, where you can ask questions. But for obvious reasons, experts in white-rabbit-dev cannot guarantee they will answer all your questions in a timely manner, or even at all. That would not be scalable. Answers on white-rabbit-dev are therefore provided on a good-will basis. The same is true for email you send privately to individual core WR developers. If you need guaranteed support, a local team or a company is the way to go.
Q: How can I test White Rabbit switch equipment? Or can I buy some test equipments from your company?
A: CERN is not a company, but a publicly funded intergovernemental
organization. We develop the technology but don't produce the equipment,
we buy WR switches from commercial vendors. Anyone can produce a WR
switch as all the sources are on our
pages (just have a careful
look at the wiki, sub-projects, ets... )
For example, there is wiki page about the WR switch on which
you can find the companies that are involved in switch development and
the companies from which you can buy a WR switch (Commercial producers,
at the bottom). On the wiki page about SPEC
card (an example of WR Node)
you will find companies producing SPEC
cards.
Q: If I don’t participate in WR workshop meetings, is the presentation material available?
A: All we do is public. All information from the workshops is directly accessible.
Q: How can I trust White Rabbit?
A: Most White Rabbit equipment is Open Hardware and Open Software. You
have full access to all design data and can inspect and recompile the
source code yourself to verify anything you like.
Many tests have been done by independent institutes to verify the timing
synchronisation and data transmission performance. Some institutes
indeed had found issues that have been solved since then. Other issues
have been found and are being worked on.
Q: How important is White Rabbit?
A1: Have a look at the many institutes and companies who are using White
Rabbit at the pate Users of White Rabbit Technology.
A2: That's a hard question. It is important enough to have been
specifically mentioned in many job offers at different institutes and
companies:
- https://www.ohwr.org/project/white-rabbit/wikis/News#23-02-2018-wr-job-vacancy-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#19-06-2017-wr-job-vacancy-at-cern (CERN)
- https://www.ohwr.org/project/white-rabbit/wikis/News#26-04-2017-wr-job-vacancy-at-opnt---nl (OPNT)
- https://www.ohwr.org/project/white-rabbit/wikis/News#23-02-2017-wr-job-vacancy-at-cern (CERN)
- https://www.ohwr.org/project/white-rabbit/wikis/News#13-06-16-a-job-with-white-rabbit (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#04-09-2014-job-offer-at-gsi (GSI)
- https://www.ohwr.org/project/white-rabbit/wikis/News#7-5-2013-job-offer-for-developer (GSI)
Q: Who to contact if I cannot find the right information about White Rabbit?
A: You may contact Erik van der Bij - CERN or you may ask your question in the white-rabbit-dev mailing list.
White Rabbit use cases
Q: I have many GPS receivers on our company site. Any advantage to use White Rabbit here?
A: The cost of the GPS receiver, although even those cost over 10 K (well above the price of a simple WR installation), may not be your problem. It's the antenna installation and big cable for it that make it all a hassle to manage GPS installations. Everyone can nowadays install fibres and likely they are already installed in buildings, so within a day you may add WR timing capabilities to other buildings or offices. GPS receivers give you only the time at one single point.
Likely you can also manage the whole thing better, including backup GPS receivers on completely other places in the network as WR can do automatic switchover between timing sources. Redundancy is taken into account in the design, much in the same way as implemented by the Ethernet standards that it follows. Also for data transfer we make sure that high priority packets 'never' get lost.
Q: Can I use copper SFPs in WR network for connecting nodes that don't require WR timing?
A: Yes. You can plug copper SFP into the Switch and use it to connect your node. You will be able to transmit Ethernet traffic, but you won't get WR timing for that node through copper SFP. However, you will be able to receive PTP time from WR Switch and synchronize to it. Of course it won't be sub-ns accuracy any more. See also Copper SFP modules.
Actually it is really not worth bothering using copper SFPs as fibre-optic SFPs are cheap (a pair is less than 100 Euro, cables are not expensive either), you are using standard equipment that most WR adapts are using and the delay parameters of the ones shown in [2] are already known by the White Rabbit switch and other equipment so that you have an out-of-the-box good performance not requiring a calibration.
See also the WR Switch FAQ question: Q: Should green and orange LEDs be lit when copper SFP is used even without Ethernet cable plugged in?
Q: Are WR devices compatible with 100 Mbit/s copper SFP?
A: No. WR switches/nodes support only 1GbE (copper/fiber). If you want to connect a non-1GbE device to a WR device, you need to do it through a switch that can speak both (most of 1GbE switches should do 10/100M)
Q: Will synchronization between WR devices in a WR network be equally accurate with and without a Grandmaster connected to a good reference (10MHz, 1PPS) ?
A: You can have your grandmaster switch free running completely, or connected only to 1PPS+10MHz but not the source of UTC. The synchronization accuracy in a WR network is guaranteed between the grandmaster and any WR device, regardless of the external source. However, having a good external frequency source is generally a good idea.
Q: Is it possible/foreseen to allow more than one Grandmaster in a WR Network?
A: It will be possible in the future using the seamless redundancy tools that are currently in a proof-of-concept state, see slide 7-11 of Redundancy presentation from WR Workshop. These mechanisms allow to connect as many Grandmasters as you want to a single switch downstream, provided the Grandmasters are synchronized at picoseconds (or tens of ps) level.
The logic to detect failure is very simple. If there is only one backup (e.g. 2 GMs), the switchover is triggered by: (1) physical failure, (2) notification from the master (i.e. if it looses the source of time). If there are two or more backups (e.g. 3 and more GMs), there is a third trigger of switchover: a kind-of majority voting based on comparison of all sources.
Byzantine faults and other more difficult problems are not handled. The main problem with that faults is that it takes considerably more time to detect them. So, even if you do detect them reliably, you are already off by much more than 1ns, so your timing does not meet the specification (sub-ns synchronization), so why bother. The first step to any attempt of solving this problem is much more stable local oscillator on the switch, so you have more time to decide and you can actually use the oscillator as a third source of time, and fall back to the majority voting...
Q: White Rabbit will give very low time differences between the devices connected to the network, in the low nano seconds. But what about UTC?
A: The time in WR network is transferred like in PTP (TAI plus
information about leap seconds). The usual scenario is that you connect
the grandmaster WR switch to 1PPS+10MHz from a GPS or Cesium clock and
you get the UTC information from
NTP.
Actually the WR system uses the PTP standard. And this standard uses TAI
and not UTC. Using UTC in WR (and also in PTP) is a dangerous strategy
as the behavior during a leap second jump is uncertain.
White Rabbit technical questions
Q: What is the jitter performance?
A: A WR Switch can easily achieve a jitter (measured from 1Hz to 100kHz)
less than 20ps with two levels of hierarchy. Most of the jitter power
(99+%) is confined in the 1Hz-2kHz bandwidth. If you are using a
WR-enabled SPEC with a digital I/O card, the jitter performance may
deteriorate but still will be better than 50ps RMS. The accuracy is
specified better than 1ns, but this specific offset will be very stable
having only the mentioned jitter on top of it. Temperature effects will
give a very slow drift.
The White Rabbit low jitter
project may show more
detailed results and shows ways of how the jitter (or phase-noise) may
be improved (May 2016)
Q: Can I get links longer than 10 km?
A: If you want to build a larger distance network, you may want to know
that there are several meteorological institutes working on extending
the 10 km range. One may replace the standard SFP's by longer distance
ones (see Non-compliant SFP types), and there may
be ways of multiplexing specific wavelengths on a dark channel in a
Telecom network. Also light amplifiers may be used. See for example the
data of Vrije Universiteit Amsterdam and VSL at Users of WR
technology. At the moment of writing (April 2013) no links
longer than 16 km have been made though. The Finnish Metrology
Institute MIKES has set up a White Rabbit link between two PCs
that are connected with a 1000 (one thousand!) kilometer link.
The NEAT-FT Workshop on Optical Networks for Accurate Time and Frequency Transfer held in 2012 may give you more ideas and you possibly can find partner institutes that can help you with such a system. In 2016, other links, spanning well over 100 km have been presented at the Ninth White Rabbit Workshop.
Q: Where can I put additional delays coming e.g. from optical amplifiers to be able to compensate them?
A: For compensating additional (constant) assymetry you can modify deltaTx and deltaRx parameters in the SFP database stored in EEPROM (for WR PTP Core) or in a config file (for Wr Switch). Please take a look on our calibration procedure draft (https://www.ohwr.org/project/white-rabbit/wikis/Documents/White-Rabbit-calibration-procedure) and the WR PTP Core User's Manual (https://www.ohwr.org/documents/192). Basically what you have to do is to use "sfp add" command to specify Tx and Rx delays for particular SFP transceiver. In fact our delay model says that those Tx/Rx delays are the sum of constant delays of transmission/reception path so you can add your correction value there.
Those values are loaded by the PTP daemon only at the initialization time, so if one wants to update them, restarting the daemon is required.
Q: What limits the timing performance for the long-distance WR links?
A: Basically, the timing performance of a WR network can be limited by any link asymmetry not taken into account in the White Rabbit link delay model:
- If, instead of WDM in a single fiber, two separate fibers are used for the transmission in both directions the latency of both links has to be measured and compensated (asymmetry caused by not equal fibers' length). Moreover, if those two fibers are not part of the same cable, their operation temperature will differ causing additional link asymmetry not foreseen in the WR link delay model.
- If optical amplifiers are used, their delays have to be measured and taken into account. Asymmetry caused by different operation temperature of each amplifier are not taken into account in WR link delay model.
- Constant delays (e.g. delays from optical amplifiers) can be taken into account during the calibration and summed together with Tx and Rx hardware delays (check White Rabbit Calibration).
- Large values of delays or asymmetry are not a problem. The PTP software uses 64-bit variables as well as 64-bit mathematical operations.
Other effects are:
- Temperature effects on the fibre itself. Over a large temperature range and 10km, this can vary up to 200ps.
- Restarts of links may give an internal difference of up to 200ps.
So at the end of the day, just count on it being better than 1ns for links up to 10km. Only with much effort one can reach much better than that.
Q: May we use optical patch panels in the network or do you need direct connections?
A: There is no need to have direct connections between WR nodes. You can use patch panels in a WR network without any problem. Just make sure you have enough optical budget as each patch attenuates a bit of the light. This usually should not be a problem, but it is good to calculate this anyway.
Q: Any restriction on which fibre type should be used?
A: In 2010 we recommended to use a G652 type, that is optimized for dispersion. I am not sure if this requirement is still needed. (EB, 19/1/16)
Q: We have existing fibres already available which are meant for 10 Gb/s networks. Can I use these?
A: White Rabbit uses a 1 Gb/s transmission over a single mode fibre. So
as long as the fibre is for single-mode transmission, any fibre is OK.
The speed of data that you transfer over a fibre is not important.
So you will get the full White Rabbit performance with the fibre you
have installed.
If you look at the datasheet of the fibre you have, it will say
somewhere 9 um (mode field diameter). This specifies that the light can
only follow a straight line through it, i.e. single mode.
Multi-mode fibres have a field diameter of 50 um or 62.5 um, meaning
that the light can follow many different routes through it, including
ones that reflect via the sides. This makes that a short light pulse
will be spread out during the transmission. That’s why you cannot run
long distances with multi-mode
fibres.
Q: WR uses a single single-mode fibre, but would a short (~5m) multi-mode fibre link work too?
A: Multi-mode fibre links work fine. However you should redo timing calibration for your particular setup. Alpha (i.e. fiber asymmetry) is now caused by the difference in fibre length, not by chromatic dispersion. So for precise timing you need to calibrate each individual link.
Q: What is the accuracy and resolution of the round-trip time that is calculated?
A: The accuracy of the round-trip time measurement will be mostly determined by the accuracy of the frequency source you feed to the Grandmaster (the switch that is assigned to distribute the time, in many cases connected to a GPS or other clock). The resolution is approximately (1/62.5MHz)*(1/2^14) = 1 ps. However, also the implementation of the calculation can make a big difference: the implementation in the WRS is better than 1 ps, while the calculation in the end nodes may introduce an error in the order of 100 ps for a 10 km link.
The measurement of round-trip time is done by counting ticks of a
62.5MHz clock (which is ultimately derived from whatever clock signal
you feed to the GM) and enhancing that result (integer number of ticks)
with a fractional part using a DDMTD phase detector. The DDMTD gives you
a zooming effect of 2^N, with N=14 for the current switch gateware.
More details can be found in the document White Rabbit clock
synchronization: ultimate limits
...
Note that the synchronization is achieved by calculating master-to-slave
delay based on the round-trip measurement and calculation. This is quite
a tricky calculation because it involves large and small values (i.e.
round-trip and alpha parameter). The implementation of this calculation
can influence accuracy of synchronization on very long links (e.g.
quantization errors and limited number of bits when doing the
calculations in fixed-point arithmetic). The current implementations of
this calculations for the WR switch and node are explained in the a
short, unpublished,
write-up
(these might change for the PTP-HA High Accuracy implementation in
future, contact Maciej
Lipinski
if you need more information).
The calculation on the WRS and on a node (using WRPC) is done
differently. The calculation on the WRS is more precise that the
calculation on WRPC. This is because on a WRS the ARM processor can use
floating point calculations that cannot be used in the WRPC that has
more limited resources (notably RAM for storing the program). The
synchronization error due to limited calculation precision on WRPC (e.g.
quantization error) may be in the order of 100 ps for a 10 km link (to
be further characterized). While White Rabbit switches will remain very
precisely synchronized, an error (in for example the PPS and 10 MHz
outputs) on an end node can be caused by this calculation. This is work
requires more study and may be improved in the PTP-HA implementation.
(CERN,
22/11/18)
See also:
Frequently Asked Questions about the White Rabbit Switch
Frequently Asked Questions about the White Rabbit PTP Core
22 November 2018