White Rabbit Newsletter, September 2018
This newsletter is based on information that was collected when writing
the "White Rabbit Applications and Enhancements" article for the
ISPCS 2018 conference.
The amount of information we received for writing that article was far
greater than could fit into a 7-page article. So, the ISPCS article is a
very concise digest while this newsletter provides more details. It
contains a snapshot of the information we had in June 2018 and, while it
is a lot, it is not an exhaustive list of all applications. Although it
is most likely out-of-date by now, it should provide a very useful
source of interesting information. This newsletter complements the list
Users of White Rabbit Technology.
The following applications of WR are described:
Accelerators, synchrotrons and spallation sources:
GSI, Germany - currently 134 nodes and 32 switches used operationally, 2000 nodes and 300 switches to be used in the final installation
At GSI, WR is used for time-triggered control and timestamping.*
GSI Helmholtz Centre for Heavy Ion Research (GSI) (GSI) is a heavy ion laboratory that performs basic and applied research in physics and related science disciplines. It is located in Darmstadt, Germany. GSI’s complex of accelerators is being extended with a new Facility for Antiproton and Ion Research (FAIR). FAIR will be one of the largest and most complex accelerator facilities in the world with its biggest accelerator of 1100m circumference. The operation of the FAIR facility requires time-triggered actions in different sub-systems of its accelerators. This is the responsibility of a timing system called General Machine Timing (GMT). Operation of FAIR requires that a central controller triggers within 500us an action in any of the 2000-3000 sub-systems. While the vast majority requires only about 1us accuracy, many subsystems like RF, beam instrumentation and experiments require 1-5 ns accuracy and 10 ps precision. This will be achieved using WR network that will consist of 300 WR switches in 5 layers and 2000 WR nodes integrated with accelerator sub-system to form the basis of FAIR's timing system, the GMT, in which the WR nodes and WR switches are connected with fibers of up to 2km length. The required reaction time of the system is 1ms which translates into an upper-bound network latency of 500us from the central controller to any node. The WR-based GMT timing system already replaced the old timing system at the existing GSI facility and has started operation in 2017 with 35 nodes and 4 switches. In 2018, there are 134 nodes and 32 switches used in the production system (not counting systems for labs and testing). The first major beam time using the new timing system starts in June 2018.
ESRF, France - currently 8 nodes and 1 switch used in validation installation, 41 nodes and 4-5 switches to be used in the final installation
At ESRF, WR is used for RF-synchronous clock and trigger distribution, and timestamping in WR or RF clock domain.*
The European Synchrotron Radiation Facility (ESRF) is a joint research facility situated in Grenoble, France, and supported by 22 countries. Research at the ESRF focuses, in large part, on the use of X-ray radiation in fields as diverse as protein crystallography, earth science, paleontology, materials science, chemistry and physics. The ESRF accelerator system consists of a “gun” that generates electron bunches, a Booster that accelerates these bunches, and a Storage Ring that stores the bunches serving experimental beamlines. The operation of the ESRF accelerator facility is controlled by a “Bunch Clock” system that delivers to accelerator sub-systems the ~352 MHz Radio Frequency (RF) signal (as well as 355 kHz, 10 kHz, and other clocks) and triggers initiating sequential actions synchronous to the RF signal, such as “gun trigger”, injection trigger”, “extraction trigger”. The jitter of the RF signal is required to be below 50ps. The RF is continuously trimmed around the 352 MHz value as the tuning parameter in the “fast orbit feedback” process. The current “Bunch Clock” system becomes obsolete while being inflexible and unable to meet new needs of the evolving accelerator complex and experiments. Thus, it has been decided to refurbish the current system, basing the new “Bunch Clock” system on White Rabbit (WR). The new “Bunch Clock” uses the WR-wide common notion of time and frequency. Using the WR time and frequency, it digitizes the input 352 MHz RF signal, sends its digital version over WR Network, and synthesizes a delayed and in-phase version of the input RF signal in the WR Node. The same WR Network is also used to distribute information about the triggers from the “master module”. The WR Nodes allow precise timestamping which is required for diagnostics and by experimental beamlines using the accelerator facility. The WR-based new “Bunch Clock” system was validated by transmitting a real-life RF signal over 6 months without failure and within specification. Phase noise measurements of the RF signal transmitted over WR showed <10ps jitter. As part of the validation, the new system has been successfully used to inject bunches in the storage ring. A system consisting of a WR switch and eight WR nodes is to be operational in July 2018. It is to partially replace the old system by the end of 2018 and will be expanded to 41 WR nodes and 4-5 WR switches by 2020. The ESRF ”Bunch Clock” system not only distributes a number of RF frequencies, it also provides timestamps and triggers that can be synchronous with these RF frequencies.
JINR, Russia - currently 50 nodes and 15 switches used operationally, 250 nodes and 30 switches to be used in the final installation
At JINR, WR is used for time/frequency distribution and timestamping.*
The Joint Institute for Nuclear Research (JINR) is an international research center for nuclear sciences in Dubna, Russia. It operates a collection of experimental physics facilities, including superconducting accelerator of nuclei and heavy ions. These facilities are now being upgraded within the Nuclotron-based Ion Collider Facility (NICE) project to study properties of dense baryonic matter. The NICA project includes one existing and two future detector facilities:
- Operational BM@N (Baryonic Matter at Nuclotron) fixed-target experiment that uses Nuclotron extracted beams. White Rabbit is the main clock and time distribution system at BM@N, WR network has been operational since 2015. It currently consists of one WR GrandMaster, one fan-out WR switch and three WR switches in DAQ racks for endpoint connections (WRPC enabled electronics, ~ 50 nodes total). For BM@N, the most demanding timing requirement comes from the Time-of-Flight (ToF) measurements: the start detectors have a precision of 30 ps (rms) each and timestamps are averaged over a few hits, thus the requirement on time reference is below 20 ps rms, and it is important to have long-term stability/repeatability during the experimental run.
- Constructed MPD (Multi-Purpose Detector) on collider. White Rabbit is also selected as main clock and time distribution system at MPD. ToF start signal accuracy requirement is 50 ps (rms) and the overall ToF resolution is expected to be below 100 ps. There are complete MPD Technical Design Reports for main subsystems. The MPD setup will have roughly 200 WR nodes and will require around 15 additional WR switches.
- Planned SPD (Spin Physics Detector) on collider. There is a Letter of Intent only thus there are no exact requirement figures yet.
CSNS, China - currently 50 nodes and 4 switches used operationally, 50 nodes and 4 switches to be used in the final installation
At CSNS, WR is currently used for timestamping, distribution of time reference, real-time data transmission for controls. In the near future, WR will be used for generation of triggers.*
The China Spallation Neutron Source (CSNS) is an accelerator-based pulsed spallation neutron source located in south China, the fourth of its kind in the world. As a "super microscope" for looking into the microstructure of materials, the CSNS has a wide range of application prospects, including in life sciences, physics, chemistry, resources and the environment, and new energy. Completed in March 2018, the facility includes a powerful linear proton accelerator, a rapid circling synchrotron, a target station and three neutron instruments: the General-Purpose Powder Diffractometer (GPPD), Small-Angle Neutron Scattering instrument (SANS), and multi-purpose reflectometer (MR). The instrument control system of CSNS is based on White Rabbit network that provides synchronization and real-time control. The control system of CSNS is in charge of target and instrument control. In CSNS, the precise time (T0) of a proton hitting the target needs to be measured. The measured time is broadcast to the target stations and the neutron instruments so that these equipment can work relative to T0. This time is also needed to measure the neutron time of flight. The required precision of T0 is 10ns, while its transmission to all neutrino instruments must be below 5us. WR has long proven to provide sub-ns accuracy of synchronization, thus meets the 10ns requirement, and the tests have confirmed WR’s suitability for the real-time controls (delay below 5us with jitter below <500ns). At CSNS, WR network is used also to synchronize “standard” IEEE1588 devices (NI, BeckHoff PLCs). The WR Network is synchronized to a GPS receiver and a rubidium clock. It is composed of 50 nodes and 4 WR Switch in 2 layers.
HIAF, China - currently 2 nodes and 3 switches used in lab validation tests, 359 nodes and 32 switches to be used in the final installation
The High Intensity heavy ion Accelerator Facility (HIAF) is a new facility planned in China for heavy ion related researches. It consists of two ion sources, a high intensity Heavy Ion Superconducting Linac (HISCL), a 45 Tm Accumulation and Booster Ring (ABR-45) and a multifunction storage ring system. HIAF will use WR as facility-wide timing subsystem that will provide synchronization for other subsystems, such as Power Supply, Beam Diagnostics, Data Acquisition, ECR and high Frequency Subsystem. The final layout of the HIAF Timing Subsystem will consist of 32 WR switches and 359 WR nodes. The Timing Subsystem has been validated in the lab with 2 WR nodes and 3 WR switches.
Neutrino detectors:
KM3NeT, Mediterranean sea - currently 36 nodes and 2 switches used in field validation tests, 6210 nodes and 400 switches to be used in the final installation
At KM3NeT, WR is used for timestamping.*
The Cubic Kilometre Neutrino Telescope
(KM3NeT) is a research infrastructure housing
the next generation neutrino telescopes located at the bottom of the
Mediterranean Sea. Once completed, the telescopes will have detector
volumes between megaton and several cubic kilometres of clear sea water.
Located in the deepest seas of the Mediterranean, KM3NeT will aim at the
discovery and subsequent study of high-energy neutrino sources in the
Universe and the determination of the mass hierarchy of neutrinos. When
completed, the KM3NeT infrastructure will consist of three detectors
located off shore the cost of France, Italy and Greece. Each detector
will consist of the same building blocks: pressure-resistant "digital
optical modules" (DOMs) detectors attached to strings. Holding 18 DOMs,
each string will be anchored to the sea floor and supported by floats.
Equally spaced strings will detect Cherekov light generated by the
charged particles from neutrino induced interactions inside or close to
the telescope. By correlating detection time from different DOMs, the
properties and trajectory, thus source, of neutrinos will be studied.
The required angular resolution of the measurement is 0.1 degree which
translates into synchronization of all the DOMs with 1 ns accuracy and a
few 100 ps of jitter. Such synchronization must be performed between the
on-shore reference and all the DOMs submerged in the see. The
ARCA
installation is located at a depth of 3500, 100km off-shore of Italy.
The
ORCA
installation is located at a depth of 2475 m, 40 km off-shore France.
The installation of the shore of Greeece is planned. While currently one
string of 18 DOMs is installed at each site, once
completed, ARCA will consist of
4140 DOMs with a volume of 1 Gton of seawater (1km3) and ORCA will
consist of 2070 DOMs spanning with a volume of about 3.7 Mton.
White Rabbit is used in KM3NeT as synchronization
system.
It synchronizes DOMs (WR Nodes) with the on-shore clock reference to
allow timestamping the arrival of photons using FPGA-based
Time-to-Digital converters with 1ns granularity. Additionally, WR is
used to synchronize a “start of run” at a predefined UTC/TAI time. WR
has been successfully operating with the already installed DOMs (one
string in ARCA and ORCA, total of 36 nodes and 2 switches). In the final
installation of ARCA and ORCA to be completed around 2020, around few
hundred WR Switches will be deployed to synchronize over 6000 DOMs. It
is yet to be decided whether some of the switches will be submerged or
all of the will be located
on-shore.
protoDUNE/WA105 at CERN, Switzerland - currently 14 nodes and 5 switches used in a prototype installation, 36 nodes and 5 switches to be used in the final installation
At WA105 (protoDUNE), WR is used for timestamping.*
In the framework of the Deep Underground Neutrino Experiment
(DUNE) to
be developed over the next decade, a number of prototype and
smaller-scale detectors are being installed at Fermilab and at CERN.
These detectors contribute to evaluating technologies and procedures to
be used in DUNE. The DUNE experiment will consist of two detectors. One
detector will record particle interactions near the source of the beam,
at the Fermi National Accelerator Laboratory (Fermilab) in Batavia,
Illinois. A second detector, the largest of its type ever built, will be
installed more than a kilometer underground at the Sanford Underground
Research Laboratory in Lead, South Dakota — 1,300 kilometers downstream
of the source. The DUNE Conceptual Design Report [2] proposes White
Rabbit (WR) as part of the timing distribution system. As such, WR is
being evaluated in some of the currently developed and deployed neutrino
detectors that are part of the DUNE initiative at Fermilab and at CERN.
While the accuracy required currently by the neutrino experiments is at
a sub-microsecond level (~400ns accuracy, page 36 of
"[1]":https://indico.cern.ch/event/493989/contributions/2005320/1254851
), the goal is to facilitate a nanosecond-level synchronization.
For the DUNE's Far Detector, WR is considered for synchronous
distribution of timing signals across multiple sub-detector systems,
with sub-ns precision. These timing signals include signals from the
beam, calibration signals, and potentially also triggers generated by
detector subsystems (e.g. light collection system). The WR system allows
to flexibly configure such multi-node timing distribution system. A
prototype of a Far Detector, called
WA105
detector, is installed at CERN. In WA105, WR is used to timestamp
events. The WR Network has 14 WR Nodes and 5 WR Switches.
SBN, USA - currently 6 nodes and 1 switch used in testing phase
At SBN, WR is used for timestamping.*
The Short-Baseline Neutrino (SBN) Program at Fermilab aims to perform sensitive searches for νe appearance and νμ disappearance in the Booster Neutrino Beam (BNB) using three LAr TPC detectors: SBND (Near Detector), MicroBooNE and ICARUS (Far Detector) located respectively at 110/470/700 meters from the primary target & magnet horns enclosure. The ICARUS detector is the first to use White Rabbit in North America. In this detector, White Rabbit is to be used to time-tag the neutrinos from their production at the beam source through to the detector at the end of the experiment. On the SBN ICARUS detector, White Rabbit is also expected to get an extremely accurate tagging of unwanted cosmic particles that come from space and get in the way of the experiment, potentially hiding the neutrino signatures.
HAWC, Mexico - currently 6 nodes and 1 switch used operationally
At HAWC, WR is used for distribution of time and frequency.*
The High Altitude Water Cherenkov observatory (HAWC) is an air shower array devised for TeV gamma-ray astronomy. HAWC is located at an altitude of 4100 m a.s.l. in Sierra Negra, Mexico. HAWC consists of 300 Water Cherenkov Detectors, each instrumented with 4 photomultiplier tubes (PMTs). WR Network consisting of 1 WR switch and 6 WR nodes distributes the clock and pulse-per-second (PPS) signals to two subsystems: HAWC main array and outrigger array for a high energy upgrade.
Cosmic ray detectors:
TAIGA (was HiSCORE), Siberia, Russia - currently 20 nodes and 4 switches used operationally, 1100 nodes and 90 switches to be used in the final installation
At TAIGA, WR is used for timestamping and their collection via WR network, sending trigger-messages and issuing time-calibration signals to light-flashers.*
The Tunka Advanced Instrument for cosmic ray physics and Gamma Astronomy (TAIGA) experiment measures air showers which are initiated by high energy gamma rays or charged cosmic rays. It is situated in Siberia in the Tunka valley close to lake Baikal. TAIGA is testing a novel, cost-effective detection principle for high energy gamma astronomy for energies beyond 10's of TeV - a hybrid detector that combines:
- TAIGA-HISCORE - a wide-angle, non-imaging Cherenkov light-front sampling array on a 100m-200m grid, and
- TAIGA-IACT - Imaging Air Cherenkov Telescopes located at distances up to 600-900m, i.e. operating effectively in mono-mode (as opposed to standard IACT setups like CTA).
Currently, 43 HiSCORE stations are operating on ~0.4 km2 in conjunction
with one IACT; the full prototype array with 110 stations and 3 IACTs
will be finished in 2019. An upgrade to 10km2 / 1100 stations and 16
IACTs is in planning. The HiSCORE-array aims at achieving a good
reconstruction of the air shower arrival direction. For a pointing
resolution of ∼ 0.1 degrees, all array stations need to be synchronized
with sub-ns time precision. Synchronization of HiSCORE
stations uses White Rabbit Network to
perform timestamping and WR-triggering, in conjunction with a custom
timing system.
WR was evaluated through field tests in Tunka between 2012 and 2017 (see
"[1]":http://inspirehep.net/record/1483337/files/PoS(ICRC2015)1041.pdf
and
"[2]":https://www.asterics2020.eu/article/testing-white-rabbit-hardware-field-conditions-siberia).
The first HiSCORE-3 prototype array (winter season 2012/13) with three
stations was composed of 1 WR Switch and 6 WR Nodes. It allowed
methodical tests, study of fiber-delay compensation in varying
temperature and gaining confidence in WR. The 9-station array HiSCORE-9
(winter season 2013/14) was arranged on a regular grid of 3 x 3 stations
with 150 m distance and it instrumented an area of 0.09 km2. It was the
first astroparticle physics setup operated using WR (1 WR Switch and 9
WR Nodes). It focused on an end-to-end functional test of the WR
time-synchronization, and included a full WR-based DAQ with DRS4-readout
"[1]":http://inspirehep.net/record/1483337/files/PoS(ICRC2015)1041.pdf.
HiSCORE-9 was upgraded in 2014 to HiSCORE-28 with 28 stations at 100 m
spacing forming a super-cell structure, on a total area of 450 m × 600 m
(0.25km2), and integrated into a hybrid DAQ. As described in
"[4]":http://inspirehep.net/record/1483337/files/PoS(ICRC2015)1041.pdf,
laboratory tests, long-term field tests and direct cross-verification
with the alternative timing system, LED-calibration runs, and operation
with Air Showers, confirmed a better than 0.5ns precision of WR and its
suitability for application at TAIGA. The full first-phase HiSCORE
detector of 1km2 is to be operational in 2019 with 40 WR nodes and 4 WR
Switches in 2 layers. Deployment of a 10 km2 installation is foreseen
from 2020, with 1100 WR nodes and 90 WR Switches in 3 layers. For
universal monitoring of major time-system components, including
redundant GM-WRS GPS clocks and GPSDO, a 4-channel dead-time-free 1
nsec-TDC is
used.
LHAASO, China - currently 40 nodes and 4 switches used operationally, 6734 nodes and 564 switches to be used in the final installation
Large High Altitude Air Shower Observatory
(LHAASO) will be one of the world’s
largest and most sensitive cosmic-ray facilities, once completed in
2020. Located at about 4410 m above sea level in the Haizi Mountain in
Sichuan Province in southwest China, the observatory will attempt to
understand the origins of high-energy cosmic rays. Cosmic rays are
particles that originate in outer space and are accelerated to energies
higher than those that can be achieved in even the largest man-made
particle accelerators. These high-energy protons and atomic nuclei
create air showers of particles such as protons and muons. Where cosmic
rays come from has remained a mystery since they were first spotted some
100 years ago. LHAASO aims to detect cosmic rays using a Cherenkov water
detector, covering a total area of 80 000 m2. The facility will also
consist of a 1.3 km2 array of 6000 scintillation detectors that will
study electrons and photons in the air showers, while an overlapping 1.3
km2 underground array of 1200 underground Cherenkov water tanks will
detect muons. Aiming to high sensitivity and wide spectrum of cosmic ray
detection, the 1 km2 complex array consists of over 7000 detectors of
different types. To reconstruct the air shower events with high angular
resolution (0.5 degree), timestamps of all detector electronics and
digitizers should be aligned better than 500 ps RMS while the
synchronization of Analog-to-Digital converters requires < 100ps
skew. Such
synchronization must be maintained in a harsh and remote environment
with large and rapid temperature variations in the range of -10 to +55
Celsius degree. The synchronization of LHAASO's detectors will be
performed using White Rabbit Network consisting of 564 WR Switches in 4
layers connecting 6734 WR Nodes embedded in each detector.
To meet LHAASO’s stringent synchronization performance requirements in
the harsh environmental conditions, two dedicated developments were
performed. First, the dependency on temperature of the WR Link Model
parameters was studied . These
parameters, the fixed delays and alpha, are assumed to be constant in
normal operation of WR while the degradation of synchronization
performance due to their variation with temperature can be up to 700 ps
under ambient temperature between -10 and 55 degrees Celsius. This
studies showed the parameters’ linear dependency on temperature which
allows easy correction. An online real-time correction method was
applied based on the result which reduced the synchronization variation
below 150ps with standard deviation below 50 ps under a varying
temperature between -10 and +50 Celsius degree. Second, to guarantee the
overall synchronization precision, individual calibration of each WR
node is essential to compensate device deviation. To facilitate
calibration of over 7000 WR nodes, a portable calibration node (PCN)
was developed that allows
auto-calibration.
It has been shown that the autocalibration procedure can achieve
calibration of 300-ps accuracy.
A small prototype based on the White Rabbit (WR) network was built at Yangbajin, Tibet, China in 2014. This prototype contains 4 WR switches and 50 WR customized nodes (48 electron detectors and 2 muon detectors). The four WR switches are used to build a four layer WR network. All the WR nodes were calibrated using PCN and perform online temperature compensation. The prototype was operational for over a year performing detection of air showers and providing performance benchmark. It showed that the long term synchronization of the White-Rabbit network is promising and 500 ps overall synchronization precision is achievable with individual calibration and temperature correction. With these promising results, the construction of LHAASO detectors started in 2016 and is due to finish in 2020. 1/4 of LHAASO facility is to be operation and produce physical data in 2018.
CTA, Spain/Chile - currently 32 nodes and 3 switches used in laboratory validation tests, 220 nodes and 10 switches to be used in the final installation
At CTA, WR is used to distribute clock/time to the telescopes and allow absolute timestamping of events triggered by the cameras.*
The Cherenkov Telescope Array - CTA
is one of the major future facilities in the field of astroparticle
physics and high-energy astrophysics, exploring the high-energy universe
with gamma rays (20 GeV to 300 TeV). The CTA observatory is planned to
start full operation in the early 2020's. It will consist of more than
100 telescopes distributed over two sites, one on La Palma (Spain) and
one in Paranal (Chile). The telescopes on each site will be located up
to several kilometers away from each other. The telescopes consist of
tessellated mirrors which focus the few nanoseconds long Cherenkov light
ashes from air showers initiated by cosmic gamma rays in the Earth’s
atmosphere onto fast-recording, pixelated cameras. In order to properly
combine the spatial and temporal information of these short light ashes
from all the telescope cameras and accurately reconstruct the properties
of the observed air shower, precise knowledge of the timing is mandatory
for CTA. It is therefore required, that the relative timing precision
between different cameras is better than two nanoseconds with less than
one nanosecond jitter (RMS). This is due to the very short duration of
the Cherenkov light flashes (order of a few 10 ns only) recorded by the
different telescopes and the need to combine the data from all
telescopes belonging to the same event based on their timestamps.
In order to achieve a high time precision, CTA will use a unified timing
system which is based on a hierarchical White Rabbit Ethernet network.
A mock setup of a Timing System similar to the final CTA Timing System
was built in the lab in
Amsterdam.
It consisted of 3 WR Switches and 32 WR Nodes and allowed validation of
system performance providing synchronization at -175 ± 12 picoseconds
and therefore well within all of the CTA requirements. The final
installation for CTA will have two hierarchical networks, one for the
northern array (with its planned 19 telescopes) and one for the southern
array (with its planned 99 telescopes). Each network will consist of a
Grandmaster switch, an appropriate number of intermediate WRS directly
connected to the GM, and one WR node per telescope which is connected to
one of the intermediate WRS. This means, that each of the 128 telescopes
will be only one hop away from its GM
switch.
National time laboratories:
MIKES, Finland - currently few nodes and 10 switches used operationally
At MIKES, WR is used for dissemination of time (ToD) and frequency to disseminate UTC (MIKES), and for comparison of reference clocks, in future generation of a fault-tolerant ensemble time-scale based on around 10 atomic clocks distributed geographically throughout the country.*
VTT MIKES Metrology is the national metrology
institute of Finland. One of its mandates is a nation-wide dissemination
of the official time in Finland, UTC (MIKE). MIKES’s customers include
both Finnish and international companies as well as the public sector.
MIKES has been evaluating application of White Rabbit for dissemination
of UTC and comparison of clocks over long-distance fiber links. Since
2013, it has been operating 950 km WR link (see
"[1]":https://www.ohwr.org/project/white-rabbit/uploads/c6b0314e91a7ce93ad556ee7afaeaa47/MIKES-CSC_WR_time_link.pdf
and "[2]":https://www.ohwr.org/project/white-rabbit/wikis/mikes) – the
longest WR-link reported currently – to study application of WR for
long-haul time transfer, and compare to GNSS-based time-transfer methods
such as GPS Precise Point Positioning. Recently, it has been also
operating a 50km WR
link
to study mid-distance low-jitter and low-asymmetry time transfer for
applications in geodesy (IGS reference station, VLBI and Satellite laser
ranging).
The 950km WR-PTP link (see
"[1]":https://www.ohwr.org/project/white-rabbit/uploads/c6b0314e91a7ce93ad556ee7afaeaa47/MIKES-CSC_WR_time_link.pdf
and "[2]":https://www.ohwr.org/project/white-rabbit/wikis/mikes) uses
unidirectional paths in a dark channel of the Finnish University and
Research Network (FUNET) between Espoo and Kajaani, Finland. The
long-haul link uses identical SFPs on ITU-T DWDM channel 60 (196.00 THz)
in both Espoo and Kajaani. It consists of 11 fiber spans between 15 and
140 km in length with amplifiers and dispersion compensation spools.
Independent verification of WR time transfer between the two ends of the
long-haul link was performed using GPS PPP postprocessing to determine
the WR fiber asymmetry. The same method was used to verify WR time
transfer, while the WR asymmetry parameter was held constant. The
results presented
in 2016
from 4-month operation of the 950km WR link show the time transfer error
within ±2 ns over the entire interval. The stability of the time
transfer difference between WR and GPS PPP (expressed as time deviation)
is shown in Fig. 6 of this
article.
A minimum of 20 ps is reached at an averaging time of 1000 s. Experience
with this link and others in conventional dual-fiber telecom networks
indicate that fiber-cuts or other link-maintenance typically occurs a
few times per year and usually results in a change of asymmetry at the
nanosecond level. For maintaining sub-ns performance of a long-haul
WR-link in a dual-fiber network one thus needs independent
calibration/verification of the asymmetry each time a fiber-cut/repair
is observed.
The 50km WR-PTP
link
is established between laboratories operating active hydrogen masers.
It is used to provide the official time of Finland, UTC-MIKE, to the
Metsähovi
Observatory.
The link uses optical DWDM transceivers on adjacent ITU DWDM channels
(optical carrier spacing 100 GHz). This limits the systematic error due
to chromatic dispersion to <700 ps without any calibration. The link
takes advantage of the recently developed Low Jitter Daughterboard
(LJD) that enhances the
performance of the WR Switch without any modifications to the WR-PTP
protocol. The results show performance at the 1e-12 level (at 1s, with
0.5 Hz BW). It also demonstrates that bidirectional optical link using
adjacent 100 GHz DWDM channels is an economical and simple solution for
reducing systematic errors on long links below 1 ns.
The MIKES WR-network has been gradually expanding since 2013, now
covering a total link-length of >1500 km, with ~10 switches in 2-3
layers. Some WR-LEN"":http://sevensols.com/index.php/products/wr-len/
nodes are also
used.
LNE-SYRTE, France - currently 2 nodes and 4 switches used operationally
At LNE-SYRTE, WR is used for distribution of time (ToD) and frequency to disseminate UTC (OP).*
The LNE-SYRTE is the French national metrology laboratory responsible for time and frequency, under mandate from the national metrology institute, LNE. It is part of the SYRTE department of Paris Observatory, which is a leader in time and frequency metrology and is also specialized in Earth rotation, celestial reference frames and history of Astronomy.
LNE-SYTRE performs experiments with White Rabbit for time and frequency
dissemination over long distance optical fiber links. A preliminary
experimental study using fiber spools was carried out over the years
2015-2017. First, the performance of the WR Switch was enhanced by
enabling direct clocking of the Grandmaster WR switch with an external
signal. The GM stability was improved by a factor of approximately 25.
Then the bandwidth of the Soft PLLs in the Slave WR Switch and
WR-ZEN node was
increased from the default value of 20 Hz to 70 Hz, and the rate of PTP
messages was increased from the default value of 1 message per second to
14 messages per second, following the approach reported by Mattia Rizzi.
Next, WR time transfer over 25 km fiber spool link was
studied.
In this study, the syntonization performance of a connection composed of
a 25 km unidirectional (dual-fibre) link using a single wavelength of
about 1541 nm was compared with a connection composed of a bidirectional
link over one 25 km fiber spool using the two wavelengths 1310 nm and
1490 nm. For both connections, a time deviation of the time intervals
about 1-2 ps at 1000 seconds of integration time and peak-peak
fluctuations of about 150 ps were reported. The main conclusion of the
study is that uni-directional links using 2 fibers and one wavelength
are suitable for WR. Finally, time and frequency transfer over a long
haul WR network spanning 500 km, composed of 4 cascaded WR switches and
a WR-ZEN Node was studied. This setup utilized a 2 × 125 km
uni-directional fiber link along with wavelength division multiplexing
to build long haul links in the laboratory. Long range SFPs in the
C-band or OSC channels close to the C-band were utilized. For the
cascaded 500 km link, the study reported a short term frequency
stability of 2 × 10−12 at one second of integration time which reached
the level of 2 × 10−15 over one day of integration time. The results
showed a time stability of 5.5 ps at one second of integration time,
reaching a minimum of 1.2 ps at 20 seconds of integration time. The peak
to peak fluctuations were about 2.5 ns over 14 days of consecutive
measurement. The limitations for the performance were due to fiber
thermal noise for long integration time. The performance of the cascaded
WR long haul link was compared with that of two good quality GPS
receivers. The cascaded 500 km link exhibited better stability by two
orders of magnitude at long term (from 1000 s of integration time
onwards).
The achieved performance of the cascaded 500 km WR link is state of the
art and surpasses the stability performance of commercial GPS
receivers.
At present, WR is used at LNE-SYTRE (Observatoire de Paris) for
in-campus dissemination of UTC (OP). Within the coming year, in-field
deployment in the Paris area should be accomplished. Specifically,
Observatoire de Paris is participating in the EU project WRITE -EURAMET
EMPiR,
which aims at disseminating frequency and time standards to industrial
partners.
VSL, The Netherlands - currently 4 nodes and 2 switches used operationally
At VSL, WR is used for distribution of time (ToD) and frequency to disseminate UTC (VSL).*
The Dutch Metrology Institute is responsible for dissemination of UTC in the Netherlands. It operates a WR link between Delft and Amsterdam (2x137km) and strives for an uncertainty well below 1 ns, which would make the UTC dissemination more accurate (and also more stable) than satellite methods (GPS, TWSTFT). While earlier work was limited to 8 ns uncertainty due to fiber chromatic dispersion specs, with a new method to determine alpha parameter 'in situ' (the alpha parameter allows to calculate delay asymmetry), it seems that this delay asymmetry (over a 2x 137km link) can be calibrated out to a few 0.1 ns or less. It has been already shown (VU, Nikhef, VSL, SURFnet) that alpha can be calibrated 'in situ' such that two identical 50 km links give consistent clock offsets of 35(6) ps and 38(7) ps, respectively (i.e. with a time error due to alpha that appears to be < 100 ps). The results on the 137 km are work in progress (Nikhef, VU, VSL, SURFnet).
NIST, USA - currently 2 nodes used operationally
At NIST, WR is used for distribution of time (ToD) and frequency to disseminate UTC (NIST).*
The National Institute of Standards and Technology (NIST) produces a real-time realization of UTC (NIST) which is used to contribute to Coordinated Universal Time (UTC) and as a source for accurate time in the USA. NIST has successfully evaluated WR for distribution of NIST's UTC realization across campus. The WR hardware has met the accuracy and stability requirements to provide UTC (NIST) to the time transfer systems used to contribute to UTC that are in different locations on the NIST campus. The current WR Network consists of two WR nodes and is to be expanded.
INRiM, Italy, currently 8 nodes and 1 switch used operationally
At INRiM, WR is used for distribution of time (ToD) to disseminate UTC (IT), RF reference transfer and timestamping.*
L'Istituto nazionale di ricerca metrologica -INRiM is a public research centre acting as Italy's national metrology institute (NMI). INRiM is offering a WR-based service to the financial district of Milan, offering a time-stamping for financial transactions within the European Mifid-II regulation. INRiM offers the UTC reference and the know-how transfer, the service is commercially provided by INRiM and consortio Topix. TOPIX have a WR distribution over 400 km fiber haul in a DWDM infrastructure and offers UTC tractability with WR. Additionally, INRIM is equipping its fibre testbed for Time and Frequency transfer to offer WR over it, in particular connecting the INAF radiotelscope in Medicina (Bologna) and Rome (fiber hauls of 500 and 800 km, respectively). INRIM also coordinates the EU-EMPIR project WRITE (White Rabbit Industrial Timing Enhancement) that will develop the metrological capacities required to accelerate the industrial adoption of PTP-WR, through improved hardware and calibration techniques, implemented in industrial environments.
Other applications:
SKA, Australia/Africa - currently 2 nodes and 1 switch used in test installation, 233 nodes and 15 switches to be used in the final installation
At SKA, WR is use to distribute time (ToD).*
The Square Kilometre Array -SKA project is an international effort to build the world’s largest radio telescope, with eventually over a square kilometre of collecting area. The SKA will consist of two large radio astronomy interferometers, each consisting of many receptors. In the SKA design, WR is used for distributing the UTC time signal as a 1PPS second tick. The UTC distribution has two main functions: 1) to allow the 'phasing up' of the interferometer by limiting the search space for the correlator, and 2) to tie the incoming radio signal to UTC with high accuracy. This is used for pulsar timing including gravitational wave detection and observations of transients (sudden, unexpected signals). There will be two SKA1 telescopes, one in Australia (SKA1-Low) and one in South Africa (SKA1-Mid).
- For SKA1-Low, the WR signal will be directly distributed to 36 locations at up to 80km of fiber distance.
- For SKA1-Mid, the WR signal will be distributed to 197 receivers, with the longest fiber distance of 173km. Most WR end points will be directly connected to the central WR switches, but the signals for the longest links will go through one or two "repeater" WR switches (Boundary Clocks).
The design goal at SKA is an overall accuracy of 2ns (1 sigma) and it has be shown that the worst case errors on the longest links are only slightly above 1ns.
ELI-ALPS/BEAMS, Hungary & Czech Republic - currently few nodes and 5 switches used operationally, 70 nodes and 16 switches to be used in the final installation at ELI-BEAMS
At ELI-ALPS, WR is used for timestamping, trigger acquisition and
re-distribution.*
At ELI-BEAMS, WR is used for timestamping, generation of triggers,
generation of RF, distribution of clock/time reference, time-triggered
control, fixed-latency.*
The Extreme Light Infrastructure
-ELI is the first civilian
large-scale high-power laser research facility to be realized with
trans-European cooperation and the worldwide scientific community.
Hungary, the Czech Republic and Romania, with a coordinated management
and research strategy, will simultaneously implement the project through
the construction of the three laser facilities with the respective
mission in the attosecond, beamline and photonuclear applications. The
main objective of ELI Attosecond Light Pulse Source (ELI-ALPS) is the
establishment of a unique attosecond facility which provides ultrashort
light pulses between THz (10^12 Hz) and X-ray (10^18-10^19 Hz) frequency
range with high repetition rate for developers and end-users.
Experimental projects demanding ultrahigh intensity light, like laser
particle acceleration or laser generated x-ray radiation will be
primarily developed at the Beamline Facility (ELI-BEAMS) in Prague,
Czech Republic, while the photo-induced nuclear experiments will be
performed at the research institute to be built in Magurele (ELI-NP),
near Bucharest, Romania. Two of the infrastructures,
ELI-ALPS
and
ELI-BEAMS,
use White Rabbit.
The
ELI-ALPS
requires ns-to-ps accuracy for timestamping, ps accuracy for Radio
Frequency (RF) systems and fs accuracy for pulsed optical
synchronization. White Rabbit is planned to provide a common timestamps
base for the whole
facility
- except for the lasers, which need a femtosecond level synchronization.
Moreover, triggers from the lasers are to be acquired and redistributed
over White Rabbit towards the experimental areas. The currently
installed WR network at ELI-ALPS consists of 3 WR Switches.
In the ELI-BEAMS, White Rabbit is the Electronic Timing System (ETS) that is part of facility level control system. The EST provides synchronization and trigger signals to all subsystems and devices during operation, while each laser experiment is driven by its local timing system. ETS is required to provide picosecond level accuracy for typical laser experiments when driving streak cameras and low level of accuracy when pulse duration and pulse rise time are greater than 100ps (typically CCD cameras, Osciloscopes etc.). ELI-BEAMS has positively evaluated White Rabbit with a test setup consisting of 2 WR Switches and two WR Nodes. Currently, ELI-BEAMS runs WR Network that is connected by 40km long WR link to WR Grandmaster at CESNET (Czech organization responsible for internet connection), with a backup connection to local GPS receiver. The WR Network has currently 3 layers, upper level near laser cavity (top level) server room, bottom layer in experimental halls (each has two WR switches). The final the WR Network at ELI-BEAMS will consist of 16 WR Switches and 70 WR nodes.
Smart Grid at EPFL, Switzerland - currently 2 nodes and 1 switch used in experimental setup
At EPFL, WR is used to timestamp the measurement of PMUs.*
The Distributed Electrical Systems Laboratory (DESL) at the Swiss Federal Institute of Technology Lausanne (EPFL), one of world’s best technical universities, operates its own experimental Smart Grid. Within the framework of technology improvements for the Smart Grid, one of the research studies evaluated White Rabbit for distributed measurement of synchrophasors via Phasor Measurement Units (PMUs) . This study investigated application of WR as a way to improve the steady state PMU accuracy and mitigate the well-known disadvantages of the currently-used GPS-base synchronization of PMUs: accessibility of clear-sky view and vulnerability of GPS signals. The results of the research show that WR-PMU can provide slightly better performance than the GPS-PMU. The performance of the WR-PMU was limited by PMU’s hardware and not WR performance. The studies also show that WR is an appropriate alternative for GPS-based PMUs. More studies are ongoing.
DLR, Germany - currently 1 node and 1 switch used operationally
The German Aerospace Center (DLR) is the national
aeronautics and space research center of the Federal Republic of
Germany. Its extensive research and development work in aeronautics,
space, energy, transport, digitalization and security is integrated into
national and international cooperative ventures. Within this scope, DLR
performs optical
reconnaissance
using Satellite Laser Ranging
(SLR). Laser ranging to objects
in space is of great importance in many different fields. By measuring
the distance to dedicated satellites various scientific phenomena such
as tectonic plate drifts, crustal deformation, the Earth’s gravity field
or ocean tides can be investigated. Furthermore, precise orbit
determination of cooperative as well as non-cooperative objects can
support evasive manoeuvres to avoid collisions with space debris.
Besides, reentry predictions and mission preparation can benefit from
satellite laser ranging. Together with Very Long Baseline Interferometry
(VLBI), LSR is one of the core technologies for the Global Geodetic
Observing System (GGOS).
The Satellite Laser Ranging is an established technology for geodesy,
fundamental science and precise orbit determination. While many SLR
stations are in operation, the SLR station that has been successfully
operated by DLR since 2016 is
unique. Apart from the fact that
it uses optical fibre rather than a coudé path (i.e. guiding the laser
beam through turning axes of an astronomical platform with a number of
mirrors), it has been assembled completely from commercial off-the-shelf
components, which increases flexibility and significantly reduces
hardware costs. To synchronize the measurement with UTC, a small White
Rabbit network is used. A WR switch coupled
to a dedicated timing GPS module serves as grandmaster clock. A WR Node
with FMC-DEL is
connected to this WR switch. The FMC-DEL pulse generator card triggers
the laser at precisely defined times and synchronizes the HydraHarp
event timer to UTC. Thus, all time tags are synchronised to UTC to
better than 100 ns, while time differences are measured with an accuracy
of better than 1
ns.
Deutsche Börse , Germany - currently 7 switches and 4 nodes (+ several clients connected to the access switches) used operationally
At Deutsche Börse, WR is used to synchronise its timestamping devices and the timestamping devices of its clients.*
Deutsche Börse is a marketplace organizer for the trading of shares and other securities, as well as a transaction services provider. It is a joint stock company with headquarters in Frankfurt that gives companies and investors access to global capital markets. Deutsche Börse uses White Rabbit internally to synchronize all network capture and timestamping devices in the co-location 2.0 network. Timestamps taken by those devices are the basis for the High Precision Timestamps Daily File Service (HPT file). To allow customers to synchronize their clocks to the same time source used by Deutsche Börse for the timestamps in the HPT file, Deutsche Börse considers offering White Rabbit Service. The first step is to run a high precision time synchronization pilot project using the White Rabbit protocol. The aim of this project is to determine whether White Rabbit is a feasible alternative or addition to the current PTP time synchronization protocol. The WR Network used by Deutsche Börse consists of 7 WR Switches (+ 1 backup) and 4 WR Nodes in 3 layers. Several clients (switches or nodes) are connect to 11 ports of the access WR switches in the Deutsche Börse WR Network.
5 September 2018