White Rabbit Newsletter - December 2015

CERN (including Alessandro Rubini)

White Rabbit Switch

During 2015 our WR Switch developments were focused mainly in two areas: improving reliability under a heavy load of traffic and ensuring ways for remote configuration and diagnostics. The former included various tests performed with a professional IT network tester and fixes to the WR Switch gateware. For the latter, we now have a set of SNMP exports that let operators determine the overall state of a device as well as provide detailed information for expert users. We have also added a possibility of fetching the main configuration file from a central TFTP/HTTP/FTP server on boot time and generating a WR-synchronized 10MHz clock on the latest v3.4 switch hardware. All these and other features and bugfixes were included in the new stable v4.2 firmware which was released on the 28th of August.

WR Switch Grandmaster phase noise analysis

Currently release 3.4 of the WR switch operating in G.M. mode (locked to a Cesium clock) achieves 8-9 ps of RMS jitter (1Hz-100kHz). Most of the noise lies in the 1Hz-2kHz bandwidth (SoftPLL Nyquist region). Mattia Rizzi found that a large amount of jitter was related to the Xilinx DCM used to multiply the 10MHz input reference clock rather than the phase detector (DDMTD). The Xilinx DCM outputs a clock with noise contributions above 20kHz. Most of the high-frequency phase noise is not filtered by the DDMTD and appears as white noise in the SoftPLL bandwidth due to aliasing. A clock with a better phase noise profile above 2 kHz may improve the G.M. jitter to less than 4ps (1Hz-100kHz).

White Rabbit PTP Core

We were also working on the WR PTP Core. Among others, main new features include the possibility of storing calibration data and init script in the Flash memory on the carrier, various bugfixes to withstand heavy Ethernet traffic and LM32 software cleanup. These and other features and bugfixes, as well as external contributions (like Kintex-7 support done by Peter Jansweijer, of Nikhef) were part of the new v3.0 WR PTP Core released on the 16th of December.

Applications

Regarding new CERN WR deployments, this year we have provided a small WR network consisting of one WR Switch and two VME nodes to synchronize vibration measurement equipment on the surface and underground in the LHC Point 1 [1]. We also developed a fixed-latency trigger distribution system to study beam instabilities in the LHC [2] and tested a proof-of-concept system for RF distribution using WR [2].

WR Reliability

A proof-of-concept implementation of switch-over between synchronization paths was developed and tested. The implementation supports multiple backup paths, it provides short term holdover and hardware-assisted notification that a BC lost connection to the Grandmaster (clockClass degradation). The results show a phase-jump of up to 800ps when switching to a single available backup after unplugging the active link, a phase-jump during switchover resulting from fiber attenuation is up to 100ps. When multiple backup links are provided, majority voting is used in failure pre-detection and the 800ps phase-jump can be reduced to around 100ps.

WR Standardization

The work on WR standardization is slowly converging. WR will likely find its way into IEEE1588 standard in the following parts:

"Layer-1 based synchronization performance enhancement" - an optional feature that allows interoperation between L1 syntonization and PTP synchronization
Port configuration from an external interface - optional feature that allows hand-configuration of PTP states
"Relative Calibration Procedures" - an informative annex that describes WR Calibrations
High Accuracy Default PTP Profile - a third PTP default profile that mandates High Accuracy features
Sub-ns synchronization using High Accuracy Default Profile - an informative annex that describes how to implement High Accuracy Default profile so that it provides sub-ns accuracy

[1] http://cds.cern.ch/journal/CERNBulletin/2015/51/News%20Articles/2111441?ln=en
[2] http://icalepcs.synchrotron.org.au/papers/wec3o01.pdf

JINR

The Data Acquisition Group in the Laboratory of High Energy Physics (LHEP) of the Joint Institute for Nuclear Research (Dubna, Russia) develops and produces readout electronics and software for data taking. This year WR-enabled devices took part in a real data taking run. The BMN [1] technical run was carried out at the extracted beam of the Nuclotron in February - March 2015.

Readout electronics included waveform digitizers [2] for Zero-Degree and Electromagnetic Calorimeters readout and time-stamping TDC for Time-of-Flight and Drift Chamber detectors. Multiple TDC boards [3] [4], Trigger Master [5] and Readout Controller were built in the VME standard. Trigger and clock are distributed within the crate over a private bus driven by the Trigger Master. Both the waveform digitizer and the trigger master run the White Rabbit Node Core for time synchronization. They have a hardware IP stack for communication.

In the waveform digitizers we had to synchronize a 62.5 MHz clock across devices. It is derived directly from the 125 MHz WR clock and its phase is aligned with timestamp. The TDC system uses a 41.667 MHz clock that is exactly 1/3 of the WR clock. A hardware divider-by-3 is reset every time WR acquires sync and timestamp is a multiple of 3. This method is not ideal and requires a complete system reset after each divider reset.

White Rabbit technology provided a stable and precise time base. The Gigabit data path has been used for readout and board control. The use of an 18-port WR switch for bulk data transfer is limited by the 1 Gb/s port bandwidth. This may be improved by either link aggregation (etherchannel) or implementing 10G interfaces.

WR technology is planned to be used for the control and monitoring systems of NICA, Nuclotron-based Ion Collider Facility in JINR. WR is considered as a timing system in the Multi-Purpose Detector (MPD) of NICA.

[1] http://nica.jinr.ru/
[2] http://afi.jinr.ru/ADC64s2
[3] http://afi.jinr.ru/TDC72VHL
[4] http://afi.jinr.ru/TDC64V
[5] http://afi.jinr.ru/FVME2TM

Anders Wallin

Anders performed a stability and phase-noise measurement of the WRS when free-running and GM-locked:
http://www.anderswallin.net/2015/09/white-rabbit-switch-noise-measurement/

Activities in the Netherlands

Xilinx Artix-7 GTP (Nikhef)

Work is in progress to create and validate VHDL sources for a deterministic PHY for the Xilinx Artix-7 family (using a GTP). This will supplement the already validated 7-series GTX (used in Kintex-7 and Virtex-7).

Calibration (Nikhef, Vrije Universiteit Amsterdam):

Some WR users want the freedom to use different SFPs (or other converters), for example for long haul applications. One would like to be able to field-exchange SFPs without the need to do re-calibration. To be able to do so, each network component should be individually calibrated and contain as set of calibration parameters. System level aspects in WR implementations call for remote (and possibly dynamic) update procedures of calibration parameters from network management entities.

Currently there is a single set of calibration parameters for a WR port (combining FPGA, PCB, FMC and SFP propagation delays). We propose to split these parameters by defining an electrical phase plane for each WR port and calibrate the SFPs and FMCs individually. Having the electrical phase plane as a reference implies calibration of the electrical/optical and optical/electrical converters. This domain crossing calibration is under development and first results are promising.

Work is in progress to enable absolute delay calibrations of WR gear. This would enable independent developers or vendors to exchange their WR gear. The latter is an important feature for standardisation. Absolute calibrated devices can be used as "golden standards" for the existing relative calibration procedure.

Uniformly defined measurement definitions, procedures and tooling are under under development. Plans for system level calibration aspects still need to be detailed.

Long haul WR link Delft - Amsterdam (Nikhef, SURFnet, VSL, Vrije Universiteit Amsterdam):

A long-haul test link of 2 x 137km between the Dutch National Metrology institute (VSL) in Delft and Nikhef in Amsterdam is used to disseminate UTC (VSL). The link includes two quasi-bidirectional semiconductor optical amplifiers. After a standard WR-calibration, first time transfer tests result in 2(8) ns time transfer accuracy on the round trip over the fibre. The accuracy and uncertainty on this link are almost completely determined by the unknown chromatic dispersion encountered. Calibration efforts aiming at sub-ns time transfer accuracy are in progress. Schemes to determine chromatic dispersion without modifying the fibre infrastructure between the sites will be tested. If successful, this will lift a barrier for implementation of WR on existing fibre infrastructure.

CTA (UVA, Nikhef):

The Cherenkov Telescope Array (CTA:http://cta-observatory.org) is the next generation very high energy gamma-ray experiment surpassing current instruments by at least an order of magnitude in sensitivity. The University of Amsterdam with the help of Nikhef in Amsterdam, the APC (http://www.apc.univ-paris7.fr/APC_CS) in Paris and DESY in Zeuthen (http://www.desy.de) are working on a prototype of the timing system for CTA using White Rabbit as the baseline technology. For Amsterdam and DESY, these developments are also integrated into the ASTERICS infrastructure cluster which is a part of the european Horizon 2020 program (http://asterics2020.eu/).

Within ASTERICS, as part of the CLEOPATRA work package, there is a joint timing effort to enhance White Rabbit to deliver higher precision over very large distances for radio telescopes for VLBI, to facilitate the precision calibration for large scale projects, to operate up to thousands of timing nodes in harsh environments, and to port White Rabbit to 10 Gb/s Ethernet. This ASTERICS timing project is a cooperation of the University of Granada in Spain, DESY in Zeuthen in Germany, JIVE, ASTRON, Vrije Universiteit Amsterdam and the University of Amsterdam, Nikhef, and SURFnet in the Netherlands.

The CTA focus is on using White Rabbit in harsh environments and improvements in White Rabbit calibration, and synergies between KM3NeT and CTA are being explored. The VLBI subtask aims for a proof-of-principle demonstration of VLBI between telescope sites at Dwingeloo and Westerbork, synchronised through a 200 km DWDM fiber link with live data traffic.

University of Granada (UGR) and Seven Solutions

UGR is contributing to the White-Rabbit community with different R&D projects involving the WR Switch, WR nodes and, together with Seven Solutions, the development of new stand-alone versions of WR nodes based on the Xilinx Zynq architecture [1]. At the same time, UGR is collaborating with other WR partners in the development of a reliable WR node with redundancy capabilities to avoid single point of failure issues.

WR-ZEN developments

The WR-ZEN [2] is an open hardware board designed by Seven Solutions. It uses the Xilinx Zynq technology that contains an ARM processor and a FPGA device in the same System on a Chip (SoC) taking the advantages of a better HW/SW partitioning and interoperability between the FPGA and the ARM processor.

UGR with the collaboration of Seven Solutions have implemented a WR node for the WR-ZEN board with two fully functional SFP ports implementing Fine Delay FMC features.

The WR-ZEN gateware has been designed by UGR and Seven Solutions including some additional IP cores such as a dual-port WRPC, an AXI-WB Bridge core and an I2C Arbiter. In addition, LM32 firmware has been updated to work properly with the new gateware and board architecture taking into account its new features and peculiarities.

In terms of software, the board runs a SMP Linux kernel [3] where several drivers have been adapted from the SPEC project to control the NIC and the Fine Delay cores. On the userspace, some tools have been modified to work with this new architecture. Some examples of these tools are the zen-vuart, zenmem and those related with the Fine Delay driver such as fmc-fdelay-input and fmc-fdelay-pulse.

White Rabbit High-availability Seamless Redundancy (WR-HSR)

UGR is working on increasing availability and robustness of the White Rabbit technology to extend its utilization to industrial networks such as Smart Grid. Applications related to this field demand high availability and very short switch over time. In order to meet these requirements, the electrical substation automation standard IEC 61850 [4] suggests the High-availability Seamless Redundancy (HSR, IEC 62439-3) [5] as one of the redundancy protocols. HSR guarantees zero-time recovery in case of failure in ring and mesh network topologies for both time and data frames and it is based on the duplication of frames in the network.
UGR is currently implementing the HSR protocol for WR devices, in particular the WR-Switch to increase availability and avoid single point of failure for time and data distribution. This work involves the development of the Peer-2-Peer mechanism [6] for clock propagation, PeerDelay [6] for the delay measurement, the utilization of HSR tags [5] in frames, and the SwitchOver mechanism developed by CERN to recover from a node failure. Furthermore, additional gateware must be developed to duplicate and forward frames. Current progress and more information can be found at the OHWR WR-HSR webpage: https://www.ohwr.org/project/wr-hsr/wiki

References

[1] Xilinx Zynq technology and devices, Available online at http://www.xilinx.com/products/silicon-devices/soc/zynq-7000.html
[2] WR-ZEN board, Available online at http://www.sevensols.com/es/productos/wr-zen.html
[3] Xilinx Linux OS, Available online at https://github.com/Xilinx/linux-xlnx
[4] IEC 61850: Power Utility Automation Standard
[5] IEC 62439-3:2012. Industrial communication networks - High availability automation networks
[6] IEEE 1588-2002. Precision Clock Synchronization Protocol for Networked Measurement and Control Systems

DESY (CTA, HiSCORE)

DESY has continued its WR-based timing support for HiSCORE, the Gamma and Cosmic Ray detector in the Tunka-valley/Siberia.

The success of this pioneering WR-timing application yielded critical input to propose for the Cherenkov Telescope Array (CTA; see http://cta-observatory.org) - the next generation very high energy gamma-ray experiment - a centralized, WR-based clock-distribution and trigger-time stamping system with sub-nsec precision and sufficient redundancy to operate in harsh conditions.
The system is now in its prototyping phase (activity in collaboration with APC, Paris and UvA, Amsterdam).

HiSCORE

The 9-station HiSCORE setup has been successfully operating since the 2013/2014 observation period. The Array has been enlarged to 28 stations covering 0.3 km^2 in 2014.

The 9-station WR-setup is based on a SPEC with DIO5Ch, which is operated as a 1-nsec time-stamping unit, with the analog PMT signals directly fed into the DIO5Ch, thus acting as a discriminator with adjustable threshold and digital pulse selection capability. Additional DAQ functionality for waveform sampling readout (DRS4-EB; trigger and busy) turned the SPEC/DIO5Ch into a complete front-end board. See contributions to ECRS2014 and ICRC2015 conferences.

The new stations (2014) have a custom-made front-end board (DRS4-based, MSU/Moscow) connected as Mezzanine to a WR-SPEC. Detailed long-term test of WR performance with a custom made time-distribution system shows excellent WR-perfomance.

DESY has worked in close collaboration with Humboldt-University Berlin (Martin Brueckner, now PSI).

Tsinghua University

WR development

1:Fixed delay constrain for WR node
The location and inter-connection of WRPC logic and the PPS signal path are all decided by the P&R engine without any constrains, thus the fixed delay varies each time the project is re-compiled. Since in our application the WRPC is integrated in the same FPGA with customer logic which can be remotely upgraded from time to time, the requirement for fixed delay re-calibration is not convenient and even not possible. Besides, the WR node suffers more temperature variation from the un-constrained long inter connection delay.
By adding additional constrains to lock the location of key blocks of the WRPC, the inter-connection delay of the WRPC would be reduced and relatively fixed. This makes the integration of WRPC with other logic much easier and releases the necessity of fixed delay temperature correction.

2:Dual-Port WR node development
The PCB design of Dual-Port WR node in FMC form (Cute-WR-DP) has been delivered for fabrication. The XC6SLX45T-3CSG324C is used to fit within the FMC form together with two SFP sockets. The two ports work in parallel to provide redundancy for high-reliability applications. More flexible topologies like daisy chain are still under consideration.

WR applications

The prototype array of 60 WR nodes and 4 WRS deployed in Tibet has operated for almost 18 months. Preliminary physics analysis from both telescope and cosmic sources confirmed that all normal detector units are aligned within nano-second precision which include the contributions from crystal, PMT, digitization electronics and WR network. Some nodes still suffered from link connection failures; nodes lost synchronization during the run which can be explained by a known WRPC bug. Further analysis is ongoing.

GSI

During 2015 the Timing Team at GSI has been heavily involved in the development and commissioning of the General Machine Timing (GMT) system [1] for the CRYRING [2] project. CRYRING is an ion storage ring, formerly located at Stockholm University. Besides it serves as a test case for developments relevant for the FAIR [3] project. So far, the injector of CRYRING is routinely operated using FAIR-style solutions like the GMT since a few months. Injection into the ring itself is planned for spring 2016.

In addition to the "production" WR network for CRYRING, two other networks have been set up at GSI: The first one is a "user" network, where other developers use WR to synchronize their systems under development (e.g. RF System). The second one is a "test-bench" network used by the Timing Team for testing and debugging the latest version of the control system and WR firmware. Monitoring tools (e.g. Nagios/Icinga) are being used in the GSI WR networks for monitoring status, stability and synchronization quality of the WR Switches as well as Altera based TRs. Moreover, the monitoring tools serve to detect and troubleshoot networking problems such as broken links or unregistered/broken nodes.

Together with the related WR activities, other essential features for the GMT have been developed during this year:

Simplified API for Timing ("SAFTlib") [4]
Upgrade of the Data Master [1]
I/O Control and clock generator VHDL modules
Integration of the timing system with higher layers of the control System (e.g FESA)

Along with the CRYRING project, the Timing Team has been developing WR-based TRs in form factors AMC and PMC in collaboration with Cosylab (FAIR in-kind contribution) based on the in-house developed WR TR (Pexaria, Vetar and Exploder [5]). The designs will be published in OHWR in due time.

[1] https://www-acc.gsi.de/wikis/pub/Timing/TimingSystemDocuments/GMT_Description_v3-1.pdf
[2] https://www.gsi.de/work/forschung/appapni/atomphysik/forschung/experimentieranlagen/cryringesr.htm
[3] https://www.gsi.de/en/research/fair.htm
[4] https://www-acc.gsi.de/wikis/Timing/TimingSystemDocumentsSaftlib
[5] https://www.gsi.de/work/fairgsi/rare_isotope_beams/electronics/digitalelektronik/digitalelektronik/module/lwl/pexaria/pexaria5.htm

Creotech

Creotech Instruments SA in cooperation with N.A.T develops a WR-enabled MCH module with dual SFP, dual WR D3S and TDC/DTC. This module can be installed on top of a standard N.A.T MCH in both single width and double width modules. Optionally, the WR-MCH gets access to the MLVDS ports via a modified PENTAIR (SCHROFF) MTCA.4 backplane (coming soon).
Creotech also ported the WR node to the AMC-FMC Carrier with Kintex 7 (AFCK) [1].
Currently, in the frame of the WR-SYNTEF project, a feasibility study is carried out on the use of WR to synchronize the work of ESA Ground Stations Network. This solution would provide optimal use of ground station resources and ground stations network synchronization.
We are also working in cooperation with Warsaw University of Technology on the next generation of WR-enabled AMC-FMC carriers based on ZynQ UltraScale+ SoC devices. This module will be optimized for recent JESD204B ADC/DAC GS/S converters.

[1] https://www.ohwr.org/project/afck/wiki

Newsletter 2015 12