White Rabbit Newsletter, May 2017
Quite a lot has happened in the WR scene since the last newsletter. With two companies producing switches, WR is now a true multi-vendor solution. We have also seen more open and proprietary hardware incorporating WR support. The WR user list keeps getting bigger and bigger. We also saw people starting to advertise WR-related jobs in the Netherlands, Switzerland and Germany. WR was even on Spanish TV!
CERN
A fiber cleaning course was recently organized to raise awareness about this important issue.
WR Switch firmware activities
We have released WR Switch firmware v5.0 with many improvements in both
software and gateware. Inside you will find an updated Buildroot and
Linux kernel, added VLANs configuration to the main dot-config file,
improved Syslog messages, added tools for reading and writing SFP
EEPROMs and much more. On the HDL side, version v5.0 brings among others
a new bandwidth throttling module for the traffic going to the ARM CPU
and re-written, more efficient multi-port linked list in the Ethernet
switching core. Please check the official
news message about the release. We are
currently in the process of integrating WR switches within the CERN
monitoring (SNMP) and configuration (dot-config generation from a
central database) infrastructure.
We plan soon to prepare a v5.0.1 release with several minor software
fixes to v5.0. Please check the
issues with a target
version v5.0.1 for more details.
WR PTP Core activities
We have released v4.0 of the WR PTP Core. This new release adds among others the SNMP and Syslog for remote monitoring as well as VLAN support. On the HDL side, we have re-written the Mini-NIC module to fix problems with handling high loads of traffic and improved a lot the synthesis time with RAM initialization for Xilinx. We took the opportunity of this new release to reorganize the wr-cores HDL repository to add platform and board support modules as well as a few reference designs for officially supported boards. Please check the release user manual for a detailed description of these new modules and instructions for how to use them in your custom designs. In the coming months we would like to expand the platform, board, reference design modules to cover also Xilinx series 7 FPGAs and work on porting the WR PTP Core to Kintex Ultrascale FPGAs.
New manufacturer of the WR Switch hardware and the official Production Test Suite
Creotech Instruments S.A. is another company that started producing WR Switch hardware last year. We have bought a few devices and successfully evaluated them together with GSI to make sure they are produced from the same design files as switches sold by 7Solutions. Currently, we are working together with INCAA to prepare an official Production Test Suite (PTS) for the WR Switch hardware. This PTS consists of a set of tests that can be used either by the companies during production, to verify that all the hardware components are soldered and working properly, or by users to test the hardware in case of failures.
SPEC and SVEC PTS-es expanded to assign official MAC addresses by manufacturers.
Till now, the Ethernet MAC address of a WR port on SPEC and SVEC cards was generated from the 1-Wire thermometer ID. Last year we updated the Production Test Suites (PTS) for SPEC and SVEC cards to allow manufacturers to assign unique MAC addresses from the official pool. The MAC address is stored in the Flash memory and can be used by the default WR PTP Core firmware. The update to PTS-es was sent to all the companies producing these cards. Please contact your supplier if you're interested in having WR Nodes with official MAC addresses.
Proof of concept demo of super low latency triggers distribution in White Rabbit.
In 2016 we made a proof of concept implementation of super low latency trigger distribution in White Rabbit. We added a module that injects special 8B/10B characters to the Ethernet data stream. These special characters are then detected on the other side and interpreted as triggers. Thanks to the fact that the injection and reception is done in the physical layer, the overall latency of the trigger is the result of only the SerDes, PCB and fiber latencies. This opens new possibilities of using White Rabbit as a trigger distribution network in high energy physics experiments. This is currently a proof of concept implementation and still requires more work to be included in the stable release of the WR Switch and WR PTP Core. If you find this interesting for your application, please let us know and have a look at branch supertrigger in the wr-cores HDL repository.
Tsinghua University
We have adapted WR technology for a pre-clinical Positron Emission tomographic Instrument. The PET ring consists of 8 detector modules, each containing 6 LYSO arrays coupled to a 8*8 SiPM matrix. The signals from the SiPM are processed by a dedicated ASIC and then digitized and time-tagged with a commercial ADC and FPGA-based TDC respectively. Each module acts as a White Rabbit slave node to synchronize the time reference of the TDC. The extracted list mode data is also transmitted via the WR link. A standard WR switch collects the data from all 8 detector modules, but the firmware has been modified. The list mode data packages from each module are extracted while other packages enter the switching matrix as normal. The list mode data are then processed by dedicated timestamp coincidence logic. The outcomes are then re-packeted and send out via a separate port to a reconstruction PC. The scheme greatly simplifies the PET design by replacing traditional off ring electronics with a standard White Rabbit switch.
The design of dual-port Cute-WR mezzinine has been released. It can act as a slave with a redundant connection to improve the reliability or in daisy chain mode to support a cascade topology. We have verified the WR functionality on each port for both master/slave. The daisy chain for PTP transmission has been verified, and the routing of data frames is still under development. With the dual port mezzanine, we are also designing a portable calibration node which can play an important role in a WR auto-calibration procedure, WR array deployment and validating the synchronization accuracy, bringing the advantage of higher integration, more convenience and lower human workload (http://ieeexplore.ieee.org/document/7820218/).
A prototype array of 100 detector nodes are running in Tibet. For quite a long period, we suffered some instabilities, randomly losing nodes during power up and normal operation. The new released WRPC 4.0 has greatly improved the situation. The LHAASO project has been finally approved and more than 500 WRS and 7000 WR nodes will be deployed at an altitude of 4300m asl. At the end of 2018,a quarter of the full array is foreseen to operate.
Nikhef and Vrije Universiteit Amsterdam
Xilinx Family-7 GTH (Nikhef)
VHDL sources are available for a deterministic PHY for the Xilinx Family-7 GTH (used by some Virtex-7 devices).
Rabbit_FX: White Rabbit Oscillators on an FMC mezzanine (Nikhef)
A White Rabbit FMC module that includes the DACs and Oscillators needed to run White Rabbit has been built (https://redmine.nikhef.nl/et/project/rabbit_fx/wiki). It enables code development on a vendor FPGA development board before new hardware is available.
White Rabbit Switch mini backplane for embedded applications. (Nikhef)
In embedded detector electronics there is usually not much space available. Work is in progress to create a small WR Switch mini backplane with two 2x5 SFP cages.
Calibration (Nikhef, Vrije Universiteit Amsterdam):
Calibration of WR gear is addressed by the following three main items:
1) Absolute delay calibration
It is proven that hardware delay calibration parameters can be derived
(https://www.ohwr.org/project/wr-calibration/wiki). These parameters
define the timing relationship between measurable external phase planes
(i.e. the PPS connector and SFP electrical connector). Additional "mode
abscal" still needs to be added to PPSi.
2) Electrical/Optical and Optical/Electrical calibration
Although still in a laboratory state, it has been proven that hardware
delay calibration parameters of electrical/optical and
optical/electrical converters (i.e. SFPs in most cases) can be measured
with pico-second precision. Storing the delay parameters in SFP EEPROM
will enable exchange of SFPs without the need for re-calibration. A
definition for storing calibration parameters can be found here:
https://www.ohwr.org/project/sfp-plus-i2c/wikis/User_EEPROM.
3) In-situ asymmetry coefficient determination
Work is in progress to determine the fiber asymmetry coefficient in-situ
using SFPs with a tunable laser.
Tools for calibration have been developed.
a) It is planned to produce a batch of "SFP+ Loop Back Modules" (https://www.ohwr.org/project/white-rabbit/uploads/f6d07a9f48cfbaf3846f0b9b985571e4/SFP__timing_calibration_module.pdf) used for absolute calibration. Currently the mechanical housing causes some issues that prevent us from production. When these issues are solved an announcement will follow via the white-rabbit-dev mailing list such that people interested can sign up for a module.
b) A 10 Gbps capable Multi SFP crate is designed including "SaFariPark" software (https://www.ohwr.org/project/sfp-plus-i2c/wiki) which enables us to exercise SFP modules, both with respect to their serial link, as well as the I2C interface. The latter is needed for accessing digital diagnostics, storing calibration parameters and enabling SFP tuning. Moreover "SaFariPark" computes SFP checksums and can correct SFPs with corrupted EEPROM content (unfortunately something we experienced too often).
c) A proposal for a generic software library to access SFPs via the I2C
bus is launched (https://www.ohwr.org/project/libsfp/wiki). When this
software is deployed in WR gear then calibration
parameters stored in EEPROM are accessible, as well as reading out SFP
Digital Diagnostics.
EPFL
The distributed measurement of synchrophasors via Phasor Measurement
Units (PMUs) represents one of the most advanced sensing layers in the
domain of Wide Area Monitoring and Control (WAMC) systems. According to
the IEEE Std. C37.118.1-2011, a PMU is a measurement device synchronized
to a common Coordinated Universal Time (UTC) reference, which reports
phase-aligned and time-stamped measurements of frequency, amplitude and
phase angle of the voltage and current phasors of a power system.
In general, the time synchronization of PMUs relies on the Global
Positioning System (GPS) as it represents a good tradeoff between
performance and cost. However, this synchronization system has three
main drawbacks: (i) accuracy, (ii) accessibility and (iii) security.
Concerning the first point, it is worth noting that, in order to
correctly support synchrophasor based applications, PMUs must be
characterized by high accuracy levels in estimating the synchrophasors,
especially when power distribution systems applications are envisaged.
Modern PMUs are adopting synchrophasor estimation algorithms exhibiting
steady state phase accuracies in the order of tens of ppm of radians.
Since these values correspond to time jitter that typically
characterizes the GPS units usually adopted by the PMU hardware, we may
conclude that one of the barriers to improve the steady state PMU
accuracy is the uncertainty of the time dissemination technology.
Concerning the second point listed before, i.e., the accessibility of
the GPS, we can observe that such a time source might not provide a
stable and reliable time reference especially in cases where underground
substations, with limited or no access to the sky, need to be equipped
with PMUs. Therefore, a more suitable time dissemination technique,
deployable over the legacy power system’s telecom infrastructure, might
be required. Concerning the third point, security, recent works have
shown that, since civilian GPS satellite signals are not authenticated,
they can be spoofed by superimposing a fake signal with a higher
signal-to-noise ratio, which would enable an attacker to manipulate the
GPS clock.
Among the possible alternatives to the GPS, White Rabbit (WR) represents
an excellent candidate as time synchronization protocol in view of its
superior peculiarities. In this project, we have developed a PMU
integrating WR technology. Using a WR network composed of a WR switch
and WR-cRIO modules, we assessed the performance of the WR-PMU by means
of a PMU calibrator. Moreover, we compared the performance of the WR-PMU
with the one of a GPS-based PMU. The two PMUs were characterized by the
same synchrophasor estimation algorithm and by the same hardware
platform, with the exception of the time synchronization module and
technique. The results exhibit similar performance between the WR-PMU
and the GPS-PMU, showing WR’s applicability for PMUs. The developed
system can be used as an alternative for specific conditions where the
GPS signal is not available.