White Rabbit Newsletter September 2014
What follows is a minimally edited set of contributions from different institutes and companies.
CERN (including Alessandro)
Switch gateware
V4 of the switch includes gateware with VLAN support, increased throughput and lower latency than V3. The new gateware was tested with a specialized IT traffic generator and analyzer. The tests show latencies well below 10us.
Switch software
Support for remote logging, review and merge of http work by University of Granada, some fixes to initialization, support for SNMP using the standard MIB files, support for a new barebox and ubifs, new improved installation procedure, with support for bigger firmware files, minor changes to build under latest Ubuntu.
Standardization
The IEEE1588 High Accuracy committee works on standardizing WR within the IEEE 1588 standard. The last PTP standardization meeting was organized at CERN in April 2014. Likely, different aspects of WR will be formed into optional features provided by the standard. The work of the committee is now focusing on the first feature: L1 Syntonization. It started by studying the way WR uses L1 syntonization and phase detection to enhance the timestamping precision. A number of potentially useful use cases were identified. Currently, the methods are being generalized. The goal is to make them applicable in a standard way to different media, syntonization methods and phase detection techniques.
PPSi
We released ppsi-v2014.07. It includes a PTP simulator that allowed us to fine-tune the servo algorithm for non-WR slaves. It also includes a configurable frame loss rate as a fault-injection system. This helped us reproduce and debug a relatively rare problem occurring to one of our users. PPSi 2014.07 is also the subject matter of a paper submitted (and accepted) for ISPCS 2014.
Node calibration procedure
After publishing two drafts for comments, we're now happy to announce that v1.0 of the "White Rabbit calibration procedure" document is available on OHWR. You can download it from the Documents section of the main White Rabbit project page [1].
The document provides step-by-step instructions to be performed to ensure your WR network is properly calibrated and synchronized with sub-ns accuracy. It also contains additional hints on how to measure the fiber link which is already deployed and how to recover a WR Calibrator parameters if your primary device got broken. Two appendices provide the mathematical proofs for the calibration correctness and the estimation of measurement errors.
[1] https://www.ohwr.org/project/white-rabbit/wikis/Documents/White-Rabbit-calibration-procedure
GSI
This is just an update to last years newsletter [1]. The GSI Timing Team is working on the General Machine Timing (GMT) system based on White Rabbit for the upcoming FAIR facility [2]. The next milestone will achieve "readiness for test operation" with the CRYRING@ESR project. CRYRING is a contribution by Sweden and presently being set up behind the Experimental Storage Ring (ESR). Besides it serves as a test case for FAIR relevant developments.
Timing Node Hardware
All hardware is based on Arria II GX or Arria V GX FPGAs. Production readiness was achieved for the form factors PCIe [3], VME [4] and SCU [5]. The PEXARIA5 PCIe module [3] serves as a reference design for future developments of timing nodes for FAIR.
Timing Node Gateware
During the last year we complemented existing VHDL cores including newer version of the Timestamp Latch Unit (TLU), an Etherbone master, Message Signaled Interrupts (MSI) and a Wishbone MSI Crossbar. Another important achievement was the unification of form factor specific top designs by providing the so-called "monster" design. In cooperation with other teams of the control system group we finally achieved agreement on the data format, which will be provided to the other layers of the control system when time critical actions of the accelerator are triggered by the timing system. Subsequently, the first production version of the Event-Condition-Action (ECA) unit has been implemented. On Arria V GX based platform we have improved the granularity for triggering actions (ECA unit) and latching timestamps (TLU) from 8 ns to 1 ns. This does not required external circuitry such as delay chips but is achieved by using differential IOs of the FPGA clocked at 1 GHz and is entirely done in gateware.
Data Master
The data master schedules events by broadcasting relevant timing messages across the timing network (remark: The ECA unit in a timing node filters incoming messages. Relevant messages are decoded and preconfigured actions on the timing node are executed on time). The data master is implemented using a cluster of lm32 softcores instantiated on a PCIe board [3]. The softcores are synchronised using either MSI or shared memory. Schedules are prepared beforehand and presently described using a XML based syntax.
Release Asterisk
In June 2014 we released gateware, firmware for the hardware platforms [3-6] under the codename "Asterisk" [6]. Please note, that this release includes many cool tools such as "eb-console", which allows accessing the "White Rabbit GUI" remotely. The gateware of all form factor also includes a flash controller that allows to flash devices via any Etherbone supported protocol such a PCIe, VME, USB and UDP.
Integration with higher layers of the control system
Integration of timing nodes into the core of the Front-End Software Architecture [7] has so far only been implemented as a prototype. Nevertheless, it allows FESA classes to receive events from the timing system by subscribing to a so-called timing event source in FESA core. A next step of prime importance is to link the data master to the Settings Management system of the accelerator control system.
Operation
So far, the timing system works reliably including the data master, a timing network of 11 active White Rabbit Switches and about 20-30 active timing nodes. Already today, the timing system serves for time stamp distribution for Data AcQuisition (DAQ) systems at GSI. A first beam time involving a DAQ system with synchronisation via White Rabbit PTP is scheduled for September 2014. Timing nodes in form factors PCIe [3] and VME [4] will be used.
[1]
https://www.ohwr.org/project/white-rabbit/wikis/Newsletter_2013_10
[2] https://www-acc.gsi.de/wikis/Timing
[3]
https://www.gsi.de/work/fairgsi/common_systems/csee_electronics/digitalelektronik/digitalelektronik/module/lwl/pexaria/pexaria5.htm
[4]
https://www.gsi.de/de/work/fairgsi/common_systems/csee_electronics/digitalelektronik/digitalelektronik/module/vme/vetar/vetar2.htm
[5] S. Rauch, R. Bär, W. Panschow, M. Thieme, ICALEPCS 2011, Grenoble,
WEPMN018
[6] https://github.com/stefanrauch/bel_projects branch "Asterisk".
[7] FESA A. Schwinn et al., PCaPAC 2010, Saskatoon, WECOAA03
MIKES
PPS output tests
This shows that the PLL between an external 10MHz+1PPS clock and the
WR-timebase is quite noisy. Tomasz W said he is aware of the issue and
will work on it when he has
time.
http://www.anderswallin.net/2014/01/white-rabbit-dio-vs-fdelay-pps-output-test/
Time-stamping test with FMC FDEL
The tdc_raw mode of the driver was used. It packs multiple time-stamps
into a ZIO data-block, instead of the default behaviour where each time
stamp is contained in one block. With very moderate PC hardware (i3 CPU)
this allows time-stamping at ca 150 kHz, while streaming the data to
disk.
http://www.anderswallin.net/2014/02/white-rabbit-fine-delay-time-stamp-testing/
Photon counting correlation test
http://www.anderswallin.net/2014/02/photon-correlation-test-with-modulated-led/
This method is widely used in the trapped-ion community - where people
work on either optical clocks or quantum computing. When ions are
trapped in RF-fields using Paul-traps they undergo secular (usually ca 1
MHz) and driven (usually 10MHz) motion in the trap. This motion is
undesirable as it shifts the 'clock-transition' frequency due to the
doppler effect. The secular motion can be minimized by laser cooling the
ion, while the driven motion is minimized by optimizing the trap design
and applying additional compensation DC electric-fields. The motion can
be detected as a correlation signal in the fluorescence signal from the
ion. Using a modulated LED as the light-source, this demonstration shows
photon-counting with a PMT, time-stamping with the Fine-Delay, and
real-time computation of an arrival-time histogram on the PC. We hope to
apply this technique to our ion clocks in the future.
University of Granada
UGR is contributing to the White-Rabbit community with different R&D projects in relation to the WR switch, WR nodes, porting WR components to other development platforms as well as stand-alone versions of current WR nodes. In addition to this, UGR has been also involved in the development of the Web Management Interface for the WR switch.
At the same time, UGR is working in conjunction with Seven Solutions in the development of a reliable WR node with redundancy capabilities to avoid single point of failure issues.
All this work is being carried out in the framework of several national and international projects, such as ACELTEC and the Square Kilometre Array (SKA). UGR contributes to both projects in synchronization and timing work-packages promoting WR as the main time&frequency distribution technology.
In addition, UGR is putting its efforts in Smart Grid technologies and the applicability of WR to this field.
Stand-alone FMC Delay Card
The Stand-alone FMC Delay Card is a fully operational stand-alone FMC Delay card based on a WR node that can be initialized remotely. It also performs periodic calibrations without requiring being plugged in a PC, leading to a reduction of final system costs, size and power consumption.
WR Port to 10 Gigabit Ethernet
UGR is working on the development of a new hardware architecture to perform an analysis about the feasibility of porting WR from 1 Gigabit Ethernet to 10Gps. We are designing and testing different FPGA architectures looking for the best solution for this purpose.
At the same time, UGR has modified a Xilinx development platform VC7 by adding external White-Rabbit oscillators to provide this platform with this feature. In addition, we have modified the Phase-locked Loop (PLL) to disseminate a frequency reference of 156,25 MHz on the platform.
This project includes a WR PTP Core port from an Altera Spartan-6 FPGA to a Xilinx Virtex-7 one to take profit of the advantages of the new Virtex-7 FPGA series.
WR Switch Web Management Interface
UGR has developed a friendly Web Management Interface for the WR switch to facilitate its configuration and monitoring for non-WR-expert users. This interface has been recently upgraded to the 4.0 switch release adding features such as VLAN configuration, network setup, PPSi clock customization, NTP Server, user login and remote flashing. Now, it is also possible to load and backup the configuration files for all services in addition to other improvements on the interface and previous features.
Seven Solutions
WR customization for KM3Net
KM3Net is a cubic-kilometer scale neutrino telescope at the bottom of the Mediterranean Sea. KM3net is a highly complex network composed by underwater sensors hosted at Digital Optical Modules (DOMs), the onshore station and the optical connections between/inside them. Every single DOM has a unidirectional 1Gb/s uplink to reach the onshore station (through this link delivers the acquired data). On the other side, the onshore station has a unique unidirectional 1 Gb/s downlink to reach all the DOMs, this is called the broadcast or slow control link (SC). The SC is shared by every single DOM, so that when a DOM receives a packet, all the other DOMs also receive it. This topology highly reduces the communication resources cost at this facility, but it requires customizing the communication elements as the switches and nodes.
A deep customization of WR switch has been done to develop it from being a synchronization point-to-point technology which uses bidirectional links between two devices to be able to synchronize 360 DOMs by asymmetric and unidirectional links. This customization allows sharing the unidirectional downlink between 360 DOMs, highly optimizing the communication resources.
WR Zynq Embedded Node (7S-UGR)
UGR and 7S are working together in the development of a new WR node in which the main SoC is a Xilinx Zynq programmable chip. This chip is composed of two Cortex ARM9 professors and a FPGA making this a promising platform for Smart Grid applications. The FPGA is planned to implement a new version of the WR PTP Core (7S) and the two ARMs will run the operating system and the WR software (UGR).
This platform has been focused to ease the development of standalone nodes by including a high number of communication ports and the possibility to expand new features using the FMC connectors. Additionally this platform has been focused for industrial markets and will provide various redundancy features such as an internal backup oscillators (MEMS) as holdover mechanism or the High-availability Seamless Redundancy (HSR) protocol to avoid single points of failure. It will also support standard timing distribution protocols (i.e, IRIGB, NMEA, ToD, SNTP) required in Smart Grid or Telecommunication markets.
The PTS Core
Seven Solutions has been improving the Production Test Suite repository to ease the creation of tests for its own products. This repository has been designed as a submodule to include in the test repository of each product. It contains documented classes to support common peripherals, internal buses (e,g, i2c, spi, 1wire) and various drivers (wishbone master). It also provides to the user a simple API to test devices through SSH, to test devices with various boards and to log this information in a clean way.
WB in EPICS
EPICS is a set of Open Source software tools, libraries and applications developed collaboratively to create distributed soft real-time control systems for large scientific experiments and it is commonly used by particle accelerators and telescopes in the USA. Seven Solutions has created an IOC driver to ease the integration of wishbone bus devices into the EPICS platform by using a standard driver to access the wishbone bus such as PCIe or VME. This solution also uses a modification of the wishbone slave generator to create automatically the proper PV (Process value) that needs to be integrated in the EPICS platform.
LLRF using WR
Seven Solutions has been working in collaboration with CIEMAT on the creation and support of a fully Digital Low-Level Radio Frequency platform to control several RF cavities such as RFQ or RF-Linac built for a high-intensity CW deuteron accelerator. In this project we have developed a CPCI-Serial carrier with two FMC HPC plugs to a loop-back acquisition board (12xADC/2xDAC) and a digital IO board for interlocks signals. The whole system is controlled through the EPICS platform and its CSS/BOY interface. Finally WR is used to synchronize all the LLRF along the accelerator.
Tsinghua University
LHAASO prototype array setup in Tibet
The LHAASO experiment has been finally located at Yading, Daocheng,
China, near the world's highest airport
(http://en.wikipedia.org/wikis/Daocheng_Yading_Airport)
A prototype array for LHAASO KM2A has been installed in YBJ, Tibet
(http://en.wikipedia.org/wikis/Yangbajain) to test the function &
stability of the white-rabbit network and the customized detector and
electronics of LHAASO.The prototype array includes 4 WR switches in a
chain topology, 50 electronic detectors and 2 muon detectors. The
Grandmaster switch gets the UTC time from a GPS receiver using NTP
service, the PPS signal from the same GPS receiver keeps the timestamps
updated. A 500ps synchronization among the units has been achieved after
calibration, more detailed tests are still ongoing.
Temperature Effect and Correction Method of White Rabbit Timing Link
The temperature effect and contributions from different components are
separately studied and analyzed. An online real-time temperature
correction method was applied based on the result which significantly
reduce the synchronization variation from 300 ps to 50 ps in a
temperature range of 50 degrees centigrade.
The related results can be found at: http://arxiv.org/abs/1406.4223
Dual-Port WR development
We are developing a WR node with two SFP ports. This node keeps most compatibility with Cute-WR (using the same FPGA and other peripheral circuits but just an additional SFP connector). The two ports can be connected to different WRS and work simultaneously to improve the reliability of the WR node in case of link failure. We are also exploring the possibility to make the node work in chain-mode to support a daisy connection topology.
Consulting for potential WR applications in China
- JUNO (Jiangmen Underground Neutrino Observatory) known as the following of Dayabay reactor neutrino experiment which had successfully measured theta13 in 2012. The experiment is scheduled for data taking in 2020. The WR network will be applied to provide synchronized clock and timestamp to hundreds of under-water electronics boxes with a high requirement for reliability.
- The beam protection system for an ADS (Accelerator Driven Sub-critical System) prototype. The scale for the prototype will consist of only tens of WR-Nodes. The precision of WR could be over-qualified for this but they raise the requirement of high reliability.
- Timing infrastructure for the Jingping under-ground laboratory, which is now the deepest under-ground lab in the world with 2800m rock overhead. 8 new experiment halls are now under construction. WR will be deployed as standard timing interface for all potential experiments.
- Timing synchronization for a modularized human TOF-PET instrument. The instrument will be divided into 18 separate DAQ sectors that will need to be synchronized with few hundred pico-second level for the TOF coincidence scheme.