White Rabbit Newsletter October 2013
What follows is a minimally edited set of contributions from different institutes and companies.
CERN (including Alessandro and Aurelio)
We are mostly working on consolidating the switch gateware, PPSi (PTP Ported to Silicon) and standardization. Alessandro is just starting to look at switch software for remote management and diagnostics, and will soon publish a master plan for the work to be done in that area over the following months.
PPSi v2013.09 has been silently released, and has been operating at the ISPCS 2013 plug fest. On the switch it supports UDP mode (missing in ptp-noposix) and can properly be a master or a slave to non-wr clocks. PPSi can run both Ethernet and UDP mode on the same Ethernet port, so it can talk with all clocks on the network. It was tested on WR switches and SPECs against a number of commercial and prototype devices from different vendors at the plug fest. Here's a report of the tests. The PTP servo and the Best Master Clock Algorithm were also tested. Some bugs were fixed, valuable feedback collected and the results are promising - PPSi can easily speak with standard PTP devices.
The revision process of the IEEE1588 Precision Time Protocol (PTP) standard started in mid-2013. The revision work is done by the p1588 Working Group, which consists of over 180 members from all over the world representing industry and science, including CERN. White Rabbit (WR), defined as an extension to the PTP protocol, enhances PTP providing sub-ns synchronization. These enhancements are now evaluated by the Working Group for inclusion in the standard. A special committee was created within the Working Group to deal with high accuracy enhancements to PTP. This committee consist of over 50 members and is lead by Maciej Lipiński representing CERN. The first face-to-face meeting of the P1588 Working Group took place during the last week of September 2013 in Lemgo, Germany. During the meeting different aspects of the White Rabbit technology were studied to decide where and how they could be included in the standard. The Working Group members agree that many aspects of WR can be generally useful in many different applications and could be integrated into the standard as optional features or put into informative annexes. Working out the the details will be one of the most challenging and interesting tasks for the Working Group in the coming months and years. Surely, we are one step closer to our goal: White Rabbit being a high-performance standard implementation of the revised Precision Time Protocol.
CERN RF group (Magnus)
I am in the process of upgrading the RF program in the PS and will
therefore be the receiving side of the B-train (real-time bending
magnetic field) information in the PS. The B-train information will be
transmitted with the White Rabbit protocol and the receiving side will
calculate the RF frequency and provide this to the beam control. Since
we are the receiving side of a link we have collaborated with Daniel
Oberson and David Giloteaux at the sending side and benefited greatly
from their knowledge and experience working with WR.
In the end we will use the
SVEC with one custom FMC board
and possibly a CO provided DIO
FMC for further
distribution of the frequency over small distances. For development we
are currently using the White Rabbit starting
kit as it contains
two SPEC's with DIO FMC's. The
VME and PCI-e carriers are very similar in design which makes it easy to
switch to the SVEC later in the process for the final implementation.
The memory solutions and the general hardware found on the carriers meet
our requirements with good margin. We will treat the "White Rabbit PTP
Core - WRPC"/project/wr-cores/wikis/Wrpc_core as a black box with the
possibility of changing some generics and implementing the "user defined
module" at the low layer of the hierarchy.
GSI
The General Machine Timing (GMT) system for the upcoming FAIR facility
is based on White Rabbit (WR) and will consist of a Timing Master
The ongoing work is dedicated to a control system for CRYRING, a small synchrotron that is presently set-up at GSI. CRYRING is an in-kind contribution by Sweden to FAIR. The related milestone has to be reached by the end of this year and the GMT must include the following aspects.
1.) Timing Master
- A clock master WRS connected to a GPS disciplined oscillator.
- A Data Master (DM) implementing a cluster of lm32 softcores in a WR node. Each lm32 controls a specific machine (like CRYRING) and has the task of generating control messages for equipment synchronization. For the upcoming milestone, about 2-4 lm32 are instantiated. The DM will be implemented using a PEXARIA5 PCIe card [5].
- A Management Master (MM) provides service like BOOTP and DHCP. Both protocols are required to provide TRs and WRSs with IP addresses. Furthermore the MM serves as a gateway from the standard controls network to the timing network and the WRS management network.
2.) Timing Network
- About 4-5 White Rabbit Switches are used.
- The network must be characterized in terms of throughput, latency, bit error rate, package loss rate... . Issues like forward error correction and redundancy are not yet addressed.
3.) Timing Receivers (TR)
- TRs decode the control messages generated by the DM following the Event Condition Action (ECA) concept [2]. For the next milestones, only timed activities (called timing-events) like IRQ or digital signal generation are considered. When comparing digital signals generated by two distinct TRs, a precision of 1ns or better is achieved.
- Present TRs are either included in the so-called Scalable Control Unit (SCU) [3] or implemented in the form factors standalone "EXPLODER2C" [4], PCIe "PEXARIA5" [5] and VME "VETAR2" [6]. All form factors are based on Altera FPGAs of the Arria II or Arria V families.
4.) Gateware
Both DM and TR share the same concept and gateware is synthesized based
on identical components. The corner stones of the
gateware have been implemented. A few examples include a) host-bus to
Wishbone bridges for the form factors Ethernet, PCIe, USB and VME b)
lm32 softcore processors c) the ECA unit [2] and d) a Timestamp Latch
Unit (TLU). The mentioned components are available via the Open Hardware
Repository [7]. Examples for integration of those components into form
factor specific gateware can be found at [9].
5.) DAQ systems at FAIR
Besides its task as event distribution system for accelerator equipment,
the GMT will provide services to other systems and users of the FAIR
facility. Specifically, this is of interest to DAQ systems of the
experiments. Free-running DAQ systems require distribution of a
high-precision 200MHz clock and distribution of the related timestamps.
Jitter and drift for GSI form factors have been characterized using a
dedicated set-up for DAQ systems [8]. The measured values are better
than the ones given in the detailed specifications of the GMT (100ps).
Globally triggered DAQ system will make use of the TLU which is part of
the TRs gateware. When a PEXARIA5 board is integrated in a typical DAQ
setup, latching
of timestamps by digital input signals and subsequent readout via PCIe
can be performed at a rate exceeding 50kHz. As an important result it
has been demonstrated, that White Rabbit links, hardware, gateware and
host-bus bridges are robust and work reliably over periods of many days.
The analysis of the above measurements is still ongoing and results will
be published later.
[1]: https://www-acc.gsi.de/wikis/Timing
[2]:
https://www-acc.gsi.de/wikis/pub/Timing/TimingSystemDocuments/eca_design_2013-05-23.pdf
[3]: S. Rauch, R. Bär, W. Panschow, M. Thieme, ICALEPCS 2011,
Grenoble, WEPMN018
[4]:
https://www-acc.gsi.de/wikis/Timing/TimingSystemDocumentsEXPLODER2C
[5]:
https://www-acc.gsi.de/wikis/Timing/TimingSystemDocumentsPEXARIA5
[6]: https://www-acc.gsi.de/wikis/Timing/TimingSystemDocumentsVetar2
[7]: https://www.ohwr.org/
[8]: C. Ugur, E. Bayer, N. Kurz and M. Traxler, A 16 channel high
resolution (<11 ps RMS) time-to-digital converter in a field
programmable gate array, 2012 JINST 7 C02004.
[9]: https://github.com/stefanrauch/bel_projects
Seven Solutions and University of Granada
UGR
The University of Granada has joined the White-Rabbit initiative to improve and extend the development of the configuration & management WR tools. Our target application scenarios are science projects such as the Square Kilometer Array (http://www.skatelescope.org/) or industrial applications focusing on Smart-Grid.
As first contribution we are close to finishing a configuration tool to extend the functionalities of the WR-NIC project. The current project allows a SPEC card to work as a NIC and the PC applications allow controlling the DIO interfaces. We include the Etherbone modules on the gateware and modify interface signals in order to work as stand-alone system. We have developed a simple tool to create generic commands to control DIO channels. It is very flexible software that can be adapted to any other FMC mezzanine cards very easily. This will allow the remote generation and reception of signals from White-Rabbit nodes with just writing a configuration file.
In addition, we participate in the SKA SaDT (Signal and Data Transport) consortium. Our role is to evaluate the White Rabbit timing solution for this telescope, focusing on improving the phase stability of WR and integration on a single network of timing and control/monitoring functionalities.
In the next period we will work on these issues as well as collaborating with CERN/Alessandro for the development of the WR network management tools.
Seven Solutions
White-Rabbit for Smart-Grid: The TIGRIS project.
TIGRIS is an R&D project to optimize the distribution in a electrical network. Seven Solutions will investigate how synchronization and high precision timing can benefit the motorization and thus enhance the capacity of the system without investing in new infrastructure.
Wishbone drivers for EPICS: the ACELTEC project.
Seven Solutions is participating in the ACELTEC project in collaboration with University of Huelva (Spain) and Cibernos.
We are currently developing a generic asynchronous driver for Wishbone peripherals in order to "plug'n'play" the devices into the EPICS platforms.
The 10MHz/PPS Fwd Kit.
This kit consists of two standalone SPEC boards plugged with a FMC DIO for the acquisition of the 10MHz/PPS signal and a FMC DEL to forward the 10MHz/PPS to non-WR equipment. This kit is designed to easily extend GPS clock signals to any part of your facility using fiber.
The SPEC-1N and SPEC-3N19U boxes.
As explained above the UGR and Seven Solutions are improving the standalone capabilities of WR devices. We have therefore created two SPEC boxes to pack the WR node for standalone configuration.
MIKES
1. stability measurement over 1km link from April-May
https://www.ohwr.org/project/white-rabbit/uploads/8ddccb79ff6ccc1563bcf3e18c194bd0/MIKES_wr_stability.png
2. using WR/SPEC with an NTP shared-memory refclock driver to
discipline NTP/system time
https://github.com/aewallin/ptp2ntpd
3. long distance WR (1000km)
example of 10 ms RTT and its daily variation due to temperature
variation:
https://www.ohwr.org/project/white-rabbit/uploads/2ac1fc3ab0a25ea55657821485e799dd/MIKES_kajaani_rtt_graph_2013sep23.png
Anders plotted an example of system time vs WR-time when using the
shared-memory refclock driver. It is now at
https://www.ohwr.org/project/white-rabbit/uploads/6496b4287ffc1a19c792725dc737392c/MIKES_ntp_graph_2013sep30.png
There is some text about it at the bottom of the page:
https://www.ohwr.org/project/white-rabbit/wikis/Mikes
HiSCORE and DESY
- The Siberian Tunka setup is expected to have 9 stations in the field in 5 weeks.
- Results, presented at the Int.CosmicRayConf in summer:
http://www.ifh.de/~wischnew/talks/cta/icrc2013/
WR has seen first light, and has been used to detect and reconstruct first CosmicRay showers. The final aim is to see the much rarer gamma-ray showers.
ESS Bilbao
At ESS Bilbao it is planned to use WR in the whole facility as the main
tool for timing and synchronization. There are two main lines of
interest at ESS Bilbao about WR: RF distribution and Timing
distribution. As first approach we have implemented a laboratory
testbench with two SPECS, a WR Switch and a Master Oscillator. It has
been configured and tested as a standalone system. Moreover some effort
on Etherbone thanks to the WR community has been started. Nowadays, the
system is close to be installed at the ESS Bilbao's Ion Source (ISHP) as
the timing system.
ESS Bilbao is also interested in WR PXI/PXIe cards as a standard to be
used in its equipment. Concerning to the RF distribution, it is under
study, but the FMC DAC 600M 12b 1cha DDS board is very promising.
Creotech
From our side we are at the stage of building a WR-MCH with integrated DDS, CPU and RTM (RTM with 8x WR enabled-SFP). It will be a combo that will WR-enable MTCA.4, improve connectivity speed and reduce the number of occupied slots in a uTCA crate. The CPU and RTM-SFP boards are already designed and produced. We do it in cooperation with LNLS.