Newer
Older
\documentclass[a4paper, 12pt]{article}
\usepackage[english]{babel}
\usepackage{fullpage}
\usepackage{pgf}
\usepackage{tikz}
\usepackage[pdftex]{hyperref} % makes cross references and URLs clickable
\usepackage[overload]{textcase}
\usepackage{listings}
\usepackage{color}
\usepackage{textcomp}
\usepackage{longtable} % table over many pages
\usepackage[document]{ragged2e} %texta djustment
\usepackage{mdwlist} % to have tight itemization
\usepackage[T1]{fontenc}
\usepackage{lmodern}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx}
\usepackage{colortbl}
\usepackage{array}
\usepackage{multirow}
\newcommand{\newparagraph}[1]{\paragraph{#1}\mbox{}\\}
\definecolor{wrlblue}{RGB}{165,195,210}
\definecolor{wrlgray}{RGB}{209,211,212}
\definecolor{light-gray}{gray}{0.95}
\hypersetup{
colorlinks,
linkcolor={red!50!black},
citecolor={blue!50!black},
urlcolor={blue!80!black}
}
% set listings as in other WR-doc(s)
\lstset{columns=flexible, upquote=true, frame=single,
basicstyle=\footnotesize\ttfamily, backgroundcolor=\color{light-gray}, label=lst:init_src}
\newcommand{\multirowpar}[2]{
\multirow{#1}{\hsize}{\parbox{\hsize}{\strut\raggedright#2\strut}}
}
\newcommand{\hdltablesection}[1]{
\multicolumn{4}{|c|}{\bf\small#1}
}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{M}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}\ttsmall}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{D}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}\ttsmall}m{#1}}
\let\underscore\_
\renewcommand{\_}{\underscore\allowbreak}
\newenvironment{hdlparamtable}{
\setlength{\extrarowheight}{1pt}
\begin{center}
\begin{longtable}{|M{.2\textwidth}|C{.09\textwidth}|D{.11\textwidth}|L{.5\textwidth}|}
\firsthline
\rowcolor{wrlblue}
\bf{name} & \bf{type} & \bf{default} & \bf{description}\\
\hline
\endhead
}{
\lasthline
\end{longtable}
\end{center}
}
\newenvironment{hdlporttable}{
\setlength{\extrarowheight}{1pt}
\begin{center}
\begin{longtable}{|M{.25\textwidth}|C{.05\textwidth}|D{.05\textwidth}|L{.55\textwidth}|}
\firsthline
\rowcolor{wrlblue}
\bf{name} & \bf{dir} & \bf{size} & \bf{description}\\
\hline
\endhead
}{
\lasthline
\end{longtable}
\end{center}
}
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\tts}[1]{
\texttt{\small{#1}}}
% same as \tts{}, without argument
\newcommand{\ttsmall}{\ttfamily\small}
\newcommand{\hrefwrpc}[1]{
\tts{\href{http://www.ohwr.org/projects/wr-cores/repository/entry/#1?rev=\wrpcrelease}{#1}}}
\title{White Rabbit PTP Core User's Manual}
\author{Grzegorz Daniluk \\Dimitrios Lampridis \\Adam Wujek \hfill CERN (BE-CO-HT)}
\begin{document}
\makeatletter
\hypersetup{pdftitle={\@title},pdfauthor={\@author}}
\raggedright
{\LARGE\bf\@title}\\[0.2 cm]
\hrule height 4pt \vspace{0.1cm}
{\large\gitrevinfo\hfill\today}\\
\vspace*{\fill}
{\large\@author}\\
\hrule height 2pt
\justify
\makeatother
\newpage
\tableofcontents
\newpage
\section{Introduction}
This is the user manual for the White Rabbit PTP Core (WRPC), part of the White
Rabbit project. It describes the building and running process, and it provides a
guide to instantiating the WRPC in your own HDL design.
If you don't want to get your hands dirty and prefer to start with the demo binaries
available at \url{http://www.ohwr.org/projects/wr-cores/files} for officially
supported boards, please skip Section~\ref{Building the Core} and move forward
directly to Section~\ref{Programming FPGA}. For help with instantiating the WRPC in
your own HDL design, see Section~\ref{sec:wrpc_hdl}.
% ##########################################################################
\subsection{Software and hardware requirements}
\label{Software and hardware requirements}
% ==========================================================================
\subsubsection{Repositories and Releases}
This manual is about the official \gitrevinfo{} stable release of the White
Rabbit PTP Core (WRPC). The code and documentation for the project is
distributed in the following places:
\begin{itemize*}
\item \url{http://www.ohwr.org/projects/wr-cores/documents}
hosts the pdf documentation for every official release.
\item \url{http://www.ohwr.org/projects/wr-cores/files}
place where you can find already synthesized demo bitstreams, ready to be
downloaded to one of the officially supported boards
\item \url{git://ohwr.org/hdl-core-lib/wr-cores.git}
repository with HDL sources of the WRPC
\item \url{git://ohwr.org/hdl-core-lib/wr-cores/wrpc-sw.git}
repository with the LM32 software running inside the WRPC
\end{itemize*}
Other tools useful for building and running the WRPC can be downloaded from the
following locations:
\begin{itemize*}
\item \url{git://ohwr.org/misc/hdl-make.git}
\textit{hdlmake} is used in the HDL synthesis process to create a Makefile and
Xilinx ISE / Altera Quartus project file
\item \url{http://www.ohwr.org/attachments/download/1133/lm32.tar.xz}
LM32 toolchain used to compile the LM32 software running inside the WRPC.
This is a 32-bit version of the toolchain. If you encounter problems running
this toolchain on modern 64bit machines, try 64 version described below.
\item \url{http://www.ohwr.org/attachments/download/3868/lm32_host_64bit.tar.xz}
LM32 toolchain used to compile the LM32 software running inside the WRPC
(64-bit version of the toolchain).
Repositories containing the WRPC gateware and software (\textit{wr-cores},
\textit{wrpc-sw}) are tagged with \texttt{wrpc-v4.0} tag. Other tools used to
build the core and load it into the FPGA should be used in their newest stable
releases, unless otherwise stated.
% ==========================================================================
\subsubsection{Required hardware}
The minimum hardware set required to run the WR PTP Core reference firmware
depends on the hardware platform you want to use. One of the following setups
can be chosen:
\begin{itemize}
\item \href{http://www.ohwr.org/projects/spec}{SPEC PCIe board} +
\href{http://www.ohwr.org/projects/fmc-dio-5chttla}{FMC DIO card} + PC
computer running Linux
\item \href{http://www.ohwr.org/projects/svec}{SVEC VME board} + VME crate with
a single board computer running Linux\footnote{\label{note_a20}In our test setup
we used MEN A20 board}
\item \href{http://www.ohwr.org/projects/vfc-hd}{VFC-HD VME board} +
\href{http://www.ohwr.org/projects/fmc-dio-5chttla}{FMC DIO card} +
VME crate with a single board computer running Linux\footref{note_a20}
\end{itemize}
To be able to test White Rabbit synchronization you would also need
additional components regardless of the reference platform chosen from the list
above:
\begin{itemize}
\item another WR node (e.g. one of the reference boards listed above) or a
White Rabbit Switch
\item a pair of WR-supported SFP transceivers\footnote{The list of supported
SFPs can be found on our wiki page
\url{http://www.ohwr.org/projects/white-rabbit/wiki/SFP}}
\item a roll of G652, single mode fiber to connect your WR devices
\end{itemize}
% ##########################################################################
\newpage
\section{Building the Core}
\textbf{Note:} you can skip this chapter if you want to use the release binaries
available from \textit{ohwr.org}.
\vspace{1em}
Depending on your needs, building the WRPC can be a one- or two-step process.
In most of the cases you only need to synthesize the FPGA firmware (section
\ref{HDL synthesis}). This way, you get a working WRPC with the default/release
LM32 software running inside the core. If, for some reasons, you need to modify
the LM32 software, please check also section \ref{LM32 software compilation}
which contains a description of the software compilation process.
% ==========================================================================
\subsection{HDL synthesis}
Before running the synthesis process you have to make sure your environment is
set up correctly. You will need a synthesis software from your FPGA vendor.
Depending on whether you want to run the WRPC on Xilinx (e.g. SPEC, SVEC boards) or
Altera/Intel (e.g. VFC-HD) FPGA, you should install either Xilinx ISE or Quartus
Prime software.
\subsubsection{Setting up Xilinx ISE on Linux}
To synthesize the FPGA firmware containing the WRPC, Xilinx ISE with free of
charge WebPack license is enough. ISE provides a set of scripts:
\texttt{settings32.sh}, \texttt{settings32.csh}, \texttt{settings64.sh} and
\texttt{settings64.csh} that configure all the system variables to let you
easily run the software. Depending on the shell you use and whether your Linux is
32 or 64-bits you should execute one of them before the other tools are used.
For 64-bit system and BASH shell you should call (assuming that ISE is installed
in the default \textit{/opt} directory):
\begin{lstlisting}
$ /opt/Xilinx/<version>/ISE_DS/settings64.sh
\end{lstlisting}
You can check if the shell is configured correctly by verifying if the
\texttt{\$XILINX} variable contains path to your ISE installation directory.
\subsubsection{Setting up Quartus Prime on Linux}
To be able to synthesize the WRPC for Arria V FPGA (which is used on the VFC-HD
board) you need at least a license for the Quartus Prime Standard Edition with
the support of Arria V family. To set up the software after it is installed, you
should add the location of its binaries to your \texttt{\$PATH} environment
variable. Assuming you have installed the software in \textit{/opt/altera}, the
following command should be executed:
\begin{lstlisting}
$ export PATH=/opt/altera/16.0/quartus/bin:$PATH
\end{lstlisting}
\subsubsection{Downloading the sources and running the synthesis}
Thanks to the \textit{hdlmake} tool, the synthesis process for the reference
designs does not differ between Xilinx and Altera/Intel based boards. The tool creates
synthesis Makefile as well as ISE/Quartus project file based on a set of
Manifest.py files that you will find in the \textit{wr-cores} repository.\\
First, please clone the \textit{hdlmake} repository from its location given in
section \ref{Repositories and Releases}:
\begin{lstlisting}
$ git clone git://ohwr.org/misc/hdl-make.git <your_location>/hdl-make
$ cd <your_location>/hdl-make
\end{lstlisting}
This checks out \textit{hdlmake} version 2.1 patched with the Arria V FPGA
support (VFC-HD board).
Then, using your favorite editor, you should create an \textit{hdlmake} script in
/usr/bin to be able to call it from any directory. The script should have the
following content:
\begin{lstlisting}
#!/usr/bin/env bash
python2.7 <path_to_hdlmake_sources>/hdl-make/hdlmake/__main__.py #@
\end{lstlisting}
Please, make your script executable:
\begin{lstlisting}
$ chmod a+x /usr/bin/hdlmake
\end{lstlisting}
Having all the tools in place, you can now clone the main WR PTP Core git
repository for the v4.0 release. The set of commands below clones the WR PTP Core
repository, checks out the release tag, and downloads other HDL repositories
(submodules) needed to synthesize the core:
\begin{lstlisting}
$ git clone git://ohwr.org/hdl-core-lib/wr-cores.git <your_location>/wr-cores
$ cd <your_location>/wr-cores
$ git submodule init
$ git submodule update
\end{lstlisting}
The local copies of the submodules are stored to:
\texttt{<your\_location>/wr-cores/ip\_cores}
\vspace{1em}
\textbf{Note:} If you use the WRPC within another project (like
\textit{wr-nic}), you may need to checkout another release tag for this
repository. Please refer to the project's documentation to find out which
version of this package you need to build.
\vspace{1em}
The subdirectory you should enter to run the synthesis depends on the hardware
platform you use:
\begin{itemize*}
\item \textbf{SPEC}: \texttt{<your\_location>/wr-cores/syn/spec\_ref\_design}
\item \textbf{SVEC}: \texttt{<your\_location>/wr-cores/syn/svec\_ref\_design}
\item \textbf{VFC-HD}: \texttt{<your\_location>/wr-cores/syn/vfchd\_ref\_design}
\end{itemize*}
After selecting a proper location from the list above, please call
\textit{hdlmake} without any arguments to create the Makefile and project file:
\begin{lstlisting}
$ hdlmake
\end{lstlisting}
After that, the actual synthesis is just the matter of executing:
\begin{lstlisting}
$ make
\end{lstlisting}
This takes (depending on your computer) about 10 minutes and should generate
bitstream files in various formats depending on your selected reference
hardware:
\begin{itemize*}
\item \textbf{SPEC}: \texttt{spec\_wr\_ref\_top.bin}, \texttt{spec\_wr\_ref\_top.bit}
\item \textbf{SVEC}: \texttt{svec\_wr\_ref\_top.bin}, \texttt{svec\_wr\_ref\_top.bit}
\item \textbf{VFC-HD}: \texttt{vfchd\_wr\_ref.sof}
\end{itemize*}
You can select the bitstream format to be downloaded to FPGA depending on the
programming method:
\begin{itemize}
\item \textbf{*.bin} files to program the Xilinx FPGA on SPEC or SVEC board
using the official software support package (\textit{spec-sw},
\textit{svec-sw}). See section \ref{Programming FPGA} for more
information.
\item \textbf{*.bit} files to program the Xilinx FPGA with Xilinx USB Platform
Cable (using e.g. Xilinx Impact tool)
\item \textbf{*.sof} file to program the Intel FPGA (VFC-HD board) using using
the Altera / Intel JTAG cable
\end{itemize}
If, you would like to clean-up the repository to start building everything from
scratch you can use the following commands:
\begin{itemize*}
\item \texttt{\$ make clean} - removes all synthesis reports and log files;
\item \texttt{\$ make mrproper} - removes \texttt{*.bin}, \texttt{*.bit} and
\texttt{*.sof} files;
\end{itemize*}
% ==========================================================================
\subsection{LM32 software compilation}
\textbf{Note:} By default, the LM32 software for a stable release is embedded
inside the FPGA bitstream you've downloaded from \textit{ohwr.org} or
synthesized in section \ref{HDL synthesis}. You can skip this section, unless
you need to make some custom changes to the LM32 software and compile it
yourself.\\
First, you need to download and unpack the LM32 toolchain from the location
mentioned in section \ref{Repositories and Releases}. The following example
uses 32bit version of a toolchain. If you encounter problems running it, please
use 64bit version.
\begin{lstlisting}
$ wget http://www.ohwr.org/attachments/download/1133/lm32.tar.xz
$ tar xJf lm32.tar.xz -C <your_location>
\end{lstlisting}
Then you need to set a \texttt{CROSS\_COMPILE} environment variable in order
to compile the software for the LM32 processor:
\begin{lstlisting}
$ export CROSS_COMPILE="<your_location>/lm32/bin/lm32-elf-"
\end{lstlisting}
To get the sources of the WRPC software, please clone the \textit{wrpc-sw} git
repository tagged with \texttt{wrpc-v4.0} tag. The commands in the listing below
clone the \textit{wrpc-sw} repository together with submodules needed for this software.\\
\begin{lstlisting}
$ git clone git://ohwr.org/hdl-core-lib/wr-cores/wrpc-sw.git \
<your_location>/wrpc-sw
$ cd <your_location>/wrpc-sw
\end{lstlisting}
\textbf{Note:} If you use WRPC within another project, you may need to checkout
a different tag or a specific commit. If this applies, please refer to a
documentation for this project.\\
Before you can compile \textit{wrpc-sw}, you can make some configuration
choices. The package uses \textit{Kconfig} as a configuration engine, so you may
run one of the following commands (the first two are text-mode, the third uses
a KDE GUI and the fourth uses a Gnome GUI):
\begin{lstlisting}
$ make menuconfig
$ make xconfig
$ make gconfig
\end{lstlisting}
Other \code{Kconfig} target applies, like \code{config}, \code{oldconfig}
and so on. A few default known-good configurations are found in
\texttt{./configs} and you choose one by \code{make}ing it by name. For all
three boards mentioned in this manual \code{spec\_defconfig} can be used.
\begin{lstlisting}
$ make spec_defconfig
\end{lstlisting}
After the package is configured, just run \code{make} without parameters to
build your binary file:
\begin{lstlisting}
$ make
\end{lstlisting}
The first time you build, the \textit{Makefile} automatically downloads
the \textit{git submodules} of this package, unless you already did that
by hand. The second and later build won't download anything
from the network.
The compilation process produces the binary in 3 different formats:
\begin{itemize*}
\item \textit{wrc.bin} can be then used with the loader from \textit{spec-sw}
or \textit{svec-sw} software package to program the LM32 inside the White
Rabbit PTP Core (see section \ref{Programming FPGA} for details).
\item \textit{wrc.bram} you can use to initialize WRPC internal RAM with LM32
software during the synthesis for Xilinx FPGAs.
\item \textit{wrc.mif} you can use to initialize WRPC internal RAM with LM32
software during the synthesis for Altera FPGAs.
\end{itemize*}
The location of \textit{wrc.bram}/\textit{wrc.mif} files should be passed to the
WR PTP Core HDL using the \texttt{g\_dpram\_initf} generic.
% ##########################################################################
\section{Programming FPGA with WRPC firmware}
\label{Programming FPGA}
% ==========================================================================
\subsection{Programming FPGA on SPEC}
\label{Programming FPGA on SPEC}
\textbf{Note:} If you use a more recent version of the \texttt{spec-sw} than the
one described in this manual, the up-to-date documentation can always be found
in \textit{doc/} subdirectory of the \texttt{spec-sw} git repository.\\
First you need to clone the SPEC board software support package
(\textit{spec-sw}) from \textit{ohwr.org}. It is a set of Linux kernel drivers and
userspace tools, that interacts with a SPEC board plugged into a PCI-Express
\begin{lstlisting}
$ git clone git://ohwr.org/fmc-projects/spec/spec-sw.git <your_location>/spec-sw
$ cd <your_location>/spec-sw
$ git checkout aff79eb
$ make
\end{lstlisting}
You have to download also the "golden" firmware for SPEC card. It is used by
the drivers to recognize the hardware:
\begin{lstlisting}
$ wget http://www.ohwr.org/attachments/download/4057/spec-init.bin-2015-09-18
$ sudo mv spec-init.bin-2015-09-18 /lib/firmware/fmc/spec-init.bin
\end{lstlisting}
Now you can load the drivers necessary to access the SPEC board from your
system:
\begin{lstlisting}
$ sudo insmod fmc-bus/kernel/fmc.ko
$ sudo insmod kernel/spec.ko
\end{lstlisting}
You can use the \textit{dmesg} Linux command to verify if this step succeeded.
Among plenty of messages you should be able to find something very similar to:
\begin{lstlisting}[basicstyle=\scriptsize\ttfamily]
[1275526.738895] spec 0000:20:00.0: probe for device 0020:0000
[1275526.738906] spec 0000:20:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[1275526.738913] spec 0000:20:00.0: setting latency timer to 64
[1275526.743102] spec 0000:20:00.0: got file "fmc/spec-init.bin", 1485236 (0x16a9b4) bytes
[1275526.934710] spec 0000:20:00.0: FPGA programming successful
[1275527.296754] spec 0000:20:00.0: mezzanine 0
[1275527.296756] Manufacturer: CERN
[1275527.296757] Product name: FmcDio5cha
\end{lstlisting}
By default, when loading the \textit{spec.ko} driver, FPGA gets programmed with
the "golden" bitstream. Starting from version 3.0, WR PTP Core uses a flash
memory chip on the carrier as a default place for storing the calibration
parameters and the init script. Also the storage format of this information is
now better organized in the files of the SDBFS filesystem. Therefore, starting
from v3.0 you have to write the SDBFS filesystem image to the flash
before running the WRPC. You can download the image from \textit{ohwr.org}:
\begin{lstlisting}
$ wget http://www.ohwr.org/attachments/download/4060/sdbfs-flash.bin
\end{lstlisting}
It contains all the files required by the core. They are empty, but have to
exist in the SDBFS structure to be written later as described in section
\ref{Writing configuration}. To write the filesystem image to flash, please
execute the following command:
\begin{lstlisting}
$ sudo tools/flash-write -c 0x0 0 1507712 < <your_location>/sdbfs-flash.bin
\end{lstlisting}
\noindent\textbf{Note:} If you have more than one SPEC board in your computer,
you can use \code{-b} parameter which takes the PCI bus address of the card you
want to program. This can be found in the list of the PCI devices installed in
the system (\code{lspci}).
\noindent\textbf{Note:} Please refer to section \ref{Writing SDBFS image in
standalone configuration} for instructions on how to write the SDBFS image to a
standalone SPEC or custom hardware.\\
Now, you are ready to program the FPGA with your synthesized bitstream from
\texttt{<your\_location>/wr-cores/syn/spec\_ref\_design/spec\_wr\_ref\_top.bin}
\begin{lstlisting}
$ sudo tools/spec-fwloader \
<your_location>/wr-cores/syn/spec_ref_design/spec_wr_ref_top.bin
\end{lstlisting}
If everything went right up to this moment you have your board running the FPGA
bitstream with the WR PTP Core. If you need to load your own \texttt{wrc.bin}
compiled in section \ref{LM32 software compilation}, you can use the
\texttt{spec-cl} tool:
\begin{lstlisting}
$ sudo tools/spec-cl <your_location>/wrpc-sw/wrc.bin
\end{lstlisting}
After all these steps, you should be able to start a Virtual-UART tool (also
part of the \textit{spec-sw} package) that will be used to interact with the
WRPC shell:
\begin{lstlisting}
$ sudo tools/spec-vuart
\end{lstlisting}
If you are able to see the shell prompt \textit{wrc\#} this means the Core
is up and running on your SPEC card. Congratulations !
% ==========================================================================
\subsection{Programming FPGA on other boards}
Currently the driver packages for SVEC and VFC-HD boards depend on other CERN
drivers and front-end infrastructure. We are working to make them exportable. If
you are a CERN user of SVEC or VFC-HD board, please check
\url{https://wikis.cern.ch/display/HT/WR+Nodes} (accessible \textbf{only} from
CERN network) for more instructions. If you are outside CERN and would like to
use one of these boards, please contact us.
% ==========================================================================
\subsection{Writing configuration}
First, you should perform a few configuration steps through the WRPC shell
before using the core.
\noindent\textbf{Note:} the examples below describe only a subset of the WRPC
Shell commands. The full list of supported commands can be found in Appendix
Before making any configuration changes, it is recommended to stop the
PTP daemon. Then, the messages from the daemon will not show up to the
console while you interact with the shell.
\begin{lstlisting}
wrc# ptp stop
\end{lstlisting}
\noindent First you should make sure your board has a proper MAC address
assigned:
\begin{lstlisting}
wrc# mac get
\end{lstlisting}
If the result of above command is \texttt{MAC-address: 22:33:ww:xx:yy:zz} (where
ww, xx, yy, zz are hex values), this
means MAC was not yet configured and stored in the Flash/EEPROM. The value is
based on thermometer serial number as is unique among SPEC devices. This is
globally accepted as ``locally assigned'', but you might want to assign your own
address. A value \texttt{22:33:44:55:66:77} is the final fallback if no
thermometer is found.
You should get the MAC for your board from its manufacturer. To configure the
address and store it into the Flash/EEPROM (so that it's automatically loaded
every time the WRPC starts) you should type two commands in the shell:
\begin{lstlisting}
wrc# mac set xx:xx:xx:xx:xx:xx
wrc# mac setp xx:xx:xx:xx:xx:xx
\end{lstlisting}
where \texttt{xx:xx:xx:xx:xx:xx} is the MAC address of your board.\\
Next, you should input calibration fixed delays values and alpha parameters. The
example below clears any existing entries and adds two Axcen transceivers with
$\Delta_{TX}$, $\Delta_{RX}$ and $\alpha$ parameters associated with them.
\begin{lstlisting}
wrc# sfp erase
wrc# sfp add AXGE-1254-0531 180625 148451 72169888
wrc# sfp add AXGE-3454-0531 180625 148451 -73685416
\end{lstlisting}
To check the content of the SFP database you can execute the \textit{sfp show}
shell command.\\
\noindent\textbf{Note:} The $\Delta_{TX}$ and $\Delta_{RX}$ parameters above are
the defaults for wrpc-v4.0 release bitstream available on \textit{ohwr.org},
running on the SPEC v4 board and calibrated to port 1 of a~WR Switch
v3.3. These values as well as the default parameters for other boards are
available on the calibration wiki page
(\url{http://www.ohwr.org/projects/white-rabbit/wiki/Calibration}). If you
re-synthesize the firmware or want to use the most accurate estimation of
the fixed delays and alpha for your fiber, you should read and perform the WR
Calibration procedure (\url{http://www.ohwr.org/documents/213}).\\
The WRPC mode of operation (GrandMaster/Master/Slave) can be set
using the \textit{mode} command:
\begin{lstlisting}
wrc# mode gm # for GrandMaster mode
wrc# mode master # for Master mode
wrc# mode slave # for Slave mode
\end{lstlisting}
This stops the PTP daemon, changes the mode of operation, but does not start it
automatically. Therefore, after calling it, you need to start the daemon
manually:
\begin{lstlisting}
wrc# ptp start
\end{lstlisting}
\noindent\textbf{Note:} For running the GrandMaster mode, you need to have the
\texttt{g\_with\_external\_clock\_input} generic enabled in your FPGA firmware.
You also must provide 1-PPS and 10MHz signal from an external source (e.g. GPS receiver or
Cesium clock). Depending on your board you should connect:
\begin{itemize*}
\item \textbf{SPEC}: DIO mezzanine LEMO No.4 for 1-PPS, LEMO No.5 for 10MHz
\item \textbf{SVEC}: SVEC LEMO No.4 for 1-PPS, LEMO No.3 for 10MHz
\item \textbf{VFC-HD}: DIO mezzanine LEMO No.4 for 1-PPS, LEMO No.5 for 10MHz
\end{itemize*}
\vspace{1em}
One option is to type by hand all the commands needed to initialize the WRPC software to
the required state every time the core starts. However, you can also write your
own initialization script and save it to the Flash/EEPROM. It will be executed every time
the WRPC software starts. A~simple script, that loads the calibration
parameters, configures the WR mode to Slave, starts the PTP daemon and sets
\begin{lstlisting}
wrc# init erase
wrc# init add ptp stop
wrc# init add sfp match
wrc# init add mode slave
wrc# init add ptp start
\end{lstlisting}
Almost exactly the same one can be used for running WRPC in the GrandMaster
or Master mode. The only difference would be changing the
\texttt{init add mode slave} line to \texttt{init add mode gm} or
\texttt{init add mode master}.
% ==========================================================================
\subsection{Running the Core}
Having the SFP calibration database, and the init script created in \ref{Writing
configuration} you can now restart the WRPC by typing the shell command:
\begin{lstlisting}
wrc# init boot
\end{lstlisting}
You should see log messages that confirm the execution of the initialization
script:
\begin{lstlisting}
executing: ptp stop
PTP stop
executing: sfp match
AXGE-3454-0531
SFP matched, dTx=180707 dRx=148323 alpha=-73685416
executing: mode slave
PTP stop
Locking PLL
executing: ptp start
PTP start
Slave Only, clock class set to 255
executing: ip set 192.168.1.5
IP-address: 192.168.1.5 (static assignment)
\end{lstlisting}
Now you should have the WR PTP Core running in WR Slave mode.\\
\textbf{Important!!!} WRPC needs to make a calibration of t24p phase transition
value. It has to be done only once for a new bitstream and is performed
automatically when WRPC runs in the Slave mode. That is why it is very
important, even if WRPC is meant to run in the Master mode, to configure it to
Slave for a moment and connect to any WR Master. This has to be repeated every
time a new bitstream (gateware) is deployed. The measured value is automatically
stored to Flash/EEPROM and used later in the Master or GrandMaster mode.\\
The Shell also contains a monitoring function which you can use to check the
\begin{lstlisting}
wrc# gui
\end{lstlisting}
The information is presented in a clear, auto-refreshing screen (see the figure blow).
The information is refreshed at every WR iteration or periodically if
nothing else happens (so you see an up-to-date timestamp). The period
defaults to 1 second and can be changed using the \texttt{refresh} command. To
exit from this console mode press <Esc>. A full description of the information
reporter by \textit{gui} is provided in Appendix \ref{WRPC GUI elements}.
\noindent\textbf{Note:} the \textit{Synchronization status} and \textit{Timing
parameters} in \texttt{gui} are available only in the WR Slave mode. When
running as WR Master, you can see only the current date and time,
link status, Tx and Rx packet counters, IP, lock and calibration status.
\vspace{1em}
\includegraphics[width=12cm]{wrpc_mon.png}
\vspace{1em}
If you want to log statistics from the WRPC operation, it is better to use a
\texttt{stat} shell command. It reports the same information as \texttt{gui},
but in a single line, which is easier to parse and analyze:
\begin{lstlisting}
wrc# stat
lnk:1 rx:172338 tx:151811 lock:1 ptp:slave sv:1 ss:'TRACK_PHASE' aux0:1 sec:6047 \
nsec:828412744 mu:836453 dms:398530 dtxm:224455 drxm:232479 dtxs:180625 drxs:149251 \
asym:39393 crtt:49643 cko:1 setp:5082 ucnt:270 hd:31734 md:46228 ad:0 temp: 52.6875 C
lnk:1 rx:172392 tx:151860 lock:1 ptp:slave sv:1 ss:'TRACK_PHASE' aux0:1 sec:6049 \
nsec:399776360 mu:836452 dms:398530 dtxm:224455 drxm:232479 dtxs:180625 drxs:149251 \
asym:39392 crtt:49642 cko:2 setp:5082 ucnt:271 hd:31730 md:46211 ad:0 temp: 52.6875 C
(...)
\end{lstlisting}
\vspace{1em}
Unlike \texttt{gui}, the \texttt{stat} command runs asynchronously: you can still
issue shell commands while stats are running. You can stop statistics by running
\texttt{stat} again. As an alternative to the toggling action of \texttt{stat}
alone, you can use ``\texttt{stat on}'' or ``\texttt{stat off}''.
Statistics are printed every time the WR servo runs; thus no statistics
are reported when the node is running in master and GrandMaster mode, nor when your node
is running as slave and the master disappeared.\\
You can verify the synchronization performance by observing the offset between
1-PPS signals from the WR Master and your WRPC running in the Slave mode. Please
remember to use oscilloscope cables of the same length and type (with the same
delay), or take their delay difference into account in your measurements.
Depending on your board, 1-PPS output is produced on:
\begin{itemize*}
\item \textbf{SPEC}: DIO mezzanine LEMO No.1
\item \textbf{SVEC}: SVEC LEMO No.1
\item \textbf{VFC-HD}: VFC-HD LEMO L3
% ==========================================================================
\newpage
\section{WRPC diagnostics}
\subsection{SNMP}
Up to the version 4.0 of WRPC the only way to perform diagnostics
of the \texttt{wrpc-sw} was to use serial console with tools like \textit{gui}, \textit{stat},
etc. For some set-ups, like standalone node, it is very inconvenient to use
external console for diagnostics.
Starting with version 4.0 of WRPC, it is possible to include the \textit{Mini
SNMP responder}, which allows to perform remote diagnostics using \textit{SNMP} via
a port connected to the \textit{Write Rabbit} network.
The configuration file of WRPC contains the following
SNMP-related options:
\begin{itemize*}
\item \texttt{CONFIG\_SNMP} -- include the \textit{Mini SNMP responder} into WRPC
\item \texttt{CONFIG\_SNMP\_SET} -- enable the support of SNMP \textit{SET} packets
\item \texttt{CONFIG\_SNMP\_VERBOSE} -- enable verbose output from the \textit{Mini SNMP
responder} on the WRPC's console
\end{itemize*}
The MIB file describing WRPC's OIDs can be found in the \texttt{lib} directory
of the \texttt{wrpc-sw} repo.
So far, the \textit{Mini SNMP responder} supports version 1 and a subset of version
2c of the SNMP protocol.
The following types of requests are supported:
\begin{itemize*}
\item GET -- get value of a given OID
\item GETNEXT -- get value of a next OID after the given OID (this is used
for \texttt{snmpwalk}s)
\item SET -- change the value of a given OID (so far used only for adding
SFP's to the database and PTP restarts)
\end{itemize*}
The \textit{Mini SNMP responder} does not support:
\begin{itemize*}
\item bulk requests packets (GETBULK)
\item more than one OID in the request packet
\item \texttt{trap} and \texttt{inform} packets
\item encryption
\item authentication
\item SNMPv2c return error types; all returned error types follows SNMPv1
\end{itemize*}
To make examples more readable, listings below use \texttt{SNMP\_OPT} environment
variable. Make sure you set it properly in your shell.
\begin{lstlisting}
$ SNMP_OPT="-c public -v 2c -m WR-WRPC-MIB -M +/var/lib/mibs/ietf:lib 192.168.1.20"
\end{lstlisting}
where:
\begin{sloppypar} % to prevent \texttt{} from going to the margine
\begin{itemize*}
\item \texttt{-c public} -- sets SNMP community as "\textit{public}"
\item \texttt{-v 2c} -- specifies SNMP version
\item \texttt{-m WR-WRPC-MIB} -- specifies MIBs to be loaded
\item \texttt{-M +/var/lib/mibs/ietf:lib} -- contains path to MIBs in the host
system (\texttt{/var/lib/mibs/ietf}) and path to \texttt{WR-WRPC-MIB} (\texttt{lib});
on Debian-like systems default MIBs can be downloaded using
\texttt{download-mibs} command (package \texttt{snmp-mibs-downloader}); on
CentOS and RedHat MIBs are included in the \texttt{libsmi} package
\item \texttt{192.168.1.20} -- the IP address of the target board
\end{itemize*}\end{sloppypar}\noindent
For example, to get the system uptime please execute the \texttt{snmpget} command:
\begin{lstlisting}
$ snmpget $SNMP_OPT wrpcTimeSystemUptime.0
\end{lstlisting}
To get a dump of all available OIDs please execute the \texttt{snmpwalk}
command:
\begin{lstlisting}
$ snmpwalk $SNMP_OPT wrpcCore
\end{lstlisting}
Part of the \texttt{snmpwalk}'s output:
\begin{lstlisting}
WR-WRPC-MIB::wrpcVersionHwType.0 = STRING: spec
WR-WRPC-MIB::wrpcVersionSwVersion.0 = STRING: wrpc-v3.0-251-g14e952e
WR-WRPC-MIB::wrpcVersionSwBuildBy.0 = STRING: Adam Wujek
WR-WRPC-MIB::wrpcVersionSwBuildDate.0 = STRING: Jun 7 2016 18:12:24
WR-WRPC-MIB::wrpcTimeTAI.0 = Counter64: 1465375022
WR-WRPC-MIB::wrpcTimeTAIString.0 = STRING: 2016-06-08-08:37:02
WR-WRPC-MIB::wrpcTimeSystemUptime.0 = Timeticks: (18186) 0:03:01.86
WR-WRPC-MIB::wrpcTemperatureName.1 = STRING: pcb
WR-WRPC-MIB::wrpcTemperatureValue.1 = STRING: 38.5625
WR-WRPC-MIB::wrpcSpllMode.0 = INTEGER: slave(3)
WR-WRPC-MIB::wrpcSpllIrqCnt.0 = Counter32: 1259605
[...]
WR-WRPC-MIB::wrpcPortSfpInDB.0 = INTEGER: inDataBase(2)
WR-WRPC-MIB::wrpcPortInternalTx.0 = Counter32: 452
WR-WRPC-MIB::wrpcPortInternalRx.0 = Counter32: 869
WR-WRPC-MIB::wrpcSfpPn.1 = STRING: AXGE-1254-0531
WR-WRPC-MIB::wrpcSfpDeltaTx.1 = INTEGER: 180625
WR-WRPC-MIB::wrpcSfpDeltaRx.1 = INTEGER: 148451
WR-WRPC-MIB::wrpcSfpAlpha.1 = INTEGER: 72169888
End of MIB
\end{lstlisting}
It is recommended to use SNMP v2c for communication with a WRPC.
Please note that when the version 1 of SNMP is used, 64 bit counters are not
supported. This makes impossible to read some WRPC's objects with
SNMPv1.
% --------------------------------------------------------------------------
\subsubsection{Managing SFP database via SNMP}
\label{Managing SFP database via SNMP}
The SFPs database can be displayed using the \texttt{sfp show} command from
\begin{lstlisting}
wrc# sfp show
1: PN:AXGE-1254-0531 dTx: 180625 dRx: 148451 alpha: 72169888
2: PN:AXGE-3454-0531 dTx: 180625 dRx: 148451 alpha: -73685416
\end{lstlisting}
The same data is exported by the \textit{Mini SNMP responder} via the table
\texttt{wrpcSfpTable}:
\begin{lstlisting}
$ snmpwalk $SNMP_OPT wrpcSfpTable
WR-WRPC-MIB::wrpcSfpPn.1 = STRING: AXGE-1254-0531
WR-WRPC-MIB::wrpcSfpPn.2 = STRING: AXGE-3454-0531
WR-WRPC-MIB::wrpcSfpDeltaTx.1 = INTEGER: 180625
WR-WRPC-MIB::wrpcSfpDeltaTx.2 = INTEGER: 180625
WR-WRPC-MIB::wrpcSfpDeltaRx.1 = INTEGER: 148451
WR-WRPC-MIB::wrpcSfpDeltaRx.2 = INTEGER: 148451
WR-WRPC-MIB::wrpcSfpAlpha.1 = INTEGER: 72169888
WR-WRPC-MIB::wrpcSfpAlpha.2 = INTEGER: -73685416
End of MIB
\end{lstlisting}
When the SET support is compiled into the \textit{Mini SNMP responder}, it is
possible to erase or add/replace SFP entires to the SFPs database via SNMP.\\
\begin{sloppypar} % to prevent \texttt{} from going to the margine
Addition (or modification) of one SFP to the database can be done by a row of
SNMP SETs. First, please set the delta Tx (\texttt{wrpcPtpConfigDeltaTx.0}), the
delta Rx (\texttt{wrpcPtpConfigDeltaRx.0}) and the alpha (\texttt{wrpcPtpConfigAlpha.0})
with new values. Then, to commit the change to the SFP database, perform the SNMP SET on
the \texttt{wrpcPtpConfigApply.0} with the value \texttt{writeToFlashCurrentSfp}. It will
add/update values for the currently plugged SFP.
\end{sloppypar}
To add or update entries for other SFPs, you shoud set deltas and alpha like
above, set PN of an SFP to the \texttt{wrpcPtpConfigSfpPn.0} and commit
the change by setting \texttt{writeToFlashGivenSfp} to the
\texttt{wrpcPtpConfigApply.0}.\\
It is also possible to update parameters of the currently used SFP without
storing them to the Flash/EEPROM. For that, please set delta Tx, delta Rx and
alpha as described above, then set \texttt{writeToMemoryCurrentSfp} to the
\texttt{wrpcPtpConfigApply.0}. Please remember that these changes are made only
in RAM and will be lost after a power cycle of a board, soft reset of WRPC or
unplug/plug of a fiber/SFP.\\
If a database entry or values in RAM of the currently used SFP are updated, it is
necessary to perform a restart of the PTP daemon
(set \texttt{wrpcPtpConfigRestart.0} with the value \texttt{restartPtp}). Such
restart is necessary because currently PTP does not support on-the-fly changes
of deltas nor alpha. It is expected that this behavior will change in the
future.\\
Each SNMP SET of \texttt{wrpcPtpConfigApply.0} or \texttt{wrpcPtpConfigRestart.0} returns
the status of a performed action. For details please check \texttt{WR-WRPC-MIB}
file.
Commands below add an SFP with PN as "\texttt{NEW-SFP}", delta Tx "\texttt{1111}",
delta Rx "\texttt{2222}" and alpha "\texttt{3333}".
\begin{lstlisting}
$ snmpset $SNMP_OPT wrpcPtpConfigDeltaTx.0 = 1111
WR-WRPC-MIB::wrpcPtpConfigDeltaTx.0 = INTEGER: 1111
$ snmpset $SNMP_OPT wrpcPtpConfigDeltaRx.0 = 2222
WR-WRPC-MIB::wrpcPtpConfigDeltaRx.0 = INTEGER: 2222
$ snmpset $SNMP_OPT wrpcPtpConfigAlpha.0 = 3333
WR-WRPC-MIB::wrpcPtpConfigAlpha.0 = INTEGER: 3333
$ snmpset $SNMP_OPT wrpcPtpConfigSfpPn.0 = NEW-SFP
WR-WRPC-MIB::wrpcPtpConfigSfpPn.0 = STRING: "NEW-SFP"
$ snmpset $SNMP_OPT wrpcPtpConfigApply.0 = writeToFlashGivenSfp
WR-WRPC-MIB::wrpcPtpConfigApply.0 = INTEGER: applySuccessful(100)
\end{lstlisting}
In case when the SFP database does not contain the currently plugged SFP, the last
\texttt{snmpset} command will return \texttt{applySuccessfulMatchFailed(101)}.
Optionally restart the PTP:
\begin{lstlisting}
$ snmpset $SNMP_OPT wrpcPtpConfigRestart.0 = restartPtp
WR-WRPC-MIB::wrpcPtpConfigRestart.0 = INTEGER: restartPtpSuccessful(100)
\end{lstlisting}
Simple verification of performed actions:
\begin{lstlisting}
wrc# sfp show
1: PN:AXGE-1254-0531 dTx: 180625 dRx: 148451 alpha: 72169888
2: PN:AXGE-3454-0531 dTx: 180625 dRx: 148451 alpha: -73685416
3: PN:NEW-SFP dTx: 1111 dRx: 2222 alpha: 3333
\end{lstlisting}
The same add can also be achieved by performing \texttt{sfp add} command in
\begin{lstlisting}
wrc# sfp add NEW-SFP 1111 2222 3333
Update existing SFP entry
3 SFPs in DB
\end{lstlisting}
Verify the result via SNMP:
\begin{lstlisting}
$ snmpwalk $SNMP_OPT wrpcSfpTable
WR-WRPC-MIB::wrpcSfpPn.1 = STRING: AXGE-1254-0531
WR-WRPC-MIB::wrpcSfpPn.2 = STRING: AXGE-3454-0531