Skip to content

W
White Rabbit Starting Kit
Commit 1d521b2a authored by Benoit Rat's avatar Benoit Rat
Browse files
Options

doc: preliminar update for release v3.0 

- update changelog
- update install
- update tools and remove sfp detect (new wrc-v4.2)
- update repos & submodules
- remove live-usb
- remove automatic write of EEPROM FRU
- remove etherbone in standalone
parent 61302ef2
Hide whitespace changes
Inline Side-by-side
Showing with 189 additions and 487 deletions
doc/wr-starting-kit.md
View file @ 1d521b2a
% WR SPEC Starting Kit
% WR SPEC Starting Kit
% Benoit RAT, Javier Diaz (Seven Solutions) & Miguel Jimenez (UGR)
......@@ -15,7 +15,7 @@ written permission by Seven Solutions.
~~~~~~~
The "WR SPEC Starting Kit" (as defined above) is provided under the terms of GPL v2.0
Copyright (C) 2013 - Seven Solutions
Copyright (C) 2019 - Seven Solutions
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
......@@ -65,7 +65,11 @@ conditions
[Seven Solutions]
2.0 15/05/2014 Benoit Rat\ Updating for v2.0 release
[Seven Solutions]
3.0 10/10/2019 Francesco Colella\ Updating for 3.0 reléase
[Seven Solutions]
------------------------------------------------------------------------
You can also check the [Changelog section](#changelog) for more information.
......@@ -102,7 +106,7 @@ and how to integrate it on your own project.
You can find more information on other components and use cases on:
* Our webpage <http://www.sevensols.com/whiterabbitsolution/>
* Our webpage <http://www.sevensols.com/>
* The official wiki page <http://www.ohwr.org/projects/white-rabbit/wiki>
......@@ -112,7 +116,7 @@ About the Starting Kit
This starting kit uses two nodes, each one composed of a [SPEC] and one
[FMC-DIO] card.
A node makes basic operations such as input timestamping or programmable output pulse generation.
A node makes basic operations such as input time-stamping or programmable output pulse generation.
Additionally, specific software and gateware layers allow to use it as a standard network
interface card implementing the White Rabbit technology functionalities.
Network packages with accurate time-stamping information are generated/timestamped at
......@@ -122,11 +126,12 @@ It is based on different projects:
* [spec-sw]: driver to communicate to the [SPEC] card through PCIe. It
also includes a set of tools to experiments.
* [wr-nic]: gateware that includes the NIC & DIO capabilities.
* [wr-nic]: gateware/software that includes the NIC capabilities.
* [fmc-dio-5chttla]: gateware/software that includes the support for FMC-DIO 5ch TTL
* [wrpc-sw]: white rabbit PTP firmware for the synchronization.
About this document:
About this document
--------------------
This document is intended to be a step by step user friendly tutorial
......@@ -134,11 +139,7 @@ to start with the White Rabbit technology. It includes the description
of some simple experiments to illustrate [WR] capabilities. Some
concepts are deliberately avoided to ease the comprehension of this document.
If you want to know more about these concepts, please access to the
related documents in the [Reference section](#references)
and especially to the following ones:
* [spec-sw.pdf]
* [wrpc.pdf]
related documents in the [Reference section](#references).
The rest of the document provides an explanation about the system setup, software driver, FPGA configuration and
some application examples.
......@@ -153,19 +154,42 @@ the different source projects and how to compile them.
Changelog
-----------------
-----------
In this section, we resume the main changes that happens at each release of the starting-kit
### wr-starting-kit-v3.0
This new release has been focused to update all different modules used by the starting kit but does not provide
specific update for the starting kit.
* Update spec-sw with new VIC core to support kernels (v4.15-v4.18)
* Use new wr-core v4.2 with their update tools (see changelog of wrc-v4.2)
* Separate NIC, FMC-DIO, SPEC-SW into different modules
* Update submodules to work with new ohwr gitlab URL
* Remove etherbone support to avoid conflict in epfilters
A detailed changelog can be found on the wiki page of each submodules:
* [wr-nic-v2.0.1-1](https://www.ohwr.org/project/wr-nic/tags)
* [fmc-bus-v2017-06](https://www.ohwr.org/project/fmc-bus/tags)
* [spec-sw-v2017-10-19](http://www.ohwr.org/projects/spec-sw/tags)
* [coht-vic-v1.2.2-3](https://www.ohwr.org/project/coht-vic/tags)
* [fmc-dio-5chttla-v3.0](https://www.ohwr.org/project/fmc-dio-5chttla/tags)
* [wrpc-v4.2](https://www.ohwr.org/project/wr-cores/wikis/Documents/WR-PTP-Core-v4.2-Files)
* [ppsi-v2016.12-91](https://www.ohwr.org/project/ppsi/tags)
### wr-starting-kit-v2.0
* The driver now support kernel from 2.8.x to 3.5.x
* A valid EEPROM data is now **mandatory** in order to start the kernel
* The gateware is now accesible remotely/standalone through ethernet using etherbone core.
* First DIO channel output is now reserved for outputing PPS
* The gateware is now accessible remotely/standalone through ethernet using etherbone core.
* First DIO channel output is now reserved for outputting PPS
* PPSi has been introduced to improve compatibility with other PTP devices.
* GrandMaster locking has been improved
* FMC bus has been updated to the stable version which is officialy included in linux kernel (>3.5)
* FMC bus has been updated to the stable version which is officially included in linux kernel (>3.5)
* Calibration has been improved using t42p procedure
......@@ -192,21 +216,13 @@ What do you need?
In order to use the white rabbit starting kit and setup the different
experiments you will need:
* An oscillocope with at least 150Mhz bandwitdh (500Mhz is recommanded).
* A PC with at least two `PCIe x4` ports (`x8` & `x16` are also compatible)
* The Operative System **Ubuntu LTS 32bit (Long Term Support)** installed.
* An oscilloscope with at least 150Mhz bandwidth (500Mhz is recommended).
* Two PCs with at least two `PCIe x4` ports (`x8` & `x16` are also compatible)
* The Operative System **Ubuntu LTS 64bit (Long Term Support)** installed.
* Two mini-USB (B) cables (not provided with the kit).
This tutorial has been tested and verified with Ubuntu LTS 12.04 and Ubuntu LTS 14.04,
this mean that standard support will only be given for these releases.
However, the `spec-sw` driver should work with other releases, distributions and architectu
that run kernel `2.8.x` to `3.5.x`.
Finally, [Seven Solutions] also provides a fully setup PC or USB-live images to skip
the instalation process and ease the introduction to the *"White Rabbit World"*.
> ***Note:*** This tutorial follow a configuration with two [SPEC+FMCDIO] boards connected in the two **PCIe x16** interfaces of the same computer. However, you can also use two different computers (if you do not have two **PCIe x4**) and follow this tutorial; you just need to replace all the commands with `wr1` or `0x0300` by the `wr0` and its corresponding bus_id on the second PC. The configuration with two separated PCs is more *"natural"* to understand how to communicate two different nodes using the starting kit,
but it requires more physical space to implement it.
This tutorial has been tested and verified with Ubuntu LTS 16.04.6 (kernel v.4.15.9) and Ubuntu LTS 18.04.2 (kernel v.4.18.0-17), this mean that standard support will only be given for these releases.
However, the `spec-sw` driver should work with other releases, distributions and architectures.
![Configuration with one or two PCs](img/ssk_configs.png)
......@@ -225,7 +241,6 @@ in the package[^standardssk] :
* 1x LC-LC cable (2m)
* 3x LEMO cable (2m)
* 3x LEMO-BNC Adaptor
* 1x Pre-installed live-USB (Ubuntu 14.04)
![The components of the starting kit](img/ssk_components.jpg)
......@@ -241,51 +256,28 @@ Physical setup
(A Virtual UART is also available, but it is safer to use the physical
one).
* Connect the two [SPEC+FMCDIO] boards using the SFPs and the optical fiber cable (LC-LC)
* In the demo we have put the violet SFP on the top SPEC board (wr0)
* ... and the blue SFP below (wr1)
* In the demo we have put the violet SFP on the PC01 SPEC board (wri1)
* ... and the blue SFP on the PC02 (wri1)
* Start Ubuntu LTS.
* Prepare an oscilloscope with at least two input channels to access
the [FMCDIO] outputs.
![Mounting two SPECs in the same PC](img/ssk_inside.jpg)
![Mounting two SPECs in two PC](img/ssk_inside.jpg)
![Connecting SFPs and fiber](img/ssk_sfp.jpg)
![The UART use a mini-USB cable to communicate](img/ssk_usb.jpg)
Pre-Installed Live-USB
========================
In our latest WR package we also provide a Live-USB (Ubuntu 14.04) pendrive with
the drivers & tools pre-installed.
In order to run the Live-USB from the pendrive you need to modify the
BIOS of your PC to enable booting from USB.
In the most recent BIOS you will find a `Boot options menu` by pressing
tipically `F10`, `F11` or `F12` during the startup of the PC.
Otherwise you might need to modify the Boot priority in the BIOS Configuration. To do so restart your computer, and watch for a message telling you which key to press to enter the BIOS setup. It will usually be one of F1, F2, DEL, ESC or F10. Press this key while your computer is booting to edit your BIOS settings. Once you are in the BIOS configuration, look for something similar as `Boot` > `Boot Order`. You should see an entry for `removable drive` or `USB media`. Move this to the top of the list to make the computer attempt to boot from the USB device before booting from the hard disk. If you still having trouble do perform this step please search on the web how to `Boot from USB` using your BIOS manufacturer as keyword in the research.
Once Ubuntu has started you should select your own language, and click
the `Try Ubuntu Live` options, then you can go directly to the
[Setting Master & Slave using White Rabbit Core Section](#setting-master-slave-using-white-rabbit-core)
If you prefer to use the WR Starting Kit with your own PC you need to
follow the installation steps in the [next section](#installation).
Installation
=======================
> Notes: If you are running the Live-USB pendrive you should skip this
section.
Once you have your system running with the boards plugged, you need to:
* Check that the boards has been detected
* Install the tools to build drivers, etc...
* The drivers : `spec-sw`
* The HDL bitstream of [SPEC+FMCDIO]: `wr-nic`
SPEC detection
......@@ -300,13 +292,12 @@ First you should check that the boards has been correctly detected on your PCIe
You should obtain something similar as:
~~~~~{.sh}
05:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
0b:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
01:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
~~~~~~~~~~
Where `05` and `0b` represent the ID of the PCIe slot given by your motherboard.
Where `01` and `0b` represent the ID of the PCIe slot given by your motherboard.
> **Note**: The PCIe slot numbering are normally ordered from top to bottom on the motherboard,
this mean that the board with ID `05` will be above the one with index `0b`.
this mean that the board with ID `01` will be above the one with index `0b`.
Tools
......@@ -319,12 +310,13 @@ Then you need to install all the tools that you will need:
* **linux-source**: Might be useful to compile drivers & kernel modules
* **minicom**: Hyperterminal for linux
* **texinfo, texlive, emacs**: Tools to build documentation
* **lib**: libreadline-dev
You can also run this command[^debian] for minimal setup:
~~~~{.sh}
sudo apt-get install git build-essential linux-headers-$(uname -r) minicom
sudo apt-get install git build-essential libreadline-dev linux-headers-$(uname -r) minicom
~~~~~~~~~
and this one if you also want to generate documentation (not mandatory):
......@@ -337,7 +329,7 @@ sudo apt-get install texinfo emacs texlive pandoc
[^debian]: This sample command is for debian's like distributions.
However similar packages exist for other distributions.
Install the project
Compile & Install the project
-------------------
The first step is to install the driver to communicate with the card using
......@@ -350,25 +342,23 @@ includes the [spec-sw] project.
>:$ cd ~/wr/
## Clone the repository
>:$ git clone git://ohwr.org/white-rabbit/wr-starting-kit.git
>:$ git clone https://ohwr.org/project/wr-starting-kit.git
>:$ cd wr-starting-kit
## Checkout the stable release
>:$ git checkout -b wr-starting-kit-v2.0 wr-starting-kit-v2.0
>:$ git checkout -b wr-starting-kit-v3.0 wr-starting-kit-v3.0
~~~~~~~~~~~~
Then you can get the project submodule by running
Configure your Git username/email
~~~~{.sh}
## Obtain the spec-sw project using submodules
>:$ git submodule init
>:$ git submodule update
## Update submodule of spec-sw
>:$ cd spec-sw
>:$ git submodule init
>:$ git submodule update
>:$ cd -
To set your global username/email configuration:
Open the command line.
Set your username:
git config --global user.name "FIRST_NAME LAST_NAME"
Set your email address:
git config --global user.email "MY_NAME@example.com"
~~~~~~~~~~~~
Or you can try our new Makefile that should perform everything!
......@@ -382,132 +372,6 @@ Or you can try our new Makefile that should perform everything!
The **master** branch might have the latest source but support is only offered for tagged release. The other branches are normally used for development and are not stable.
Structure of the spec-sw project
---------------------------------
Most of the information on how the project is structured can be found in the [spec-sw.pdf] in the `/doc` folder,
however in the following paragraphs we briefly summarize it:
### Folders layout
Each of the following folders contain:
* **doc**: documentation of the project
* **gateware**: downloaded HDL binaries also called gateware.\
(This folder is created while downloading gateware)
* **kernel**: Kernel modules (drivers) to connect the [SPEC] to the PC through PCIe.
* **tools**: Set of tools used for the experiments in the Starting Kit
### Structure of the driver and HDL
The Starting Kit contains 3 different drivers:
* **fmc.ko**: Define a generic FMC-bus[^fmc-bus] used to identify and connect to FMC boards independently
of the carrier being used (e.g. [SPEC], SVEC, ...)
* **spec.ko**: It is the device driver that connects to the [SPEC] through PCI. It also loads the spec-init.bin “golden” gateware file.
* **wr-nic.ko**: The wr-nic driver is basically an Ethernet driver with
support for hardware time stamping. It also loads the following:
* `fmc/wr_nic_dio.bin` gateware for the NIC and DIO properties.
* `fmc/wr_nic_dio-wrc.bin` software for the LM32[^lm32inc].
[^fmc-bus]: More information about the [FMC]-bus can be found within [fmc-bus.pdf] in the `doc/` folder.
[^lm32inc]: In future version, the program file for the LM32 will be included into the gateware.
Compile & install
------------------------
A Makefile in the [spec-sw] project has been written to compile and install easily the drivers and the tools used below.
~~~~{.sh}
## Go to spec-sw project
>:$ cd spec-sw
## In the root folder (spec-sw), run
>:$ make
## then, install the driver in your system so that they load automatically
>:$ sudo make install
~~~~~~~~~~~
If everything works well you should see the driver in
~~~~{.sh}
>:$ ls -l /lib/modules/$(uname -r)/extra
~~~~~~~~~~~
> ***Notes:*** The procedure below is specific to ubuntu distribution so you
might want to find a similar way to perform it if you use another
distribution.
You should add the `extra` folder to the search of module so that at
next reboot the kenel modules are easily founded.
~~~~{.sh}
## Open the configuration file
>:$ sudo nano /etc/depmod.d/ubuntu.conf
~~~~~~~~~~~
Check if you already have the extra folder or otherwise add the `extra` keyword
before built-in.
The first line of this file should now be similar as:
search updates ubuntu extra built-in
Finally, you must regenerate the map of dependencies with this new path
by calling:
~~~~{.sh}
>:$ sudo depmod -a
~~~~~~~~~~~
Download, install & load the gateware
--------------------------------------
You need to install the gateware[^version] to `/lib/firmware/fmc`
### Automatic procedure
You can also use the script `wr-ssk-get` in `wr-starting-kit/scripts` folder
to ease the installation. `sudo` is required
~~~~{.sh}
## Fetch and install the firmware
>:$ sudo scripts/wr-ssk-get --all
~~~~~~~~~~~~~~
### Manual procedure
First you need to download the gateware/firmware files from our website:
<http://www.sevensols.com/dl/wr-starting-kit/bin/latest_stable.tar.gz>
~~~~{.sh}
## Create the gateware folder
>:$ mkdir firmware
## Extract to the gateware folder
>:$ tar -xzf wr-starting-kit-v2.0_gw.tar.gz -C ./firmware
~~~~~~~~~~~
Once you have the file you need to install them to your system in order to make them load automatically
~~~~{.sh}
## Install the HDL binaries in /lib/firmware/fmc (require sudo)
>:$ sudo cp -v firmware/*.bin /lib/firmware/fmc
~~~~~~~~~~~~~
[^version]: The HDL binaries (gateware) came from other project, you can find in
the [Developers Section](#quick-start-guide-for-developers) where to obtain them and
how to compile them.
Loading the driver
------------------------
......@@ -515,7 +379,9 @@ To enable the wr-nic you should execute the modprobe[^errmodprobe] command to lo
~~~~{.sh}
>:$ sudo modprobe spec
>:$ sudo modprobe htvic
>:$ sudo modprobe wr-nic
>:$ sudo modprobe wr-dio
~~~~~~~~
[^errmodprobe]: We have found a problem in some distribution with `modprobe` command.
......@@ -526,7 +392,7 @@ You should expect to obtain two (or one) new interface(s):
~~~~{.sh}
>:$ ifconfig -a | grep wr
wr0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX
wri1 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX
~~~~~~~~~~~~
If it is not the case you might have an empty EEPROM that need to be
......@@ -555,40 +421,15 @@ you might obtain the following warning on `dmesg`:
~~~~~{.sh}
>:$ dmesg
...
spec 0000:05:00.0: FPGA programming successful
spec 0000:05:00.0: mezzanine 0
EEPROM has no FRU information
11.949966] spec 0000:01:00.0: mezzanine 0
[ 11.949967] Manufacturer: CERN
[ 11.949967] Product name: FmcDio5cha
...
~~~~~~~~~~
This information, called the FRU, contains the type of FMC board, its serial number, etc.
You can generate in two different way:
### Automatic
A small script has been created for this procedure and thus you just need to call it as below:
scripts/wr-ssk-get -w
Then the script will detect if you have one or more [FMC-DIO] with empty
eeprom. If it is the case it will ask to enter a S/N.
> ***Tip***: The S/N is labeled on the back side of the
[FMC-DIO] board and the index of PCIe slots on the bus "normally" respect
the physical plug order from top to bottom on the motherboard.
If for instance you have one with the label `7S-FMCDIO5ch-v1.0-S3-058` and
you can't read the label on the second one, we suggest you to fill the input as below:
Found 2 FMC board(s) with empty EEPROM
Enter S/N for board fmc-0500 in the format xx-XXX (or 0) : 03-058
...
Enter S/N for board fmc-0b00 in the format xx-XXX (or 0) : 0
### Manual
If you prefer to perform these steps manually you can try the
following procedure:
In order to write it in case you have an old FMC board without any EEPROM written,
you should execute the following commands
~~~~~{.sh}
##First you need to go in the fru-generator folder
......@@ -599,29 +440,29 @@ following procedure:
## Then, find out on which bus id you have the boards
>:$ ls /sys/bus/fmc/devices/
fmc-0500 fmc-0b00
FmcDio5cha-0100
### Enter root mode in the terminal
>:$ sudo bash
## Write the eeprom for fmc-0500: the first/the top one fmc board.
>:# FRU_VENDOR="CERN" FRU_NAME="FmcDio5cha" FRU_PART="EDA-02408-V2-0" \
./fru-generator -s 7S-DIO-v2-S03-058 > /sys/bus/fmc/devices/fmc-0500/eeprom
## Write the eeprom for fmc-0b00: the 2nd/the bottom one fmc board.
## Write the eeprom for fmc-0100: the first/the top one fmc board.
>:# FRU_VENDOR="CERN" FRU_NAME="FmcDio5cha" FRU_PART="EDA-02408-V2-0" \
./fru-generator -s 7S-DIO-v2-S00-000 > /sys/bus/fmc/devices/fmc-0b00/eeprom
./fru-generator -s 7S-DIO-v2-S03-058 > /sys/bus/fmc/devices/FmcDio5cha-0100/eeprom
## Finally, Go back to user and starting-kit root directory
>:# exit
>:$ cd -
~~~~~~~~~~
Don't forget to modify the bus id: `fmc-XXXX` and the S/N
Don't forget to modify the bus id: `FmcDio5cha-XXXX` and the S/N
`7S-DIO-v2-Sxx-XXX` with your specific devices setup.
### Reload the kernel driver
> ***Notes:*** Removing the kernel is actually proving a kernel panic (core-dump) and
your PC will be freezed. This is a known bug that you be corrected in a release fix.
Once you have correctly written (Auto or Manual) the EEPROM of the [FMC-DIO], you need
to reload the kernel driver by doing this
......@@ -634,13 +475,9 @@ you should now obtain something like this on `dmesg`:
~~~~~{.sh}
...
[269290.520027] spec 0000:05:00.0: mezzanine 0
[269290.520035] Manufacturer: CERN
[269290.520038] Product name: FmcDio5cha
...
[269291.086221] spec 0000:0b:00.0: mezzanine 0
[269291.086228] Manufacturer: CERN
[269291.086230] Product name: FmcDio5cha
[ 10.901519] spec 0000:01:00.0: mezzanine 0
[ 10.901521] Manufacturer: CERN
[ 10.901521] Product name: FmcDio5cha
...
~~~~~~~~
......@@ -679,7 +516,7 @@ This depends on how you have plugged the USB cable to your USB of your machine.
### Virtual UART
An easier way to perform this operation is to access to the virtual
UART of the [SPEC] board by using the spec-vuart tool.
UART of the [SPEC] board by using the wrpc-vuart tool.
You first need to know the `bus_id` of your board
> ***Notes:*** The VUART can not work if the USB is already connected as the Physical UART
......@@ -687,26 +524,24 @@ has the priority.
~~~~~{.sh}
>:$ lspci | grep CERN
05:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
0b:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
01:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
~~~~~~~~~~
This means that your first board (wr0) is at **05**:00.0 and the second one (wr1) is at **0b**:00.0
This means that your first board (wri1) is at **01**:00.0
Thus, for each board you can open a terminal and run the following command:
~~~~~{.sh}
>:$ sudo ./tools/spec-vuart -b 0x05 #For the first SPEC board
>:$ sudo ./tools/spec-vuart -b 0x0b #For the second SPEC board
>:$ sudo wrpc-vuart -f /sys/bus/pci/devices/0000\:00\:01.0/0000\:01\:00.0/resource0 -o 0x20500
~~~~~~~~~~
> ***Warning:*** Be aware that if you try to remove the spec kernel modules with the virtual UART
activated, your PC will hang and thus you will need to reboot. You must close
the virtual UART before doing this kind of operation.
the virtual UART before doing this kind of operation.
[^sudomc]: On Ubuntu LTS, you need sudo permission to access to a
ttyUSB*X* device. It might be a good idea to add your user to the dialout group to
obtain appropriate permisssion on the device: `sudo usermod -a -G dialout $USER`
obtain appropriate permission on the device: `sudo usermod -a -G dialout $USER`
The White Rabbit Core Shell
......@@ -726,28 +561,26 @@ The most useful commands are repeated here for your convenience
For the tutorial we will use the following names:
* `wrc1#` for `wrc#` console of the main board (wr0/busID=0x0500)
* `wrc2#` for `wrc#` console of the second board (wr1/busID=0x0b00)
* `wrc1#` for `wrc#` console of the main board (wri1/busID=0x0100) on the PC1
* `wrc2#` for `wrc#` console of the main board (wri1/busID=0x0100) on the PC2
Configure in slave & master mode
--------------------------------
Now we can perform the following operation:
Now we can perform the following operation on both PCs:
* Check if you have the correct SFP and its corresponding value[^sfpdetectbug]
~~~~~{.sh}
wrc1# sfp detect
AXGE-3454-0531 #purple
wrc1# sfp match
AXGE-3454-0531 #purple
SFP matched, dTx=46407, dRx=167843, alpha=-73622176
~~~~~~~
~~~~~{.sh}
wrc2# sfp detect
wrc2# sfp match
AXGE-1254-0531 #blue
wrc1# sfp match
SFP matched, dTx=46407, dRx=167843, alpha=73622176
~~~~~~~
......@@ -759,7 +592,7 @@ and check the [Calibration Section](#calibration) that explain how to add SFPs p
~~~~~{.sh}
wrc1# mode master
SPLL_Init: running as Free-running Master, 1 ref channels, 2 out channels
PTP stop
Locking PLL...
~~~~~~~
......@@ -768,7 +601,8 @@ mode, this should not be necessary but you might have set it before in master mo
~~~~~{.sh}
wrc2# mode slave
slave
PTP stop
Locking PLL
~~~~~~~
### Start the synchronization with PTP daemon
......@@ -781,17 +615,6 @@ wrc1# ptp start
wrc2# ptp start
~~~~~~~
And you should obtain the following message on the slave board:
SPLL_Init: running as Slave, 1 ref channels, 2 out channels
Enabling ptracker channel: 0
Enabling ptracker channel: 0
servo:pre: -1885143040/2ps
servo:Deltas: Master: Tx=46407ps, Tx=175043ps, Slave: tx=46407ps, Rx=167843ps.
Adjust: counter = seconds [+0]
Adjust: counter = nanoseconds [+343914800]
servo:busy
Finally you can run the WRC shell in `gui` mode to obtain more information
~~~~~{.sh}
......@@ -799,70 +622,56 @@ wrc2# gui
~~~~~~~~~~
~~~~~{.sh}
WR PTP Core Sync Monitor v 1.0
WR PTP Core Sync Monitor wr-btrain-v1.1-17-g5a72198
Esc = exit
TAI Time: Thu, Jan 1, 1970, 0:30:15
TAI Time: Thu, Jan 1, 1970, 02:01:57
wru1: Link up (RX: 766, TX: 340), mode: WR Slave Locked Calibrated
Link status:
wru1: Link up (RX: 12860, TX: 8181) IPv4: BOOTP running
Mode: WR Slave Locked Calibrated
Synchronization status:
PTP status: slave
Synchronization status:
Servo state: TRACK_PHASE
Phase tracking: ON
Synchronization source: wru1
Aux clock status:
Aux clock 0 status: enabled
Timing parameters:
Round-trip time (mu): 698557 ps
Master-slave delay: 345660 ps
Master PHY delays: TX: 46407 ps, RX: 175043 ps
Slave PHY delays: TX: 46407 ps, RX: 167843 ps
Total link asymmetry: 7237 ps
Cable rtt delay: 262857 ps
Clock offset: 2 ps
Phase setpoint: 7268 ps
Skew: 2 ps
Manual phase adjustment: 0 ps
Update counter: 117
Round-trip time (mu): 684638 ps
Master-slave delay: 354511 ps
Master PHY delays: TX: 121751 ps, RX: 99027 ps
Slave PHY delays: TX: 0 ps, RX: 1600 ps
Total link asymmetry: -24384 ps
Cable rtt delay: 462260 ps
Clock offset: 1 ps
Phase setpoint: 561 ps
Skew: 0 ps
Update counter: 7
--
~~~~~~~~~~~~~~~~~~
You can see that the slave node is locked, calibrated and that the phase tracking is enabled.
> ***Notes:*** We also recommand you to set up an init script if you do not want to repeat these operations at each
reboot. You can look at the [wrpc.pdf] for more information or use the `init show` command to check the
one you have running.
> ***Notes:*** We also recommend you to set up an init script if you do not want to repeat these operations at each reboot. You can look at the [wrpc.pdf] for more information or use the `init show` command to check the one you have running.
Bring up the network interface
Check the network interface
-------------------------------
Once the drivers are loaded and the PTP has started you might need to bring the new network interfaces up.
Once the drivers are loaded and the PTP has started you might check if the new network interfaces is up.
In order to do this you should just execute:
~~~~{.sh}
>:$ sudo ifconfig wr0
>:$ sudo ifconfig wr1
~~~~~~~~~~~~
And then check if they are mounted by doing
~~~~{.sh}
>:$ sudo ifconfig | grep wr
wr0 Link encap:Ethernet HWaddr 08:00:30:0d:e8:6b
wr1 Link encap:Ethernet HWaddr 08:00:30:0d:e4:cd
wri1 Link encap:Ethernet HWaddr 22:33:07:00:f1:a7
~~~~~~~~~~~~
> ***Notes:*** On the distributions that use gnome desktop (i.e, Ubuntu), the interfaces are automatically mounted by *gnome
network manager*, so you might not need to perform the last step.
Some Illustrative Experiments
==================
......@@ -875,23 +684,15 @@ the same order as the experiments.
> **Notes:** Before each experiment you should make sure that you set
up your boards in *master/slave* mode and that the PTP daemons are running on both.
You cand find how to do it in subsection [4.3](#configure-in-slave-master-mode).
A basic explanation is included in [Frequently Added Questions](#frequently-added-questions) section (first one).
Playing with the DIO channels
----------------------------------------------------------------------------------------
The first step to perform is to check if the [FMCDIO] channels are accessible using the installed driver.
In the `tools` subdirectory you will you find the `wr-dio-cmd` program,
that let you quickly test the I/O features of the [FMCDIO] boards
This is the general syntax of the command[^sudo]: `wr-dio-cmd <ifname> <cmd> [<arg> ...]`
The main commands correspond to: `mode, pulse & stamp`, and you can
find an extended description of
them in the [spec-sw.pdf:6.7][spec-sw.pdf] document.
The `wr-dio-cmd` program, let you quickly test the I/O features of the [FMCDIO] boards.
This is the general syntax of the command[^sudo]: `wr-dio-cmd: use "wr-dio-cmd [-V] <dev> <cmd> [...]"`
[^sudo]: As the application use `ioctl` you should call with root privilege by using sudo.
......@@ -922,7 +723,7 @@ For example you can set it up like this
~~~~~{.sh}
## Configure channel 0 as input with termination, 1 as input, 4 as fixed low
>:$ sudo ./tools/wr-dio-cmd wr0 mode Ii--0
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode Ii--0
~~~~~~~~~~~~~~~~
......@@ -941,7 +742,7 @@ To reset to the default mode you can reset/reprogram the FPGA or set it back wit
~~~~~{.sh}
## Revert back to default mode
>:$ sudo ./tools/wr-dio-cmd wr0 mode p00ic.
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode p00ic.
~~~~~~~~~~~~~~~~
......@@ -959,18 +760,18 @@ Finally you need to active a trigger pulse in your oscilloscope. Then you can tr
~~~~~{.sh}
## Set channel 4 as Input
>:$ sudo ./tools/wr-dio-cmd wr0 mode 4 D
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 4 D
## Pulse channel 4 for 0.1 seconds now
>:$ sudo ./tools/wr-dio-cmd wr0 pulse 4 .1 now
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .1 now
## Pulse for 10 microseconds in the middle of the next second
>:$ sudo ./tools/wr-dio-cmd wr0 pulse 4 .00001 +1.5
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .00001 +1.5
## Pulse for 1ms at 17:00 today
## Set the datetime of the next event (60 seconds from now)
## NOTE: this only work if the date is correctly set in the master SPEC,
>:$ sudo ./tools/wr-dio-cmd wr0 pulse 4 .001 $(date +%s --date 17:00)
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .001 $(date +%s --date 17:00)
~~~~~~~~
> ***Note:*** please note that the `pulse` command activates the [FMCDIO] mode (without changing the termination)
......@@ -981,7 +782,7 @@ Finally you need to active a trigger pulse in your oscilloscope. Then you can tr
Once you have generated the pulse you can retrieve its timestamp by executing:
~~~~~{.sh}
>:$ sudo ./tools/wr-dio-cmd wr0 stamp 4
>:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
ch 4, 378.788588536
ch 4, 381.268701864
ch 4, 387.284885816
......@@ -1008,27 +809,29 @@ The configuration is done as indicated in the figure below:
![Time-stamping configuration](img/ssk_playdio.png)
~~~~~{.sh}
## Configure #2 & #5 (ch1 & ch4) as ouput and #3 (ch2) as input with termination on wr0.
>:$ sudo ./tools/wr-dio-cmd wr0 mode -DI-D
## Configure #4 (ch3) as input on wr1
>:$ sudo ./tools/wr-dio-cmd wr1 mode 3 I
## Configure #2 & #5 (ch1 & ch4) as ouput and #3 (ch2) as input with termination on wri1 (spusa).
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode -DI-D
## Then flush the previous timestamp
>:$ sudo ./tools/wr-dio-cmd wr0 stamp &> /dev/null
>:$ sudo ./tools/wr-dio-cmd wr1 stamp &> /dev/null
## Configure #4 (ch3) as input on wri1 (tornado)
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 3 I
## Then flush the previous timestamp (spusa and tornado)
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp &> /dev/null
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp &> /dev/null
## Schedule the pulse to the next common event on two outputs
>:$ sudo ./tools/wr-dio-cmd wr0 pulse 1 .00001 +2
>:$ sudo ./tools/wr-dio-cmd wr0 pulse 4 .00001 +2
## Then (after 60s), you should run stamp on the wr0
>:$ sudo ./tools/wr-dio-cmd wr0 stamp
ch 1, 2267.500000000
ch 2, 2267.500000008
ch 4, 2270.500000000
>:$ sudo ./tools/wr-dio-cmd wr1 stamp
ch 3, 2270.500000008
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 1 .00001 +2
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 4 .00001 +2
## Then (after 60s), you should run stamp on the wri1
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp
ch 1, 186789.999999992
ch 2, 186790.000000000
ch 4, 186792.999999992
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp
ch 3, 186792.999999992
~~~~~~~~~~~~~~~~~
......@@ -1068,11 +871,11 @@ of both boards.
If you don't see this pulse, you might want to force the pulse mode:
~~~~~~{.sh}
## run pps on channel 0 of the "first" card
>:$ sudo ./tools/wr-dio-cmd wr0 mode 0 p
## run pps on channel 0 of the "first" card (spusa)
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p
## run pps on channel 0 of the "second" card
>:$ sudo ./tools/wr-dio-cmd wr1 mode 0 p
## run pps on channel 0 of the "second" card (tornado)
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p
~~~~~~~~~~
......@@ -1093,8 +896,8 @@ The Network Interface Card (NIC) Synchronization
### Introduction
The *wr-nic* driver registers a Linux network interface card for
each [SPEC] device that it drives. The cards are called `wr%d` (i.e.,
*wr0*, *wr1*, ...).
each [SPEC] device that it drives. The cards are called `wri%d` (i.e.,
*wri1*, *wri2*, ...).
The [SPEC] can carry normal data traffic in
addition to the PTP frames of *White Rabbit*, that remain
......@@ -1141,11 +944,11 @@ sends the forward frame and reports data as soon as it gets a reply.
~~~~~{.sh}
## On a terminal run
>:$ sudo ./tools/stamp-frame wr0 listen
tools/stamp-frame: Using interface wr0, with all timestamp options active
>spusa:$ sudo stamp-frame wri1 listen
tools/stamp-frame: Using interface wri1, with all timestamp options active
## On another terminal (maybe on another host) run
>:$ sudo ./tools/stamp-frame wr1
>tornado:$ sudo stamp-frame wri1
tools/stamp-frame: Using interface wr1, with all timestamp options active
timestamp T1: 1891.948736656
timestamp T2: 1892.038390176
......@@ -1159,8 +962,8 @@ backward time: -0.089544832
This is done four times: departure and arrival of the forward frame, followed
by departure and arrival of the backward frame. Thus,
time stamps T1 and T4 are collected at the
original sender (here: *wr1*) while T2 and T3 are collected at the
remote host (here: *wr0*). The times above are all consistent
original sender (here: *wri1 "tornado"*) while T2 and T3 are collected at the
remote host (here: *wri1 "spusa"*). The times above are all consistent
because the two [SPEC] cards are synchronized with *White Rabbit*. The
reported forward and backward times match the fact that we used
a 10km fiber to connect the two cards; the difference between them
......@@ -1267,7 +1070,7 @@ delay of 50 microseconds).
Then we should run the "dumb" agent on the slave board in charge of forwarding *ioctl* packet to the *DIO* core:
~~~~~~{.sh}
>:$ sudo ./tool/wr-dio-agent wr1 &
>tornado:$ sudo wr-dio-agent wri1 /dev/fmc-dio-1:0 &
~~~~~~~~~~
Then you need to connect the output of the generator[^fake100hz] to channel 1 of the master [SPEC+FMCDIO] board.
......@@ -1275,7 +1078,7 @@ The generated waveform should be a 0-5V pulse at ~100Hz (5ms at 5V then 5ms at 0
[^fake100hz]: If you lack a wave form generator, you can make an output pulse with
`wr-dio-cmd <if> mode <ch> P` or other means and use it as a trigger. Check the [Two PCs example](#two-pcs-example).
`wr-dio-cmd <dev> mode <ch> P` or other means and use it as a trigger. Check the [Two PCs example](#two-pcs-example).
......@@ -1283,10 +1086,10 @@ You should setup the channel 0 as input and check if the 100Hz signal is correct
~~~~~~{.sh}
## Set channel 0 of master board as Input and disable PPS signal (if the FPGA is reflashed).
>:$ sudo ./tools/wr-dio-cmd wr0 mode I-0--
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode I-0--
## Retrieved the time stamped value
>:$ sudo ./tools/wr-dio-cmd wr0 stamp
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp
ch 0, 3573.281851462
ch 0, 3573.291851460
ch 0, 3573.301851460
......@@ -1309,13 +1112,13 @@ on both the local (channel 3) and the remote card (channel 1)
~~~~~~{.sh}
## Set local3 as Output
>:$ sudo ./tools/wr-dio-cmd wr0 mode 3 D
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 3 D
## Set remote1 as Output (and disable 2)
>:$ sudo ./tools/wr-dio-cmd wr1 mode -D0--
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 mode -D0--
## Creating the ruler to forward from input 0 to local3 and remote1
>:$ sudo ./tools/wr-dio-ruler wr0 IN0 L3+0.001 R1+0.001
>spusa:$ sudo wr-dio-ruler wri1 /dev/fmc-dio-1:0 IN0 L3+0.001 R1+0.001
wr-dio-ruler: configured for local channel 3, delay 0.001000000
wr-dio-ruler: configured for remote channel 1, delay 0.001000000
~~~~~~~~~~~~~
......@@ -1325,16 +1128,16 @@ wr-dio-ruler: configured for remote channel 1, delay 0.001000000
To check if the experiment works well you can compare the timestamp of the two outputs
~~~~~~~~{.sh}
## Checking the timestamp on local ouput
>:$ sudo ./tools/wr-dio-cmd wr0 stamp 3
## Checking the timestamp on local output
>spusa:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
ch 3, 5340.004864960
ch 3, 5340.014864960
ch 3, 5340.024864968
ch 3, 5340.034864964
...
## Checking the timestamp on remote ouput
>:$ sudo ./tools/wr-dio-cmd wr1 stamp 1
## Checking the timestamp on remote output
>tornado:$ sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 1
ch 1, 5340.004864962
ch 1, 5340.014864960
ch 1, 5340.024864968
......@@ -1360,7 +1163,7 @@ we verified the tools and we think this is a good explanation of how to use them
It shows how to use the *ruler* and *agent* on
two hosts, called `spusa` and `tornado`. It is pretty the same thing as the previous
experiment only that the board are now both named `wr0`.
experiment only that the board are now both named `wri1`.
![Transmit PPS between two PCs](img/ssk_txpps.png)
......@@ -1370,28 +1173,28 @@ channel 0.
~~~~~{.sh}
>spusa:# sudo ./tools/wr-dio-cmd wr0 mode 0 P
>spusa:# sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 P
>tornado:# sudo ./tools/wr-dio-agent wr0 &
>tornado:# sudo wr-dio-agent wri1 /dev/fmc-dio-1:0 &
>spusa:# ./tools/wr-dio-ruler wr0 IN4 L3+.001 R4+.001 R2+.001
>spusa:# sudo wr-dio-ruler wri1 /dev/fmc-dio-1:0 IN4 L3+.001 R4+.001 R2+.001
wr-dio-ruler: configured for local channel 3, delay 0.001000000
wr-dio-ruler: configured for remote channel 4, delay 0.001000000
wr-dio-ruler: configured for remote channel 2, delay 0.001000000
[... wait a few seconds ...]
>spusa:# ./tools/wr-dio-cmd wr0 stamp 3
>spusa:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 3
ch 3, 385.001000000
ch 3, 386.001000000
ch 3, 387.001000000
ch 3, 388.001000000
>tornado:# ./tools/wr-dio-cmd wr0 stamp 2
>tornado:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 2
ch 2, 385.001000000
ch 2, 386.001000000
ch 2, 387.001000000
ch 2, 388.001000000
>tornado:# ./tools/wr-dio-cmd wr0 stamp 4
>tornado:# sudo wr-dio-cmd /dev/fmc-dio-1:0 stamp 4
ch 4, 385.001000000
ch 4, 386.001000000
ch 4, 387.001000000
......@@ -1460,10 +1263,10 @@ captured by the oscilloscope.
to your gateware release. When this document was generated we refer as
wr-nic-v2.0 (starting kit). You can also check using the `ver` command
if the corresponding embedded LM32 corresponds to your gateware version
(i.e, wrpc-v2.1-for-wrnic lm32 is embedded in wr-nic-v2.0).
(i.e, wrpc-v2.1-for-wrnic lm32 is embedded in wr-nic-v2.0).
~~~~~{.sh}
wrc# ver
wrc# ver
WR Core build: wrpc-v2.1-for-wrnic
...
......@@ -1488,7 +1291,6 @@ yourself the calibration procedure.
in its EEPROM. This value must be obtained by first connecting this
board in slave mode to a working WR master. For more information read the
[wrpc-v2.1.pdf] page 7.
SPEC+DIO as grandmaster
------------------------
......@@ -1602,110 +1404,6 @@ Please refer to *Writing EEPROM and calibration* Section of the [wrpc.pdf] docum
Manage standalone node using etherbone
----------------------------------------
You can also use SPEC card in standalone mode as we have seen before
but... how can you configure spec card if you do not use drivers?
In wr-nic project, Etherbone core has been added to the design and it allows
a direct access to the memory map[^specmem] (wishbone registers) using
UDP/TCP packets.
In order to connect you to the [SPEC+FMCDIO] in standlone you must
first configure it with a valid IP. This can be done automatically (complex) using bootp protocol,
or manually (easy) through the WRC mini-USB UART.
[^specmem]: If you are still connected through the PCIe you can also use the tools
`spec-sw/tools/specmem` to directly read/write on the WB registers as you will
do with the etherbone tool.
### Set IP using BootP
This is the more complex way to do it because you need to create a BootP server
on your LAN, but then the IP of each standalone board can be automatically
asigned.
On Ubuntu the `dnsmasq` package can be used as BootP deamon. The configuration
file should be something similar as:
~~~~~~~{.sh}
## Specified interfaces (10.10.10.1 is your IP)
interface=eth1
## DHCP setup
dhcp-range=10.10.10.1,10.10.1.15,12h
## BootP setup
dhcp-boot=pxe.0,tfpservname,10.10.10.1
## Set a specific IP for a MAC address (optional)
dhcp-host=08:00:30:0d:f8:23,targetsysname,10.10.10.10
~~~~~~~~~~~~
### Set IP through WRC UART
An easier way to set the IP of your standalone board is through the WRC Shell by
connecting the mini-USB UART to your PC.
Then you should just write in the terminal:
~~~~~~~{.sh}
wrc# ip set 10.10.10.10
~~~~~~~~~~~~
If you want this IP to be kept after a power cycle you must add this command in the
init EEPROM script as explained [previously](#eeprom-boot-script).
### Access to the wishbone device
At this step, you need to connect the standalone device with your PC. If you
do not have any SFP interface on your PC you can use an SFP-RJ45 adapter
to connect to it.
You need to check that your interface is on the same LAN as the standalone
[SPEC+FMCDIO] board or you should set it manually if it is not the case:
~~~~~~~{.sh}
>:$ sudo ifconfig eth1 10.10.10.1
~~~~~~~~~~~~
Finally you can try the etherbone library to connect to your standalone
[SPEC+FMCDIO] board. Please check that you have compiled the library for
your platform by doing `make && sudo make install` in the main folder or by going to
`etherbone/api` and perform a `make clean && make && sudo make install`.
To ease the communication using etherbone, we have added the `eb-mem.sh` tool in the `scripts/`
folder that access through the etherbone library to the device and perform
read/write operations.
Below we display a quick example on how to use it.
~~~~~~{.sh}
## It shows you Device memory map
>:$ scripts/eb-mem.sh --scan --ip 10.10.10.10
## Read a memory address (DIO I/O mode register <=> 62000 + 300 + 3C)
>:$ scripts/eb-mem.sh --read --ip 10.10.10.10 --address 0x6233C
## Write a memory address (DIO I/O mode register <=> 62000 + 300 + 3C)
## Force only ch2 (#3) into P mode (forbidden operation while using wr-dio-cmd)
>:$ scripts/eb-mem.sh --write --ip 10.10.10.10 --address 0x6233C --value 0x00A00
## Show help
>:$ scripts/eb-mem.sh --help
~~~~~~~~~~~~~
<!--TODO: Caloe -->
A library called CALoE (Configuration Abstraction Layer over Etherbone)
can also be used to ease the task of accesing to various register through
etherbone.
You can find more information on the following page
<https://github.com/klyone/caloe/wiki>
Quick Start Guide For Developers
=================================
......@@ -1844,7 +1542,7 @@ git submodule init
git submodule update
## Go to the main directory
cd wr-nic/syn/spec/
cd syn/specdio/
## Synthetize using hdlmake
hdlmake --make-ise --ise-proj
......@@ -1853,6 +1551,10 @@ make
You should finally obtain the bitstream to import in your [FMC] driver folder.
> **Notes**: You must have the proper version of hdlmake installed, please
check the manual [wr-nic.pdf] for more information on how to synthesize the
gateware.
Frequently Added Questions
===========================
......@@ -2026,11 +1728,11 @@ WRC / WRPC
[WR]:http://www.sevensols.com/whiterabbitsolution
[WRS]: http://www.sevensols.com/en/products/wr-switch.html
[WRSSK]: http://www.sevensols.com/en/products/wr-starting-kit.html
[SPEC+FMCDIO]: http://www.sevensols.com/en/products/wr-starting-kit.html
[SPEC]: www.sevensols.com/en/products/spec.html
[SPEC+FMCDIO]: http://sevensols.com/index.php/kit-spec/
[SPEC]: http://sevensols.com/index.php/products/spec/
[FMC]: http://www.ohwr.org/projects/fmc-projects
[FMC-DIO]: http://www.sevensols.com/en/products/fmc-dio.html
[FMCDIO]: http://www.sevensols.com/en/products/fmc-dio.html
[FMC-DIO]: http://sevensols.com/index.php/fmc-dio/
[FMCDIO]: http://sevensols.com/index.php/fmc-dio/
[FMC fine-delay]: http://www.sevensols.com/en/products/fmc-del.html
[OHWR]: http://www.ohwr.org/projects/white-rabbit
......@@ -2052,7 +1754,7 @@ References
[wr-switch-guide.pdf]: http://www.ohwr.org/attachments/download/2263/wr-switch-sw-v3.3-20130725_userguide.pdf
[wr-switch-guide.pdf]: http://www.sevensols.com/dl/wr-switch-sw/startupguide/latest_stable.pdf
[wr-nic]: http://www.ohwr.org/projects/wr-nic/
[wr-starting-kit]: http://www.ohwr.org/projects/wr-starting-kit/
[spec-sw]: http://www.ohwr.org/projects/spec-sw/
......@@ -2060,7 +1762,7 @@ References
[spec-golden]: http://www.ohwr.org/projects/spec/repository/show/trunk/hdl/golden
[spec-2-spec]: http://www.ohwr.org/projects/wr-cores/wiki/spec-to-spec
[spec-getting-started.pdf]:http://www.ohwr.org/projects/spec-getting-started/wiki
[wr-nic.pdf]: http://www.sevensols.com/dl/wr-nic/latest_stable.pdf
[wr-nic.pdf]: http://www.sevensols.com/dl/wr-nic/doc/latest_stable.pdf
[fmc-bus.pdf]: http://www.ohwr.org/attachments/download/2685/fmc-bus-2014-02-release.pdf
[spec-sw.pdf]:http://www.sevensols.com/dl/spec-sw/latest_stable.pdf
[wr_external_reference.pdf]: http://www.ohwr.org/attachments/1647/wr_external_reference.pdf
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment