.. Copyright (c) 2019 CERN (home.cern)
   SPDX-License-Identifier: CC-BY-SA-4.0

.. _devel:

-----------
Development
-----------

This section provides guidelines on how to develop a new WRTD Node (or, modify an :ref:`existing one
<ref_nodes>`), including :ref:`hw_dev`, :ref:`gw_dev` and :ref:`fw_dev`, using the resources
provided by the `WRTD project <https://www.ohwr.org/project/wrtd/>`_. If you simply need to control
an existing WRTD Node, please refer to :numref:`usage` instead.

It should be noted that a new Node does not necessarily need to use the resources provided by the
WRTD project itself. In fact, the only hard requirement is that the Node must be able to send and/or
receive :ref:`Event Messages <message>` over an Ethernet-based LAN, using multicast UDP on address
``224.0.23.159``, port ``5044``. It should also respect the :ref:`event_id` naming conventions of
WRTD.

Of course, the above is the absolute minimum requirement, which is already fullfilled by any LXI
device supporting the relevant LXI extended functions (see also :ref:`lxi`).

Moving one step further, the network interface of the Node should be White Rabbit enabled, to allow
for accurate timestamping and sub-ns synchronisation between the Nodes. This has implications for
your :ref:`hw_dev`.

If your :ref:`node` contains an FPGA and you choose to use the WRTD-provided :ref:`gw_dev`
resources, then this also opens up the possibility of using the WRTD-provided :ref:`fw_dev`
resources as well, to develop the application(s) running inside the FPGA. A WRTD :ref:`node`
developed with the WRTD :ref:`gw_dev` and :ref:`fw_dev` resources will be accessible via the
provided :ref:`clib`, :ref:`pywrap` and :ref:`tools`.


.. _dev_setup:

Setting Up the Environment
==========================

The WRTD development environment has been tested and works best under Linux.

The source code of WRTD is maintained in a `Git <https://git-scm.com/>`_ repository. To create a
local copy of it, please use:

.. code-block:: console

   $ git clone "https://ohwr.org/project/wrtd.git"

And then `checkout <https://git-scm.com/docs/git-checkout>`_ the correct commit, branch or tag that
you wish to work on. Typically, the latest stable commit will be tracked by the `master
<https://www.ohwr.org/project/wrtd/tree/master>`_ branch, while the latest development commit will
be tracked by the `proposed_master <https://www.ohwr.org/project/wrtd/tree/proposed_master>`_
branch.

Once this is done, you need to bring in the various dependencies of WRTD (which are present as `Git
submodules <https://git-scm.com/book/en/v2/Git-Tools-Submodules>`_ within the WRTD repository):

.. code-block:: console

   $ cd wrtd
   $ git submodule update --init

.. important::

   Despite the fact that many of WRTD's dependencies contain their own submodules, you should not
   bring in those dependencies recursively, as they may create conflicts (e.g. both WRTD and one of
   its submodules depending on another submodule but on a different version of it). WRTD's top-level
   dependencies (which are brought in with the above command) contain everything that is needed for
   development.

Furthermore, you will need a RISC-V cross-compilation toolchain for compiling the firmware for the
Nodes.

.. note:: Because at the time of the release of WRTD v1.0, the OHWR deployment procedures were
          undergoing significant changes (in particular with respect to packaging), up-to-date
          installation instructions for all the necessary development tools will be available
          through the `project Wiki
          <https://www.ohwr.org/project/wrtd/wikis/Development-Tools-Installation>`_. Once these
          procedures have been finalised and tested, the contents of the wiki page will be merged
          here.

Alternatively, you can try to build your own toolchain like this:

.. code-block:: console

   $ git clone "https://ohwr.org/project/soft-cpu-toolchains.git"
   $ cd soft-cpu-toolchains/riscv
   $ sh ./build-riscv.sh

This will checkout the ``master`` branch of the repository and start the build process, which takes
a long time to complete.

Once the process is complete, the resulting RISC-V toolchain binaries will be placed under
``soft-cpu-toolchains/riscv/riscv-toolchain/bin``. You should either copy them to somewhere within
your system's path (typicallly ``/usr/local/bin``), or set the environment variable
``CROSS_COMPILE_TARGET`` to point to the correct location, with the ``riscv32-elf-`` suffix appended
to the path, like this:

.. code-block:: console

   $ export CROSS_COMPILE_TARGET="<path_to_repository>/riscv/riscv-toolchain/bin/riscv32-elf-"

You either need to configure the above command to be run automatically on a new terminal (typically
by putting the command in ``~/.profile`` or ``~/.bash_profile`` but this depends ultimately on the
shell you are using), or you should run it every time you start a new terminal and need to compile
RISC-V firmware.

.. _hw_dev:

Hardware
========

Hardware-wise, WRTD does not impose any particular requirements. It is assumed that the device is
capable of producing and/or consuming "events" (e.g. trigger in, trigger out), otherwise there is no
point in using WRTD.

Other than that, the only requirement on the hardware is that it supports `White Rabbit (WR)
<https://www.ohwr.org/project/white-rabbit>`_.

If you are designing your own board and you would like to have something as a reference, here's a
page which can `help you <https://www.ohwr.org/project/white-rabbit/wikis/WRReferenceDesign>`_, or,
more specifically, here's a list with a few open-source boards that already support WR:

* `SPEC <https://www.ohwr.org/projects/spec/wikis>`_ (Xilinx Spartan6 FPGA)
* `FASEC <https://www.ohwr.org/projects/fasec/wikis>`_  (Xilinx Zynq SoC FPGA)
* `VFC-HD <https://www.ohwr.org/projects/vfc-hd/wikis>`_ (Intel Arria V)

All of the above examples make use of the `WR PTP Core
<https://www.ohwr.org/project/wr-cores/wikis/wrpc-core>`_ inside their FPGA.

.. _gw_dev:

Gateware
========

There is no HDL template yet for WRTD gareware development. Users who need to develop WRTD gateware
for a new :ref:`node` are advised to use the top-level VHDL modules of the :ref:`ref_nodes` as a
starting point. Their source code is available on the WRTD Git repository:

https://www.ohwr.org/project/wrtd/tree/master/hdl/top

Every :ref:`node` should include at a minimum an instance of `MockTurtle
<https://www.ohwr.org/project/mock-turtle/wikis/home>`_ and the `WR PTP Core
<https://www.ohwr.org/project/wr-cores/wikis/wrpc-core>`_.

Furthermore, the MockTurtle configuration should define at least one *Host Message Queue* per CPU,
for communication between the WRTD library and the WRTD :ref:`application` running on the CPU, as
well as one *Remote Message Queue* per CPU for conencting the WRTD :ref:`application` to the WR
network.

Every :ref:`node` should also include a *MockTurtle Ethernet Endpoint*, to interface the *Remote
Message Queue* of the MockTurtle with the *Fabric Interface* of the WR PTP Core. If in doubt, have a
look at how these three modules are instantiated and connected to each other in the
:ref:`ref_nodes`.

.. _fw_dev:

Firmware
========

WRTD provides a common firmware development framework for :ref:`Applications <application>`. To use
it, users must also use a compatible :ref:`gw_dev`.

Similar to :ref:`gw_dev` development, users are advised to use the firmware of the :ref:`ref_nodes`
as a starting point:

https://www.ohwr.org/project/wrtd/tree/master/software/firmware

Setup
-----

To start developing firmware for WRTD, users must first include the relevant header file:

.. code-block:: c

 #include "wrtd-rt-common.h"

Users must also describe the various features of their application, by means of C preprocessor
directives. The following needs to be copied to the application source code and adjusted as
necessary:

.. code-block:: c

 /* Number of MockTurtle CPUs. */
 #define NBR_CPUS xxx
 /* Index of this CPU. */
 #define CPU_IDX xxx
 /* Maximum number of Rules allowed */
 #define NBR_RULES xxx
 /* Maximum number of Alarms allowed */
 #define NBR_ALARMS xxx
 /* Number of Devices. A "Device" is a unidirectional set
    of Local Channels.
    Most Applications will have one Device per direction (in/out).
    Maximum allowed number of Devices = 4. */
 #define NBR_DEVICES xxx
 /* Number of Local Channels per Device. */
 #define DEVICES_NBR_CHS { xxx, 0, 0, 0}
 /* Direction of Local Channels per Device. Direction can be either
    WRTD_CH_DIR_IN (from the environment to WRTD) or
    WRTD_CH_DIR_OUT (from WRTD to the environment). */
 #define DEVICES_CHS_DIR { WRTD_CH_DIR_IN, 0, 0, 0}
 /* A unique number to identify this Application. */
 #define APP_ID xxx
 /* Application version (major, minor). */
 #define APP_VER RT_VERSION(xxx, yyy)
 /* A string name for this Application. */
 #define APP_NAME xxx
 /* 1 if the Application can receive Events over the network,
    0 otherwise. */
 #define WRTD_NET_RX x
 /* 1 if the Application can send Events over the network,
    0 otherwise. */
 #define WRTD_NET_TX x
 /* 1 if the Application can receive Events from Local Channels,
    0 otherwise. */
 #define WRTD_LOCAL_RX x
 /* 1 if the Application can send Events to Local Channels,
    0 otherwise. */
 #define WRTD_LOCAL_TX x

Apart from the preprocessor definitions, users must also provide implementations for a set of
functions. These include:

* :ref:`fw_api_init`

  + :cpp:func:`wrtd_user_init`
  + :cpp:func:`wrtd_io`

* :ref:`fw_api_wr_link`

  + :cpp:func:`wr_link_up`
  + :cpp:func:`wr_time_ready`
  + :cpp:func:`wr_enable_lock`
  + :cpp:func:`wr_aux_locked`
  + :cpp:func:`wr_sync_timeout`

* :ref:`fw_api_event_in`

  + :cpp:func:`wrtd_local_output`

.. _fw_api_init:

Initialisation and Main Loop
----------------------------

WRTD firmware runs in a continuous loop. All such code should be put in the :cpp:func:`wrtd_io`
function. Additionally, the :cpp:func:`wrtd_user_init` offers the possibility to execute code once,
after reset, before entering the main execution loop.

.. doxygenfunction:: wrtd_user_init
.. doxygenfunction:: wrtd_io

.. _fw_api_wr_link:

WR Link Control and Status
--------------------------

It is expected (but not required) that WRTD firmware has access to the WR link. In particular, most
applications should be able to detect whether the link is up, the timecode is valid, etc. The way
this is done is application-specific. The following functions allow users to describe how this is
done.

.. doxygenfunction:: wr_link_up
.. doxygenfunction:: wr_time_ready
.. doxygenfunction:: wr_enable_lock
.. doxygenfunction:: wr_aux_locked
.. doxygenfunction:: wr_sync_timeout

Event I/O
---------

Ultimately, the purpose of a WRTD firmware is to relay :ref:`Events <event>` from/to the outside
world.

Internally, WRTD represents an Event using the :cpp:class:`wrtd_event` structure.

.. doxygenstruct:: wrtd_event
                   :members:

.. _fw_api_event_in:

Sending Events
++++++++++++++

When an incoming :ref:`Event <event>` has been matched to a :ref:`rule` with a :ref:`local_channel`
output, WRTD will call the user-provided :cpp:func:`wrtd_local_output` function. This function
should perform all the application-specific actions to program the relevant :ref:`local_channel` to
generate the actual output.

.. doxygenfunction:: wrtd_local_output

.. _fw_api_event_out:

Receiving Events
++++++++++++++++

Receiving an :ref:`event` is also application-specific. The monitoring of :ref:`local_channel`
inputs is typically done periodically in the main execution loop, using the :cpp:func:`wrtd_io`
function.

Once the firmware detects in incoming :ref:`event` and fills in a :cpp:class:`wrtd_event` structure,
it should simply pass this to WRTD using the :cpp:func:`wrtd_route_in` function. The rest (rule
matching, event forwarding, etc.) will be handled by WRTD.

.. doxygenfunction:: wrtd_route_in

.. hint:: Contrary to the rest of the functions presented in :numref:`fw_dev`,
          :cpp:func:`wrtd_route_in` is not a user-defined function. It is already provided by the
          WRTD firmware development framework. Users should simply create and fill in the
          :cpp:class:`wrtd_event` ``ev`` structure with the details of the received :ref:`event` and
          then pass it on to WRTD by calling this function.
