Interrupts

Versions 1.0 and 1.1 of SDB don't describe interrupts. Lack of interrupt
description is a design choice (that may change in the future,
though), and here we try to explain it.

??Most of the text in this page is from a message Wesley Terpstra sent
to the mailing list of the project, slightly edited to fit this context.??

SDB was born as a description of address spaces. Access to address
spaces is usually done through bus signals, according to the
bus specification: signal lines, protocols, timings. Interrupts
strictly-speaking are not part of the bus: they are a sort of a
``secondary'' bus, where only one signal line connects one core to the
``interrupt controller'' core. The same applies to the out-of-band
DMA channels used in some designs, where two devices are connected via
a path not part of the normal bus interconnect fabric.
In that case the DMA controller can act as a master on behalf of the slave.

We currently don't offer such descriptions because we really think
designs should be converted to the concept of MSI interrupts, and
this section summarizes the reasoning.

A bus lets you control slave
devices from master devices. This communication is always initiated by
the master, a request-response messaging pattern. Then one day you have
a slave device which needs to initiate communication.
The traditional answer has been interrupt lines, that
let you reuse the bus lines. Your
slave card just needs to "wake-up" the master, who can then use the
usual bus lines to find out what's the matter and take appropriate
action. So, for the cost of a single line, they enable two-way
communication.

Unfortunately, this simple solution comes at a significant cost. The
core problem is that you've actually introduced a completely new (albeit
superficially simple) bus to your system. All of the following problems
stem from this change:

• like in a normal bus protocol, an interrupt needs to be
  acknowledged. This is typically done by writing some register on the
  slave device which causes it lower the interrupt line when you've dealt
  with all the things that needed processing;

• like in a normal bus protocol, you need flow control. Maybe the
  master is busy and wants to deal with some critical section code before
  it gets around to dealing with your interrupt. This is why interrupt
  systems need enable flags, mask registers, and so on;

• like in a normal bus protocol, you need addressing. The master
  should be able to tell what the interrupt was about. Usually you use
  multiple interrupt lines, a one-hot addressing scheme. Interrupt 1 means
  NIC is ready and interrupt 2 means the printer is burning;

• like in a normal bus protocol, you need routing/arbitration.
  With interrupts, the master/slave relationship is inverted. Here the
  slaves send requests to the masters. So you need to decide which master
  receives the interrupt from a given slave. Unlike the preceding points,
  this doesn't really have a standard solution. Every system is different.
  If you have one master, it's simple. Once you have multiple masters you
  need to decide how the interrupts are wired to the masters. Often you
  need this to be reconfigurable at runtime. For example, the NIC might be
  handled by the LM32 in-chip or the host CPU off-chip;

• like in a normal bus protocol, you need a description language.
  When your software starts up it needs to determine which slave is
  connected to which master

• you lose plugability with the original bus protocol: interrupts
  are a completely distinct bus and they cannot be plugged together with
  devices that just use the vanilla bus protocol.
  \end{itemize}

Recently, the PCI SIG decided to obsolete "legacy interrupts" and
replace them with "message-signalled interrupts". MSI is a procedure for
using the existing PCIe bus for sending interrupts. It essentially says:
if you need a slave to talk to a master just send requests as a
master yourself.

This approach has a few advantages: it allows an unlimited
number of interrupt lines (addressing), without the need to share lines. It
makes possible for all participants on the PCIe bus to receive
interrupts if they desired (routing). There is a performance
benefit since it requires less messaging than legacy interrupts
(acknowledgement). Indeed, now that PCIe uses a message-oriented
protocol, the pin count advantage of legacy interrupts is gone.
Finally, legacy interrupts can be implemented on top of the
MSI scheme.

FPGA designers face exactly the same decision that the PCI SIG faced:
they are designing a SoC. Pin count is not an issue because synthesis
eliminates any pins that are not used, and inside the chip there is plenty of
freedom.

In the core that is expected to handle the ``interrupt'',
we can add a bus slave interface instead of a "raise
interrupt" pin. At first this may seem just a clever hack that
saves some time. However, we realized this brings in more and more
advantages:

• acknowledgements are handled just like normal bus acknowledgment.
  For Wishbone, for example, when the host
  system is done processing the interrupt, it raises the ack line. This
  means interrupts are discrete, like in MSI. You can say "I have 5 things
  for you to do" and not have to squish them into a "I raise this line
  until you do everything I want". This means you can choose to handle
  some of events that generated interrupts, but delay processing the
  others. If the master stops handling interrupts, acks stop flowing
  and no code is needed.

• there's no need for special interrupt masking etc. If the
  master doesn't
  want to process an interrupt right now, it just lets the stall line go high.
  From the software side, this requires nothing special at all. If the master
  is not reading, the queue fills and the line goes high. Again no node
  is needed;

• there is an unlimited number of interrupts. Each address+data
  pair can be interpreted differently;

• nothing special at all is needed to route interrupts. They
  are just another master on the crossbar. If a master wants to receive
  interrupts from a particular slave, it just writes its address to a
  register on the device. When that slave generates an interrupt, it is a
  write to the specified address and the master gets the interrupt. This completely
  solves the problem of determining which of multiple slaves
  raises which interrupts;

• by using the same protocol for interrupts as for the normal bus,
  modularity is improved. For example we can make software bus slaves and
  hardware masters can read/write
  memory from software programs in userspace on the host system;

• since interrupts are generated by bus writes, they are compatible
  with a remote protocol like Etherbone or other ones.
  You can remotely trigger an interrupt on any device in the
  network.

The only argument we can see in favour of legacy interrupts on a SoC is
that they are "simpler". A slave just needs to write a '1' or a '0' to
an output as opposed to adding a bus master interface. However,
this ``simplicity'' argument ignores the costs that happen once you have
enough of these devices, once the
interrupt bus becomes non-trivial. To keep the code simple, one could
easily imagine a small HDL component that takes an address register and
a generates a write upon request.

To use MSI and achieve compatibility with a legacy master like
a soft-core with an ``interrupt'' input line, you just need an
"interrupt unit" which is a bus slave that raises different pins when it
is written to at different offsets. In other words, just like in PCIe,
implement legacy interrupts using the MSI approach is pretty simple.

To summarize: interrupt lines are not strictly in the scope of SDB and
we currently don't offer such description because we really think FPGA
designs should be converted to the concept of MSI interrupts -- which need
no description support within SDB as a side effect.

On the other hand, it may make sense to define SDB structures to
describe this special wiring, to help existing ``legacy'' projects to
benefit from SDB and avoid doping the software source code with static
information about device wiring. Maybe future releases of this specification
will allow description of legacy interrupts.