Introduction
In production, once the SPEC7 is deployed as a PCIe slave and in order to perform proper remote upgrades, we would need to access the QSPI from the host computer via the PCIe. Unfortunately, only the Processing System in the Zynq-7000 is able to access the QSPI, while the PCIe in the Zynq-7000 is located in the Programmable Logic.
In the SPEC7 reference gateware designs from Nikhef, we have that PCIe BAR0 is 1 MiB and is mapped to 0x4000_0000 address range, i.e. the AXI bus HPC0 addressing space that has Programmable Logic as slave:
- the lower 512KiB are connected to the White Rabbit complex.
- the upper 512KiB are intended to be used to communicate the Host and the Zynq PS.
In these reference designs, the Vivado block designs performs a direct connection between the translated bus from BAR0 to the Processing System, but this cannot be done directly because the address range doesn't point to any actual PS peripheral or physical memory.
PL Block RAM Bridge
As a quick and dirty test, we modify the provided Vivado Block Designs to introduce a 512KiB Block RAM in PL acting as a bridge between PS and PCIe.
Note that we could access 0x40080000
from PS once we have an actual peripheral in the mapped address region, and we have checked this actually works, but we are using an additional AXI interface at 0x40100000
to avoid for potential bus collisions if PCIe and PS try to access the same address at the same time.
Testing the bridge
In order to test the block RAM bridge in PL, we will use:
- U-Boot at SPEC7: the commands to write and read to the AXI address space
-
PCIe Host Computer: the userspace
pcimem
program to write and read in the associated BAR.
From U-Boot to PCIe Host
Write from U-Boot
SPEC7> mw.l 0x40100000 0xdeadbeef 0x1
SPEC7> md.l 0x40100000 0x1
40100000: deadbeef ....
Read from PCIe Host
jdgarcia@cfc-774-cdv35:~>sudo ./pcimem /sys/bus/pci/devices/0000\:01\:00.0/resource0 0x80000 w
/sys/bus/pci/devices/0000:01:00.0/resource0 opened.
Target offset is 0x80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000)
PCI Memory mapped to address 0x7faa8ebbe000.
0x80000: 0xDEADBEEF
From PCIe Host to U-Boot
Write from PCIe Host
jdgarcia@cfc-774-cdv35:~>sudo ./pcimem /sys/bus/pci/devices/0000\:01\:00.0/resource0 0x80000 w 0xCAFEBABE
/sys/bus/pci/devices/0000:01:00.0/resource0 opened.
Target offset is 0x80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000)
PCI Memory mapped to address 0x7f98b8a73000.
0x80000: 0xDEADBEEF
Written 0xCAFEBABE; readback 0xCAFEBABE
Read from U-Boot
SPEC7> md.l 0x40100000 0x1
40100000: cafebabe ....