- Introduction
- Gateware modifications
- DDR tests driven by the Processing System
- DDR tests driven by the Host Computer via PCIe DMA
Introduction
In this section, we will introduce the design modifications and procedures that we have conducted in order to verify the DDR3 memory attached to the Zynq-7000 Programmable Logic in the SPEC7 board.
To make the things simpler, we have used the Xilinx Memory Interface Generator (MIG) for 7th Series FPGA in the Vivado Block Design IP Integrator. For a more straightforward integration, we let the IP-Core to generate a proper AXI slave interface that can be easily attached to both the Processing System and the XDMA PCIe subsystem. In this way, we will be able to perform:
- Memory tests driven by the Processing System using simple write/read operations
- Memory tests driven by the Host Computer using PCIe DMA based file transferences.
In both the SPEC7 Golden Boot gateware and the Nikhef SPEC7 Reference and Write designs, the gatewares we have modified to test the DDR3, we have peripherals attached to the General Purpose AXI Master Interface 0 (GPM0). As the addressing range for this bus is 1 GiB, exactly the same size the DDR3 attached to the Programmable Logic has, we cannot connect the PL memory to this bus without overlapping the previously existing AXI slaves addressing spaces. For this reason, we will enable the General Purpose AXI Master Interface 1 (GPM1) and will use its entire 1 GiB addressing range (Low Address 0x8000_0000
, High Address 0xBFFF_FFFF
) to cover the whole DDR3 attached to the Programmable Logic.
For our tests, we have used a SPEC7 card powered by a XC7Z030, but all of the work we have done is intended to be compatible with designs mounting the XC7Z035 and XC7Z045.
Before moving forward, we advance that all of the tests described in this page have been successfully executed in the DDR3 attached to the Programmable Logic without finding any error, so we can conclude that the design is working as expected.
Gateware modifications
In this section we explain the modifications that we have introduced to the SPEC7 gateware designs in order to connect the DDR3 attached to the Programmable Logic and perform the validation tests.
Configuration for the Memory Interface Generator (MIG)
As previously commented, we are integrating the DDR3 controller as an IP-Core in the Vivado Block Design. Once the properly configured IP-Core has been generated, it can simply be exported as part of the TCL script used to regenerate the entire Block Design.
In order to allow for easier modification of future gateware designs, we will go through the entire process of configuring the MIG IP-Core to generate a controller matching the DDR3 attached to the Programmable Logic.
First we need to add a Memory Interface Generator (MIG 7 Series) in our Block Design. Then, we double click in the freshly added core to introduce the configuration. This will show the following introductory screen that we can safely skip by pressing Next
:
Then, we choose to create a new design with a single controller and AXI interface:
After this, we will be offered to select the pin compatible FPGAs that we want to support with our memory controller. As we are using the XC7Z030 device for our tests but the production SPEC7 will mount the XC7Z035 and XC7Z045, we select to support the three devices:
Then, we need to specify that our memory selection is of DDR3 class:
Once done, we proceed with the controller options, that will match the dual MT41K25616XX-107
configuration attached to the SPEC7 Programmable Logic. Note that we will use 3000ps as the clock period as this is the maximum allowed frequency for this device working at 1.35V:
Next, we select the proper AXI configuration that will allow us to integrate the memory controller into the block design. We choose 32bits as the Data Width as this will allow us to connect easily with both the Processing System and the XDMA PCIe:
Now we need to configure the memory options. We will left all of the values unchanged but the Input Clock Period, that we set to 4000ps (250MHz) so that we can easily generate this clock from several sources such as the Processing System to Programmable Logic clocks:
In the FPGA options menu, we will set the System and Reference clocks to No Buffer
as our design has not dedicated external clock sources for the DDR3 attached to the Programmable Logic. We also left the System Reset to Active Low
so we can easily connect this to different reset signals such as the ones generated by the Processing System:
Next, in the extended FPGA options, we can left the values unchanged as the Digital Controlled Impedance applies and the 50 Ohms Internal Termination Impedance matches our design:
Now we need to select the IO Planning Options between bank and pink selection modes. Because our design is fixed, we can proceed with the Fixed Pin Out
option:
Once we have selected the fixed pin out, we need to proceed with the proper Pin Selection that matches our specific design. When we do this for the first time, we need to manually introduce the Bank, Byte and Pin Numbers for each of the signals in the physical DDR3 interface. When the process is complete, we can click the save pin out
button to export the pin selection as an UCF file so that we can use this later in new configurations by just using the Read XDC/UCF
button. Specifically, to make things easier, this is the UCF file containing the proper DDR3 pin selection for the SPEC7:
Finally, we need to configure the System Signals Selection. For the SPEC7 design, we can safely left sys_rst
, init_calib_complete
and tg_compare_error
unconnected as there are no dedicated pins in the board for them:
From now onward, we will just be offered several informative screens and a license acceptance dialog until we reach the Generate button that we will use to create the DDR3 controller from our custom configuration.
Modifications for SPEC7 Golden Boot gateware
In our first tests for the DDR3 attached to the Programmable Logic, we used the Golden Boot gateware. In order to do this, we performed several changes to the default Vivado Block Design.
First, we need to create the DDR3 controller following the procedure depicted in the previous section.
Then, in the Processing System block, we need to enable the following additional signals:
-
Enable the General Purpose AXI Master Interface 1 (GPM1): this will be attached to the AXI interface of the DDR3 controller, so the 1GiB DDR will be mapped to the GPM1 address range (we use
Run Connection Automation
so that the companion signals are connected too). -
Enable the PL Fabric Clock 1 (FCLK_CLK1): we set the configuration for this clock as Clock Source
IO PLL
and Requested Frequency to250 MHz
so that we can drive both the System and Reference clocks in the DDR3 controller. -
Enable the General Purpose Reset Signal 1 for PL logic (FCLK_RESET1_N): this signal will be used to drive the
sys_rst
signal in the DDR3 controller.
By doing this, we will be able to access from the Processing System to the DDR3 memory connected in the Programmable Logic by just writing and reading in the following AXI bus address range spanning the whole GPM1 space:
- Low Address:
0x8000_0000
- High Address:
0xBFFF_FFFF
In order to do this, we just need to remove this line in the spec7_wrapper.tcl
generation script:
set_property -dict [list CONFIG.Pcie_fast_config {Tandem_PROM (Refer PG054)}] [get_bd_cells pcie_7x]
Unfortunately, the rationale behind using the Boot Golden gateware is to enable the Tandem PROM mode in a configuration that mimics the one from XDMA enabled designs that are loaded in a later stage, allowing in this way the reliable enumeration of the SPEC7 as a PCIe slave. For this reason, it makes no sense to include the DDR3 controller as an AXI peripheral in the stable Boot Golden gateware.
In any case, if we want to use the DDR3 controller in XDMA enabled design loaded in a second stage, we will need to be sure that the First Stage Bootloader (FSBL) has been generated from an exported hardware enabling the additional interfaces and signals in the Processing System included in this section. By doing this, we will be sure that the Zynq-7000 is able to use the required resources; otherwise, it would lead to catastrophic failures, e.g. the Processing System will hang when trying to access the GPM1 addressing space if this bus is not enabled in the FSBL.
Modifications for Nikhef SPEC7 Reference and Write designs
At the time of performing more in depth tests for the DDR3 attached to the Programmable Logic, we used both Nikhef Reference and Write designs. In order to do this, we performed several changes to the shared Vivado Block Design that were extended to several other files.
We are introducing two modified design versions:
- A limited bandwidth version in which the DDR at PL can be accessed from the Processing System and the XDMA.
- A improved bandwidth version in which the DDR at PL can only be accessed from the XDMA.
DDR3 at Programmable Logic accessible from Processing System and XDMA
First, we need to create the DDR3 controller following the procedure depicted in the previous section.
Then, in the Processing System block, we need to enable the following additional signals:
-
Enable the General Purpose AXI Master Interface 1 (GPM1): this will be attached to the AXI interface of the DDR3 controller, so the 1GiB DDR will be mapped to the GPM1 address range (we use
Run Connection Automation
so that the companion signals are connected too). -
Enable the PL Fabric Clock 1 (FCLK_CLK1): we set the configuration for this clock as Clock Source
IO PLL
and Requested Frequency to250 MHz
so that we can drive both the System and Reference clocks in the DDR3 controller. -
Enable the General Purpose Reset Signal 1 for PL logic (FCLK_RESET1_N): this signal will be used to drive the
sys_rst
signal in the DDR3 controller.
By doing this, we will be able to access from the Processing System to the DDR3 memory connected in the Programmable Logic by just writing and reading in the following AXI bus address range spanning the whole GPM1 space:
- Low Address:
0x8000_0000
- High Address:
0xBFFF_FFFF
FCLK_CLK1
generated by the Processing System. In order to solve this, we can introduce an intermediate global clock buffer by using the Utility Buffer IP:
[DRC PLIDC-9] IDELAYCTRL REFCLK driver not buffer check: REFCLK pin of IDELAYCTRL cell 'Pcie/processing_system_pcie_i/mig_7series_0/u_processing_system_pcie_mig_7series_0_0_mig/u_iodelay_ctrl/u_idelayctrl_200' is driven by 'Pcie/processing_system_pcie_i/processing_system7_0/inst/PS7_i' {PS7}. This will lead to an unroutable situation. The REFCLK pin of an IDELAYCTRL cell should always be driven by clock buffer.
Finally, in order to test the DDR3 attached to the Programmable Logic with DMA write and read requests initiated by the host computer through the XDMA PCIe core, we need to:
- Disconnect the established AXI Stream loopback interface.
- Reconfigure the XDMA IP Core to select the AXI Memory Mapped option in the DMA Interface.
- Connect the newly created and DMA driven
M_AXI
interface to the DDR3 memory controller slave using theRun Connection Automation
menu.
By doing this, the XDMA AXI Memory Mapped interface will be able to access the following address range in which the DDR3 resides:
- Low Address:
0x0000_0000_8000_0000
- High Address:
0x0000_0000_BFFF_FFFF
Once we have introduced all of the proposed changes, the resulting Block Design will look similar to this:
After propagating the changes introduced in the block design, we need to modify the following files, all of them coming from the wr-cores
repository:
./hdl/wr-cores/board/spec7/wr_spec7_pkg.vhd
./hdl/wr-cores/platform/xilinx/wr_pcie/processing_system_pcie.tcl
./hdl/wr-cores/top/spec7_ref_design/spec7_wr_ref_top.vhd
./hdl/wr-cores/top/spec7_write_design/spec7_write_top.vhd
All of the introduced changes for Nikhef SPEC7 Reference and Write designs for DDR3 testing purposes with PS and XDMA are now stored in the jgarcia-spec7-ddr-pl
branch of the wr-cores
repository, that has been created straight from the proposed_spec7
branch.
In this way, in order to generate the modified Nikhef designs supporting the DDR3 attached to the Programmable Logic, we need to follow this procedure. First, we clone the sources and fetch the appropriated submodules:
git clone https://gitlab.nikhef.nl/peterj/spec7.git
cd spec7
git checkout proposed_master
git submodule init
git submodule update
cd hdl/wr-cores
git checkout jgarcia-spec7-ddr-pl
git submodule init
git submodule update
cd ip_cores/general-cores
git checkout peter_proposed_spec7_v5
Now, we get into the appropriated syn
folder for the Reference or Write design:
# SPEC7 Reference design
cd hdl/spec7_ref_design/syn
# SPEC7 Write design
cd hdl/spec7_write_design/syn
Then, we change this file to select the target Zynq device we want to use:
vi proj_properties.tcl
e.g. we enable the XC7Z030
based design (note that this supported by Vivado Webpack license, while the default XC7Z035
is not supported):
#set device xc7z035fbg676-1
set device xc7z030fbg676-1
Once done, we launch Vivado from inside the syn
folder:
source /tools/Xilinx/Vivado/2019.2/settings64.sh
export LC_ALL="C"
vivado
And finally, from the TCL console in Vivado, we source the following script so that the chosen design is automatically generated:
source ../../../sw/scripts/viv_do_all.tcl
DDR3 at Programmable Logic accessible from XDMA only with Improved Bandwidth
In those designs in which we are allowing access to the DDR3 at Programmable Logic from the Processing System, we are using a 32 Bit data width in the AXI DDR3 controller. This introduces a bottleneck that avoids using the full bandwidth provided by the combined PCIe and DMA system at the time of accessing the DDR3 at Programmable Logic.
In order to allow for higher bandwidth and perform a more intensive test on the DDR3 attached to the Programmable Logic, we can use a 256 bits data width in the AXI interface of the DDR3 controller and limit the access to the XDMA interface only.
First, in the Processing System block, we need to enable the following additional signals:
-
Enable the PL Fabric Clock 1 (FCLK_CLK1): we set the configuration for this clock as Clock Source
IO PLL
and Requested Frequency to250 MHz
so that we can drive both the System and Reference clocks in the DDR3 controller (remember that we need to introduce an intermediate global clock buffer by using the Utility Buffer IP). -
Enable the General Purpose Reset Signal 1 for PL logic (FCLK_RESET1_N): this signal will be used to drive the
sys_rst
signal in the DDR3 controller.
Then, in the MIG generator, we use the same configuration previously explained but modify the AXI DATA WIDTH value from 32 to 256 in the AXI Parameter dialog. In this way, we will dramatically increase the allowed bandwidth for the DDR3 controller:
Finally, in order to test the DDR3 attached to the Programmable Logic with DMA write and read requests initiated by the host computer through the XDMA PCIe core, we need to:
- Disconnect the established AXI Stream loopback interface.
- Reconfigure the XDMA IP Core to:
- Select the AXI Memory Mapped option in the DMA Interface.
- In the AXI Clock Frequency select the 250 MHz checkbox.
- Connect the newly created and DMA driven
M_AXI
interface to the DDR3 memory controller slave using theRun Connection Automation
menu.
By doing this, the XDMA AXI Memory Mapped interface will be able to access the following address range in which the DDR3 resides:
- Low Address:
0x0000_0000_8000_0000
- High Address:
0x0000_0000_BFFF_FFFF
Once we have introduced all of the proposed changes, the resulting Block Design will look similar to this:
After propagating the changes introduced in the block design, we need to modify the following files, all of them coming from the wr-cores
repository:
./hdl/wr-cores/board/spec7/wr_spec7_pkg.vhd
./hdl/wr-cores/platform/xilinx/wr_pcie/processing_system_pcie.tcl
./hdl/wr-cores/top/spec7_ref_design/spec7_wr_ref_top.vhd
./hdl/wr-cores/top/spec7_write_design/spec7_write_top.vhd
All of the introduced changes for Nikhef SPEC7 Reference and Write designs for DDR3 testing purposes with high-bandwidth XDMA are now stored in the jgarcia-spec7-ddr-pl-256-bit
branch of the wr-cores
repository, that has been created straight from the proposed_spec7
branch.
In this way, in order to generate the modified Nikhef designs supporting the DDR3 attached to the Programmable Logic, we need to follow this procedure. First, we clone the sources and fetch the appropriated submodules:
git clone https://gitlab.nikhef.nl/peterj/spec7.git
cd spec7
git checkout proposed_master
git submodule init
git submodule update
cd hdl/wr-cores
git checkout jgarcia-spec7-ddr-pl-256-bit
git submodule init
git submodule update
cd ip_cores/general-cores
git checkout peter_proposed_spec7_v5
Now, we get into the appropriated syn
folder for the Reference or Write design:
# SPEC7 Reference design
cd hdl/spec7_ref_design/syn
# SPEC7 Write design
cd hdl/spec7_write_design/syn
Then, we change this file to select the target Zynq device we want to use:
vi proj_properties.tcl
e.g. we enable the XC7Z030
based design (note that this supported by Vivado Webpack license, while the default XC7Z035
is not supported):
#set device xc7z035fbg676-1
set device xc7z030fbg676-1
Once done, we launch Vivado from inside the syn
folder:
source /tools/Xilinx/Vivado/2019.2/settings64.sh
export LC_ALL="C"
vivado
And finally, from the TCL console in Vivado, we source the following script so that the chosen design is automatically generated:
source ../../../sw/scripts/viv_do_all.tcl
DDR tests driven by the Processing System
In all of the previously introduced gatewares, we have mapped the DDR3 memory attached to the Programmable Logic as an AXI peripheral covering the complete GPM1 addressing range, i.e.:
- Low Address:
0x8000_0000
- High Address:
0xBFFF_FFFF
In this way, we can use software running in the Zynq-7000 ARM Cortex-A9 processors at the Processing System to perform different tests by directly executing write and read operations across the full DDR3 memory range.
U-Boot based tests
From the U-Boot customized for the SPEC7, we can easily verify that we are able to successfully perform read and write operations in the DDR3 by using the provided commands:
-
mw
: memory write. -
md
: memory dump.
In this way, we can write a known pattern to a known DDR3 location and then dump the contents to verify that the values are the same, e.g.:
U-Boot 2019.01-ge305ce8 (Feb 18 2021 - 15:50:32 +0100) Nikhef SPEC7
CPU: Zynq 7z030
Silicon: v3.1
Model: Nikhef SPEC7 board
DRAM: ECC disabled 1 GiB
MMC: mmc@e0100000: 0
Loading Environment from SPI Flash... SF: Detected n25q256 with page size 512 Bytes, erase size 8 KiB, total 64 MiB
*** Warning - bad CRC, using default environment
In: serial@e0000000
Out: serial@e0000000
Err: serial@e0000000
Net: ZYNQ GEM: e000b000, phyaddr 0, interface rgmii-id
Warning: ethernet@e000b000 using MAC address from ROM
eth0: ethernet@e000b000
464 bytes read in 10 ms (44.9 KiB/s)
Importing environment from SD ...
Hit any key to stop autoboot: 0
SPEC7> mw.l 0x80000000 0xdeadbeef 0x1
SPEC7> md.b 0x80000000 0x1
80000000: ef .
SPEC7> md.l 0x80100000 0x1
80000000: deadbeef ....
SPEC7> mw.l 0x90000000, 0xdeadbeef 0x1
SPEC7> md.l 0x90000000 0x1
90000000: deadbeef ....
SPEC7> mw.l 0xA0000000, 0xdeadbeef 0x1
SPEC7> md.l 0xA0000000 0x1
a0000000: deadbeef ....
SPEC7> mw.l 0xB0000000, 0xdeadbeef 0x1
SPEC7> md.l 0xB0000000 0x1
b0000000: deadbeef ....
SPEC7>
Once we have verified that we can access the DDR3, we can proceed with an automated test. In orden to do this, we can use the mtest
command provided by U-Boot too:
mtest [start [end [pattern [iterations]]]]
In which the rationale for the different arguments are:
-
start
is the first address of the memory range we want to test. -
end
needs to be adjusted to the last 4-bytes address that will be actually tested. see this comment from Denx -
pattern
is the known 32 bit data we want to write and then read from memory. -
iterations
is the number of times we want to repeat the test of the provided memory range.
mtest
arguments need are read by the tool as hexadecimal numbers.
As an example, this is the required command to test the whole memory address once with a known pattern:
SPEC7> mtest 0x80000000 0xBFFFFFFC 0XA5A5A5A5 1
Testing 80000000 ... bffffffc:
Pattern A5A5A5A5 Writing... Reading...Tested 1 iteration(s) with 0 errors.
By running mtest
with several pattern values, we verified the DDR3 attached to the Programmable Logic for the first time.
In any case, mtest
is too slow and non a very powerful option. This is an excerpt from U-Boot docs:
2. The "mtest" command.
This is probably the best known memory test utility in U-Boot.
Unfortunately, it is also the most problematic, and the most
useless one.
There are a number of serious problems with this command:
- It is terribly slow. Running "mtest" on the whole system RAM
takes a _long_ time before there is any significance in the fact
that no errors have been found so far.
- It is difficult to configure, and to use. And any errors here
will reliably crash or hang your system. "mtest" is dumb and has
no knowledge about memory ranges that may be in use for other
purposes, like exception code, U-Boot code and data, stack,
malloc arena, video buffer, log buffer, etc. If you let it, it
will happily "test" all such areas, which of course will cause
some problems.
- It is not easy to configure and use, and a large number of
systems are seriously misconfigured. The original idea was to
test basically the whole system RAM, with only exempting the
areas used by U-Boot itself - on most systems these are the areas
used for the exception vectors (usually at the very lower end of
system memory) and for U-Boot (code, data, etc. - see above;
these are usually at the very upper end of system memory). But
experience has shown that a very large number of ports use
pretty much bogus settings of CONFIG_SYS_MEMTEST_START and
CONFIG_SYS_MEMTEST_END; this results in useless tests (because
the ranges is too small and/or badly located) or in critical
failures (system crashes).
Because of these issues, the "mtest" command is considered depre-
cated. It should not be enabled in most normal ports of U-Boot,
especially not in production. If you really need a memory test,
then see 1. and 3. above resp. below.
Vitis standalone Memory Tests template
At the time of validating the DDR3 attached to the Programmable Logic by using the ARM Cortex-A9 in the Zynq-7000 Processing System, we can use the Memory Tests standalone application template provided by Xilinx Vitis.
In order to do this, we just need to create a Vitis Platform from an exported Vivado design supporting the DDR3 in the Programmable Logic as an AXI peripheral, e.g. the modified SPEC7 golden boot or Nikhef Reference and Write designs. Then we need to create a FSBL and a standalone application from the Memory Tests template.
Once the Memory Tests application has been created, we can see in the memorytest.c
file that the procedure consists in writing a known pattern and then read and check the result across the whole memory range for 32, 16 and 8 bits words.
status = Xil_TestMem32((u32*)range->base, 1024, 0xAAAA5555, XIL_TESTMEM_ALLMEMTESTS);
print(" 32-bit test: "); print(status == XST_SUCCESS? "PASSED!":"FAILED!"); print("\n\r");
status = Xil_TestMem16((u16*)range->base, 2048, 0xAA55, XIL_TESTMEM_ALLMEMTESTS);
print(" 16-bit test: "); print(status == XST_SUCCESS? "PASSED!":"FAILED!"); print("\n\r");
status = Xil_TestMem8((u8*)range->base, 4096, 0xA5, XIL_TESTMEM_ALLMEMTESTS);
print(" 8-bit test: "); print(status == XST_SUCCESS? "PASSED!":"FAILED!"); print("\n\r");
Now, the devices and memory ranges that the memory test application checks is automatically extracted from the Vitis platform we previously created. For any of the gateware designs covered in this wiki, this information is contained in the automatically generated memory_config_g.c
file:
/* This file is automatically generated based on your hardware design. */
#include "memory_config.h"
struct memory_range_s memory_ranges[] = {
{
"mig_7series_0_memaddr",
"mig_7series_0",
0x80000000,
1073741824,
},
{
"ps7_ddr_0",
"ps7_ddr_0",
0x00100000,
1072693248,
},
/* ps7_ram_0 memory will not be tested since application resides in the same memory */
{
"ps7_ram_1",
"ps7_ram_1",
0xFFFF0000,
65024,
},
};
int n_memory_ranges = 3;
We can see that there initially are three different memory ranges to be checked plus a discarded memory range:
-
mig_7series_0_memaddr
: the DDR3 memory attached to the Programmable Logic, to be checked. -
ps7_ddr_0
: the DDR memory attached to the Processing System, to be checked. -
ps7_ram_0
: low-mapped On Chip Memory (OCM) blocks, where the application will reside. -
ps7_ram_1
: high-mapped On Chip Memory (OCM) blocks, to be checked.
Unfortunately, the application doesn't fit in the ps7_ram_0
once built. In order to solve this, we can:
- perform the test only for the
mig_7series_0_memaddr
. - move the memory test application to
ps7_ddr_0
.
In this way, we first modify the memory_config_g.c
file content to this:
/* This file is automatically generated based on your hardware design. */
#include "memory_config.h"
struct memory_range_s memory_ranges[] = {
{
"mig_7series_0_memaddr",
"mig_7series_0",
0x80000000,
1073741824,
},
};
int n_memory_ranges = 1;
Then, by double clicking the lscript.ld
file, a GUI menu will be opened so that we can edit the location for all of the components of our standalone application to the now not tested Processing System DDR. Specifically, we need to assign ps7_ddr_0
as the Memory Region for all of the application Sections:
lscript.ld
file to use the desired memory.
Once we have built the memory test standalone application, we can include it in a boot image to be run by the SPEC7. In this way, we can check the results of the performed tests from the SPEC7 UART, e.g.:
--Starting Memory Test Application--
NOTE: This application runs with D-Cache disabled.As a result, cacheline requests will not be generated
Testing memory region: mig_7series_0_memaddr
Memory Controller: mig_7series_0
Base Address: 0x80000000
Size: 0x40000000 bytes
32-bit test: PASSED!
16-bit test: PASSED!
8-bit test: PASSED!
--Memory Test Application Complete--
DDR tests driven by the Host Computer via PCIe DMA
In this section, we will introduce the more demanding DMA based tests driven by the host computer via the PCIe that we have conducted to validate the DDR3 memory attached to the Programmable Logic in the SPEC7.
These test have been successfully executed in the modified Nikhef SPEC7 Reference and Write designs, as the XDMA IP Core is required to access the DDR3 from the Front End Computers.
In order to do this, we have built and used in the host computer the Kernel and User Space components provided by the Xilinx PCIe DMA drivers for Vivado 2019.2.
Once we have build the Xilinx drivers (the specific procedure depends on the host Linux Operating System distribution), we can load the required XDMA kernel module by:
sudo insmod dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
Once the Kernel module is loaded, we can check the SPEC7 is recognized and using the appropriated module by reviewing the output from the lspci
command, e.g.:
05:00.0 Serial controller: Xilinx Corporation Device 7022 (prog-if 01 [16450])
Subsystem: Xilinx Corporation Device 0007
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at a1600000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at a1700000 (32-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: xdma
In addition, if everything works properly, we can verify that the following associated device nodes have been created:
jdgarcia@cfc-774-cdv35:~>ls /dev/xdma*
/dev/xdma0_c2h_0 /dev/xdma0_events_12 /dev/xdma0_events_4 /dev/xdma0_h2c_0
/dev/xdma0_control /dev/xdma0_events_13 /dev/xdma0_events_5 /dev/xdma0_user
/dev/xdma0_events_0 /dev/xdma0_events_14 /dev/xdma0_events_6 /dev/xdma0_xvc
/dev/xdma0_events_1 /dev/xdma0_events_15 /dev/xdma0_events_7
/dev/xdma0_events_10 /dev/xdma0_events_2 /dev/xdma0_events_8
/dev/xdma0_events_11 /dev/xdma0_events_3 /dev/xdma0_events_9
The most interesting of these device nodes are:
-
/dev/xdma0_control
: internal control register space for the XDMA core. -
/dev/xdma0_user
: memory mapped character device that provides access to the XDMA AXI Lite interface. -
/dev/xdma0_h2c_0
: Host to Card descriptor that provides access to the DMA capable XDMA AXI Master in memory mapped mode. -
/dev/xdma0_c2h_0
: Card to Host descriptor that provides access to the DMA capable XDMA AXI Master in memory mapped mode.
We can found more detailed information on the intended use of each of the devices nodes created by the XDMA driver here:
In any case, /dev/xdma0_h2c_0
and /dev/xdma0_c2h_0
are the devices in which we are interested at the time of conducting DMA based tests driven by the host computer. At the time of using this devices, we can use the dma_to_device
and dma_from_device
userspace tools that Xilinx provides in the same software package.
The dma_to_device
tool will be used to write a binary file from the host computer to the SPEC7 DDR3:
dgarcia@cfc-774-cdv35:~/spec7_dma> ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_to_device -h
./dma_ip_drivers/XDMA/linux-kernel/tools/dma_to_device
usage: ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_to_device [OPTIONS]
Write via SGDMA, optionally read input from a file.
-d (--device) device (defaults to /dev/xdma0_h2c_0)
-a (--address) the start address on the AXI bus
-k (--aperture) memory address aperture
-s (--size) size of a single transfer in bytes, default 32,
-o (--offset) page offset of transfer
-c (--count) number of transfers, default 1
-f (--data infile) filename to read the data from.
-w (--data outfile) filename to write the data of the transfers
-h (--help) print usage help and exit
-v (--verbose) verbose output
Return code:
0: all bytes were dma'ed successfully
< 0: error
The dma_from_device
tool will be used to read a binary file from the SPEC7 DDR3 to the host computer:
jdgarcia@cfc-774-cdv35:~/spec7_dma> ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_from_device -h
./dma_ip_drivers/XDMA/linux-kernel/tools/dma_from_device
usage: ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_from_device [OPTIONS]
Read via SGDMA, optionally save output to a file
-d (--device) device (defaults to /dev/xdma0_c2h_0)
-a (--address) the start address on the AXI bus
-k (--aperture) memory address aperture
-s (--size) size of a single transfer in bytes, default 32.
-o (--offset) page offset of transfer
-c (--count) number of transfers, default is 1.
-f (--file) file to write the data of the transfers
-e (--eop_flush) end dma when ST end-of-packet(eop) is rcved
* streaming only, ignored for memory-mapped channels
* acutal # of bytes dma'ed could be smaller than specified
-h (--help) print usage help and exit
-v (--verbose) verbose output
Return code:
0: all bytes were dma'ed successfully
* with -e set, the bytes dma'ed could be smaller
< 0: error
root
user and sudo
commands are only allowed to write in the /tmp
folder on CERN's Front End Computers.
Manual DMA based tests
As a first approach of testing the DDR attached to the Programmable Logic from the host computer using the DMA, we can use a manual procedure consisting in the following steps:
- Create an input sample binary file in the host.
- Calculate the hash for the input sample binary file using
md5sum
. - Upload the input file to the SPEC7 DDR3 using the Host-to-Card DMA device node.
- Download the DDR3 contents to a new output file using the Card-to-Host DMA device node.
- Calculate the hash for the output sample binary file using
md5sum
. - Compare the hashes for the input and output binary files to detect potential errors.
- If an error is detected, compare the contents of the input and output files to identify the error location.
Now, we can go through this steps, starting with the creation of an input sample binary file of known size. In order to do this, we can use the dd
command.
As a first example, we can create a 64 KiB file with random content by using the following command:
dd if=/dev/urandom of=/tmp/data_64K_in.bin bs=64K count=1 iflag=fullblock
In the same way, we can create a 64 KiB file with a known 32 bit pattern by using the following command. Note that, at the time of creating the pattern, we need to perform byte swapping so that the values are written in the correct order, e.g. for the 0xDEADBEEF
pattern:
while true ; do printf "\xEF\xBE\xAD\xDE"; done | dd of=/tmp/data_64K_in.bin bs=64K count=1 iflag=fullblock
In this way, here we have a complete example of executing the whole procedure for manual testing using the DMA captured from an actual test in which we are writing and reading 64KiB (65536 bytes) of data at the physical address 0xA0000000
:
jdgarcia@cfc-774-cdv35:~/spec7_dma>while true ; do printf "\xEF\xBE\xAD\xDE"; done | dd of=/tmp/data_64K_in.bin bs=64K count=1 iflag=fullblock
jdgarcia@cfc-774-cdv35:~/spec7_dma>md5sum /tmp/data_64K_in.bin
a15fd4979e6b14373d35827cd10d4833 /tmp/data_64K_in.bin
jdgarcia@cfc-774-cdv35:~/spec7_dma>sudo ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_to_device /dev/xdma0_h2c_0 -f /tmp/data_64K_in.bin -s 65536 -a 0xA0000000 -c 1
/dev/xdma0_h2c_0 ** Average BW = 65536, 233.433548
jdgarcia@cfc-774-cdv35:~/spec7_dma>sudo ./dma_ip_drivers/XDMA/linux-kernel/tools/dma_from_device /dev/xdma0_c2h_0 -f /tmp/data_64K_out.bin -s 65536 -a 0xA0000000 -c 1
/dev/xdma0_c2h_0 ** Average BW = 65536, 204.368271
jdgarcia@cfc-774-cdv35:~/spec7_dma>md5sum /tmp/data_64K_out.bin
a15fd4979e6b14373d35827cd10d4833 /tmp/data_64K_out.bin
We can check that the hashes for the input and output files match, so the upload and download DMA transactions have worked properly without any error.
Note that, in order to verify the whole DDR3 memory attached to the Programmable Logic, we would need to write and read data to the complete address range associated to it, i.e.:
- Low Address:
0x8000_0000
- High Address:
0xBFFF_FFFF
We can cover the whole address range by using two different approaches:
- Write and read a single 1GiB binary file at offset
0x8000_0000
- Write and read a smaller binary file at consecutive offsets till the whole memory is checked.
Before going forward, as the DDR3 connected to the Programmable Logic can be accessed from the software running at the Processing System too, we can check that the data has been actually written to the SPEC7 memory. For example, if we used the 0xDEADBEEF
pattern we can read the contents from U-Boot and check if they are as expected, e.g. check the content of the 64 KiB written at 0xA0000000
:
- 65536 bytes / 4 bytes = 16384 read cycles = 0x4000 read cycles
SPEC7> md.l 0xA0000000 0x4000
a0000000: deadbeef deadbeef deadbeef deadbeef ................
a0000010: deadbeef deadbeef deadbeef deadbeef ................
a0000020: deadbeef deadbeef deadbeef deadbeef ................
...
a000ffd0: deadbeef deadbeef deadbeef deadbeef ................
a000ffe0: deadbeef deadbeef deadbeef deadbeef ................
a000fff0: deadbeef deadbeef deadbeef deadbeef ................
SPEC7>
In the same way, if we are using random patterns, we can verify the contents of the uploaded binary file by using hexdump
and then check against the correspondent physical memory addresses from U-Boot.
As an example, the contents of a random 1 GiB binary file at offsets 0x00000000, 0x10000000, 0x20000000, 0x30000000
are:
jdgarcia@cfc-774-cdv35:~/spec7_dma>hexdump -C -s 0x00000000 -n 4 /tmp/data_1G_in.bin
00000000 5d fc b9 f1 |]...|
00000004
jdgarcia@cfc-774-cdv35:~/spec7_dma>hexdump -C -s 0x10000000 -n 4 /tmp/data_1G_in.bin
10000000 46 3a ea e9 |F:..|
10000004
jdgarcia@cfc-774-cdv35:~/spec7_dma>hexdump -C -s 0x20000000 -n 4 /tmp/data_1G_in.bin
20000000 35 24 b1 7f |5$..|
20000004
jdgarcia@cfc-774-cdv35:~/spec7_dma>hexdump -C -s 0x30000000 -n 4 /tmp/data_1G_in.bin
30000000 c0 38 ad b2 |.8..|
30000004
If the 1 GiB file is copied to the DDR base address, the same offsets are loaded to AXI physical addresses 0x80000000, 0x90000000, 0xA0000000, 0xB0000000
that can be checked with U-Boot:
SPEC7> md.l 0x80000000 0x1
80000000: f1b9fc5d ]...
SPEC7> md.l 0x90000000 0x1
90000000: e9ea3a46 F:..
SPEC7> md.l 0xA0000000 0x1
a0000000: 7fb12435 5$..
SPEC7> md.l 0xB0000000 0x1
b0000000: b2ad38c0 .8..
Finally, if we detect that the md5sum
for the input and output files don't match, we can detect the specific error offset by using a combination off:
-
xxd
: hexdump of binary files -
diff
: difference between files
In this way, xxd
creates a dump with 16 bytes per line and hex data, e.g.:
jdgarcia@cfc-774-cdv35:~>xxd /tmp/data_64K_in.bin
00000000: efbe adde efbe adde efbe adde efbe adde ................
00000010: efbe adde efbe adde efbe adde efbe adde ................
00000020: efbe adde efbe adde efbe adde efbe adde ................
00000030: efbe adde efbe adde efbe adde efbe adde ................
...
0000ffd0: efbe adde efbe adde efbe adde efbe adde ................
0000ffe0: efbe adde efbe adde efbe adde efbe adde ................
0000fff0: efbe adde efbe adde efbe adde efbe adde ................
And the diff
output for a changed hex value would be (note that this is a simulated diff, as we don't have detected any error in any DMA test!):
jdgarcia@cfc-774-cdv35:~>diff <(xxd /tmp/data_64K_in.bin) <(xxd /tmp/data_64K_out.bin)
146c146
< 00000910: efbe adde efbe adde efbe adde efbe adde ................
---
> 00000910: efbe adde efbe afde efbe adde efbe adde ................
jdgarcia@cfc-774-cdv35:~>diff <(xxd /tmp/data_1G_in.bin) <(xxd /tmp/data_1G_out.bin)
diff: memory exhausted
Automated DMA based tests
As we commented previously, if we use binary data sizes that are a fraction of the DDR3 attached to programmable Logic total size, we need to write, read and check the data sequentially in across the whole memory offset.
In order to perform the test for the complete DDR3 memory in an automated way, we can use a simple bash
script to execute this task. As an example, here we have an example script that allows for configuring the following parameters:
-
pattern_array
: list of all the 32 bit hexadecimal patterns that we want to test. -
LOG_FILE
: file containing the test outputs for the different patterns and offsets. -
DATAFILE_PREFIX
: path and name prefix for the input and output binary data files. -
DATAFILE_SIZE_MIB
: size in MiB for the generated datafiles (it must be an integer divisor of the memory size) -
DDR_SIZE_MIB
: size of the memory in MiB, i.e. 1024 for the SPEC7. -
XDMA_TOOLS_PATH
: path to the folder where the Xilinxdma_to_device
anddma_from_device
reside.
Note that we have executed this script several times with different block sizes and patterns and we have not detected any error.
pattern_array=(
a5a5a5a5
5a5a5a5a
ffffffff
00000000
f00ff00f
0ff00ff0
00000001
00000002
00000004
00000008
00000010
00000020
00000040
00000080
00000100
00000200
00000400
00000800
00001000
00002000
00004000
00008000
00010000
00020000
00040000
00080000
00100000
00200000
00400000
00800000
01000000
02000000
04000000
08000000
10000000
20000000
40000000
80000000
)
LOG_FILE=/tmp/ddr_pl_test.log
DATAFILE_PREFIX=/tmp/data_test
DATAFILE_SIZE_MIB=8
DATAFILE_SIZE_BYTES=$((1024*1024*$DATAFILE_SIZE_MIB))
DDR_SIZE_MIB=1024
BASE_OFFSET=2147483648
XDMA_TOOLS_PATH=/user/jdgarcia/spec7_dma/dma_ip_drivers/XDMA/linux-kernel/tools
function check_ddr {
pattern=$1
while true ; do printf "\x${pattern:6:2}\x${pattern:4:2}\x${pattern:2:2}\x${pattern:0:2}"; done \
| dd of=${DATAFILE_PREFIX}_in.bin bs=1M count=${DATAFILE_SIZE_MIB} iflag=fullblock
HASH_INPUT=($(md5sum ${DATAFILE_PREFIX}_in.bin))
echo "" >> $LOG_FILE
echo "TEST PATTERN: 0x$pattern" >> $LOG_FILE
IMAX=$(($DDR_SIZE_MIB/$DATAFILE_SIZE_MIB))
for n_block in $(seq $IMAX);
do
BLOCK_OFFSET_DEC=$(($BASE_OFFSET+$DATAFILE_SIZE_BYTES*($n_block-1)))
BLOCK_OFFSET=$( printf "0x%x" $BLOCK_OFFSET_DEC )
BW_H2C=$(sudo $XDMA_TOOLS_PATH/dma_to_device /dev/xdma0_h2c_0 -f ${DATAFILE_PREFIX}_in.bin -s $DATAFILE_SIZE_BYTES -a $BLOCK_OFFSET -c 1)
BW_C2H=$(sudo $XDMA_TOOLS_PATH/dma_from_device /dev/xdma0_c2h_0 -f ${DATAFILE_PREFIX}_out.bin -s $DATAFILE_SIZE_BYTES -a $BLOCK_OFFSET -c 1)
HASH_OUTPUT=($(md5sum ${DATAFILE_PREFIX}_out.bin))
sudo rm ${DATAFILE_PREFIX}_out.bin
if [ ${HASH_INPUT} == ${HASH_OUTPUT} ]; then
TEST_RESULT="PASS"
else
TEST_RESULT="FAIL"
fi
echo "OFFSET: $BLOCK_OFFSET, RESULT: $TEST_RESULT" >> $LOG_FILE
#echo "$BW_H2C MB/s" >> $LOG_FILE
#echo "$BW_C2H MB/s" >> $LOG_FILE
done
sudo rm ${DATAFILE_PREFIX}_in.bin
}
echo "SPEC7 DDR PL TEST: `date`" > $LOG_FILE
sudo rm ${DATAFILE_PREFIX}_in.bin
sudo rm ${DATAFILE_PREFIX}_out.bin
for pattern in "${pattern_array[@]}"; do
echo "Testing the $pattern pattern"
if [ "${#pattern}" -eq "8" ]; then
if ! [[ $pattern =~ ^[0-9A-Fa-f]{8,}$ ]] ; then
echo "PATTERN ERRROR: string is not hex"
else
echo $(check_ddr "$pattern")
fi
else
echo "PATTERN ERROR: string has not 8 hexadecimal positions"
fi
done
DMA bandwidth measurement
At the time of performing intensive tests driven by the XDMA to check the DDR3 connected to the SPEC7 Programmable Logic, we should ideally employ all of the bandwidth that the PCIe provides. For the SPEC7, we have:
- PCIe Gen2: This offers a maximum bandwidth in the physical layer of 5GT/s or 500MB/s for each lane.
- 2x Lanes: This provides a total maximum bandwidth in the physical layer of 10GT/s or 1000MB/s.
- Encoding: The encoding used in the PCIe Gen2 is 8b/10b, providing an actual bandwidth of about 800MB/s.
To check how close we are of the available bandwidth when performing the DDR tests, we can use the output from Xilinx XDMA userspace tools at the time of performing DMA read/write operations. Specifically, the reported bandwidth is in MBytes/sec, e.g.:
-
Average BW = 65536, 204.596039
: 65536 Bytes @ 204.596039 MBytes/sec
By using this feature, we have compiled the Host-to-Card (H2C) and Card-to-Host (C2H) bandwidths for several binary file / data package sizes in the low and high bandwidth versions of the XDMA enabled designs that we have used to verify the DDR3 connected to the Programmable Logic, i.e.:
- DDR3 at Programmable Logic accessible from Processing System and XDMA
- DDR3 at Programmable Logic accessible from XDMA only with Improved Bandwidth
Finally, by using the -c, --count
flag in the Xilinx userspace tools for DMA, we identified that the first read operation from a batch, i.e. Card-to-Host, experiences an extra delay before being actually executed, while the following ones work properly. This is a quite usual behavior in PCIe interfaces, and can be verified from this example log:
dev /dev/xdma0_h2c_0, addr 0x80000000, aperture 0x0, size 0x4000000, offset 0x0, count 4
host buffer 0x4001000 = 0x7f2365c08000
#0: CLOCK_MONOTONIC 0.079478903 sec. write 67108864 bytes
#1: CLOCK_MONOTONIC 0.079427894 sec. write 67108864 bytes
#2: CLOCK_MONOTONIC 0.079441935 sec. write 67108864 bytes
#3: CLOCK_MONOTONIC 0.079428242 sec. write 67108864 bytes
** Avg time device /dev/xdma0_h2c_0, total time 317776974 nsec, avg_time = 79444240.000000, size = 67108864, BW = 844.729126
/dev/xdma0_h2c_0 ** Average BW = 67108864, 844.729126
dev /dev/xdma0_c2h_0, addr 0x80000000, aperture 0x0, size 0x4000000, offset 0x0, count 4
host buffer 0x4001000, 0x7efe23cd8000.
#0: CLOCK_MONOTONIC 0.089676834 sec. read 67108864/67108864 bytes
#1: CLOCK_MONOTONIC 0.077182836 sec. read 67108864/67108864 bytes
#2: CLOCK_MONOTONIC 0.077198346 sec. read 67108864/67108864 bytes
#3: CLOCK_MONOTONIC 0.077259361 sec. read 67108864/67108864 bytes
** Avg time device /dev/xdma0_c2h_0, total time 321317377 nsec, avg_time = 80329344.000000, size = 67108864, BW = 835.421509
/dev/xdma0_c2h_0 ** Average BW = 67108864, 835.421509
In order to account for this effect, we will measure the bandwidth for the 1st and 2nd transaction in both the Host-to-Card (H2C) and Card-to-Host (C2H) channels.
DMA Bandwidth for DDR3 at PL accessible from PS and XDMA (32 BIT AXI DDR)
In the modified Nikhef reference designs in which the DDR3 at PL can be accessed from both the Processing System and the XDMA, we have an 32 bit data width in the AXI interface of the MIG generated memory controller. This introduces a bottleneck that makes the measured bandwidth to fall far below the theoretical maximum for the SPEC7 PCIe interface.
BLOCK SIZE | DATA SIZE [Bytes] | H2C 1st [MB/s] | H2C 2nd [MB/s] | C2H 1st [MB/s] | C2H 2nd [MB/s] |
---|---|---|---|---|---|
4 KiB | 4096 | 45.811430 | 92.208640 | 44.828718 | 113.412338 |
8 KiB | 8192 | 78.079280 | 77.692738 | 76.737171 | 75.138040 |
16 KiB | 16384 | 121.702829 | 134.194986 | 121.873931 | 129.755758 |
32 KiB | 32768 | 189.647190 | 193.634548 | 170.297688 | 178.317606 |
64 KiB | 65536 | 237.095350 | 243.984706 | 206.538799 | 218.645742 |
128 KiB | 131072 | 249.678073 | 265.646819 | 220.773293 | 232.730579 |
256 KiB | 262144 | 298.086020 | 301.034669 | 235.542745 | 247.370070 |
512 KiB | 524288 | 297.998764 | 306.236014 | 249.169850 | 269.026919 |
1 MiB | 1048576 | 311.044467 | 314.210484 | 259.879218 | 269.217073 |
2 MiB | 2097152 | 317.096987 | 318.626236 | 262.128764 | 276.358463 |
4 MiB | 4194304 | 320.266457 | 321.760388 | 263.616048 | 277.753265 |
8 MiB | 8388608 | 322.249937 | 323.388346 | 264.220883 | 277.801923 |
16 MiB | 16777216 | 323.074429 | 323.686474 | 264.714435 | 277.853535 |
32 MiB | 33554432 | 323.232498 | 323.489815 | 264.355168 | 277.892781 |
64 MiB | 67108864 | 323.223210 | 323.536329 | 264.366254 | 278.184430 |
128 MiB | 134217728 | 323.376871 | 323.452376 | 264.386761 | 278.079917 |
DMA Bandwidth for DDR3 at PL accessible from XDMA only (256 BIT AXI DDR)
In the modified Nikhef reference designs in which the DDR3 at PL can only be accessed from the XDMA engine, we have an 256 bit data width in the AXI interface of the MIG generated memory controller. This allows the DDR3 controller to perform burst access to the memory and makes the measured bandwidth to stay around the theoretical maximum for the SPEC7 PCIe interface.
BLOCK SIZE | DATA SIZE [Bytes] | H2C 1st [MB/s] | H2C 2nd [MB/s] | C2H 1st [MB/s] | C2H 2nd [MB/s] |
---|---|---|---|---|---|
4 KiB | 4096 | 43.801397 | 43.801397 | 44.392422 | 47.717794 |
8 KiB | 8192 | 90.115064 | 96.223645 | 96.223645 | 105.446073 |
16 KiB | 16384 | 164.179852 | 185.936720 | 184.034057 | 189.357866 |
32 KiB | 32768 | 273.130397 | 303.486089 | 282.585074 | 322.402275 |
64 KiB | 65536 | 426.042581 | 445.489770 | 424.093392 | 442.326642 |
128 KiB | 131072 | 562.874149 | 588.072719 | 530.346680 | 609.651387 |
256 KiB | 262144 | 688.302140 | 700.440340 | 629.120938 | 660.252471 |
512 KiB | 524288 | 726.173740 | 753.303591 | 640.795519 | 725.244566 |
1 MiB | 1048576 | 788.691206 | 781.138420 | 720.127849 | 795.946546 |
2 MiB | 2097152 | 800.662474 | 813.749152 | 723.903480 | 838.386609 |
4 MiB | 4194304 | 827.278895 | 834.062936 | 738.342680 | 856.978876 |
8 MiB | 8388608 | 843.301711 | 844.679544 | 749.561090 | 865.335301 |
16 MiB | 16777216 | 839.550953 | 844.654581 | 745.797801 | 867.204512 |
32 MiB | 33554432 | 842.928647 | 845.895286 | 747.164912 | 870.135899 |
64 MiB | 67108864 | 844.225261 | 845.045147 | 747.852031 | 869.567883 |
128 MiB | 134217728 | 844.725603 | 845.908796 | 748.661325 | 869.611530 |