Skip to content

S
Simple PCIe FMC carrier SPEC

Faq

Last edited by Erik van der Bij
Page history

Frequently Asked Questions

about the SPEC board

Hardware

Q: Is the SPEC compatible to all motherboards?

A: We have heard that the SPEC board does not seem to be detected in certain modern motherboards. It may be that certain motherboards that have PCIe Gen 3 slots that are not compatible to the GN4124 that is PCIe 1.1. Apparently PCIe Gen 2 slots work OK.
If you have more info, please let us know. See Issue 958. (June 2014).

We bought a new PC HP Pavillion 500-232d. I tried to setup there my SPEC plus fine delay. I installed Ubuntu 12.04 LTS with Kernel version 3.11.0-23. However when I plug in my SPEC card and do lspci my spec card is not shown.

We had similar problems with a Supermicro X9SAE-V-4U. It could only detect the SPEC if there was a single card inserted. As soon as we inserted another one, both disappeared. We tried all possible permutations (it has 3 or 4 pcie slots) and BIOS configurations. The final fix was that we changed to another motherboard.

Table of known incompatible motherboards:

Model BIOS Slots Problem Notes Issue
HP 500-232D n/a all 2014: The System does not recognize the SPEC Issue 958
Supermicro X9SAE-V-4U n/a all 2014: The System does not recognize more than one SPEC It seems to work fine with a single SPEC Issue 958
PCIE-Q57A-R10 Rev 1.0 n/a all 2014: The System does not booth with a SPEC Model uses 8K2 pull-up on PERST line. Solved by removing R208 on SPEC V4-0. Issue 1067
Siemens IPC 847E (and 647) n/a all 2020: The BIOS does not always recognise the card in slot 4 and 6, 16x and 32x PCIe slots. Workaround by programming PCI Device Class ID to 0xFF00 instead of 0x0000 https://its.cern.ch/jira/browse/SIEM-165 - CERN only

Q: The SPEC uses a 4-lane PCIe bus (PCIe x4). Can it work in a x1 slot?

A: The SPEC can work in principle in a x1 slot if it would mechanically fit. Of course it would work only at x1 speed and this is handled fully automatically by the Gennum PCIe interface chip. You can open up the x1 connector on your motherboard (this has been tested on two different systems) or one could cut the SPEC board (we even had foreseen to have no power planes in that area for this purpose). This latter option hasn't been tried out yet.

Q: Do I need to connect the external power supply connector when used in a PCIe slot?

No the power supply is not necessary while connecting through the PCIe slot. The board gets its power from the PCIe card edge connector. This exernal connector is only used in stand-alone applications where the card is not plugged into a PCIe slot.

Q: Does the board comply to CE regulations (EMC)?

A: Yes it does. In June 2012 extensive EMC tests have been performed on the SPEC with a digital I/O card. These tests have been made by AT4 wireless, a testing laboratory accredited by the Spanish National Accreditation Body (ENAC -Entidad Nacional de Acreditación) to perform the tests indicated in the Certificate No. 51/LE 203.
The test report is publicly available and shows that the card has passed the most restrictive EMC tests of each class (domestic and industrial) successfully:

  • EM Radiated Emission: Class B (Domestic) EN 55022 (2006) / A1 (2007)
  • EM Immunity (Industrial): EN 61000-4-3 (2006) / A1 (2008) / A2 (2010), EN 61000-4-4 (2004) / A1 (2010) / Corr (2010), EN 61000-4-6 (2009), EN 61000-4-2 (2009), EN 61000-4-8 (2010).

Q: Is there a users manual available?

A: There is the Getting Started with the SPEC project that provides a detailed tutorial on how to get ready to work with the (SPEC), including hardware deployment instructions, full required toolchain setup and a collection of step-by-step demonstrative tutorials.

All good starting points are:

Q: The card contains the several oscillators, which ones can I use?

A: the board contains:

  1. 1x 10-280 MHz I2C Programmable XO Oscillator, starts up at 100 MHz (OSC4, Silicon Labs Si570, freely usable)
  2. 1x 25 MHz TCXO controlled by a DAC with SPI interface (IC3 or IC7, AD5662, used by White Rabbit PTP core)
  3. 1x 20 MHz VCXO controlled by a DAC with SPI interface (OSC2, AD5662, used by White Rabbit PTP core)
  4. 1x low-jitter frequency synthesizer (IC6, TI CDCM61004, fixed configuration, Fout=125 MHz, used by White Rabbit PTP core)

The Si570 (1) is free for you to use in your application. I don’t believe it is used in our Fine delay, ADC or TDC FMC cards either as these FMC mezzanines have their own local oscillator that feeds it clock from the mezzanine to the carrier. Although it stays programmable, the Si570 that is on the board starts up at 100 MHz.

The 25 MHz and 20 MHz ones (2, 3) are used in application using White Rabbit (see White Rabbit Node Reference Design).
As White Rabbit is a really nice thing (sub ns synchronisation, data link to the board, UTC time), I think that you should not try to use these. Even if you believe right now you don’t need White Rabbit, you will in the future.

The 25 MHz is used to generate an 125 Mhz clock with the TI CDCM61004 (4). You may use that clock as the WR core will vary it only by 10 ppm or so.

Q: What is the average current load on each power rail?

A: For a SPEC with the DIO 5Cha mezzanine:
1.2V - 0.6A
1.8V - 0.22A
2.5V - 0.22A
3.3V - 0.7A
(Data from Henk Peek, 28/1/16)

Q: The SPEC also exists with larger FPGA types. How can I detect the difference?

A: The SPEC is available with larger FPGAs too. A PCI driver can read the PCI Device ID of the card, which has (or should be actually) been programmed to be differently between the types. The PCI Device ID has been registered at CERN, with CERN's Vendor ID 0x10DC. (See PCI, xTCA, USB & VME64x Board ID Allocation - CERN only)

Type FPGA type FLASH type PCI Device ID Remarks
SPEC Xilinx Spartan6 XC6SLX45T-3FGG484C M25P32 (4MB) 0x18D standard version
SPEC100 Xilinx Spartan6 XC6SLX100T-3FGG484C M25P32 (4MB) 0x1A2
SPEC150 Xilinx Spartan6 XC6SLX150T-3FGG484C M25P64 (8MB) 0x1A3 SPEC150 OHWR page

Q: What is the PCIe performance I can expect?

A: with the setup below we were able to reach a transfer speed of 477MB/s.

Board: SPEC
Transfer: from FPGA Dual port RAM through the Gennum PCIexpress interface to the motherboard memory
OS: LabVIEW RT
DMA: no linked list, one DMA transfer of 200MB
MotherBoard: INTEL DH77DF
CPU: G1610 Intel Celeron 2.6 GHz
Memory: DDR3 1333MHz

(Data from INCAA, 19/10/16)

Q: Can I use the front-panel drawings for other purposes?

A: Maybe we have not made that explicit on the mechanical drawings, but the complete SPEC design, including the front-panel, is licenced under the CERN OHL v1.1 as is shown on the SPEC project page (top-right). So yes, you may use these CERN drawings of the SPEC front-panel in any way; modify, produce, sell. There is no need to ask for permission, despite it is written on the drawing "This drawing may not be used for commercial purposes without written authorization".

Q: The Gennum PCIe bridge is at end-of-life. What will you do?

A: In January 2018 we started to work on defining a specification of a new board, the SPEC7, that has the PCIe bridge integrated in the main FPGA.
As introducing a new design can take three years from start to production, it is likely that we stay until the year 2020 with the current SPEC design.

Firmware

Q: Is there a basic ucf file (all connected IOs and their standards) for the SPECv4?

A: you can use the one we've used for the FmcAdc100M14b4cha: spec_top_fmc_adc_100Ms.ucf.
You'll just have to rename the signals going to the FMC slot according to you needs.

Q: If I use the SPEC in stand-alone mode, can it boot from the SPI Flash memory or do I need a JTAG programmer?

A: In stand-alone mode the Xilinx will boot directly from the SPI Flash memory (which in turn you must have programmed before with a JTAG programmer or with an application that programs it via the Gennum PCIe interface chip).
In applications where the board is used in a Linux PC, the Xilinx will first load its configuration from the SPI Flash memory with a "golden firmware" which allows the driver to detect the carrier board type but also that of the mezzanine. Dependent on that, the driver will then reconfigure the Xilinx with other, mezzanine (and sometimes application) dependent software.

Schematics page 10 (V4) describes:

FPGA boot method selection, BOOT_SELx signals are driven by GPIO of the GN4124. If the GPIO are not configured (high-Z), the pull-up defines the default mode.
1) From SPI Flash (default mode)
The FPGA is in Master SPI mode and takes its configuration from the SPI Flash memory.
BOOT_SEL0=high
BOOT_SEL1=high
2) From GN4124
The FPGA is in Slave SPI mode and is configured via the GN4124 by the driver at startup
BOOT_SEL0=high
BOOT_SEL1=low
3) Access SPI Flash from GN4124
This mode is not used to boot the FPGA, but only to program the SPI Flash memory from the GN4124.
BOOT_SEL0=low
BOOT_SEL1=low (or high)

Q: Can you give me an idea of the Xilinx resources used in a typical application?

The SPEC uses the modestly sized XC6SLX45T. Typically resource use is given below.

FMC TDC 1ns 5cha (without White Rabbit support):

  • 6% FF
  • 14% LUT
  • 9% RAMB16, 0% RAMB8

FMC TDC 1ns 5cha (with White Rabbit support):

  • 15% FF
  • 39% LUT
  • 53% RAMB16, 3% RAMB8

FMC ADC 100M 14b 4cha (without White Rabbit support):

  • 12% FF
  • 19% LUTs
  • 21% RAMB16, 1% RAMB8

FMC DEL 1ns 4cha (with White Rabbit support):

  • 29 % FF (16132)
  • 57 % LUTs (15801)
  • 77 % BRAMs (90 x RAMB16BWER, most of it used by the LM32 CPU in the WR Core)

Q: What is the spec-init.bin (the golden gateware) and where are the HDL sources for it stored?

A: The spec-init.bin or the golden gateware for SPEC board is a default binary downloaded to FPGA by the SPEC kernel drivers. It provides a Wishbone access to GPIO lines for bit-banging the I2C interface and accessing the EEPROM on FMC Mezzanine board. The driver reads its content to recognize the mezzanine and decide which kernel module and gateware to load (e.g. for Fine Delay, WR-NIC,...). You can find the binary in the files section of the spec-sw project Project-Attachments section of the spec-sw project and HDL sources in the SPEC svn repository.(https://www.ohwr.org/project/spec/repository/trunk/hdl/golden) SPEC git repository. (EB - updated links on 20/1021)

Q: does the card have a MAC address stored in a memory?

A: No. There is no memory where the MAC address is stored. So SPEC boards do not come with a MAC address that is set or programmed by the manufacturer.
When you use the White Rabbit PTP core, it uses the ID of digital 1-wire thermometer available on the SPEC board to generate unique MAC address for each board. There is a specific [WRPTP FAQ entry about MAC addresses(https://www.ohwr.org/project/wr-cores/wikis/Wrpc-faq#mac-addresses).

Actually, in 2017 the SPEC production test system (PTS) has been expanded with the possibility to program a MAC address during production. That address is then stored in the flash eeprom. (April 2020)

Software

Q: Can you provide us the original Gennum GN412x RDK Software and the GenDiag program?

A: We have not had the need ourselves to use the RDK software in the SPEC or SPEXI projects. I’m also not even sure that we have it and if we (CERN) have the right to give this software from Gennum out either.
The Simple PCIe FMC carrier- Software project is where you can find software to communicate with the Gennum chip.
As IP core to communicate with the Gennum we use the Gennum GN4124 core bus interface project.

Q: How do I build dependencies?

A: The SPEC depends on the following projects that you have to clone, build, and install:

git clone https://ohwr.org/project/general-cores.git
make -C general-cores/software
make -C general-cores/software modules_install
git clone https://ohwr.org/project/fmc-sw.git
make -C fmc-sw
make -C fmc-sw modules_install
git clone https://gitlab.cern.ch/coht/fpga-manager.git
make -C fpga-manager/drivers/fpga
make -C fpga-manager/drivers/fpga modules_install

Matthieu Cattin, Erik van der Bij, Grzegorz Daniluk, Federico Vaga - 20/10/2021