Commit 531b7e14 authored by Theodor-Adrian Stana's avatar Theodor-Adrian Stana

Added MTF import template

parent f37c936c
extract_pts_data.py
===================
This tool helps in grouping together PTS data for storage to MTF. It assumes
all the data about one module is stored in one file. It reads PTS files as
instructed in code and outputs data to two output files:
* equip-data.csv -- data to appear under the Equipment Data tab in MTF
* logfiles.csv -- the log files for each board to appear under the Documents
tab in MTF
The format of the data is CSV, each parameter is separated by a comma; the
first parameter in the list is the LHC equipment name, HC*.
The data output in each of the CSV files is to be copied into the templates
obtained from EDMS support (edms.support@cern.ch). To make use of the script,
it should be modified (so knowledge of a bit of Python is necessary) so as to
output a .csv file following the import template obtained from EDMS support.
See the guide below and the provided templates for a more in-depth
understanding.
Importing PTS data into MTF
---------------------------
1. Follow the guide at the following page to upload your documents to DFS, so
that the MTF import tool can find them:
https://espace.cern.ch/edms-services/faq/EDMS%20related%20tools/Batch%20Import%20-%20%20Documents.aspx
The easiest way to perform this step is under Windows, as indicated on the
page. In case you don't have Windows, use one of the VMs at CERN.
The logfiles.xls template contains a known-to-work filled-in template, so you
should fill in at least the same parameters for your board.
2. Contact edms.support@cern.ch for an equipment data template for your board.
Each board belongs to a certain EDMS class and take note that if you intend
to add new parameters to the class, the new parameter will be added under
Equipment Data to ALL boards in the class, so you should ask EDMS support which
boards belong to your board's class.
3. Consult the template file obtained from EDMS and see in which column you
want to add the data in your PTS file.
See an example setting in the equip-data.xls file in this directory.
4. Based on the data in your PTS log file, decide in which column in the EDMS
class template you want to add the PTS data to.
5. Copy the extract_pts_data.py script to where your log files are.
6. Now that you have decided which columns to change in the documents batch
import template and the equipment data template, change the
extract_pts_data.py script as shown below (the steps are clearly marked in the
code of extract_pts_data.py):
i. Change the DFS_DIR path to where you uploaded the files in step 1 above
ii. Change the fields to write to the logfiles.csv file, adding commas between
each field to add a column separation.
iii. Copy the equipment data you want in your PTS log file to the equip-data.csv
file by adding lines as appropriate under Step 3 in the file
7. Run the extract_pts_data.py script, it will output the data you requested in
the steps above to the two output files. Note that it will only look in files
that have passed a test (the script runs a grep for PASS in the current folder)
8. Using LibreOffice or Word, copy the data from the equip-data.csv file to the
.xls template for uploading equipment data you received from EDMS support.
9. Using LibreOffice or Word, copy the data from the logfiles.csv file to the
.xls template for uploading the document
10. Since it may be that there are two tests which have passed, check the
equipment data template after having copied the files for duplicates and remove
the duplicate lines.
11. Upload the output files to new documents in EDMS as specified at:
https://espace.cern.ch/edms-services/faq/EDMS%20related%20tools/Batch%20Import%20-%20Creating%20the%20Import%20Request.aspx
#!/usr/bin/python
#===============================================================================
# CERN (BE-CO-HT)
# Extract PTS data for storage to MTF
#===============================================================================
# author: Theodor Stana (t.stana@cern.ch)
#
# date of creation: 2014-06-30
#
# version: 1.0
#
# description:
#
# This script can be used to generate a CSV file containing data about the
# board to be stored to MTF.
#
# It should be run after a batch of boards are tested using PTS.
#
# The steps below show the modifications to be made to the code in case of
# new boards.
#
#===============================================================================
# GNU LESSER GENERAL PUBLIC LICENSE
#===============================================================================
# This source file is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by the
# Free Software Foundation; either version 2.1 of the License, or (at your
# option) any later version. This source is distributed in the hope that it
# will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
# of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU Lesser General Public License for more details. You should have
# received a copy of the GNU Lesser General Public License along with this
# source; if not, download it from http://www.gnu.org/licenses/lgpl-2.1.html
#===============================================================================
# last changes:
# 2014-06-30 Theodor Stana File created
#===============================================================================
# TODO: -
#===============================================================================
import os
import glob
import subprocess
import time
#===============================================================================
# Step 1: Change log file directory and glob pattern
#===============================================================================
# Log file DFS directory and pattern for file name in which to look for the data.
DFS_DIR = "./"
FINNAME = "*.inf"
def extract_and_store():
# Open the output file in the logfile directory
fdata = open("equip-data.csv", 'w')
flogs = open("logfiles.csv", 'w')
fnull = open(os.devnull, 'w')
# grep each file in the logfile dir for FAIL message and write entry for
# those that didn't fail
for fname in sorted(glob.glob(FINNAME)):
#===============================================================================
# Step 2: Add each log file of a board to a new item in the logfiles.csv file.
# The items are separated by commas, so a comma is placed after each
# entry.
# Base your entry on the template received from EDMS support.
#===============================================================================
# Get a brand new entry string
entry = ""
# Get board serial number and append to the entry table
sn = fname.find("HC")
entry += fname[sn:sn+19]
entry += ','
entry += "Theodor-Adrian Stana"
entry += ','
entry += "t.stana@cern.ch"
entry += ','
entry += ','
entry += ','
entry += fname[sn:sn+19]
for i in range(8):
entry += ','
entry += "AB-CO-MTF"
entry += ','
entry += "Specification"
for i in range(5):
entry += ','
entry += "DOC-OWNER-MTF"
entry += ','
entry += "Board PTS test log"
for i in range(9):
entry += ','
entry += time.strftime("%Y-%m-%d", time.localtime())
entry += ','
entry += DFS_DIR + '/' + fname
flogs.write(entry + '\n')
# Only add to equip-data.csv for files which haven't failed
if (subprocess.call("grep FAIL " + fname, shell=True, stdout=fnull, stderr=fnull) == 1):
# Get a brand new entry string
entry = ""
# Add board code to the first column
entry += fname[sn:sn+19]
entry += ','
# Add number of columns to PCB ID
for i in range(6):
entry += ','
#===============================================================================
# Step 3: Add any parameters to look for and the number of chars to store as
# equipment data. The parameters should be based on the template received
# from EDMS support.
# Make sure to add a comma after each parameter but last.
#===============================================================================
# Add parameters from PTS log to equip-data.csv
relgw = None
goldgw = None
with open(fname, 'r') as fin:
for l in fin:
# Unique board ID from DS18B20 thermometer chip
if "Unique ID" in l:
entry += l[-17:-1]
# Add number of columns to cost
for i in range(10):
entry += ','
# Release & golden gateware from flash test
if ("Release gateware" in l):
relgw = "release v" + l[-4:-1]
if ("Golden gateware" in l):
goldgw = "golden v" + l[-4:-1]
if (relgw and goldgw):
entry += goldgw + " + " + relgw
# Golden + release gateware is last parameter to read in
# file, so end reading the file
break
# Last, add the value of goods parameter
for i in range(5):
entry += ','
entry += "392.4"
# Write to the equip-data.csv file
fdata.write(entry + '\n')
fdata.close()
flogs.close()
extract_and_store()
Run:1 Begin:test01
INFO:Found TTLBL board in slot:4 OK
tests : Front Panel LEDs with Schmitt trigger driver IC5, bicolor LEDs with transceiver IC1
uses : conv_ttl_blo_v2.bit and leds.py
PASS: test01
Run:1 End:test01
Run:1 Begin:test02
INFO:Found TTLBL board in slot:4 OK
tests : Thermometer IC12 and unique ID
uses : conv_ttl_blo_v2.bit and therm_id.py
Unique ID: ED0000046F1AB828
Current temperature: 26.812
PASS: test02
Run:1 End:test02
Run:1 Begin:test03
INFO:Found TTLBL board in slot:4 OK
tests : TTL pulse repetition, buffers IC2, IC3, Schmitt trigger
inputs IC6, IC11, NAND gate IC10
uses : conv_ttl_blo_v2.bit and ttl_pulse_switch.py
CH1 received 3 pulses and sent 3 pulses - good
CH2 received 3 pulses and sent 3 pulses - good
CH3 received 3 pulses and sent 3 pulses - good
CH4 received 3 pulses and sent 3 pulses - good
CH5 received 3 pulses and sent 3 pulses - good
CH6 received 3 pulses and sent 3 pulses - good
CH7 received 3 pulses and sent 3 pulses - good
CH8 received 3 pulses and sent 3 pulses - good
CH9 received 3 pulses and sent 3 pulses - good
CH10 received 3 pulses and sent 3 pulses - good
Switches readout as expected: 0x7f
PASS: test03
Run:1 End:test03
Run:1 Begin:test04
INFO:Found TTLBL board in slot:4 OK
tests : Blocking pulse repetition; input optocouplers IC22-IC27
and Schmitt trigger buffer IC16; output buffer IC4, NAND
gate IC10; blocking output transistors T1-T18 and
transformers L1-L12, L17-L22; reset generator IC29 and
blocking power supply generator IC32; Schmitt triggers
IC21, IC28
uses : conv_ttl_blo_v2.bit and blo_pulse_rtm.py
CH1 sent 2 pulse(s) and received 4 pulse(s) - good
CH2 sent 1 pulse(s) and received 4 pulse(s) - good
CH3 sent 1 pulse(s) and received 4 pulse(s) - good
CH4 sent 1 pulse(s) and received 3 pulse(s) - good
CH5 sent 1 pulse(s) and received 3 pulse(s) - good
CH6 sent 1 pulse(s) and received 3 pulse(s) - good
RTM lines have expected value (0x3f)
PASS: test04
Run:1 End:test04
Run:1 Begin:test05
INFO:Found TTLBL board in slot:4 OK
tests : DAC IC17, VCXO OSC3, DAC IC18, VCXO OSC1, PLL IC13
uses : conv_ttl_blo_v2.bit and dac_vcxo_pll.py
VCXO OSC3: Frequency OK; Counter value after 2sec: 40132887 is within range[38000000 .. 42000000]
PLL IC13: Frequency OK; Counter value after 2sec: 250938400 is within range[237500000 .. 262500000]
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=0.80V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=1.65V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=2.40V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=3.20V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=0.80V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=1.65V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=2.40V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=3.20V
PASS: test05
Run:1 End:test05
Run:1 Begin:test06
INFO:Found TTLBL board in slot:4 OK
tests : SFP J1 EEPROM
uses : conv_ttl_blo_v2.bit and sfp_eeprom.py
SFP type: 0x3
SFP type is correct
PASS: test06
Run:1 End:test06
Run:1 Begin:test07
INFO:Found TTLBL board in slot:4 OK
tests : SFP J1
uses : conv_ttl_blo_v2.bit and sfp_test.py
Begin SFP link test
SFP link up OK
PASS: test07
Run:1 End:test07
Run:1 Begin: test08
Board ID correct
Run:1 End: test08
Run:1 Begin:test01
INFO:Found TTLBL board in slot:4 OK
tests : Front Panel LEDs with Schmitt trigger driver IC5, bicolor LEDs with transceiver IC1
uses : conv_ttl_blo_v2.bit and leds.py
PASS: test01
Run:1 End:test01
Run:1 Begin:test02
INFO:Found TTLBL board in slot:4 OK
tests : Thermometer IC12 and unique ID
uses : conv_ttl_blo_v2.bit and therm_id.py
Unique ID: 360000046F31E428
Current temperature: 25.188
PASS: test02
Run:1 End:test02
Run:1 Begin:test03
INFO:Found TTLBL board in slot:4 OK
tests : TTL pulse repetition, buffers IC2, IC3, Schmitt trigger
inputs IC6, IC11, NAND gate IC10
uses : conv_ttl_blo_v2.bit and ttl_pulse_switch.py
CH1 received 3 pulses and sent 3 pulses - good
CH2 received 3 pulses and sent 3 pulses - good
CH3 received 3 pulses and sent 3 pulses - good
CH4 received 3 pulses and sent 3 pulses - good
CH5 received 3 pulses and sent 3 pulses - good
CH6 received 3 pulses and sent 3 pulses - good
CH7 received 3 pulses and sent 3 pulses - good
CH8 received 3 pulses and sent 3 pulses - good
CH9 received 3 pulses and sent 3 pulses - good
CH10 received 3 pulses and sent 3 pulses - good
Switches readout as expected: 0x7f
PASS: test03
Run:1 End:test03
Run:1 Begin:test04
INFO:Found TTLBL board in slot:4 OK
tests : Blocking pulse repetition; input optocouplers IC22-IC27
and Schmitt trigger buffer IC16; output buffer IC4, NAND
gate IC10; blocking output transistors T1-T18 and
transformers L1-L12, L17-L22; reset generator IC29 and
blocking power supply generator IC32; Schmitt triggers
IC21, IC28
uses : conv_ttl_blo_v2.bit and blo_pulse_rtm.py
CH1 sent 2 pulse(s) and received 4 pulse(s) - good
CH2 sent 1 pulse(s) and received 4 pulse(s) - good
CH3 sent 1 pulse(s) and received 4 pulse(s) - good
CH4 sent 1 pulse(s) and received 3 pulse(s) - good
CH5 sent 1 pulse(s) and received 3 pulse(s) - good
CH6 sent 1 pulse(s) and received 3 pulse(s) - good
RTM lines have expected value (0x3f)
PASS: test04
Run:1 End:test04
Run:1 Begin:test05
INFO:Found TTLBL board in slot:4 OK
tests : DAC IC17, VCXO OSC3, DAC IC18, VCXO OSC1, PLL IC13
uses : conv_ttl_blo_v2.bit and dac_vcxo_pll.py
VCXO OSC3: Frequency OK; Counter value after 2sec: 40132304 is within range[38000000 .. 42000000]
PLL IC13: Frequency OK; Counter value after 2sec: 250934687 is within range[237500000 .. 262500000]
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=0.80V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=1.65V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=2.40V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=3.20V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=0.80V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=1.65V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=2.40V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=3.20V
PASS: test05
Run:1 End:test05
Run:1 Begin:test06
INFO:Found TTLBL board in slot:4 OK
tests : SFP J1 EEPROM
uses : conv_ttl_blo_v2.bit and sfp_eeprom.py
SFP type: 0x3
SFP type is correct
PASS: test06
Run:1 End:test06
Run:1 Begin:test07
INFO:Found TTLBL board in slot:4 OK
tests : SFP J1
uses : conv_ttl_blo_v2.bit and sfp_test.py
Begin SFP link test
SFP link up OK
PASS: test07
Run:1 End:test07
Run:1 Begin: test08
Board ID correct
Run:1 End: test08
Run:1 Begin:test01
INFO: Found CONV-TTL-BLO board in slot 4
tests : DAC IC17, VCXO OSC3, DAC IC18, VCXO OSC1, PLL IC13
uses : pts.bit and dac_vcxo_pll.py
VCXO OSC3: Frequency OK; Counter value after 2sec: 40141488 is within range[38000000 .. 42000000]
PLL IC13: Frequency OK; Counter value after 2sec: 251003262 is within range[237500000 .. 262500000]
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=0.80V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=1.65V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=2.40V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=3.20V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=0.80V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=1.65V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=2.40V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=3.20V
PASS: test01
Run:1 End:test01
Run:1 Begin:test02
INFO: Found CONV-TTL-BLO board in slot 4
tests : Front Panel LEDs with Schmitt trigger driver IC5, bicolor LEDs with transceiver IC1
uses : pts.bit and leds.py
PASS: test02
Run:1 End:test02
Run:1 Begin:test03
INFO: Found CONV-TTL-BLO board in slot 4
tests : TTL pulse repetition, buffers IC2, IC3, Schmitt trigger
inputs IC6, IC11, NAND gate IC10
uses : pts.bit and ttl_pulse_switch.py
Ch1 received 3 pulses and sent 3 pulses - good
Ch2 received 3 pulses and sent 3 pulses - good
Ch3 received 3 pulses and sent 3 pulses - good
Ch4 received 3 pulses and sent 3 pulses - good
Ch5 received 3 pulses and sent 3 pulses - good
Ch6 received 3 pulses and sent 3 pulses - good
ChA received 3 pulses and sent 3 pulses - good
ChB received 3 pulses and sent 3 pulses - good
ChC received 3 pulses and sent 3 pulses - good
ChD received 3 pulses and sent 3 pulses - good
Switches readout as expected: 0x80
PASS: test03
Run:1 End:test03
Run:1 Begin:test04
INFO: Found CONV-TTL-BLO board in slot 4
tests : Blocking pulse repetition; input optocouplers IC22-IC27
and Schmitt trigger buffer IC16; output buffer IC4, NAND
gate IC10; blocking output transistors T1-T18 and
transformers L1-L12, L17-L22; reset generator IC29 and
blocking power supply generator IC32; Schmitt triggers
IC21, IC28
uses : pts.bit and blo_pulse_rtm.py
CH1 sent 3 pulse(s) and received 9 pulse(s) - good
CH2 sent 3 pulse(s) and received 9 pulse(s) - good
CH3 sent 3 pulse(s) and received 9 pulse(s) - good
CH4 sent 3 pulse(s) and received 9 pulse(s) - good
CH5 sent 3 pulse(s) and received 9 pulse(s) - good
CH6 sent 3 pulse(s) and received 9 pulse(s) - good
RTM lines have expected value (0x3F)
PASS: test04
Run:1 End:test04
Run:1 Begin:test05
INFO: Found CONV-TTL-BLO board in slot 4
tests : Thermometer IC12 and unique ID
uses : pts.bit and therm_id.py
Unique ID: 030000046F072228
Current temperature: 27.062
PASS: test05
Run:1 End:test05
Run:1 Begin:test06
INFO: Found CONV-TTL-BLO board in slot 4
tests : SFP J1 EEPROM
uses : pts.bit and sfp_eeprom.py
SFP type: 0x3
SFP type is correct
PASS: test06
Run:1 End:test06
Run:1 Begin:test07
INFO: Found CONV-TTL-BLO board in slot 4
tests : SFP J1
uses : pts.bit and sfp_test.py
Begin SFP link test
SFP link up OK
PASS: test07
Run:1 End:test07
Run:1 Begin: test08
Board ID correct: TBLO
Release gateware version correct: 3.0
Board ID correct: TBLO
Golden gateware version correct: 0.2
PASS: test08
Run:1 End: test08
Run:1 Begin:test01
INFO: Found CONV-TTL-BLO board in slot 4
tests : DAC IC17, VCXO OSC3, DAC IC18, VCXO OSC1, PLL IC13
uses : pts.bit and dac_vcxo_pll.py
VCXO OSC3: Frequency OK; Counter value after 2sec: 40142730 is within range[38000000 .. 42000000]
PLL IC13: Frequency OK; Counter value after 2sec: 251011432 is within range[237500000 .. 262500000]
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=0.80V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=1.65V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=2.40V
VCXO OSC3: Frequency responds to change in DAC IC17 OK. DAC=3.20V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=0.80V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=1.65V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=2.40V
PLL IC13: Frequency responds to change in DAC IC18 OK. DAC=3.20V
PASS: test01
Run:1 End:test01
Run:1 Begin:test02
INFO: Found CONV-TTL-BLO board in slot 4
tests : Front Panel LEDs with Schmitt trigger driver IC5, bicolor LEDs with transceiver IC1
uses : pts.bit and leds.py
PASS: test02
Run:1 End:test02
Run:1 Begin:test03
INFO: Found CONV-TTL-BLO board in slot 4
tests : TTL pulse repetition, buffers IC2, IC3, Schmitt trigger
inputs IC6, IC11, NAND gate IC10
uses : pts.bit and ttl_pulse_switch.py
Ch1 received 3 pulses and sent 3 pulses - good
Ch2 received 3 pulses and sent 3 pulses - good
Ch3 received 3 pulses and sent 3 pulses - good
Ch4 received 3 pulses and sent 3 pulses - good
Ch5 received 3 pulses and sent 3 pulses - good
Ch6 received 3 pulses and sent 3 pulses - good
ChA received 3 pulses and sent 3 pulses - good
ChB received 3 pulses and sent 3 pulses - good
ChC received 3 pulses and sent 3 pulses - good
ChD received 3 pulses and sent 3 pulses - good
Switches readout as expected: 0x80
PASS: test03
Run:1 End:test03
Run:1 Begin:test04
INFO: Found CONV-TTL-BLO board in slot 4
tests : Blocking pulse repetition; input optocouplers IC22-IC27
and Schmitt trigger buffer IC16; output buffer IC4, NAND
gate IC10; blocking output transistors T1-T18 and
transformers L1-L12, L17-L22; reset generator IC29 and
blocking power supply generator IC32; Schmitt triggers
IC21, IC28
uses : pts.bit and blo_pulse_rtm.py
CH1 sent 3 pulse(s) and received 9 pulse(s) - good
CH2 sent 3 pulse(s) and received 9 pulse(s) - good
CH3 sent 3 pulse(s) and received 9 pulse(s) - good
CH4 sent 3 pulse(s) and received 9 pulse(s) - good
CH5 sent 3 pulse(s) and received 9 pulse(s) - good
CH6 sent 3 pulse(s) and received 9 pulse(s) - good
RTM lines have expected value (0x3F)
PASS: test04
Run:1 End:test04
Run:1 Begin:test05
INFO: Found CONV-TTL-BLO board in slot 4
tests : Thermometer IC12 and unique ID
uses : pts.bit and therm_id.py
Unique ID: 400000046F439428
Current temperature: 26.000
PASS: test05
Run:1 End:test05
Run:1 Begin:test06
INFO: Found CONV-TTL-BLO board in slot 4
tests : SFP J1 EEPROM
uses : pts.bit and sfp_eeprom.py