extract_pts_data.py
===================

This tool helps in grouping together PTS data for storage to MTF. It assumes
all the data about one module is stored in one file. It reads PTS files as
instructed in code and outputs data to two output files:

* equip-data.csv -- data to appear under the Equipment Data tab in MTF
* logfiles.csv   -- the log files for each board to appear under the Documents
                    tab in MTF

The format of the data is CSV, each parameter is separated by a comma; the
first parameter in the list is the LHC equipment name, HC*.

The data output in each of the CSV files is to be copied into the templates
obtained from EDMS support (edms.support@cern.ch). To make use of the script,
it should be modified (so knowledge of a bit of Python is necessary) so as to
output a .csv file following the import template obtained from EDMS support.

See the guide below and the provided templates for a more in-depth
understanding.

Importing PTS data into MTF
---------------------------

1. Follow the guide at the following page to upload your documents to DFS, so
that the MTF import tool can find them:
https://espace.cern.ch/edms-services/faq/EDMS%20related%20tools/Batch%20Import%20-%20%20Documents.aspx

The easiest way to perform this step is under Windows, as indicated on the
page. In case you don't have Windows, use one of the VMs at CERN.

The logfiles.xls template contains a known-to-work filled-in template, so you
should fill in at least the same parameters for your board.

2. Contact edms.support@cern.ch for an equipment data template for your board.

Each board belongs to a certain EDMS class and take note that if you intend
to add new parameters to the class, the new parameter will be added under
Equipment Data to ALL boards in the class, so you should ask EDMS support which
boards belong to your board's class.

3. Consult the template file obtained from EDMS and see in which column you
want to add the data in your PTS file.

See an example setting in the equip-data.xls file in this directory.

4. Based on the data in your PTS log file, decide in which column in the EDMS
class template you want to add the PTS data to.

5. Copy the extract_pts_data.py script to where your log files are.

6. Now that you have decided which columns to change in the documents batch
import template and the equipment data template, change the
extract_pts_data.py script as shown below (the steps are clearly marked in the
code of extract_pts_data.py):

i. Change the DFS_DIR path to where you uploaded the files in step 1 above
ii. Change the fields to write to the logfiles.csv file, adding commas between
each field to add a column separation.
iii. Copy the equipment data you want in your PTS log file to the equip-data.csv
file by adding lines as appropriate under Step 3 in the file

7. Run the extract_pts_data.py script, it will output the data you requested in
the steps above to the two output files. Note that it will only look in files
that have passed a test (the script runs a grep for PASS in the current folder)

8. Using LibreOffice or Word, copy the data from the equip-data.csv file to the
.xls template for uploading equipment data you received from EDMS support.

9. Using LibreOffice or Word, copy the data from the logfiles.csv file to the
.xls template for uploading the document 

10. Since it may be that there are two tests which have passed, check the
equipment data template after having copied the files for duplicates and remove
the duplicate lines.

11. Upload the output files to new documents in EDMS as specified at:
https://espace.cern.ch/edms-services/faq/EDMS%20related%20tools/Batch%20Import%20-%20Creating%20the%20Import%20Request.aspx

