\PassOptionsToPackage{usenames,dvipsnames,svgnames,table,xcdraw}{xcolor}
\documentclass[11pt]{hllhcedms}   % One can enable the "twoside" option.

\usepackage{ifxetex}
\iftutex{}
  \usepackage{fontspec}
  \setmainfont{Calibri}
\else
  \usepackage[sfdefault]{carlito}
\fi

\usepackage{etoolbox}
\usepackage{microtype}
\usepackage{titling}
\usepackage[hidelinks,unicode]{hyperref}
\usepackage{varioref}
\usepackage[binary-units=true,group-digits=false,detect-all]{siunitx}
\usepackage[calc]{datetime2}
\usepackage[endianness=big]{bytefield}
\usepackage[justification=centering]{caption}
\usepackage{subcaption}
\usepackage[inline]{enumitem}
\usepackage{tcolorbox}
\usepackage{multirow}
\usepackage{tikz}
\usepackage{circuitikz}
\usepackage{xparse}
\usepackage{cleveref}
\usepackage{placeins}
\usepackage[bottom,flushmargin]{footmisc}
\usepackage[normalem]{ulem}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% DOCUMENT REFERENCE NUMBERS %%%
\edmsno{0000000}		% EDMS number
\rev{0.1} 			% Revision number
\validity{DRAFT}		% If Draft or released...
\reference{[NO REFERENCE]}	% Reference number within the project 
% Not mandatory: 
\templateedmsno{2061471}	% You can define a template number to be added to the bottom right of the page

\title{DI/OT MoniMod PSI Radiation Test Report}
\date{\today}
\author{Christos Gentsos}

\hypersetup{colorlinks=true,citecolor=MidnightBlue,linkcolor=MidnightBlue,urlcolor=blue}
\hypersetup{pdftitle=\thetitle,pdfauthor=\theauthor}
\hypersetup{pdfsubject={Radiation Report},pdfkeywords={DIOT,MoniMod,SAMD21,rad-tol,report,radiation,SEU,TMR}}

\usetikzlibrary{calc,decorations.text,shapes,positioning,arrows,patterns}

\renewcommand{\textfraction}{0.3}
\renewcommand{\bottomfraction}{0.45}
\setcounter{bottomnumber}{2}
\setcounter{totalnumber}{4}

\renewcommand\footnotelayout{%
  \advance\leftskip 5mm
  \advance\rightskip 5mm
}

\crefformat{equation}{(#2#1#3)}
\crefname{equation}{}{}
\crefname{lstlisting}{code listing}{code listings}
\crefname{figure}{figure}{figures}
% \crefname{subfigure}{subfigure}{subfigures}
\crefname{table}{table}{tables}
\crefname{chapter}{chapter}{chapters}
\crefname{section}{section}{sections}
\crefname{subsection}{subsection}{subsections}
\crefformat{ilenumi}{#1}
\crefformat{ilenumci}{#1}

\sisetup{exponent-product=\!\cdot{}\!}
\sisetup{range-units=single}
\sisetup{number-unit-product=\kern0.1em}

\newlist{ilenum}{enumerate*}{1}
\setlist[ilenum]{label= (\roman*), after={. }, itemjoin={{; }}, itemjoin*={{; and }}}

\newlist{enum}{enumerate}{1}
\setlist[enum]{label= (\arabic*), after={. }, itemjoin={{; }}, itemjoin*={{; and }}}

\newcommand{\ie}{\textit{i}.\textit{e}.}
\newcommand{\eg}{\textit{e}.\textit{g}.}
\newcommand{\etc}{\textit{etc}.}

\newcommand\hex[1]{\texttt{0x#1}}
\newcommand\bin[1]{\texttt{0b#1}}

\newcommand\ptwelve{\SI{12}[+]{V}}
\newcommand\pfive{\SI{5}[+]{V}}

\newcommand{\MM}[1]{\emph{MM#1}}

\definecolor{light-gray}{gray}{0.93}
\colorlet{codebg}{light-gray}
\tcbuselibrary{listings,breakable,skins,minted}
\AtBeginDocument{%
  \renewcommand\lstlistingname{Code Listing}
  \newtcbinputlisting[blend into=listings]{\codelisting}[4][]{%
    listing file={#2},
    label={#3},
    title={#4},
    fonttitle=\small,
    enhanced,
    listing only,
    frame hidden,%
    attach boxed title to bottom,%
    boxed title style={colback=codebg, frame hidden},%
    width=0.8\textwidth,%
    coltitle=black,%
    colback=codebg,%
    arc=3mm,%
    boxrule=0.2pt,%
    code={\singlespacing},
    minted options={%
      breaklines=true,breakbefore=\(\[\)\],breakbeforesymbolpre=,breaksymbolleft=,breaksymbolright=\tiny\ensuremath{\hookleftarrow},% chktex 40
      fontsize=\scriptsize},%
    #1}
}
% chktex-file 13

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% DOCUMENT STARTS HERE: %%%
\begin{document}

\begin{maintable}{C{1}}
\maintabletitle{\thetitle}
\end{maintable}

\begin{maincontentstyled}
  \section*{General Information and Responsibilities}
\end{maincontentstyled}

\begin{maincontentstyled}
  \begin{center}
    \begin{tabular}{ll}
      \toprule Concerned equipment: & MoniMod \\
      \midrule Group: & BE/CO \\
      \midrule Device Under Test (DUT): & MoniMod v0.2p \\
      \midrule Test Dates: & July 1--2 2020 \\
      \midrule Specifications by: & Grzegorz Daniluk, Christos Gentsos, Evangelia Gousiou \\
      \midrule Tested by: & Christos Gentsos, Evangelia Gousiou \\
      \midrule HW development by: & Christos Gentsos \\
      \midrule Report by: & Christos Gentsos \\
      \midrule SW development by: & Christos Gentsos \\
      \bottomrule
    \end{tabular}
  \end{center}

  \section*{Overview}

  This document presents the results derived from the DI/OT MoniMod radiation tests performed at the
  Proton Irradiation Facility (PIF) of the Paul Scherrer Institute (PSI) using a \SI{200}{\MeV}
  proton beam. The Devices Under Test (DUT) are version 0.2p MoniMod prototypes. These modules
  feature fan control circuitry and PMBus-compatible power rail monitoring, with the I2C, ADC and
  PWM functions these features require being provided by peripherals in a SAMD21 Cortex-M0+
  microcontroller.

  As part of the DI/OT platform, the MoniMod is to be used in future instrumentation operating in
  radiation-exposed areas around the HL-LHC, with the worst-case being the top floor (L1) of the RR
  alcoves at P1 and P5, with TID levels of \SI{25}{Gy/yr}, HEH fluence of \SI{1.4e10}{cm^{-2}/yr},
  and a \SI{1}{MeVn\text{-}eq} thermal neutron fluence of
  \SI{7e10}{cm^{-2}/yr}.\footnote{Information taken from the \emph{Radiation level specifications
      for HL-LHC} document:
    \href{https://edms.cern.ch/ui/file/2302154/0.9/HL-LHC\_specification\_document\_draft\_1\_July\_2020.pdf}{https://edms.cern.ch/ui/file/2302154/0.9/HL-LHC\_specification\_document\_draft\_1\_July\_2020.pdf}}
  Consequently, the purpose of these radiation tests was to evaluate the behavior with respect to
  Total Ionizing Dose (TID) and Single Event Effects (SEEs). Furthermore, the results of these tests
  will help with quantifying the efficiency of a software technique that has been used to mitigate
  Single Event Upsets (SEUs) in the data stored in the microcontroller RAM.

  All presented test data were recorded in the campaign of the week of June 29 – July 3, 2020, at
  the PSI PIF facility.

  \clearpage{}

  \renewcommand{\contentsname}{Outline}%
  \tableofcontents
  \clearpage{}

  \section{DUT description and Operating Conditions}\label{sec:dut_description}

  The Distributed I/O Tier project (HL-LHC Work Package 18.2) Monitoring Module (DI/OT MoniMod) is a
  monitoring module developed for the DI/OT project's power supply and (optional) fan tray and based
  on the ATSAMD21G18 Cortex-M0+ microcontroller. It can monitor voltage and current consumption for
  up to three power rails, host up to three temperature sensors, and control up to three fans
  without requiring them to support PWM. The module is accessed and managed through a PMBus
  interface.

  A MoniMod module can be seen on \vref{fig:pic:mm_nopatch}: the microcontroller is marked by a
  \textcolor{Mahogany}{red} square, the gate drivers are pointed out by the
  \textcolor{NavyBlue}{blue} one, and the MOSFETs are highlighted in \textcolor{Rhodamine}{purple}.
  The fan control circuitry needed extensive modifications to work properly, these were implemented
  by adding a couple of patch wires, removing components from the board and adding extra components
  on two little ``patch boards'', seen on \vref{fig:pic:mm_patch}. Unfortunately, patching the bugs
  found in the fan control circuits of this prototype has led to issues with signal integrity with
  the fan PWM adversely affecting the PT100 driver circuits to the point where it is impossible to
  use them while the fans are spinning.

  \begin{figure}[b]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=0.7\textwidth]{images/MoniMod_before}
      \caption{Before the necessary patches}%
      \label{fig:pic:mm_nopatch}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=0.95\textwidth]{images/MoniMod_after}
      \caption{After applying the necessary patches}%
      \label{fig:pic:mm_patch}
    \end{subfigure}
    \caption{A MoniMod module before and after the necessary patches}%
    \label{fig:pic:mm}
  \end{figure}

  \begin{figure}[t]
    \centering
    \begin{tikzpicture}[->,>=stealth,
      font=\tiny,
      base/.style = {rounded corners=1pt, draw=black,
        on grid, minimum width=12mm, minimum height=6mm,
        text width=12mm, inner sep=-6pt, text centered},
      pro/.style = {base, rectangle, fill=orange!30},
      dec/.style = {base, diamond, fill=teal!30},
      anchor=center,
      node distance=10mm and 20mm]

      \node (loop) [pro] {loop};
      \node (init) [pro, left=of loop] {init};
      \node (i2ci) [dec, right=of loop] {I2C int};
      \node (adci) [dec, right=of i2ci] {ADC int};
      \node (timi) [dec, right=of adci] {\SI{100}{Hz} int};
      \node (exti) [dec, right=of timi] {ext. int};

      \draw (init) -- (loop);
      \draw (loop) -- (i2ci);
      \draw (i2ci) -- node[above,pos=0.1] {no} (adci);
      \draw (adci) -- node[above,pos=0.1] {no} (timi);
      \draw (timi) -- node[above,pos=0.1] {no} (exti);
      \draw (exti) -- node[right,pos=0.1] {no} +(0, 7mm) -| (loop);

      \node (i2c1) [pro, below=of i2ci] {proc I2C req};
      \node (i2c2) [pro, below=of i2c1] {update I2C state};
      \draw (i2ci) -- node[right,pos=0.1] {yes} (i2c1);
      \draw (i2c1) -- (i2c2);
      \draw (i2c2) -| (loop);

      \node (adc1) [pro, below=of adci] {read ADC};
      \node (adc2) [dec, below=of adc1] {ch=last};
      \node (adcl) [pro, below left=of adc2] {ch$\leftarrow$0};
      \node (adcs) [pro, left=of adcl] {ADC\_r$\leftarrow$1};
      \node (adc3) [pro, below=of adc2] {inc ADC ch};
      \node (adc4) [pro, below=of adc3] {trig ADC};
      \draw (adci) -- node[right,pos=0.1] {yes} (adc1);
      \draw (adc1) -- (adc2);
      \draw (adc2) -- node[right,pos=0.1] {no} (adc3);
      \draw (adc2.west) -- node[above,pos=0.1] {yes} +(-4mm, 0) |- (adcl.east);
      \draw (adcl) -- (adcs);
      \draw (adcs) -- (loop);
      \draw (adc3) -- (adc4);
      \draw[-] (adc4) -- +(-50mm, 0);

      \node (tim1) [pro, below=of timi] {glide PWM};
      \node (tim2) [dec, below=of tim1] {cnt=99};
      \node (tim3) [pro, below=of tim2] {cnt$\leftarrow$0 upd. PID, PWM};
      \node (timr) [pro, below left=of adc4] {++cnt};
      \node (timw) [pro, left=of timr] {feed WD};
      \node (tim4) [dec, below=of tim3] {ADC\_r};

      \draw (timi) -- node[right,pos=0.1] {yes} (tim1);
      \draw (tim1) -- (tim2);
      \draw (tim2) -- node[right,pos=0.1] {yes} (tim3);
      \draw (tim2.west) -- node[above,pos=0.1] {no} +(-4mm, 0) |- (timr);
      \draw (timr) -- (timw);
      \draw[-] (timw) -- +(-10mm, 0) |- ($(adcs)+(0, 10mm)$);
      \draw (tim3) -- (tim4);
      \draw (tim4) -- node[above,pos=0.1] {yes} (adc4);
      \draw[-] (tim4.south) |- node[right,pos=0.1] {no} +(0, -2mm) -- +(-70mm, -2mm);

      \node (ext1) [pro, below=of exti] {upd. RPM cnt};
      \draw (exti) -- node[right,pos=0.1] {yes} (ext1);
      \draw[-] (ext1) |- +(-90mm, -45mm) |- ($(timw)+(-10mm, 0)$);

      \node (wd) [pro, below left=of loop] {WD};
      \draw (wd) -- (init);

    \end{tikzpicture}
    \caption{The control flow of the software}%
    \label{fig:dia:sw_flow}
  \end{figure}


  \subsection{DUT Functionality}

  \subsubsection{PMBus Interface}

  The firmware implements a PMBus command subset that enables voltage, current and temperature
  monitoring, as well as both fan monitoring and control. To improve data transfer reliability,
  Packet Error Checking (PEC), as defined by the SMBus standard, is supported.

  Non-standard functionality such as resetting the microcontroller, toggling PEC or setting up fan
  speed control via temperature curves can be accessed using extended PMBus commands.

  \subsubsection{Power Rail Monitoring}

  The ADC peripheral integrated in the microcontroller is used to enable the MoniMod module to
  monitor the voltage and the current draw of up to three power rails. For the voltage monitoring a
  simple voltage divider is used in place of an analog front-end as the measurement requirements are
  relaxed.

  The current draw is measured using high-side shunt resistors and a simple circuit based on a
  rail-to-rail opamp and a BJT that references the voltage differential of the shunt to ground
  level.

  \subsubsection{Temperature Monitoring}

  The current MoniMod prototypes implement circuits to use either PT100 or LM45-type temperature
  sensors, allowing a comparison of the two approaches in a radiation environment in order to choose
  which one will be used eventually; each channel is assigned a switch by which the user selects the
  type of sensor connected. A voltage reference (the same one that is used for the microcontroller
  ADC peripheral) is used to form a current source and bias the PT100; an amplifier is used to bring
  the voltage level to the appropriate range.

  \subsubsection{Fan Control}

  Three buck converters offer DC fan speed control, increasing the reliability of fans by modulating
  their speed according to the cooling needs without requiring them to be PWM-enabled which, in a
  radiation environment, might severely limit the available options.

  The fan tachometer can be used to feed a PID controller, implemented in SW, which allows setting
  the fan speeds.

  Temperature monitoring data from the three sensors can be combined using user-provided
  coefficients to provide an aggregate temperature for each fan, which can then be used to set their
  speed using user-provided fan curves.

  \subsubsection{Custom Bootloader}

  A custom bootloader allows the remote programming of the MoniMod's FW through its PMBus
  interface. Even though the remote programming functionality is only intended to be used when
  radiation levels are down, the bootloader should reliably bring up the FW in radiation.

  \subsection{Software mitigation}

  Although the MoniMod's function is not critical (hence the use of Atmel START and the ASF4
  drivers), some measures have been taken in an effort to reduce interruptions and data corruption,
  improving the QoS.

  The firmware uses TMR for variables stored in the RAM, using the COAST LLVM passes (for more
  information please visit the \href{https://coast-compiler.readthedocs.io}{COAST readthedocs
    page}).  With this approach, each protected variable is stored three times and voting logic is
  employed when the variable is used. Protection can be enabled individually for each variable and,
  indeed, one has to cull the number of protected variables as the instruction overhead for the
  extra stores and the voting might outweigh the benefits in variables that are used too
  intensively.

  The microcontroller also integrates a watchdog peripheral: this is fed every time the main timer
  callback runs, \ie{} every \SI{10}{ms}. The watchdog is set to trigger if it doesn’t get fed for
  \SI{20}{ms} --- as soon as the main loop skips a beat. That should lead to a quick revival of the
  microcontroller and, consequently, to minimal downtime.

  Another measure was employed and even though its impact in robustness would be minimal, its
  overhead is zero so it was kept:
  \begin{ilenum}
  \item placing the main loop instruction in an address that is a power of two and after the rest of
    the firmware
  \item filling the rest of the memory with NOPs
  \item adding a trampoline at the end of the memory space
  \end{ilenum}
  Since the main loop instruction address is a power of two, only one of the bits that make it up is
  set (it is of the form \bin{1000000}). Any Program Counter (PC) bit flips in a bit lower than the
  one that is set will make the resulting PC value greater (\eg{} \bin{1001000}) and lead execution
  to the trampoline through the NOPs that populate the higher addresses.

  \subsection{Normal Operating Conditions}

  Operationnaly, two variants of the MoniMod module will be provided. One, space-optimised and
  without the fan control circuitry, will be integrated in the RaToPUS (DI/OT radiation-tolerant
  power supply) to provide PMBus-compatible monitoring. The other will provide full functionality of
  the tested prototype to be integrated in the optional fan tray of the DI/OT crate.

  \begin{figure}[t]
    \centering
    \begin{tikzpicture}
      % \draw [step=0.5,gray] (-5, -5) grid (5, 5);
      % \draw [step=1,blue,very thick] (-5, -5) grid (5, 5);

      \node (bb) at (0, 0) {\includegraphics[width=3cm,angle=90]{images/breakout_board_3d}};
      \node [align=center,above=0cm of bb,xshift=0.3cm] {placed below the\\experiment table};

      \node (feather) [below=of bb] {\includegraphics[width=2cm]{images/m0feather}};
      \draw[<->,>=stealth,thick] (feather) -- (bb);

      \node (mm0) at (5,  2) {\includegraphics[width=2cm,angle=-90]{images/monimod_3d}};
      \node (mm1) at (5, -2) {\includegraphics[width=2cm,angle=-90]{images/monimod_3d}};

      \draw[<->,>=stealth,thick] (2, 0.2) to[out=0,in=-135] (3, 1.0);
      \draw[<->,>=stealth,thick] (2, -0.2) to[out=0,in=135] (3, -1.0);

      \node (psu) at (-5, 0) {\includegraphics[width=4cm]{images/psu}};
      \node [below=0cm of psu] {placed at the maze};

      \draw[->,>=stealth,thick] (-3,  0.8) -- (-2,  0.8);
      \draw[->,>=stealth,thick] (-3,    0) -- (-2,    0);
      \draw[->,>=stealth,thick] (-3, -0.8) -- (-2, -0.8);

      \fill[fill=red!80,fill opacity=0.3,rounded corners] (2.5, 3.5) rectangle (7.5, -3.5);
      \coordinate (betweenmm) at ($(mm0)!0.5!(mm1)$);
      \node at (betweenmm) {stacked and irradiated};

      \node (fan1_1) at (9, 3) {\includegraphics[width=1cm]{images/fan}};
      \node (fan1_2) at (9, 1) {\includegraphics[width=1cm]{images/fan}};
      \node (fan2_1) at (9, -1) {\includegraphics[width=1cm]{images/fan}};
      \node (fan2_2) at (9, -3) {\includegraphics[width=1cm]{images/fan}};

      \draw[->,>=stealth,thick] (7.2,  2.3) to[out=0,in=180] (8.4,  3);
      \draw[->,>=stealth,thick] (7.2,  1.8) to[out=0,in=180] (8.4,  1.1);
      \draw[->,>=stealth,thick] (7.2,  2.3-4) to[out=0,in=180] (8.4,  3-4);
      \draw[->,>=stealth,thick] (7.2,  1.8-4) to[out=0,in=180] (8.4,  1.1-4);

      \node [rotate=-90] at (10, 0) {placed below the experiment table};
    \end{tikzpicture}
    \caption{The test setup}%
    \label{fig:dia:test_setup}
  \end{figure}

  \section{Test Setup and Procedure}

  \subsection{Test Setup Description}

  The test setup used to validate a MoniMod's monitoring functionality consists of:
  \begin{enum}
  \item a PC capable of running Python that has two free USB ports
  \item two USB extenders to reach components in the maze from the control room
  \item a test coordinator module, implemented on a ``Feather M0 Basic Proto'' SAMD21-based
    development board, that gives access to the MoniMod's PMBus interface (through the USB extender)
  \item a programmable PSU, controlled by the PC (through the second USB extender)
  \end{enum}

  \begin{figure}[t]
    \centering
    \includegraphics[height=0.25\pageheight]{images/connection_board}
    \caption{DUT powering and connection board}%
    \label{fig:pic:connection_board}
  \end{figure}

  \begin{table}[b]
    \centering
    \caption{Connectivity of the DUT sensing inputs}%
    \label{tab:con}
    \begin{subtable}[t]{0.4\textwidth}
      \centering
      \caption{DUT1 connections}%
      \label{tab:con1}
      \begin{tabular}{ll}
        \toprule
        Channel & Assignment \\
        \midrule
        VCh1 & \ptwelve{} \\
        VCh2 & \pfive{} \\
        VCh3 & \pfive{} \\
        ICh1 & Is$_{\ptwelve{}}$ (DUT1) \\
        ICh2 & Is$_{\pfive{}}$ (DUT1) \\
        ICh3 & Is$_{\pfive{}}$ (DUT2) \\
        TCh1 & LM45 \\
        TCh2 & PT100 \\
        TCh3 & --- \\
        \bottomrule
      \end{tabular}
    \end{subtable}
    \begin{subtable}[t]{0.4\textwidth}
      \centering
      \caption{DUT2 connections}%
      \label{tab:con2}
      \begin{tabular}{ll}
        \toprule
        Channel & Assignment \\
        \midrule
        VCh1 & \ptwelve{} \\
        VCh2 & \pfive{} \\
        VCh3 & \pfive{} \\
        ICh1 & Is$_{\ptwelve{}}$ (DUT2) \\
        ICh2 & Is$_{\pfive{}}$ (DUT2) \\
        ICh3 & Is$_{\pfive{}}$ (DUT1) \\
        TCh1 & LM45 \\
        TCh2 & --- \\
        TCh3 & PT100 \\
        \bottomrule
      \end{tabular}
    \end{subtable}
  \end{table}

  \subsubsection{DUT Connectivity}

  A simple board has been designed to provide power to the DUTs and connect them to the test
  coordinator; it also hosts some temperature sensors for the DUTs to monitor. Banana jacks are used
  to connect the PSU and power the DUTs with \ptwelve{} and \pfive{}. The board's schematic can be
  seen on \vref{fig:sch:connection_board} --- it should be noted that for these tests the
  programming headers (and the voltage regulator that powers them) were not
  soldered. \Vref{fig:dia:test_setup} gives a broad overview of the test setup connectivity and
  component placement, highlighting the central role of this board, and a picture of it can be seen
  on \vref{fig:pic:connection_board}.

  The DI/OT crate allows a dual redundant power supply. Therefore, the two DUTs are connected to the
  same PMBus lines, in line with the MoniMods of the two redundant RaToPUS PSUs sharing the same
  bus, and irradiated together. In theory, this leaves open the possibility of a failure in one DUT
  taking down the whole bus and rendering the other DUT inaccessible; however, if this is a
  possibility, it is imperative to be aware of it and have an idea of the cross-section.

  The monitoring port connections of the two DUTs are shown on \vref{tab:con1,tab:con2}. It should
  be noted that, while both can read the same LM45-type temperature sensor, only one of the two can
  be connected to a PT100 sensor as there is a biasing circuit (a current source) and in such a case
  the readings would be double. Finally, also noteworthy is the fact that each DUT also monitors the
  current consumption on the other DUT's \pfive{} power rail.


  \subsubsection{Test Script Procedure}\label{sec:test_proc}

  Through a simple serial protocol running through the USB-provided tty, the Python script
  periodically orders the test coordinator to initiate PMBus transactions, querying the DUT for
  monitoring data and setting the fan speeds; in these tests this period has been set to \SI{5}{s}.
  The PSU-reported values for voltage and current are also read out and logged so that they can be
  compared against the values measured by the DUTs. In an effort to better cover the range of
  allowed operating conditions, the script also makes the PSU slowly oscillate the \ptwelve{} and
  \pfive{} power rail voltages and the fan speeds around their set values, within a predefined
  margin.

  The MoniMod can be remotely power-cycled. The test coordinator will automatically power-cycle both
  DUTs
  \begin{ilenum}
  \item once communicating with a DUT has consecutively failed a user-provided number of times (here
    set to 3)
  \item at pre-determined intervals, in order to gather more statistics on the robustness of the
    bootloader when irradiated
  \end{ilenum}

  Timestamped logs are saved in CSV format, easy to be parsed by the pandas libary for offline
  processing. Each line in the log contains the following information:
  \begin{ilenum}
  \item the timestamp
  \item a binary identifier of which of the two DUTs the line concerns
  \item a transaction count
  \item the software TMR error counter
  \item the I2C return code
  \item the monitoring data (voltages, currents, temperatures, tachometer readouts)
  \end{ilenum}
  The return code of a successful I2C transaction is zero. To collect one line of data many
  transactions are involved, so a single return code aggregate is logged by combining all the return
  codes with a bitwise OR operation. From these logs, an attempt to quantify the SEUs in the
  microcontroller will be made. As a first-order approximation, only serious (``obvious'')
  deviations from the PSU-reported values (or from the mean of the measured values for the
  temperatures) will be counted as SEUs.


  \subsection{DUT Positioning}\label{sec:test_pos}

  Two functionally identical boards are irradiated simultaneously; the two DUTs are hereby referred
  to as \MM{0} and \MM{1}. Although they are equivalent, there is a visual cue that sets them apart,
  the ``patch boards'' on \MM{1} are somewhat uneven. Although their function is the same there is a
  significant difference in their FW, as \MM{1}'s FW features the SW TMR technique to (hopefully)
  reduce SEUs.\footnote{It follows that the TMR error counter is only valid for \MM{1}.}

  \begin{figure}[bt]
    \centering
    \includegraphics[height=0.3\pageheight]{images/angled_support}
    \caption{The MoniMod stack mounted on the angled support (which was not used, eventually)}%
    \label{fig:pic:support}
  \end{figure}

  Initially an angled support was developed (see \vref{fig:pic:support}) in order to irradiate a
  two-module stack at an angle, requiring a single irradiation for the whole test. However, owing to
  the reduced beam time availability, other DUTs had to be irradiated in the same run; mounting
  those made it impossible to use the angled support.

  Without the angled support, the two-module stack required two irradiations: one to cover the
  microcontrollers and one for the fan control circuitry --- the distance between the centers of the
  two irradiations is \SI{45}{mm}. The impact one board would have on the beam intensity should be
  negligible considering our purposes but, for the sake of completeness, it has to be mentioned that
  the beam crosses \MM{1} first and then \MM{0}. The latter sits at the back of the stack, with its
  orientation represented by that of \MM{1} after a \SI{180}{\degree} rotation about the module's
  long axis (the microcontroller of \MM{0} is located behind that of \MM{1}).

  \begin{figure}[t]
    \centering
    \includegraphics[angle=90,height=0.95\textheight]{images/breakout_board}
    \caption{Schematics of the DUT powering and connection board}%
    \label{fig:sch:connection_board}
  \end{figure}

  \begin{figure}[p]
    \begin{centering}
      \codelisting[]
      {code/script_usage.txt}{script_usage}{Python Test Script usage}
    \end{centering}
  \end{figure}

  \subsection{Beam Details and Instrumentation Positioning}\label{sec:test_beam}

  \Vref{fig:dia:pif} shows a coarse diagram of the PIF facility irradiation room and the maze, where
  the instrumentation is placed. The initial proton beam for PIF is delivered from the PROSCAN
  accelerator with the help of the primary energy degrader, which allows setting a few discrete
  initial beam energies in the range from \SI{200}{MeV} down to \SI{30}{MeV}. The beam is
  subsequently guided to the Experimental Area where the PIF facility is located. For these tests,
  the primary energy was set to \SI{200}{MeV}.  The beam flux is measured by means of two ionization
  chambers IC1 and IC2, placed upstream and downstream the collimators, respectively.

  The counting of the IC1 and IC2 are calibrated to measure the beam flux at the position of the
  DUT, which is \SI{5}{cm} far away from the IC2 (downstream the collimator). That operation is done
  before the test by comparing the counting of the IC1 and IC2 with respect to the output of a
  scintillator detector, which is mounted in place of the DUT (\SI{5}{cm} from the IC2). The TID is
  provided by the facility and is given in \si{Gy(Si)}. A \SI{5}{cm} collimator is used to make the
  beam homogenous before exposing the DUTs to it.

  The location of the DUT in front of the beamline can also be seen on \vref{fig:dia:pif}. The test
  coordinator is placed near the DUT, but well outside of the influence of the proton beam, under
  the experiment table. Its USB port is plugged into the far end of an Ethernet USB extender, which
  is connected to the other end of the extender, at the control room computer, via the patch board
  on the wall. The PSU is located in the Maze and is also reached with the help of a USB extender
  via a cable that runs through the same patch board.

  \begin{figure}[t]
    \centering
    \begin{tikzpicture}
      \fill[pattern=north west lines] (-0.5, 0.5) rectangle (9.5, 0);
      \fill[pattern=north west lines] (9, 0) rectangle ++(0.5, -6.5);
      \fill[pattern=north west lines] (9, -6.5) rectangle ++(6.5, -0.5);
      \fill[pattern=north west lines] (15, -7) rectangle ++(0.5, -4.5);
      \fill[pattern=north west lines] (15, -11.5) rectangle ++(-2.5, 0.5);
      % \fill[pattern=north west lines] (12, -11.5) rectangle ++(0.5, 2);
      \fill[pattern=north west lines] (12.5, -10) rectangle ++(1, 0.5);
      \fill[pattern=north west lines] (13.5, -10) rectangle ++(0.5, 1.5);
      \fill[pattern=north west lines] (14, -8.5) rectangle ++(-7.5, 0.5);
      \fill[pattern=north west lines] (6.5, -8) rectangle ++(0.5, 1.5);
      \fill[pattern=north west lines] (7, -6.5) rectangle ++(-7, 0.5);
      \fill[pattern=north west lines] (-0.5, -6.5) rectangle (0, 0);

      \draw[thick] (0, 0) -- ++(9, 0) -- ++(0, -7) -- ++(6, 0) -- ++(0, -4) -- ++(-2.5, 0) -- ++(0, 1) -- ++(1.5, 0) -- ++(0, 2) -- ++(-7, 0) -- ++(0, 2) -- ++(-7, 0) -- (0, 0);

      \draw (1, -2) rectangle ++(3, -2);

      \begin{scope}
        \clip (0, 0) rectangle (9, -7);
        \filldraw[rotate=-7.5,fill=white] (3.5, -2.75) rectangle ++(10, 0.5);
      \end{scope}

      \draw[->,>=stealth] (11, -2) node[above] {Beam line} to[out=240,in=45] (8, -3);
      \draw[->,>=stealth] (5, -2) node[right] {Experiment table} to[out=180,in=30] (4, -2.5);
      \draw[->,>=stealth] (5, -1) node[right] {DUT} to[out=180,in=45] (2.4, -2.55);
      \filldraw[pattern=dots] (2.2, -2.75) circle[radius=0.25];

      \node[] at (2.5, -5) {\textbf{PIF cavern}};
      \node[] at (11, -7.5) {\textbf{Maze}};
      \node[] at (11, -10.5) {\textbf{Maze entrance}};

      \filldraw[pattern=crosshatch] (13.5, -7) rectangle ++(1, -0.4);
      \filldraw[pattern=crosshatch] (0.5, -6) rectangle ++(1, 0.4);

      \draw[->,>=stealth] (12, -6) node[above,align=center] {Instrumentation\\(Ethernet switch, PSU)} to[out=270,in=180] (13.5, -7.2);
      \draw[->,>=stealth] (3, -7) node[below] {Patch board} to[out=90,in=0] (1.5, -5.8);

      \node (ctrl) at (2, -10) {\includegraphics[width=3cm]{images/computer}};
      \node [right=0cm of ctrl] {\textbf{Control room}};
      \draw[<->,>=stealth] (0.75, -6.75) to[out=270,in=135] (1.25, -8.8);
    \end{tikzpicture}
    \caption{The PIF facility}%
    \label{fig:dia:pif}
  \end{figure}


  \FloatBarrier

  \section{Experimental Results}

  \subsection{Overview of the Runs}\label{sec:runs_overview}

  \begin{table}[b]
    \centering
    \caption{Microcontroller irradiation run details}%
    \label{tab:runs:uc}
      \begin{tabular}{lccccc}
        \toprule
        Run \# & Flux       & Fluence       & Cum. Fluence  & TID & Cum. TID \\
        \midrule
        Run 1 & \num{1e8}   & \num{1.69e11} & \num{1.69e11} & 100 & 100 \\
        Run 2 & \num{1e8}   & \num{1.69e11} & \num{3.38e11} & 100 & 200 \\
        Run 3 & \num{5e7}   & \num{5.07e10} & \num{3.89e11} &  30 & 230 \\
        Run 4 & \num{2.5e8} & \num{4.56e11} & \num{8.45e11} & 270 & 500 \\
        \multicolumn{6}{c}{\SI{18}{hrs} of annealing} \\
        Run 6 & \num{2.3e8} & \num{4.95e11} & \num{1.34e12} & 293 & 793 \\
        Run 7 & \num{2.3e8} & \num{1.47e11} & \num{1.49e12} &  87 & 880 \\
        \bottomrule
      \end{tabular}
  \end{table}

  \begin{table}[b]
    \centering
    \caption{Fan circuit irradiation run details}%
    \label{tab:runs:fans}
      \begin{tabular}{lccccc}
        \toprule
        Run \# & Flux       & Fluence       & Cum. Fluence  & TID & Cum. TID \\
        \midrule
        Run 5 & \num{2.7e8} & \num{5.66e11} & \num{5.66e11} & 335 & 335 \\
        \bottomrule
      \end{tabular}
  \end{table}

  Runs 1--5 were carried out on the first day of the MoniMod tests, July 1. The first four runs
  reached the end-of-life point of the microcontrollers. As the examination of the gafchromic film
  after the irradiation showed, \emph{the gate drivers were also accidentally irradiated in the
    first four runs} (see \vref{fig:pic:gaf2} for a clear view of this). Then, two more runs were
  carried out on the second day, July 2, after approximately 18 hours of annealing. For these last
  runs we decided to only irradiate and use in the setup a single DUT, in order to discount the
  possibility of any complex interactions between the two microcontrollers affecting the PMBus
  failures we were seeing. An overview of the beam conditions for each run with the fluence and TID
  figures can be seen on \vref{tab:runs:uc,tab:runs:fans} for the microcontroller and the fan
  circuit irradiations, respectively.

  \begin{figure}[p]
    \centering
    \begin{subfigure}[b]{\textwidth}
      \centering
      \includegraphics[height=0.3\pageheight]{images/After_irrad_2}%
      \caption{View 1, the microcontroller sits well inside the irradiation circle}%
      \label{fig:pic:gaf1}
    \end{subfigure}\vspace{1cm}
    \begin{subfigure}[b]{\textwidth}
      \centering
      \includegraphics[height=0.3\pageheight]{images/After_irrad_3}
      \caption{View 2, the gate drivers were also inside the circle of the microcontroller
        irradiation}%
      \label{fig:pic:gaf2}
    \end{subfigure}
    \caption{Two slightly different views of the gafchromic film after the two irradiations on the
      first day}%
    \label{fig:pic:gaf}
  \end{figure}

  Runs 1 and 2 are set up the same except that the voltage and fan oscillation ranges in the test
  script have been set to \SI{10}{\percent} for Run 1 and \SI{5}{\percent} for Run 2. For Run 3, the
  flux has been lowered to observe whether the SEFI cross-section would be affected. The log plots
  for these runs have been combined to \vref{fig:plt:day1}. I2C errors for \MM{0} are indicated by
  \textcolor{red}{red} vertical dotted lines; errors for \MM{1} are indicated by
  \textcolor{orange}{orange} dotted lines. Power-cycles are marked by thicker,
  \textcolor{NavyBlue}{blue} dotted lines.

  For Run 4, since the SEFI cross-section did not seem to be affected by the flux, we set the flux
  close to the maximum attainable value to quickly irradiate the microcontrollers up to
  \SI{500}{Gy}. Five minutes into this run, the fans stop spinning; this cannot be explained by a
  malfunction of the tachometer readout, as the \ptwelve{} current, as reported by the PSU, drops to
  zero. It can be explained by either the PWM peripheral or the gate driver chip being sensitive to
  radiation.

  Moving on to the MOSFETs and Run 5: here, since the beam seemed quite stable, we managed to set
  the flux even higher, to \num{2.7e8}, to quickly irradiate the fan control MOSFETs. The \ptwelve{}
  current starts increasing after \SI{\sim{} 150}{Gy} and an interesting event happens, the second
  fan on each MoniMod starts spinning again.

  After the end of run 5, at the end of the day, the script was restarted to obtain another short
  log. At that point, there was another very interesting occurrence: the \ptwelve{} power rails of
  \emph{both} boards started consuming as much current as they were given, hitting the PSU's current
  limiter of \SI{3.3}{A}, but the fans weren't spinning (as reported by the two DUTs). Monitoring
  data from the MoniMods also shows that this current is distributed equally. The PSU was quickly
  switched off to prevent any catastrophic damage to the DUTs and, at the next power-up, the setup
  behaved normally --- what's more, both MoniMods had their second fan working again.

  On the second day of irradiations, \MM{0} was removed from the test setup and only the
  microcontroller of \MM{1} was irradiated. Starting run 6, one can see on \vref{fig:plt:day1} that
  the fans don't seem to work (again) but there are no shorts observed, either. Around \SI{100}{Gy}
  in, the \pfive{} power consumption dropped to \SI{14}{mA} (from the \SI{18}{mA} it was hovering
  about to that point) and the DUT stopped responding to remote power-cycling. After a full PSU
  power cycle, however, it came back to life. One could assume that the regulator started
  misbehaving after the total dose of \SI{600}{Gy} it had been irradiated with thus far. The beam
  went out after providing us with a dose of \SI{293}{Gy}.

  In run 7, at \SI{87}{Gy} we lost communication with the microcontroller again, with a slightly
  different signature with respect to the previous run: as evidenced on \vref{fig:plt:day2i5}, the
  \pfive{} current went slightly up to \SI{19}{mA} instead of falling. The microcontroller never
  recovered after this point, having sustained a total dose of \SI{880}{Gy}.

  An interesting development emerged when trying to see whether the microcontroller would eventually
  wake up after this last run: even when shutting down the \pfive{} rail, the \ptwelve{} rail would
  report a \SIrange{600}{800}{mA} current draw. This cannot be explained by TID-induced damage in
  the MOSFETs, as those were irradiated on the first day.

  \begin{figure}[p]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_v12}
      \caption{\ptwelve{} rail voltage measurements}%
      \label{fig:plt:day1v12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_v5}
      \caption{\pfive{} rail voltage measurements}%
      \label{fig:plt:day1v5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_i12}
      \caption{\ptwelve{} rail current measurements}%
      \label{fig:plt:day1i12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_i5}
      \caption{\pfive{} rail current measurements}%
      \label{fig:plt:day1i5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_t1}
      \caption{LM45 temperature measurements}%
      \label{fig:plt:day1t1}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_f}
      \caption{Fan speed measurements}%
      \label{fig:plt:day1f}
    \end{subfigure}
    \caption{Day 1 runs}%
    \label{fig:plt:day1}
  \end{figure}

  \begin{figure}[p]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_v12}%
      \caption{\ptwelve{} rail voltage measurements}%
      \label{fig:plt:r5v12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_v5}
      \caption{\pfive{} rail voltage measurements}%
      \label{fig:plt:r5v5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_i12}%
      \caption{\ptwelve{} rail current measurements}%
      \label{fig:plt:r5i12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_i5}
      \caption{\pfive{} rail current measurements}%
      \label{fig:plt:r5i5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_t1}
      \caption{LM45 temperature measurements}%
      \label{fig:plt:r5t1}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_f}
      \caption{Fan speed measurements}%
      \label{fig:plt:r5f}
    \end{subfigure}
    \caption{Run 5 (fan driver irradiation) measurements}%
    \label{fig:plt:run5}
  \end{figure}

  \begin{figure}[tbp]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run_s_i12}
      \caption{\ptwelve{} rail current measurements}%
      \label{fig:plt:rsi12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run_s_f}
      \caption{Fan speed measurements}%
      \label{fig:plt:rsf}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run_as_i12}
      \caption{\ptwelve{} rail current measurements}%
      \label{fig:plt:rasi12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run_as_f}
      \caption{Fan speed measurements}%
      \label{fig:plt:rasf}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run4_t2}
      \caption{Run 4 PT100 temperature measurements}%
      \label{fig:plt:r4t2}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/run5_t2}
      \caption{Run 5 PT100 temperature measurements}%
      \label{fig:plt:r5t2}
    \end{subfigure}
    \caption{Short condition incident after MOSFET irradiation ended (subfigures
      \subref{fig:plt:rsi12} and \subref{fig:plt:rsf}) and\\ a short log at the power-up following
      that (subfigures \subref{fig:plt:rasi12} and \subref{fig:plt:rasf}); run 4 and 5 PT100\\
      temperature measurements (subfigures \subref{fig:plt:r4t2} and \subref{fig:plt:r5t2})}%
    \label{fig:plt:run_s}
  \end{figure}

  \begin{figure}[p]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_v12}
      \caption{\ptwelve{} rail voltage measurements}%
      \label{fig:plt:day2v12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_v5}
      \caption{\pfive{} rail voltage measurements}%
      \label{fig:plt:day2v5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_i12}
      \caption{\ptwelve{} rail current measurements}%
      \label{fig:plt:day2i12}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_i5}
      \caption{\pfive{} rail current measurements}%
      \label{fig:plt:day2i5}
    \end{subfigure}

    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_t1}
      \caption{LM45 temperature measurements}%
      \label{fig:plt:day2t1}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_f}
      \caption{Fan speed measurements}%
      \label{fig:plt:day2f}
    \end{subfigure}
    \caption{Day 2 runs}%
    \label{fig:plt:day2}
  \end{figure}

  \FloatBarrier

  \subsection{Analysis of the Test Results}

  To analyze various aspects of the log data and produce the plots, a Python script was written.

  \subsection{Functional interrupts and Interference Errors}

  Some upsets will cause the microcontroller to stop responding properly to I2C transactions until
  it is reset. It has not been investigated whether the microcontroller software can ``know'' that
  the transaction has failed so it can reset itself after a failed transaction, or if the I2C
  peripheral starts silently failing to properly perform transactions. Assuming the latter, we will
  consider those events to be Single Event Functional Interrupts (SEFIs),\footnote{As many other
    radiation reports, this document eschews the JEDEC definition of SEFI and instead takes it to
    mean that a processor either locks up, goes to standby mode, produces exceptions continuously,
    or goes into an unknown state.} and after three consecutive failed transactions both MoniMods
  are remotely power-cycled together.

  \begin{figure}[tbp]
    \centering
    \includegraphics[height=0.3\pageheight]{plots/run1_ierrs_zoom}
    \caption{Interference errors within a SEFI pattern}%
    \label{fig:plt:ierrs_zoom}
  \end{figure}

  If one takes a closer look at some of the plots, it can be observed that there are instances where
  right after a DUTs starts producing I2C transaction errors the other one follows, until they are
  power-cycled; one such instance is illustrated by \vref{fig:plt:ierrs_zoom}. Since both units
  share the same I2C bus, it is a reasonable assumption that the erratic MoniMod can sometimes
  interfere with the other one's transactions. With the help of a Python script, these instances are
  detected and counted separately as IERRs (Interference Errors). The consecutive failures that make
  up the SEFIs are also counted separately.

  The counts of SEFIs, isolated errors and interference errors, along with their cross-sections,
  calculated using the fluence figures from \vref{tab:runs:uc}, are shown on
  \vref{tab:runs:sefis}.\footnote{For runs 6 and 7 the non-responsive periods were removed to get
    the cross-sections with the rationale that since the microcontroller had already reached a dose
    greater than \SI{500}{Gy}, it would make sense that it would start misbehaving.} It can be seen
  that \MM{1} features \SI{30}{\percent} lower SEFI cross-sections, an order of magnitude less
  isolated errors, and causes an order of magnitude less interference errors to \MM{0}. This could
  be attributed to the mitigation measures implemented in its firmware.

  \subsubsection{I2C Error Codes}

  The possible I2C error codes that are defined on the ASF I2C driver header can be seen on
  \vref{tab:i2c_err}. First, a few words on the error codes are warranted. Three of the error codes
  on that table, \textbf{I2C\_ACK}, \textbf{I2C\_ERR\_ARBLOST} and
  \textbf{I2C\_ERR\_PACKAGE\_COLLISION}, aren't actually used by the driver code (hence the
  strikethrough). When the ``arbitration lost'' flag is set by the peripheral, however, one of
  \textbf{I2C\_ERR\_BAD\_ADDRESS} or \textbf{I2C\_ERR\_BUS} is returned, depending on whether the
  ``bus error'' flag is also set. Then, the \textbf{I2C\_ERR\_BUSY} can be returned when the bus is
  already busy when the test coordinator attempts to start a transaction; this would be seen in case
  a DUT were to hijack the bus. Finally, the driver has been extended to support PMBus-compliant
  Packet Error Checking (PEC) in I2C transactions; an error in the PEC checksum is indicated by an
  error code of \textbf{ERR\_BAD\_DATA}.

  As mentioned in \vref{sec:test_proc}, the error codes from all the transactions of a monitoring
  readout cycle are combined through a bitwise OR, to simplify logging. In retrospect, that was not
  a great choice: since the error codes are negative and in two's complement, there are too many
  combinations that will return -1 (any combination of \textbf{I2C\_NACK} or
  \textbf{I2C\_ERR\_BAD\_ADDRESS} with either \textbf{I2C\_ERR\_BUS} or
  \textbf{I2C\_ERR\_BAD\_DATA}), hampering any sort of extensive analysis, \eg{} on the error
  signature of SEFIs.

  In any case, the (combined) error codes have been retreived from the logs and plotted on
  \vref{fig:plt:i2c_errs_detail}. From their distribution, it it quite possible that all the -1
  errors (showing up as \textbf{COMB}) are actually \textbf{I2C\_NACK} combined with
  \textbf{I2C\_ERR\_BUS}, but the possibility of other errors being involved as well cannot be
  discounted.

  \begin{table}[b]
    \centering
    \caption{I2C error codes}%
    \label{tab:i2c_err}
    \begin{tabular}{ll}
      \toprule
      Error & Code \\
      \midrule
      I2C\_OK &  0 \\
      \sout{I2C\_ACK} & -1 \\
      I2C\_NACK & -2 \\
      \sout{I2C\_ERR\_ARBLOST} & -3 \\
      I2C\_ERR\_BAD\_ADDRESS & -4 \\
      I2C\_ERR\_BUS & -5 \\
      I2C\_ERR\_BUSY & -6 \\
      \sout{I2C\_ERR\_PACKAGE\_COLLISION} & -7 \\
      ERR\_BAD\_DATA & -9 \\
      \bottomrule
    \end{tabular}
  \end{table}

  \begin{figure}[tbp]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day1_i2c_errs_detail}
      \caption{Day 1 (two MoniMods on the bus)}%
      \label{fig:plt:i2c_errs_detail_day1}
    \end{subfigure}\hspace{5mm}%
    \begin{subfigure}[b]{0.45\textwidth}
      \centering
      \includegraphics[width=\columnwidth]{plots/day2_i2c_errs_detail}
      \caption{Day 2 (only one MoniMod on the bus)}%
      \label{fig:plt:i2c_errs_detail_day2}
    \end{subfigure}
    \caption{I2C error code distribution, broken down by error type}%
    \label{fig:plt:i2c_errs_detail}
  \end{figure}

  \begin{table}[b]
    \centering
    \caption{Microcontroller Single Event Functional Interrupts}%
    \label{tab:runs:sefis}
      \begin{tabular}{lcccccc}
        \toprule
        Run \# & MM0 SEFIs & MM1 SEFIs & MM0 ERRs  & MM1 ERRs & MM0 IERRs & MM1 IERRs \\
        \midrule
        Run 1  & 6         & 7         & 0         & 0        & 1         & 2 \\
        Run 2  & 6         & 6         & 0         & 1        & 0         & 0 \\
        Run 3  & 4         & 0         & 0         & 0        & 0         & 0 \\
        Run 4  & 16        & 10        & 14        & 0        & 0         & 18 \\
        \midrule
        Day 1 total:    & 32            & 23            & 14            & 1             & 1             & 20 \\
        Cross-sections: & \num{3.8e-11} & \num{2.7e-11} & \num{1.6e-11} & \num{1.2e-12} & \num{2.4e-12} & \num{2.4e-11} \\
        \midrule
        Run 6  & ---       & 15        & ---       & 3        & ---       & --- \\
        Run 7  & ---       & 3         & ---       & 4        & ---       & --- \\
        \midrule
        Day 2 total:    & --- & 18            & --- & 7             & --- & --- \\
        Cross-sections: & --- & \num{2.8e-11} & --- & \num{1.1e-11} & --- & --- \\
        \bottomrule
      \end{tabular}
  \end{table}

  \subsection{Fan Driver Short}

  The fan driver short observed after the end of run 5 (see \vref{fig:plt:rsi12,fig:plt:rsf})
  somehow resembles a Single Event Latch-up (SEL): there is a constant high current flow that only
  stops with a power cycle. However, the two DUTs report equal current draws; many devices must have
  suffered a SEL at the same time for that to happen, which should be near impossible.

  One could perhaps try to attribute that to the threshold voltage shift of the fan driver MOSFETs
  but there are some points that would require an explanation:
  \begin{ilenum}
  \item at a rate of \SI{6.3}{mV\per{}Gy}\footnote{As measured in the radiation report at
      \href{https://edms.cern.ch/document/2207602}{https://edms.cern.ch/document/2207602} for a
      V\textsubscript{gs} of \SI{10}{V}.} that shouldn't happen yet
  \item the threshold must have continued shifting after the irradiation ended, as the current was
    double what it was at the end of run 5
  \item annealing appears to have happened in a \emph{very} short time, as there was no such
    behavior observed after powering up the setup again
  \end{ilenum}
  It might be plausible for the quick annealing to have occured as the MOSFETs reached extremely
  high temperatures, due to the high current passing through them, but the two remaining arguments
  are left unexplained, making this theory less likely. For now, this shall remain unexplained.

  \subsection{Early Fan Driver Failure}

  Five minutes into run 4, the fans stop spinning; this cannot be explained by a malfunction of the
  tachometer readout, as the \ptwelve{} current (as reported by the PSU) drops to zero. It can be
  explained by either the PWM peripheral or the gate driver chip being sensitive to radiation.

  As mentioned in \vref{sec:runs_overview}, during run 4, the fans stopped spinning after a dose of
  \SI{273}{Gy}. Two explanations can be offered: either the PWM peripheral is sensitive to
  radiation, or the gate driver (which also happened to be in the irradiated area) is. The position
  of the latter in the fan control circuit can be seen on the schematic diagram on
  \vref{fig:sch:fan_ctrl}.

  \begin{figure}[t]
    \centering
    \includegraphics[height=0.35\pageheight]{images/fan-control-schematics-screen}
    \caption{Fan control circuit schematics}%
    \label{fig:sch:fan_ctrl}
  \end{figure}

  Unfortunately, both of these explanations are problematic. The PWM should be a relatively simple
  peripheral and since the rest of the microcontroller behaves uniformly well there is no obvious
  reason for the PWM to differ. However, fan driver functionality briefly returns on run 5 (see
  \vref{fig:plt:run5}), when we take the beam away from the microcontroller, allowing annealing
  effects to explain this but also failing to explain the consecutive loss of functionality on the
  second day. Concerning the gate driver chip, this has been tested to work fine up to \SI{500}{Gy}
  in a previous campaign.\footnote{this document can be found at
    \href{https://edms.cern.ch/document/2333834}{https://edms.cern.ch/document/2333834}} A
  difference in behavior in this setup might be explained by the use of a different (and
  problematic) lot in the prototypes but other explanations should be exhausted, first. There is a
  number of ways the radiation test differs that can be pointed out:
  \begin{ilenum}
  \item \SI{14}{V} are used instead of \SI{12}{V}
  \item a \SI{5}{V} input level is used instead of \SI{3.3}{V}
  \item an external bootstrap diode is used
  \item the \textbf{OUT} pin is grounded, changing the conditions for the the high-side driver
    internal level shifter
  \end{ilenum}
  For now, this shall also remain unexplained.

  \subsection{ADC Behavior}\label{sec:adc}

  Between the time the prototypes were received and the date of the radiation campaign there wasn't
  enough time to fully characterize them. Thus, a portion of that happened during the analysis of
  the radiation test data, along with the discovery of certain flaws in the design of these
  prototypes. First, the voltage measurements obtained on the first runs needed some scaling to
  match the PSU-reported values. These ADC inputs being unbuffered, it was assumed that the input
  resistance of the ADC was simply too low, skewing the results (indeed, it is less than
  \SI{3.5}{\kilo\ohm}, which is a bit too low). A multivariate fit was performed using the Python
  script and scaling the measurements with the results of that leading to values that match the
  ones the PSU reports perfectly. However, things got more complicated when looking at data from run
  4 and later: when the fan driving circuits stop drawing current, the adjusted measurements of the
  \pfive{} rail drift away from the PSU-reported values; no scaling is required to produce the right
  values anymore. It follows that this prototype suffers from some interference between channels, as
  the \ptwelve{} current goes up, the \pfive{} and LM45 channels report lower measurements. With
  that effect in mind, the scaling coefficients that were computed were eschewed and the
  measurements are plotted as-is, leading to the voltage measurement mismatches mostly observed in
  the plots of the first three runs --- and the end of run 5, as the current at the \ptwelve{} rail
  starts going up.

  The interference phenomenon aside, the ADC and its reference behaved very well with respect to
  TID, producing practically perfectly consistent results (as far as this testing method can
  discern) up to the end of the tests.

  One interesting observation regarding the ADC is its apparent occasional misconfiguration. One can
  notice this by looking at \MM{1}'s measurements at the beginning of run 1, around \SI{5}{Gy} in,
  and \MM{0}'s values just before the \SI{200}{Gy} mark. Many values shift to invalid ranges
  concurrently, but if one notices closely they still track the waveform shape --- they are really
  just shifted. This could be easily solved by having the firmware occasionally re-write the ADC
  configuration (blind scrubbing).


  \FloatBarrier

  \section{Conclusions}

  The cross-section of SEFIs in the form of communication getting lost is \num{2.8e-11}, a figure
  perfectly consistent with the estimates of the SAMD21 radiation campaign. Functionality can be
  quickly restored after a SEFI by remotely power-cycling the microcontroller.

  The software fault mitigation approach has a measurable effect, reducing the cross-section of
  SEFIs by \SI{30}{\percent} and that of other I2C errors (including interference with the other
  device) by an order of magnitude.

  Fan driving capability became erratic and then subsequently lost after a dose of \SI{\sim{}
    275}{Gy}. Either the PWM peripheral or the gate driver is responsible and there is not enough
  data to infer which one with any confidence. A short appeared in both MoniMods --- at the same
  time --- and current explanations for it are lacking.

  The ADC and its reference are stable enough to continue gathering valid measurements well after
  \SI{500}{Gy} and only suffer from the occasional configuration corruption, which can be fixed by
  implementing blind scrubbing.

  Some flaws in the prototype design were found and need to be fixed for the next revision, \eg{}
  the fan control interference in voltage and PT100 measurements.


\end{maincontentstyled}

\end{document}

% Local Variables:
% TeX-engine: luatex
% TeX-command-extra-options: "-shell-escape"
% fill-column: 100
% End:
