\subsection{Timing error}
As a timing error we define WR Switch not being able to provide its slave
nodes/switches with correct timing information consistent with the rest of the
WR network.

\noindent This section contains the list of faults leading to a timing error.

\subsubsection{\bf \emph{PTP/PPSi} went out of \texttt{TRACK\_PHASE}}
		\label{fail:timing:ppsi_track_phase}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Slave}
			\item [] \underline{Description}:\\
				If the \emph{PTP/PPSi} WR servo goes out of the \texttt{TRACK\_PHASE}
				state, this means something bad has happened and switch lost the
				synchronization to its Master.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPtpServoState.<n>} -- PTP servo state as string\\
				\texttt{WR-SWITCH-MIB::wrsPtpServoStateN.<n>} -- PTP servo state as number\\
				\texttt{WR-SWITCH-MIB::wrsPtpServoStateErrCnt}\\
				\texttt{WR-SWITCH-MIB::wrsPTPStatus}
			\item [] \underline{Note}: PTP servo state is exported as a string and a number.
		\end{packed_enum}

\subsubsection{\bf Offset jump not compensated by Slave}
		\label{fail:timing:offset_jump}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Slave}
			\item [] \underline{Description}:\\
				This may happen if Master resets its WR time counters (e.g. because it
				lost the link to its Master higher in the hierarchy or to external
				clock), but Slave switch does not follow the jump.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPtpClockOffsetPs.<n>} -- value of the offset in ps\\
				\texttt{WR-SWITCH-MIB::wrsPtpClockOffsetPsHR.<n>} -- 32-bit signed value of the offset in ps; with
				saturation on overflow and underflow\\
				\texttt{WR-SWITCH-MIB::wrsPtpClockOffsetErrCnt}\\
				\texttt{WR-SWITCH-MIB::wrsPTPStatus}
		\end{packed_enum}

\subsubsection{\bf Detected jump in the RTT value calculated by \emph{PTP/PPSi}}
		\label{fail:timing:rtt_jump}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Slave}
			\item [] \underline{Description}:\\
				Once WR link is established round-trip delay (RTT) can change smoothly
				due to the temperature variations. If a sudden jump is detected, that
				means erroneous timestamp was generated either on Master or Slave side.
				One cause of that could be the wrong value of the t24p transition point.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPtpRTT.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPtpRTTErrCnt}\\
				\texttt{WR-SWITCH-MIB::wrsPTPStatus}
		\end{packed_enum}

\subsubsection{\bf Wrong $\Delta_{TXM}$, $\Delta_{RXM}$, $\Delta_{TXS}$,
		$\Delta_{RXS}$ values are reported to the \emph{PTP/PPSi} daemon}
		\label{fail:timing:deltas_report}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If \emph{PTP/PPSi} doesn't get the correct values of fixed hardware delays,
				it won't be able to calculate a proper Master-to-Slave delay. Although
				the estimated offset in \emph{PTP/PPSi} is close to 0, WRS won't be
				synchronized to Master with the sub-nanosecond accuracy.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPtpDeltaTxM.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPtpDeltaRxM.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPtpDeltaTxS.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPtpDeltaRxS.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPTPStatus}
		\end{packed_enum}

\subsubsection{\bf \emph{SoftPLL} became unlocked}
		\label{fail:timing:spll_unlock}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If \emph{SoftPLL} loses lock, for any reason, Slave or Grand Master
				switch can no longer be syntonized and phase aligned with its time
				source. WRS in Free-running mode without properly locked Helper PLL is
				not able to perform reliable phase measurements for enhancing Rx
				timestamps resolution. For Grand Master the reason of \emph{SoftPLL}
				going out of lock might be disconnected 1-PPS/10MHz signals or external
				clock down. In that case, the switch goes into Free-running mode and
				resets WR time. Later we will have a holdover to keep the Grand Master
				switch disciplined in case it loses external reference.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsSpllMode}\\
				\texttt{WR-SWITCH-MIB::wrsSpllSeqState}\\
				\texttt{WR-SWITCH-MIB::wrsSpllAlignState}\\
				\texttt{WR-SWITCH-MIB::wrsSpllHlock}\\
				\texttt{WR-SWITCH-MIB::wrsSpllMlock}\\
				\texttt{WR-SWITCH-MIB::wrsSpllDelCnt}\\
				\texttt{WR-SWITCH-MIB::wrsSoftPLLStatus}
		\end{packed_enum}

\subsubsection{\bf \emph{SoftPLL} has crashed/restarted}
		\label{fail:timing:spll_crash}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(depends on SoftPLL mem read), (require changes in lm32 software)}
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If LM32 software crashes or restarts for some reason, its state may be
				either reseted or random (if for some reason variables were overwritten
				with junk values). In such case PLL becomes unlocked and switch is not
				able to provide synchronization to other devices.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsSpllIrqCnt}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntSPLL} \emph{(not yet implemented)}
			\item [] \underline{Note}: We have a similar mechanism as in the
				\emph{wrpc-sw} to detect if the LM32 program has restarted because of
				the CPU following a NULL pointer. However, LM32 program hangs on
				re-initialization phase. 
				In addition to that, we can detect if
				\emph{SoftPLL} is hanging (but not restarted) based on irq counter.
		\end{packed_enum}

\subsubsection{\bf Link to WR Master is down for slave}
		\label{fail:timing:master_down}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR (will become WARNING with the
				switch-over)
			\item [] \underline{Mode}: \emph{Slave}
			\item [] \underline{Description}:\\
				In that case, WR Switch loses timing reference, resets counters
				responsible for keeping the WR time, and starts operating in a
				Free-Running Master mode.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsSlaveLinksStatus}
		\end{packed_enum}

\subsubsection{\bf Link to WR Master is up for master}
		\label{fail:timing:master_up}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Grand Master}, \emph{Free-Running Master}
			\item [] \underline{Description}:\\
				In that case there is probably wrong configuration. Neither the
				\emph{Grand Master} nor the \emph{Free-Running Master} should be
				connected to another WR Master.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsSlaveLinksStatus}
		\end{packed_enum}

\subsubsection{\bf PTP frames don't reach ARM}
		\label{fail:timing:no_frames}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				In this case, \emph{PTP/PPSi} will fail to stay synchronized and provide
				synchronization. Even if WR servo is in the \texttt{TRACK\_PHASE} state,
				it calculates new phase shift based on the Master-to-Slave delay
				variations. To calculate these variations, it still needs timestamped
				PTP frames flowing. There could be several causes of such fault:
				\begin{itemize}
					\item HDL problem (e.g. SwCore or Endpoint hanging)
					\item \emph{wr\_nic.ko} driver crash
					\item wrong VLANs configuration
				\end{itemize}
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusPtpTxFrames.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusPtpRxFrames.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusLink.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPTPFramesFlowing}
			\item [] \underline{Note}: If the kernel driver crashes, there is not much
				we can do. We end up with either our system frozen or a reboot. For
				wrong VLAN configuration and HDL problems we can monitor if PTP frames
				are flowing on Slave port(s) of WRS and raise an alarm (change status
				word) if they don't flow anymore. We should combine this with the link
				status (up/down). If VLANs are mis configured, we don't receive PTP
				frames, but the link is still up. This could let us distinguish from a
				lack of frames due to the link down (which is a separate issue).
		\end{packed_enum}

\subsubsection{\bf Detected SFP not supported for WR timing}
		\label{fail:timing:wrong_sfp}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				By not supported SFP for WR timing we mean a transceiver that doesn't
				have the \emph{alpha} parameter and fixed hardware delays defined in the
				SFP database (\texttt{CONFIG\_SFPXX\_PARAMS} parameters in dot-config). The consequence is
				\emph{PTP/PPSi} not having the right values to estimate link asymmetry.
				Despite \emph{PTP/PPSi} offset being close to 0 \emph{ps}, the device won't
				be properly synchronized.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusConfiguredMode.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpVN.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpPN.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpVS.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpInDB.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpGbE.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpError.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsSFPsStatus}
			\item [] \underline{Note}: WRS configuration allow to disable this check on some ports.
				That is because ports may be used for regular (non-WR) PTP
				synchronization or for data transfer only (no timing). In that case any
				Gigabit SFP can be used (also copper). Detecting if a non-Gigabit
				Ethernet SFP is plugged into the cage is covered in a separate issue
				\ref{fail:other:sfp}.
		\end{packed_enum}

\subsubsection{\bf \emph{PTP/PPSi} process has crashed/restarted}
		\label{fail:timing:ppsi_crash}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If the \emph{PTP/PPSi} daemon crashes we lose any synchronization
				capabilities. Then \texttt{Monit} restarts the missing process.
				The number of process starts is stored in a corresponding object.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsStartCntPTP}\\
				\texttt{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\texttt{HOST-RESOURCES-MIB::hrSWRunName.<n>}
		\end{packed_enum}

\subsubsection{\bf \emph{HAL} process has crashed/restarted}
		\label{fail:timing:hal_crash}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If \emph{HAL} crashes, \emph{PTP/PPSi} is not able to communicate with
				the hardware i.e. read phase shift, get timestamps, phase shift the
				clock etc. When \emph{HAL} crashes, \texttt{Monit} will restart it.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsStartCntHAL}\\
				\texttt{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\texttt{HOST-RESOURCES-MIB::hrSWRunName.<n>}
		\end{packed_enum}

\subsubsection{\bf Wrong configuration applied}
		\label{fail:timing:wrong_config}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(to be done later)}
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{all}
			\item [] \underline{Description}:\\
				If there is a wrong configuration applied to the \emph{PTP/PPSi} or HAL
				(i.e.  wrong fixed delays, mode of operation etc.) there is not much we
				can do. The responsibility of WR experts (or person deploying the
				system) is to make sure that all the devices have a correct
				configuration. Later we can only generate warnings, if the key
				configuration options are changed remotely (e.g. Grand Master mode to
				Free-running Master or updated fixed hardware delays values).\\
				For misconfigured VLANs, we can monitor if PTP frames are flowing on
				Slave port(s) of the switch.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: monitor remote updates of key configuration
				options (PTP/WR mode, fixed hardware delays)
		\end{packed_enum}

\subsubsection{\bf Switchover failed}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Mode}: \emph{Slave}, \emph{Grand Master}
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				In case the primary timing link breaks, switchover is responsible for
				seamless switching to the backup one to keep the device in sync. If WRS
				operates in a \emph{Slave} mode, switchover is about switching
				between two (or more) WR links to one or multiple WR Masters. If it
				operates in a \emph{Grand Master} mode, it is about broken/lost
				connection to an external reference and switching to a backup WR link
				(another WR Master). Regardless of the configuration, if we fail to
				switch-over to a backup link (e.g. because it is down), WRS resets
				the time counters and continue the operation as a Free-Running Master.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: we should probably use parameters reported by
				the backup channel(s) of the SoftPLL and the backup PTP servo to be able
				to detect and report that something went wrong.
		\end{packed_enum}

\subsubsection{\bf Holdover for too long}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Mode}: \emph{Grand Master}
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				Signaling active holdover is one thing, but if a Grand Master switch is
				kept in holdover for too long, it may drift away from the ideal external
				reference too much. All devices in a WR network will be still
				synchronized, but no longer in sync with the external reference.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{packed_enum}

\newpage
\subsection{Data error}
As a data error we define WR Switch not being able to forward Ethernet traffic
between devices connected to the ports.\\

\noindent This section contains the list of faults leading to a data error.

\subsubsection{\bf Link down}
		\label{fail:data:link_down}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE  \emph{(to be changed later for switchover)}
			\item [] \underline{Severity}: ERROR (will be WARNING with the
				switch-over)
			\item [] \underline{Description}:\\
				This obviously stops the flow of frames on an Ethernet port and there is
				not much we can do besides reporting an error. Topology redundancy is a
				cure for that (if backup link is fine, and reconfiguration does not
				fail). There might be several causes of a link down:
				\begin{itemize}
					\item unplugged fiber
					\item broken fiber
					\item broken SFP
					\item wrong(non-complementary) pair of WDM SPFs used
				\end{itemize}
				However, we are not able to distinguish between them inside the switch.
			\item [] \underline{SNMP objects}:\\
				\texttt{IF-MIB::ifOperStatus.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusLink.<n>}
		\end{packed_enum}

\subsubsection{\bf Fault in the Endpoint's transmission/reception path}
		\label{fail:data:ep_txrx}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				This fault covers various errors reported by the Endpoint, e.g. FIFO
				underrun in the Tx PCS or FIFO overrun in the Rx PCS, receiving invalid
				\emph{8b10b} code, CRC error etc.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPstatsTXUnderrun.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXOverrun.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXInvalidCode.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXSyncLost.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXPfilterDropped.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXPCSErrors.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXCRCErrors.<n>}
		\end{packed_enum}

\subsubsection{\bf Problem with the SwCore or Endpoint HDL module}
		\label{fail:data:swcore_hang}
		\begin{packed_enum}
    \item [] \underline{Status}: TODO (add monitoring of the Endpoint hangs, depend on
      HDL)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If the SwCore is hanging, then the Ethernet forwarding is not
				performed on one or multiple ports. We have a HDL watchdog module which
				constantly monitors if the SwCore is not stuck. If such a situation is
				detected the whole SwCore is reset, all the frames enqueued in the
				Endpoints are acknowledged and lost. After this the switch can continue
				its operation and the watchdog triggers counter is incremented.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsGwWatchdogTimeouts}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsTXFrames.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPstatsForwarded.<n>}
			\item [] \underline{Note}: For Endpoint monitoring we could compare
				per-port \emph{RTUfwd} counter with the \emph{Tx} Endpoint counter for
				each port. \emph{RTUfwd} counts all forwarding decisions from RTU to the
				port $<$n$>$ (excluding PTP frames from NIC). If the sum of this number
				and RTU decisions generated from NIC is equal to the number of frames
				actually transmitted by the Endpoint, then everything works fine.
		\end{packed_enum}

\subsubsection{\bf RTU is full and cannot accept more requests}
		\label{fail:data:rtu_full}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If RTU is full for a given port, it's not able to accept more requests
				and generate new responses. In such case frames are dropped in the
				Rx path of the Endpoint.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCh-MIB::wrsPstatsRXDropRTUFull.<n>}
		\end{packed_enum}

\subsubsection{\bf Too much HP traffic / Per-priority queue full}
		\label{fail:data:too_much_HP}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(depends on HDL)}
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If we get too much High Priority traffic, then SwCore will be busy all
				the time forwarding HP frames. This way regular/best effort traffic
				won't be flowing through the switch. In the extreme case, HP traffic
				queue may become full and we start losing HP frames, which is
				unacceptable.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPstatsFastMatchPriority.<n>} - HP frames on a port\\
				\texttt{WR-SWITCH-MIB::wrsPstatsRXFrames<n>} - Total number of Rx frames on
				the port\\
				\texttt{WR-SWITCh-MIB::wrsPstatsRXPrio0.<n>} - Rx priorities 0-7\\
				\texttt{[..]}\\
				\texttt{WR-SWITCh-MIB::wrsPstatsRXPrio7.<n>}
			\item [] \underline{Note}: we need to get from SwCore the information
				about per-priority queue utilization, or at least an event when it's
				full.
		\end{packed_enum}

\subsubsection{\bf \emph{RTUd} has crashed}
		\label{fail:data:rtu_crash}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If \emph{RTUd} crashed, traffic would be still routed between the WRS ports, but
				only based on the already existing static and dynamic rules. There would be
				no learning or aging functionality. This means, MAC addresses wouldn't be
				removed from the RTU table if a device is disconnected from a port.
				Without learning, each frame with yet unknown destination MAC would be
				broadcast to all ports (within a VLAN). When \emph{RTUd} crashes,
				\texttt{Monit} will restart it.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsStartCntRTUd}\\
				\texttt{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}\\
				\texttt{HOST-RESOURCES-MIB::hrSWRunName.<n>}
		\end{packed_enum}

\subsubsection{\bf Network loop - two or more identical MACs on two or more ports}
		\label{fail:data:net_loop}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(to be done later)}
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				In such case we have a ping-pong situation. If two ports receive frames
				with the same source MAC, it is learned on one of these ports. Then if
				it comes on a second port, it is learned on a second port, and removed
				from the first one. Later, MAC is learned again on the first port, and
				removed from the MAC table for the second port, and so on. This
				situation is a network configuration problem or eRSTP failure.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: we need to monitor the \emph{rtu\_stat} to
				detect ping-pong in the RTU table.
		\end{packed_enum}

\subsubsection{\bf Wrong configuration applied (e.g. wrong VLAN config)}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(to be done later)}
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				The same problem as described in the timing fault
				\ref{fail:timing:no_frames}
		\end{packed_enum}

\subsubsection{\bf Topology Redundancy failure}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				Topology redundancy let's us prevent from losing data when the primary
				uplink is down for some reason. However, if a backup link is also down
				or reconfiguration to backup link fails, we start losing data and an
				alarm should be raised.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: One thing we need to report is a backup link(s)
				going down, but we should also think about how to determine if there is
				some problem with eRSTP and if it may fail/has failed if the primary
				link is down.
		\end{packed_enum}

\newpage
\subsection{Other errors}
\label{sec:other_fail}

\subsubsection{\bf WR Switch did not boot correctly}
		\label{fail:other:boot}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO (add rebooting system when boot is
				 not successful, add stop restarting system after defined number of restarts)
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				Every time the switch boots, we verify that all the services have
				started and are running correctly. If any of them fails, an alarm is
				raised.

				The SNMP object \texttt{wrsBootSuccessful} says if a WRS has booted
				correctly, FPGA is programmed, all kernel drivers are loaded and all
				daemons are up and running. If it's not the case, we report what went
				wrong:
				\begin{itemize}
					\item status of reading HW information from dataflash
					\item status of programming FPGA and LM32
					\item status of loading kernel modules
					\item status of starting userspace daemons
				\end{itemize}
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsBootSuccessful} -- status word informing whether switch booted correctly\\
				\texttt{WR-SWITCH-MIB::wrsRestartReason}\\
				\texttt{WR-SWITCH-MIB::wrsRestartReasonMonit}\\
				\texttt{WR-SWITCH-MIB::wrsConfigSource}\\
				\texttt{WR-SWITCH-MIB::wrsConfigSourceUrl}\\
				\texttt{WR-SWITCH-MIB::wrsBootHwinfoReadout}\\
				\texttt{WR-SWITCH-MIB::wrsBootLoadFPGA}\\
				\texttt{WR-SWITCH-MIB::wrsBootLoadLM32}\\
				\texttt{WR-SWITCH-MIB::wrsBootKernelModulesMissing}\\
				\texttt{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing}
			\item [] \underline{Note}: 
				The idea is to reboot the system if it was not able to boot correctly.
				Then we use the scratchpad registers of the processor to keep
				the boot count. If the value of this counter is more than X we stop
				rebooting and try to have a system running with at least \emph{dropbear}
				for SSH and \emph{net-snmp} to allow remote diagnostics. If on the other
				hand the switch has booted correctly, we set the boot count to 0.
		\end{packed_enum}

\subsubsection{\bf Dot-config error}
		\label{fail:other:dot-config}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				Dot-config file used to configure the switch can be stored locally or
				retrieved from a central server. Additionally URL to the remote dot-config
				can be retrieved via DHCP request. When dot-config is fetch from the server
				it has to be verified before being applied. If downloading or verification has
				failed an alarm is raised.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsBootSuccessful} -- status word informing
					whether switch booted correctly\\
				\texttt{WR-SWITCH-MIB::wrsConfigSource} -- source of a dot-config,
					local, remote or get URL to the dot-config via DHCP. When
					\texttt{wrsConfigSource} is set to the \texttt{tryDhcp}, then failure of
					getting dot-config's URL via DHCP does not rise an error in
					\texttt{wrsBootSuccessful}\\
				\texttt{WR-SWITCH-MIB::wrsConfigSourceUrl} -- path to the dot-config
					on a server (if not local)\\
				\texttt{WR-SWITCH-MIB::wrsBootConfigStatus} -- result of the dot-config verification
		\end{packed_enum}

\subsubsection{\bf Any userspace daemon has crashed/restarted}
		\label{fail:other:daemon_crash}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO \emph{(depends on monit)}
			\item [] \underline{Severity}: ERROR / WARNING (depending on the process)
			\item [] \underline{Description}:\\
				Running processes are monitored by \texttt{Monit}. When any of them
				crashes, \texttt{Monit} restarts a missing process and increments a
				corresponding start counter. If a process is restarted 5 times within
				100 seconds, then the entire switch is restarted.
			\item [] \underline{SNMP objects}:\\
				\texttt{HOST-RESOURCES-MIB::hrSWRunName.<n>} - list of processes in standard MIB\\
				\texttt{WR-SWITCH-MIB::wrsStartCntHAL}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntPTP}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntRTUd}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntSshd}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntHttpd}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntSnmpd}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntSyslogd}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntWrsWatchdog}\\
				\texttt{WR-SWITCH-MIB::wrsStartCntSPLL} \emph{(not implemented)}\\
				\texttt{WR-SWITCH-MIB::wrsBootUserspaceDaemonsMissing} - number of missing processes\\
				\texttt{WR-SWITCH-MIB::wrsBootSuccessful} - status word informing whether switch booted correctly
			\item [] \underline{Note}: We shall distinguish between crucial
				processes - error should be reported if one of them crashes; and less
				important processes (warning should be reported if they crash). If any
				of the processes has crashed, we need to restart it and increment a
				per-process counter reported through the SNMP.

				Crucial processes (Error report if any of them crashes):
				\begin{itemize}
					\item \emph{PTP/PPSi}
					\item \emph{wrsw\_rtud} - after adding configuration preserving code
						on restart, RTUd could be crossed out from this list
					\item \emph{wrsw\_hal}
				\end{itemize}
				Less critical processes (Restarting them and Warning generation is
				enough):
				\begin{itemize}
					\item \emph{dropbear}
					\item \emph{udhcpc}
					\item \emph{rsyslogd}
					\item \emph{snmpd}
					\item \emph{lighttpd}
					\item \emph{TRUd/eRSTPd} - not yet implemented
				\end{itemize}

				\emph{wrsw\_rtud} - we need to set the flag informing the process has
				crashed so that when it runs again it knows that HDL is already
				configured. It should not erase static entries in the RTU table (e.g.
				multicasts for PTP), the static entries set by-hand as well as VLANs.
				Dynamic entries are not a problem. RTUd can learn all MACs after
				restarting. The only consequence will be increased network traffic due
				to frames broadcast until all the MACs are learned. In general,
				the source code has to be checked to make sure what is cleared on the
				startup and modified to preserve the configuration.\\

				\emph{TRUd/eRSTPd} - topology reconfiguration is done in hardware if
				needed, the daemon is used only to configure the TRU/RTU HDL module.
				However, the story is similar as with the RTUd. If eRSTPd crashes, we
				need to store this information so that when it runs again, it does not
				erase the whole configuration. Also if a topology reconfiguration
				happens while eRSTPd is down, HDL should keep the flag for the eRSTPd so
				that it's aware the backup link is active.
		\end{packed_enum}

\subsubsection{\bf Kernel crash}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: ERROR
			\item [] \underline{Description}:\\
				If the Linux kernel has crashed, system reboots. Until the next boot we
				have no synchronization, no SNMP to report the status, FPGA may be still
				forwarding Ethernet traffic, but based on dynamic and static routing
				rules from before the crash. Based on the SNMP objects below it is
				possible to figure out that reboot took place and what was the reason of
				the last reboot.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsBootCnt}\\
				\texttt{WR-SWITCH-MIB::wrsRebootCnt}\\
				\texttt{WR-SWITCH-MIB::wrsRestartReason}\\
				\texttt{WR-SWITCH-MIB::wrsFaultIP} \emph{(not implemented)}\\
				\texttt{WR-SWITCH-MIB::wrsFaultLR} \emph{(not implemented)}
			\item [] \underline{Note}:
				Unfortunately, right now it is not possible to distinguish whether the
				reboot was caused by the kernel panic function or the \texttt{reboot}
				command. Preserving the state of IP and LR registers has to be
				implemented.
		\end{packed_enum}
\subsubsection{\bf System nearly out of memory}
		\label{fail:other:no_mem}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				We need to monitor the amount of free memory, report it through SNMP and
				raise an alarm if it's extremely low (but still enough to keep the
				system running).
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsMemoryTotal}\\
				\texttt{WR-SWITCH-MIB::wrsMemoryUsed}\\
				\texttt{WR-SWITCH-MIB::wrsMemoryUsedPerc} - percentage of used memory\\
				\texttt{WR-SWITCH-MIB::wrsMemoryFree}\\
				\texttt{WR-SWITCH-MIB::wrsMemoryFreeLow} - warning or error on low memory
		\end{packed_enum}
\subsubsection{\bf Disk space low}
		\label{fail:other:no_disk}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				We need to monitor the amount of free disk space, report it through SNMP
				and raise an alarm if it's extremely low (but still enough to keep the
				system running).
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsDiskMountPath.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskSize.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskUsed.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskFree.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskUseRate.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskFilesystem.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsDiskSpaceLow} - warning or error on low disk space\\
				\texttt{HOST-RESOURCES-MIB::hrStorageDescr.<n>}\\
				\texttt{HOST-RESOURCES-MIB::hrStorageSize.<n>}\\
				\texttt{HOST-RESOURCES-MIB::hrStorageUsed.<n>}
			\item [] \underline{Note}:
				Objects like \texttt{HOST-RESOURCES-MIB::hrStorage*.<n>} are available
				via standard MIB. The same functionality is implemented in
				\texttt{WR-SWITCH-MIB} objects \texttt{wrsDisk*.<n>} (to ease the
				implementation of \texttt{wrsDiskSpaceLow}).
		\end{packed_enum}

\subsubsection{\bf CPU load too high}
		\label{fail:other:cpu}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				On a healthy switch the average CPU load should be below \emph{0.1}.
				Some actions like SNMP queries or web interface activity may increase
				the average system load. The system load averages for the past 1, 5 and
				15 minutes are exported via SNMP objects. Additionally
				\texttt{wrsCpuLoadHigh} alerts when the load is too high.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsCPULoadAvg1min}\\
				\texttt{WR-SWITCH-MIB::wrsCPULoadAvg5min}\\
				\texttt{WR-SWITCH-MIB::wrsCPULoadAvg15min}\\
				\texttt{WR-SWITCH-MIB::wrsCpuLoadHigh} - warning or error when CPU load too high
		\end{packed_enum}

\subsubsection{\bf Temperature inside the box too high}
		\label{fail:other:temp}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If the temperature raises too high we might break our electronics inside
				the box. It also means that most probably one or both of the fans inside
				the box are broken and should be replaced. There are 4 temperature
				sensors monitored:
				\begin{itemize}
					\item \emph{IC19} - temperature below the FPGA
					\item \emph{IC20}, \emph{IC17} - temperature near the SCB power supply
						circuit
					\item \emph{IC18} - temperature near the VCXO and PLLs (AD9516,
						CDCM6100)
				\end{itemize}
				\texttt{wrsTemperatureWarning} is raised when the temperature read from
				any of these sensors exceeds a threshold configured in the
				\emph{dot-config}. When at least one threshold temperature is not set
				\texttt{wrsTemperatureWarning} is set to \emph{Threshold-not-set}.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsTempFPGA}\\
				\texttt{WR-SWITCH-MIB::wrsTempPLL}\\
				\texttt{WR-SWITCH-MIB::wrsTempPSL}\\
				\texttt{WR-SWITCH-MIB::wrsTempPSR}\\
				\texttt{WR-SWITCH-MIB::wrsTempThresholdFPGA}\\
				\texttt{WR-SWITCH-MIB::wrsTempThresholdPLL}\\
				\texttt{WR-SWITCH-MIB::wrsTempThresholdPSL}\\
				\texttt{WR-SWITCH-MIB::wrsTempThresholdPSR}\\
				\texttt{WR-SWITCH-MIB::wrsTemperatureWarning}
		\end{packed_enum}

\subsubsection{\bf Not supported SFP plugged into the cage (especially non 1-Gb SFP)}
		\label{fail:other:sfp}
		\begin{packed_enum}
			\item [] \underline{Status}: DONE
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If a not supported Gigabit optical SFP is plugged into the cage, then
				it's a timing issue \ref{fail:timing:wrong_sfp}. However, if a non 1-Gb
				SFP is used, then no Ethernet traffic would be flowing on that port.
				It's due to the fact, that we don't have 10/100Mbit Ethernet implemented
				inside the WRS.
			\item [] \underline{SNMP objects}:\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpVN.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpPN.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpVS.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpGbE.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsPortStatusSfpError.<n>}\\
				\texttt{WR-SWITCH-MIB::wrsSFPsStatus} - status word for SFPs' status
		\end{packed_enum}

\subsubsection{\bf File system / Memory corruption}
		\label{fail:other:memory}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
			\item [] \underline{SNMP objects}: \emph{(none)}
			\item [] \underline{Note}: how shall we detect this? Based on the
				\emph{dmesg} errors reported by UBI and system in general?  This is bad,
				crazy things may happen, we can't do much about it.
		\end{packed_enum}

\subsubsection{\bf Kernel freeze}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
				If kernel freezes we can do nothing. It can freeze e.g. due to some
				infinite loop in the irq handler. It's like with the power failure,
				somebody has to go to the place where WRS is installed and
				investigate/restart the device.
			\item [] \underline{SNMP objects}: \emph{(none)}
			\item [] \underline{Note}:
				If we have watchdog in our CPU it should be used.
		\end{packed_enum}

\subsubsection{\bf Power failure}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
				Power failure may be either a WRS problem (i.e. broken power supply
				inside the switch) or an external problem (i.e. providing voltage to the
				device). There is not much reporting we can do in such case. It's up to
				the Network Management Station to raise an alarm if the SNMP Agent does
				not respond to the SNMP requests.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{packed_enum}

\subsubsection{\bf Hardware problem}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
				If any crucial hardware part breaks we'll most probably notice it as one
				(or multiple) timing / data errors described previously. Besides that,
				we don't have any self-diagnostics on-board. Few examples:
				\begin{itemize}
					\item DAC / VCO - problems with synchronization
					\item cooling	fans - rise of the temperature inside the WRS box
						(failure \ref{fail:other:temp})
					\item power supply, ARM, FPGA - booting problem (failure
						\ref{fail:other:boot})
					\item memory chip - data corruption (failure \ref{fail:other:memory})
				\end{itemize}
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{packed_enum}

\subsubsection{\bf Management link down}
		\label{fail:other:management_link}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
				For obvious reasons we are not able to report through SNMP that the
				management link is down. This should be detected and reported by the NMS
				if it does not receive SNMP and ICMP responses from the WRS.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{packed_enum}

\subsubsection{\bf No static IP on the management port \& failed to DHCP}
		\begin{packed_enum}
			\item [] \underline{Description}:\\
				From operator's point of view it is similar to the issue
				\ref{fail:other:management_link}. WRS is not accessible through the
				management port, so its status cannot be reported. This should be
				detected and reported by the NMS if it does not receive SNMP and ICMP
				responses from the WRS. In such case WR expert should make a physical
				connection to the management USB port of the WRS to diagnose the
				problem.
			\item [] \underline{SNMP objects}: \emph{(none)}
		\end{packed_enum}

\subsubsection{\bf IP address on the management port has changed}
		\begin{packed_enum}
			\item [] \underline{Status}: TODO
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				I'm not yet sure how we should report this. Probably SNMP is not the
				best choice because if the IP changes we're no longer able to poll SNMP
				objects (until IP is updated also in the Network Management Station). We
				should either generate SNMP trap to NMS or send Syslog message to a
				central server.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{packed_enum}

\subsubsection{\bf Multiple unauthorized access attempts}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}:\\
				If we observe many attempts to gain a root access through the ssh (or
				the web interface) this might be somebody trying to do something nasty.
				We should report such situation as a Warning.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
			\item [] \underline{Note}: Bad password event is reported by Syslog as a
				warning. We should probably use this information to add an SNMP object.
		\end{packed_enum}

\subsubsection{\bf Network reconfiguration (RSTP)}
		\label{fail:other:rstp}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				If topology reconfiguration occurs because of the primary link failure,
				this fact should be reported through SNMP as a warning. It's not
				critical situation, WR network still works. However, further
				investigation should be performed to repair the broken link.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{packed_enum}

\subsubsection{\bf Backup link down}
		\begin{packed_enum}
			\item [] \underline{Status}: for later
			\item [] \underline{Severity}: WARNING
			\item [] \underline{Description}: \emph{(not yet implemented)}\\
				This is related to the issue \ref{fail:other:rstp}. If the WRS uses
				primary uplink, but the backup one fails, it's not a critical fault. WR
				Network still works, but the link should be diagnosed and repaired to
				have the backup link operational in case the primary one fails.
			\item [] \underline{SNMP objects}: \emph{(not yet implemented)}
		\end{packed_enum}

%\subsection{Switch out of sync to Master}
%
%\subsection{Switch made a big offset jump to follow Master}
%
%\subsection{Unsupported SFP plugged to one of the cages}
%
%\subsection{Lost lock to external 1-PPS \& 10 MHz}
%
%\subsection{Switch wasn't able to fetch initial time from NTP}
%
%\subsection{Suspicious value of any PTP parameter}
%e.g. bitslide > 16000;  dTx/dRx = 0, etc.
%
%\subsection{PPSi/HAL/SNMP/any other userspace daemon has crashed}
%
%\subsection{LM32 software has crashed/restarted}
%
%\subsection{Cooling fan broken}
%
%\subsection{Power supply broken}
%
%\subsection{Switch not reachable after power cut}
%
%\subsection{Switch not reachable through SNMP}
%
%\subsection{One of the links went down}
%
%\subsection{Ethernet frames being dumped}
%
%\subsection{Linux is out of memory}
%
%\subsection{Filesystem error/corruption}
%
%\subsection{HW version not recognized, FPGA bitstream not loaded}
%
%\subsection{Frames storm coming from one or multiple ports to CPU}
