servo not updating on slave (after ~49.7days)
Servo stuck on slave WRS. Looks like regular, but update counter is not increased. On master's side, the port is marked as EXT_OFF.
wr_mon's output on slave:
WR Switch Sync Monitor v6.0-82-g1debde40 [q = quit]
WR time (TAI) : 2021-06-24 14:46:33.006139 Leap seconds: 37
Switch time (UTC): 2021-06-24 14:45:56.006130 TAI-UTC : +37.000009
TimingMode: BC PLL locking state: LOCKED
----- HAL ---|---------------------------------- PPSI --------------------------------------------------------
Iface| Freq |Inst| Name | Config | MAC of peer port | PTP/EXT/PDETECT States | Pro | VLANs
------+------+----+--------------+------------+-------------------+------------------------------+-----+------
wri1 | | 0 |wri1-1-wr-raw |slave | XX:XX:XX:XX:XX:XX | SLAVE /IDLE /EXT_ON | V-W | 31
wri2 | | 1 |wri2-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri3 | | 2 |wri3-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri4 | | 3 |wri4-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri5 | | 4 |wri5-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri6 | | 5 |wri6-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri7 | | 6 |wri7-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
wri8 | | 7 |wri8-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| R-W |
*wri9 | | 8 |wri9-1-wr-raw |master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri10| | 9 |wri10-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri11| | 10 |wri11-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri12| | 11 |wri12-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri13| | 12 |wri13-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri14| | 13 |wri14-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri15| | 14 |wri15-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri16| | 15 |wri16-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri17| | 16 |wri17-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
*wri18| | 17 |wri18-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
Pro - Protocol mapping: V-Ethernet over VLAN; U-UDP; R-Ethernet
--------------------------- Synchronization status ----------------------------
Servo state: wri1:White-Rabbit: TRACK_PHASE
+- Timing parameters ---------------------------------------------------------
| meanDelay : 80.519 nsec
| delayMS : 80.508 nsec
| delayMM : 1063.492 nsec
| delayAsymmetry : 0.011 nsec
| delayCoefficient : -0.000267798264878874 fpa : -1235001513901056
| ingressLatency : 225.959 nsec
| egressLatency : 224.295 nsec
| semistaticLatency: 0.000 nsec
| offsetFromMaster : 0.005 nsec
| Phase setpoint : -15.047 nsec
| Skew : -0.012 nsec
| Update counter : 4212013 times <--------------------------- does not increase
| Master PHY delays TX: 224.879 nsec RX: 227.321 nsec
| Slave PHY delays TX: 224.295 nsec RX: 225.959 nsec
wr_mon's output on master:
WR Switch Sync Monitor v6.0-82-g1debde40 [q = quit]
WR time (TAI) : 2021-06-23 00:02:38.000217 Leap seconds: 37
Switch time (UTC): 2021-06-23 00:02:01.000210 TAI-UTC : +37.000007
TimingMode: BC PLL locking state: LOCKED
----- HAL ---|---------------------------------- PPSI --------------------------------------------------------
Iface| Freq |Inst| Name | Config | MAC of peer port | PTP/EXT/PDETECT States | Pro | VLANs
------+------+----+--------------+------------+-------------------+------------------------------+-----+------
wri1 | Lock | 0 |wri1-1-wr-raw |slave | XX:XX:XX:XX:XX:XX | SLAVE /IDLE /EXT_ON | V-W | 31
*wri2 | | 1 |wri2-1-wr-raw |master | XX:XX:XX:XX:XX:XX | DISABLED /IDLE /NONE | V-W | 31
*wri3 | | 2 |wri3-1-wr-raw |master | XX:XX:XX:XX:XX:XX | DISABLED /IDLE /NONE | V-W | 31
wri4 | | 3 |wri4-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_ON | V-W | 31
wri5 | | 4 |wri5-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_OFF| V-W | 31 <--------- slave connected to this port, note the EXT_OFF
wri6 | | 5 |wri6-1-wr-raw |master | XX:XX:XX:XX:XX:XX | MASTER /IDLE /EXT_ON | V-W | 31
*wri7 | | 6 |wri7-1-wr-raw |master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri8 | | 7 |wri8-1-wr-raw |master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri9 | | 8 |wri9-1-wr-raw |master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri10| | 9 |wri10-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri11| | 10 |wri11-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri12| | 11 |wri12-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri13| | 12 |wri13-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri14| | 13 |wri14-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri15| | 14 |wri15-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri16| | 15 |wri16-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri17| | 16 |wri17-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | V-W | 31
*wri18| | 17 |wri18-1-wr-raw|master | 00:00:00:00:00:00 | DISABLED /IDLE /NONE | R-W |
Pro - Protocol mapping: V-Ethernet over VLAN; U-UDP; R-Ethernet
--------------------------- Synchronization status ----------------------------
Servo state: wri1:White-Rabbit: TRACK_PHASE
+- Timing parameters ---------------------------------------------------------
| meanDelay : 2527.601 nsec
| delayMS : 2527.263 nsec
| delayMM : 5973.412 nsec
| delayAsymmetry : 0.338 nsec
| delayCoefficient : -0.000267798264878874 fpa : -1235001513901056
| ingressLatency : 225.959 nsec
| egressLatency : 224.295 nsec
| semistaticLatency: 0.000 nsec
| offsetFromMaster : 0.002 nsec
| Phase setpoint : 1.839 nsec
| Skew : 0.003 nsec
| Update counter : 495091 times
| Master PHY delays TX: 227.016 nsec RX: 240.940 nsec
| Slave PHY delays TX: 224.295 nsec RX: 225.959 nsec
-------------------------------- Temperatures ---------------------------------
FPGA: 58.69 PLL: 55.06 PSL: 32.62 PSR: 37.75
When servo is stuck it is necessary to restart PPSI or do link down/up (by ifconfig or unplug/plug fiber).
Only firmware v6.0 is affected with this bug.
Related to #239 (closed), #235 (closed). The details are described in ppsi#32.