HLAB h-gw-iso-l2vni-rt: server-6 loses all fabric reachability for the whole Multi-VPC Single-Subnet suite

[pau-hedgehog]

https://github.com/githedgehog/fabricator/actions/runs/33510314760/job/99867920759

Run 33510314760, job 99867920759, variant h-gw-iso-l2vni-rt, PR #1730, commit 61b4da5 (merge of refs/pull/1730/merge), build v0.50.0-9-g61b4da50-2r1526, fabric/agent v0.130.2, gateway dataplane v0.26.0. Leaves ds3000-03 and sse-c4632-01 both run Enterprise SONiC 4.5.2-Enterprise_Base, HwSKU Seastone_2, Broadcom ASIC. Companion repo githedgehog/lab-ci was checked out at master 95ca77defb1c42397de0426e65c99c56d983ef30 with no pinned ref.

Seven tests fail in the Multi-VPC Single-Subnet Suite. Every failing assertion in all seven involves server-6. No other pair of servers fails, in either direction, anywhere in the run. The four preceding suites pass in full, including ESLAG Failover, ESLAG Fallback and all 28 Gateway NAT and ACL tests.

2026-09-01T17:22:51.1273522Z 17:22:51 ERR FAIL test="Only Externals" error="curl from server-6: should be reachable but curl failed (expected true, because: switch-peering)"
2026-09-01T17:32:46.3954022Z 17:32:46 ERR FAIL test="Full Mesh All Externals" error="ping server-6 -> server-1: unexpected ping result (sent 5, rcvd 1, lost=[2 3 4 5], expected true, because: switch-peering \"vpc-01--vpc-05\")\nping server-3 -> server-6: unexpected ping result (sent 5, rcvd 1, lost=[2 3 4 5], expected true, because: switch-peering \"vpc-03--vpc-05\")\ncurl from server-6: should be reachable but curl failed (expected true, because: switch-peering)\nping server-6 -> server-7: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")\nping server-6 -> server-9: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-08\")\nping server-6 -> server-5: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-04--vpc-05\")\nping server-6 -> server-2: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-02--vpc-05\")\nping server-5 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-04--vpc-05\")\nping server-9 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-08\")\nping server-7 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")\nping server-1 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-01--vpc-05\")\nping server-6 -> server-3: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-03--vpc-05\")\nping server-2 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-02--vpc-05\")"
2026-09-01T17:42:08.9874335Z 17:42:08 ERR FAIL test="Full Loop All Externals" error="curl from server-6: should be reachable but curl failed (expected true, because: switch-peering)\nping server-5 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-04--vpc-05\")\nping server-6 -> server-5: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-04--vpc-05\")\nping server-6 -> server-7: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")\nping server-7 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")"
2026-09-01T17:49:44.6782685Z 17:49:44 ERR FAIL test="Sergei's Special Test" error="ping server-7 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-06--vpc-05\")\nping server-6 -> server-7: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-06--vpc-05\")"
2026-09-01T17:59:58.7951405Z 17:59:58 ERR FAIL test="Gateway Peering Loop" error="testing gateway loop connectivity: ping server-7 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-05--vpc-06\")\nping server-5 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-04--vpc-05\")\nping server-6 -> server-5: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-04--vpc-05\")\nping server-6 -> server-7: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-05--vpc-06\")"
2026-09-01T18:05:29.3965811Z 18:05:29 ERR FAIL test="Mixed VPC and Gateway Peering Loop" error="testing mixed peering loop connectivity: ping server-5 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-04--vpc-05\")\nping server-7 -> server-6: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")\nping server-6 -> server-5: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: gateway-peering \"vpc-04--vpc-05\")\nping server-6 -> server-7: should be reachable but ping failed (sent 5, rcvd 0, expected true, because: switch-peering \"vpc-05--vpc-06\")"
2026-09-01T18:11:41.3327910Z 18:11:41 ERR FAIL test="Mixed Gateway and Fabric External Peering" error="testing mixed peering loop connectivity: curl from server-6: should be reachable but curl failed (expected true, because: switch-peering)"

server-6 is dual-homed with EVPN multihoming (ESLAG) to ds3000-03 on Ethernet41 and sse-c4632-01 on Ethernet39, both members of PortChannel2, in vpc-05 with address 10.20.5.2/24 on VLAN 1005.

The last successful probe involving server-6 anywhere in the run is logged at 17:31:35. There are no successful probes involving server-6 in the remaining 46 minutes of the suite. Every curl from server-6 where external reachability was expected failed, at 17:22:46, 17:31:36, 17:41:50 and 18:11:35.

Triage detail in the first comment.

[pau-hedgehog]

Triage against the artifact from https://github.com/githedgehog/fabricator/actions/runs/33510314760/job/99867920759, which carries a full show-tech capture for each of the seven failing tests.

Topology of the affected path

server-6 is dual-homed with EVPN multihoming (ESLAG) to ds3000-03 on Ethernet41 and sse-c4632-01 on Ethernet39, both members of PortChannel2, in vpc-05 with address 10.20.5.2/24 on VLAN 1005. server-5 shares the same pair of leaves on PortChannel1 in vpc-04. server-7 and server-8 are single-homed to ds3000-03. server-9 is single-homed to sse-c4632-01. server-1, server-2 and server-3 attach to ds3000-01 and ds3000-02 and are reached across spines ds4000-01 and ds4000-02.

The break has a sharp instant

The probe output carries per-packet epoch timestamps. Packet 1 crosses two switch hops and is answered. Packets 2 through 5 are not.

2026-09-01T17:31:35.7112721Z 17:31:35 DBG Ping result from=server-6 to=server-1 expected=true ok=false fail=false err=<nil> stdout="PING 10.20.1.2 (10.20.1.2) 56(84) bytes of data.\n[1788283891.779476] 64 bytes from 10.20.1.2: icmp_seq=1 ttl=62 time=0.241 ms\n[1788283892.786441] no answer yet for icmp_seq=2\n[1788283893.290569] no answer yet for icmp_seq=3\n[1788283893.795488] no answer yet for icmp_seq=4\n--- 10.20.1.2 ping statistics ---\n5 packets transmitted, 1 received, 80% packet loss, time 2017ms\nrtt min/avg/max/mdev = 0.241/0.241/0.241/0.000 ms\n" stderr=""

Epoch 1788283891.779476 is 2026-09-01 17:31:31.779 UTC. Every probe involving server-6 that started at or after 17:31:32.6 returned zero replies:

2026-09-01T17:31:37.0358838Z 17:31:37 WRN Ping failed (expected reachable) from=server-6 to=server-7 sent=5 rcvd=0 lost=[] stdout="PING 10.20.6.2 (10.20.6.2) 56(84) bytes of data.\n[1788283893.106485] no answer yet for icmp_seq=1\n[1788283893.610405] no answer yet for icmp_seq=2\n[1788283894.114598] no answer yet for icmp_seq=3\n[1788283894.618618] no answer yet for icmp_seq=4\n--- 10.20.6.2 ping statistics ---\n5 packets transmitted, 0 received, 100% packet loss, time 2018ms\n" stderr=""

The condition persists across six full peering teardown and rebuild cycles and several agent configuration generations. The harness made no assertion about server-6 external reachability before 17:22:46, so the earliest point at which this could have been observed is that curl, and it was already failing there.

What was checked and is correct

All of the following were read from the per-test show-tech captures and hold in every one of the seven failing tests.

Routing tables on both of server-6's leaves carry every peer prefix in vpc-05, including the ones that fail. Peer VRFs on both leaves carry server-6's host route. The remote leaf ds3000-01 carries server-6's host route with the same tunnel endpoint as a destination that works:

 B>*   10.20.5.2/32       via 172.30.12.2                 Vlan3003(vrf VrfVvpc-05)                   200/0         00:02:43 ago
 B>*   10.20.7.2/32       via 172.30.12.2                 Vlan3006(vrf VrfVvpc-07)                   200/0         00:03:18 ago

The address resolution entry for server-6 is present and marked forwarding on both leaves, in all four captures checked:

10.20.5.2           a2:bd:86:2d:c1:01   Vlan1005             PortChannel2                Dynamic            Fwd       

Both leaves have server-6's MAC programmed in the ASIC layer 2 table on the internal VLAN backing VLAN 1005:

mac=a2:bd:86:2d:c1:01 vlan=29677 GPORT=0xb0000010 port=0xb0000010(flow) Mesh

The MAC address tables, EVPN Ethernet Segment state (ds3000-03 designated forwarder, sse-c4632-01 non designated forwarder), VXLAN VNI maps and tunnel state are all correct and symmetric. Both link aggregation members are selected and up on both leaves.

server-6 itself is healthy: the bond is up in 802.3ad mode with both members at 10 Gbps and a link failure count of zero, its DHCP lease is valid, it holds 10.20.5.2 throughout the run with no address change, it pings its default gateway in 0.339 ms, and it sustained a 9.9 Gbps iperf3 transfer to server-8 at 17:31:38, after the cutover.

What was ruled out

There are no receive or transmit errors and no discard counters on any server port or fabric port on the path, on any leaf. There are no non-zero queue drop counters on either of server-6's leaves. There are no driver-level drops on any server network interface. There are no SAI, syncd or orchagent errors in the failure window on either leaf, only telemetry noise about unsupported drop counters. Neither leaf restarted or crashed FRR or zebra at any point in the run. Both leaf agents were quiescent when the probes ran, having logged "Config applied" at 17:31:10 and 17:31:15, 17 to 22 seconds before the connectivity matrix started. The runner shows all pressure stall metrics at zero. Bond transmit hashing on server-6 does not separate the one destination that worked from the ones that failed, because failures span both hash outcomes.

Four known failure classes were checked against their own causal chains and do not apply. The external peering readvertisement race does not apply: at collection time server-6's subnet was installed on both its leaves before the curl ran, with the same route age as every working VPC, and the external BGP sessions had been up for 50 and 88 minutes without a reset. The agent mid-apply loss class does not apply: connected routes were installed before the failure and both agents finished applying well before the probe. The DHCP reassignment convergence race does not apply: server-6 never changed address and there is no dropping address resolution entry. The zebra sync MAC hold timer crash does not apply: neither leaf restarted FRR.

The one asymmetry found

On sse-c4632-01, the multihoming-synced neighbour entries for the two servers on its Ethernet Segments track each other exactly until 17:17:26, when both are removed:

Sep 01 17:17:26.106399+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeNextHop: Removed next hop 10.20.4.2|Vlan1004
Sep 01 17:17:26.106399+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeNeighborInternal: Removed neighbor ip 10.20.4.2, 66:ab:8f:70:ec:0f on Vlan1004
Sep 01 17:17:26.106399+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeMactoIp: remove neighbor 10.20.4.2 from macToNeigh mapping, key:(Vlan1004, 66:ab:8f:70:ec:0f)
Sep 01 17:17:26.106399+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeMactoIp: delete remote mac 66:ab:8f:70:ec:0f on interface Vlan1004
Sep 01 17:17:26.615419+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeNextHop: Removed next hop 10.20.5.2|Vlan1005
Sep 01 17:17:26.615419+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- removeNeighborInternal: Removed neighbor ip 10.20.5.2, a2:bd:86:2d:c1:01 on Vlan1005

After this point server-6's entry is never re-created on sse-c4632-01 again. Its Ethernet Segment mate server-5 is re-created eleven times over the following 54 minutes, at 17:22:42, 17:31:33, 17:41:47, 17:49:32, 17:55:21, 17:56:36, 17:59:37, 18:05:07 and onward to 18:11:33, each time as a multihoming-synced entry:

Sep 01 17:22:42.757610+00:00 2026 sse-c4632-01 NOTICE swss#orchagent: :- addNeighborInternal: Created neighbor ip 10.20.4.2, 66:ab:8f:70:ec:0f on Vlan1004, proto:11, nbr_tbl_add:TRUE

On the other ESLAG leaf pair, ds3000-02 re-creates both of its servers' entries at every one of those same timestamps, which is the expected pattern:

Sep 01 17:31:33.038701+00:00 2026 ds3000-02 NOTICE swss#orchagent: :- addNeighborInternal: Created neighbor ip 10.20.1.2, 6e:27:d4:e2:6b:f7 on Vlan1001, proto:11, nbr_tbl_add:TRUE
Sep 01 17:31:33.039138+00:00 2026 ds3000-02 NOTICE swss#orchagent: :- addNeighborInternal: Created neighbor ip 10.20.2.2, 66:8f:54:6e:8e:55 on Vlan1002, proto:11, nbr_tbl_add:TRUE

Meanwhile the command line view on sse-c4632-01 keeps reporting server-6's address resolution entry as present and forwarding in every capture, so the operational view and orchagent's own programming record disagree for this one host from 17:17:26 to the end of the run.

What is not known

Inference, stated as such: a neighbour entry that is present in the operational view but absent from hardware on the non-designated-forwarder leaf would black-hole traffic delivered to server-6 by way of sse-c4632-01. This is not confirmed, and it does not by itself account for server-7 failing to reach server-6 at 17:31:33.21 while server-8 succeeded from 17:31:31.89 to 17:31:33.90, given that both are single-homed on ds3000-03 with identical routing table entries.

The table that would settle this is the ASIC IPv4 host table. The collector attempts to capture it, but on this platform the command it issues is rejected by the SDK and the section records the command usage text instead, so the data is not in the artifact. That is filed separately as a collector bug. Once it is fixed, the check is whether 10.20.5.2 is absent from the ASIC host table on sse-c4632-01 while 10.20.4.2 is present.