Skip to content

felix nftables (Auto): bundled nft v1.1.0 in hardened-calico v3.32.1 segfaults on arp-family (calico-arp) transactions — retry loop wrote ~370GB of core dumps #13987

Description

@OutsrYingq

Current Behavior

On a fresh single-node RKE2 cluster shipping rancher/hardened-calico:v3.32.1-build20260827, with the default FelixConfiguration (nftablesMode: Auto), every nftables transaction targeting the arp family table (calico-arp) crashes the bundled nft binary with SIGSEGV, and felix retries in a tight loop (~10 attempts/sec) from the moment the cluster is installed:

2026-09-19 18:01:06.877 [WARNING] felix/table.go 924: Failed to program nftables, will retry error=error performing nft transaction: signal: segmentation fault (core dumped) family="arp" ipVersion=4 table="calico-arp"
2026-09-19 18:01:07.071 [ERROR]   felix/table.go 1192: Failed to load nftables table state error=exit status 1 family="arp" ipVersion=4 table="calico-arp"
2026-09-19 18:01:07.093 [WARNING] felix/table.go 916: Failed to delete table, continuing anyway error=signal: segmentation fault (core dumped) family="arp" ipVersion=4 table="calico-arp"

host dmesg (thousands of these):

nft[2345678]: segfault at 0 ip 0000000000000000 sp 00007ffd... error 14
Code: Unable to access opcode bytes at 0xffffffffffffffd6.

Notable: the v4/v6 calico tables programmed fine on the same node — only the arp family transactions crashed. The failing transaction contains add rule arp calico-arp ... arp operation reply arp saddr ip <addr> counter drop statements plus add/delete element ops on the filter-cali-arp-dispatch map (full transaction text from the felix error log available on request).

Expected Behavior

  • arp-family nftables programming succeeds with the bundled nft version, or the image bundles an nft version that works on current Debian 13 cloud kernels; and/or
  • Auto mode falls back / backs off when dataplane tooling fails consistently instead of retrying ~10/sec indefinitely.

Steps to Reproduce

  1. Debian 13 (trixie) host, kernel 6.12.38+deb13-cloud-amd64 (host nft v1.1.3)
  2. Install RKE2 v1.36.4-rke2r1 — canal ships rancher/hardened-calico:v3.32.1-build20260827; in-container nft --version → nftables v1.1.0
  3. Leave FelixConfiguration at defaults (nftablesMode: Auto)
  4. Watch calico-node logs and dmesg

Blast radius (for other operators)

With the node's default kernel.core_pattern=core, each segfaulting nft child (spawned by felix, inheriting felix's runit service cwd) dropped a ~5.6MB core dump into the container writable layer. Over ~26h this accumulated ~74,000 core files ≈ 370GB, filled the node disk and took down containerd plus every workload on the node. Disabling core dumps node-wide and/or nftablesMode: Disabled are effective mitigations.

Workaround

kubectl patch felixconfiguration default --type merge -p '{"spec":{"nftablesMode":"Disabled"}}'

Takes effect after a couple of felix dataplane restarts; verified stable afterwards (zero segfaults, arp filtering reprogrammed via iptables).

Your Environment

Additional node networking context

Beyond calico's tables, this node's kernel nftables also carries a hand-managed host firewall — an origin-protection allowlist that accepts tcp dport {80, 443} only from Cloudflare address sets (IPv4/IPv6) and drops all other traffic to those ports.

One operational consequence on the same node (unrelated to the segfaults above, noted for environment completeness): a directly-connected TLS service listening on IPv6 :443 that is not fronted by Cloudflare is unreachable from the internet — connections are dropped before reaching the process (client-side symptom: TLS handshake timeouts / SSL_ERROR_SYSCALL).

If it turns out the segfaults depend on interaction with other rulesets/sets present in the kernel nftables, the full nft list ruleset output of this node is available on request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions