Calico fails to load on clusters with external DNS names for the API server

[larsskj]

I create this ticket by request from Slack where the issue has been tracked so far. Unfortunately, history of this issue has been lost due to age on Slack, but I'll try to recreate information as well as possible.

I run two bare metal clusters set up for HA operation using the still current recommendation for kubeadm clusters:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
where you find a link to
https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing

The latter document pretty describes what I've done: One cluster has four nodes, the other eight nodes, both with three control plane nodes. The small cluster has a single host running HA Proxy in front, the larger cluster has two hosts running HA Proxy with yet another hardware load balancer in front.

For both clusters, I've created an external DNS name for the API server like https://api.mycluster.mydomain.dk. During setup with kubeadm this DNS name was specified as recommended in the document above. This setup has been working flawlessly with Calico for a very long time - the small cluster has been in use for more than six years.

However, with a Calico upgrade this setup breaks: I cannot remember the exact version anymore, but I believe it was with 3.30.0, though the specific change that triggers the problem cannot be found in the release notes.

I had a long session on Slack with primarily Brian McMahon were we did a lot of troubleshooting and found a workaround - but not without downtime in a production cluster. I still have part of the conversation available, but unfortunately the part where Brian McMahon showed me an undocumented flag to the operator has been lost - and that's required for the hack to work.

What happens is basically that the operator deploys a deamonset for the Calico nodes: I've forgotten the specific DNS settings, but with the settings provided by the operator, one of the initContainers fail as it cannot lookup the external name for the K8s API host. I need to modify the DNS settings in the daemonset by hand to get things running - and here the "secret" flag is required in order to prevent the operator from resetting the values.

My custom values, on the other hand, prevents Goldmane from working - however, we can live with that.

We would love to see this issue resolved - and maybe it already is? I just hope for the same level of support as we received last summer - we cannot afford to have a production cluster go down for a prolonged period.

Both clusters are on Kubernetes 1.36.4 and has upgraded since last year without hickups. Calico is maintained using the Tigera operator and installed using the Helm chart. Current version is 3.30.3 for the large cluster, the small cluster is still at 3.29.1 which works fine without hacks.

[larsskj]

I found the undocumented hack: Our deamonset for Calico nodes is annotated with unsupported.operator.tigera.io/ignore: "true" in order to prevent the operator from resetting my custom DNS settings.

The settings that work for us are:

      dnsConfig:
        nameservers:
        - 172.21.0.10
        - 127.0.0.53
      dnsPolicy: None
      hostNetwork: true

172.21.0.10 may look strange - but the cluster is installed to use 172.16.0.0/12 for internal IPs as the default 10.0.0.0/8 is in widespread use on the corporate network.

[tomastigera]

3.29 is now an unsupported release and we think this issue was fixed in later releases.

operator (master) now defaults the node DaemonSet's dnsPolicy to the host's own resolver instead of routing through cluster CoreDNS, which can't work before CNI is up. That matches the operator fix merged via #13823, feeding the 3.31 line

check that DaemonSet's dnsPolicy

[larsskj]

I do not have issues with 3.29 - the problem is 3.30.

Sounds like the problems have been solved by 3.31 - I'll attempt an upgrade from 3.30 to 3.31 in the weekend.

For the other cluster: Can I upgrade directly from 3.29 to 3.31 - or must I do 3.30 first?

As soon as it works again, I'll upgrade to 3.32.2 and keep both clusters updated to the latest and greatest.

[tomastigera]

Can I upgrade directly from 3.29 to 3.31 - or must I do 3.30 first?

There is a reasonable confidence that it works well, with some caveats to watch for:

• Use the latest 3.31 patch. It has all the 3.30.x upgrade fixes, including the policy-name one.

• Default-tier policy names. Policies created on v3.29.0–3.29.3 get renamed from default.foo to foo. Policies created before v3.28 keep their names. This happens whether or not they stop at 3.30. It matters if they use GitOps (ArgoCD/Flux) or scripts that refer to policies by the default.-prefixed name.

• Operator: since 3.30, the operator installs the CRDs when it starts, so tigera-operator.yaml no longer contains them. Apply the 3.31 manifests as the guide says.

  • In 3.31, calico-apiserver moves from the calico-apiserver namespace to calico-system. Anything that targets the old namespace needs updating, such as NetworkPolicies, monitoring, or RBAC.

• Some other changes that don't break anything:

  • Felix now programs IPIP routes instead of BIRD.
  • The HostEndpoint controller may update auto-created HEPs to add newly detected interfaces.
  • Goldmane and Whisker (new in 3.30) stay off unless they apply those CRs.

• Kubernetes version - check that cluster's Kubernetes version is in the 3.31 support matrix.

cc @Brian-McM