/ 
Rex-Rancher-0.003
/ Rex::Rancher::Uninstall

NAME

Rex::Rancher::Uninstall - Take RKE2/K3s and Cilium's datapath off a host, and refuse to install over what Cilium left

VERSION

version 0.003

SYNOPSIS

use Rex::Rancher::Uninstall;

# Take RKE2/K3s and Cilium's datapath off the host (destructive)
task 'wipe', 'old-node.example.com', sub {
  uninstall_node();
};

# The same line through another channel than Rex
my $cmd = Rex::Rancher::Uninstall->uninstall_cmd;
my $r   = $my_ssh->run($cmd);
warn "$_\n" for Rex::Rancher::Uninstall->uninstall_warnings($r->{stdout}, $r->{stderr});
die Rex::Rancher::Uninstall->uninstall_failure($r->{exit}, $r->{stderr})
  if $r->{exit};

# Refuse a host Cilium left state on, before preparing it
Rex::Rancher::Uninstall->check_cilium_residue;

DESCRIPTION

Cilium's datapath outlives the vendor uninstall scripts. rke2-uninstall.sh and k3s-uninstall.sh stop the agent, but not what it attached to the kernel: the socket load balancer on the root cgroup, tc programs, maps and links pinned in bpffs, the cilium_* devices, iptables chains and ip rules. Until the host is rebooted, the socket load balancer keeps translating the old cluster's service addresses to pods that are gone, so a connection to one of them hangs instead of being refused. When that address is a registry mirror, every image pull of a new install on the host stalls for minutes, and RKE2 misses its start deadline without saying why.

"check_cilium_residue" is the guard "install_server" in Rex::Rancher::Server and "install_agent" in Rex::Rancher::Agent run first: a host with that state and no RKE2 or K3s on it is refused before anything is written. With a distribution on the host the state belongs to its running cluster (a re-run, an upgrade) and is left alone.

"uninstall_node" is the other side: it runs the vendor uninstall scripts and then clears the datapath as Cilium's own cleanup does, and says when a reboot is still needed -- or, as a warning, when a missing tc or iptables tool left part of the datapath unchecked. Both test the same "cilium_residue".

Both come from kubernetes-ocp, which measured the problem (a fresh RKE2 after rke2-uninstall.sh without a reboot) and runs the same uninstall line. Whether the kernel really detaches every program when its pins go has not been verified live through Rex::Rancher; the outcome check only sees the pins, the mount and the device.

distribution_class

Rex::Rancher::Distribution: the class whose "distribution_classes" in Rex::Rancher::Distribution say which distributions are looked for and uninstalled. A subclass may name another one.

cilium_residue

my @residue = Rex::Rancher::Uninstall->cilium_residue;
# ( [ '[ -e /sys/fs/bpf/cilium ]', '/sys/fs/bpf/cilium' ], ... )

What still being on a host means Cilium's datapath outlived its cluster, as pairs of a shell test (true when it is there) and the name it is reported by: the pins in /sys/fs/bpf/cilium, the cgroup2 mount on /run/cilium/cgroupv2 (read from /proc/mounts), the cilium_host device. Pure. The one list both "check_cilium_residue" (through "cilium_residue_probe_cmd") and the outcome check of "uninstall_cmd" test.

cilium_residue_probe_cmd

The shell line "check_cilium_residue" runs on the host. Pure. When no distribution is there -- no "binary" in Rex::Rancher::Distribution on PATH and none of their server and agent units active -- it prints the name of every "cilium_residue" it finds, one per line; with a distribution it prints nothing. It only reads, and always exits 0.

cilium_residue_in

my @found = Rex::Rancher::Uninstall->cilium_residue_in($probe_output);

The "cilium_residue" names in the output of "cilium_residue_probe_cmd", in the order found; empty for a clean host. Pure. Only whole lines that are one of those names count: a login banner or a shell's complaint on the channel is not evidence of Cilium state.

check_cilium_residue

Rex::Rancher::Uninstall->check_cilium_residue;

Run "cilium_residue_probe_cmd" on the host (the exec channel, no SFTP) and die when it finds anything, naming what it found, what it does to a new install, and that the host has to be rebooted first. Returns 1 on a clean host or one with RKE2 or K3s on it. A probe that cannot run at all (a non-zero exit with nothing found) only warns that the host was not checked.

"install_server" in Rex::Rancher::Server and "install_agent" in Rex::Rancher::Agent call it before anything is written or installed; a caller that prepares the node first can call it earlier.

uninstall_cmd

The shell line "uninstall_node" runs, for a caller that runs it through its own channel. Pure. In this order:

  • every "uninstall_scripts" in Rex::Rancher::Distribution that is on the host (rke2-uninstall.sh, k3s-uninstall.sh, k3s-agent-uninstall.sh); one that fails does not keep the next from running;

  • the PATH files for the NVIDIA runtime lookup, which "ensure_nvidia_runtime_path" in Rex::Rancher::Distribution writes and no uninstaller removes ("env_files" in Rex::Rancher::Distribution: /etc/default/rke2-server and /etc/default/rke2-agent; K3s has none). One goes only when it holds nothing but "runtime_path_line" in Rex::Rancher::Distribution -- the file written on a host that had none -- and the distribution's "binary" in Rex::Rancher::Distribution is off PATH. A file with anything else in it is the admin's and stays as it is, our PATH line included, as does one while the distribution is still installed; neither is a failure;

  • what the uninstallers leave of Rex::Rancher's and the distribution's own: the Cilium CLI /usr/local/bin/cilium (Rex::Rancher::Cilium installs it), /opt/cni (the CNI binaries, Cilium's among them), /run/k3s (the runtime dir of both distributions), removed with --one-file-system: a mount the uninstaller left below them (containerd's task mounts live in /run/k3s) is skipped, never recursed into;

  • Cilium's datapath: the clsact qdisc of every device carrying a Cilium tc program (with tc installed, see below); the pins in /sys/fs/bpf/cilium and /sys/fs/bpf/tc/globals/cilium_* (unpinning a link no agent holds any more is what detaches its program); the devices cilium_host, cilium_net, cilium_vxlan, cilium_geneve, cilium_wg0; the CILIUM_* (and OLD_CILIUM_*) iptables chains in every backend present (iptables, ip6tables, their -legacy and -nft variants): jumps into them deleted, then flushed and removed, one --noflush restore per table, every other rule untouched; the /run/cilium/cgroupv2 mount, then /run/cilium with --one-file-system;

  • Cilium's ip rules: those looking up tables 2004 and 2005, then the local table lookup Cilium moved from priority 0 to 100: put back at 0 first, and the one at 100 deleted only once a priority 0 one is there, so the host never lacks a local lookup.

Each of those steps is guarded: a step with nothing to do, or a tool that is not installed, is not a failure. Two missing tools leave part of the datapath unchecked, and each says so with a warning, one line on stderr starting with rex-rancher-uninstall-warning: :

  • no tc (on Rocky/RHEL it comes with the iproute-tc package): Cilium's legacy tc attachments on the host's devices were not checked or removed;

  • no iptables backend with both -save and -restore: the CILIUM_* chains were not checked or removed.

A reboot clears either. A warning never changes the exit status. What decides is the outcome, checked last: the line exits 1 when an "binary" in Rex::Rancher::Distribution is still on PATH (... is still installed after the uninstall: PATH), or when a "cilium_residue" is still there (Cilium datapath state is still on the host after the uninstall: NAMES -- reboot the host ...), with that on stderr. "uninstall_warnings" picks the warnings out of the output, "uninstall_failure" turns a failed outcome into a message.

rke2-uninstall.sh also removes /etc/rancher/node, the node password with it; the K3s uninstall scripts keep it. For joining the same cluster again afterwards, see "uninstall_node".

uninstall_warnings

my @warnings = Rex::Rancher::Uninstall->uninstall_warnings($stdout, $stderr);

The warnings in the output of "uninstall_cmd", in the order written, each its text without the marker, trimmed; empty when every cleanup step could run. Pure. Takes any number of outputs -- pass stdout and stderr: the line writes its warnings to stderr, a channel with a pty merges them into stdout. Only whole lines starting with rex-rancher-uninstall-warning: and a text count; the vendor uninstallers' output, a login banner, a shell's complaint or a set -x trace are not warnings.

Warnings are not a failure: the exit status alone decides that (see "uninstall_failure"). "uninstall_node" logs them; a caller running the line through its own channel reports them there.

uninstall_failure

my $message = Rex::Rancher::Uninstall->uninstall_failure($exit, $stderr);

The message for an "uninstall_cmd" that exited $exit with $stderr, ending in a newline; nothing for exit 0, warnings or not. Pure. The warning lines (see "uninstall_warnings") are not part of the reason.

uninstall_node

uninstall_node();

Take RKE2 and K3s, server or agent, off the current host and clear what Cilium left in the kernel: runs "uninstall_cmd" over the exec channel (no SFTP) and returns 1 when the host is clean afterwards. Destructive: the distribution's data (/var/lib/rancher, etcd with it) goes with its uninstall script, and Cilium's devices, iptables chains, tc attachments and ip rules are removed, as is the /etc/default/rke2-server or -agent file Rex::Rancher wrote for the NVIDIA runtime -- only when it holds nothing but that PATH line; an admin's file stays (see "uninstall_cmd"). It runs only when called; no install function calls it.

Logs every "uninstall_warnings" as a warning -- no tc on the host, or no iptables backend with both -save and -restore, so that part of the datapath was not checked or removed; a reboot clears it. A warning does not fail it.

Dies with "uninstall_failure" when RKE2 or K3s is still installed afterwards (an uninstall script missing or failing), or when Cilium's datapath state survived: that host needs a reboot before anything is installed on it again, or "install_server" in Rex::Rancher::Server and "install_agent" in Rex::Rancher::Agent refuse it (see "check_cilium_residue"). Running it on a host with nothing installed changes nothing and returns 1.

Joining the host to the same cluster again afterwards, under the same node name: rke2-uninstall.sh removes /etc/rancher/node, and the node password in it, so the new RKE2 registers with a new password, which the cluster refuses (Node password rejected) while it keeps the old one's hash in the secret kube-system/NODE.node-password.rke2. Delete the node from the cluster, which takes that secret with it, or the secret itself, before the host joins again. Nothing here does that. The K3s uninstall scripts leave /etc/rancher/node in place: a K3s node rejoins with the password it had.

SEE ALSO

Rex::Rancher, Rex::Rancher::Server, Rex::Rancher::Agent, Rex::Rancher::Cilium, Rex::Rancher::Distribution

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/rex-rancher/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.