Too many open files caused by stuck hypervisor

[praiskup]

Stuck hypervisor caused that attempts to delete the VMs there repeatedly hang; resalloc is starting more and more processes ... and eventually ends up with:

ERROR 2025-05-14 05:52:11,326 PID:989:140299030574784(GarbageCleaner-<resallocserver.manager.Pool object at 0x7f9afc7c8d70>) Worker exception, pool=<resallocserver.manager.Pool object at 0x7f9afc7c8d70> resource=None
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/resallocserver/manager.py", line 208, in run
    self.job()
    ~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/resallocserver/manager.py", line 357, in job
    all_resources = self._list_all_resources()
  File "/usr/lib/python3.13/site-packages/resallocserver/manager.py", line 383, in _list_all_resources
    result = run_command(
        self.pool.id,
    ...<6 lines>...
        catch_stdout_lines_securely=True,
    ) 
  File "/usr/lib/python3.13/site-packages/resallocserver/manager.py", line 78, in run_command
    sp = subprocess.Popen(command, env=env, shell=True,
                          stdout=subprocess.PIPE, stderr=logfile) 
  File "/usr/lib64/python3.13/subprocess.py", line 1004, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr) 
                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 1730, in _get_handles 
    c2pread, c2pwrite = os.pipe()
                        ~~~~~~~^^
OSError: [Errno 24] Too many open files

[praiskup]

There were thousands of processes in that process group.

ssh -l user -T -e none -- hostname "sh -c 'which virt-ssh-helper 1>/dev/null 2>&1; if test \$? = 0; then     virt-ssh-helper 'qemu:///system'; else    if 'nc' -q 2>&1 | grep \"requires an argument\" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'nc' \$ARG -U /var/run/libvirt/libvirt-sock; fi'"

[praiskup]

We should have a timeout limit here:

resalloc/resallocserver/manager.py

Lines 78 to 79 in 08cb2a0

	sp = subprocess.Popen(command, env=env, shell=True,
	stdout=subprocess.PIPE, stderr=logfile)

But probably not for every run_command() instance. The guilty call is here:

resalloc/resallocserver/manager.py

Line 868 in 08cb2a0

TerminateWorker(event, self, int(res.id)).start()

[praiskup]

Deployed.

[praiskup]

Something similar is happening now, early 2026 (same traceback).

[FrostyX]

Is it possible this is not a bug and we have so many VMs that we need to bump ulimit -n?

[praiskup]

seems related to #178