Resources staying in STARTING/DELETING state

[praiskup]

268847 - aws_aarch64_spot_prod_00268847_20231218_005905 pool=aws_aarch64_spot_prod tags= status=STARTING releases=0 ticket=NULL
401511 - aws_aarch64_normalreserved_prod_00401511_20231228_232914 pool=aws_aarch64_normalreserved_prod tags= status=STARTING releases=0 ticket=NULL

These machines are STARTING for multiple days. The fact that the allocator failed should be detected.

[praiskup]

This may happen in two situations:

• allocator script fails, and disappears from ps aux -> I'm not sure why/how this can happen
• the allocator script hangs... e.g. on indefinitely running ansible command (e.g. subscription manager, or alike)

[praiskup]

A similar thing happens when deleting OpenStack instances, from time to time, after (not 100% this is triggering the problem)

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/resalloc_openstack/helpers.py", line 74, in best_effort_delete
    self.delete()
  File "/usr/lib/python3.12/site-packages/resalloc_openstack/helpers.py", line 184, in delete
    self.nova_o.detach()
  File "/usr/lib/python3.12/site-packages/cinderclient/v3/volumes_base.py", line 69, in detach
    return self.manager.detach(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/cinderclient/v3/volumes_base.py", line 285, in detach
    return self._action('os-detach', volume,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/cinderclient/v3/volumes_base.py", line 257, in _action
    resp, body = self.api.client.post(url, body=body)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/cinderclient/client.py", line 223, in post
    return self._cs_request(url, 'POST', **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/cinderclient/client.py", line 211, in _cs_request
    return self.request(url, method, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/cinderclient/client.py", line 197, in request
    raise exceptions.from_response(resp, body)
cinderclient.exceptions.ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-1e419934-999b-4256-a07e-a6d5e369b9c5)
failed to delete in #1 attempt

[praiskup]

No, that would be different, I'm not sure what happened, starting of the instance in DELETING state failed:

+ ansible-playbook init.yml -i 10.0.150.201,
ERROR! the playbook: init.yml could not be found
running cleanup
cleaning 05_copr_vm_production_psi_os_00544952_20240229_172705_1
cleaning 10_server
deleting server 9e95963e-642b-41f3-b771-82411eba2386
Traceback (most recent call last):
  File "/usr/bin/resalloc-openstack-new", line 22, in <module>
    main() 
  File "/usr/lib/python3.12/site-packages/resalloc_openstack/new/main.py", line 131, in main
    check_call(args.command, env=env, shell=True, stdin=DEVNULL)
  File "/usr/lib64/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -x ; ansible-playbook init.yml -i "$RESALLOC_OS_IP," >&2 </dev/null' returned non-zero exit status 1.

... probably stayed in STARTING becuase of this bug. Then I restarted resalloc, and it stayed in DELETING state after:

=== /var/log/resallocserver/hooks/544952_terminate ===
initializing <class 'resalloc_openstack.helpers.Server'>
vm copr_vm_production_psi_os_00544952_20240229_172705 not found
initializing <class 'resalloc_openstack.helpers.Server'>
vm copr_vm_production_psi_os_00544952_20240229_172705 not found
initializing <class 'resalloc_openstack.helpers.Server'>
vm copr_vm_production_psi_os_00544952_20240229_172705 not found

[pkking]

Hi @praiskup, meet a same issue here, how can we delete theres orphans resource?
In my case, the ECS AK/SK is expired so the underlying resource is not existed, but no way to delete the STARTING and DELETING resource

[praiskup]

ECS AK/SK is expired

uhm, I'm not sure what that means?

but no way to delete the STARTING and DELETING resource

First, it is important to realize why they stay in STARTING that long (indefinitely). We don't seem to have these problems these days, so not sure.

For DELETING, again - check logs for the failure reason. If you think another attempt to delete resolves the problem, just restart resalloc server.

[pkking]

Some feedbacks:

1. the reason why a resource stay STARTING is because the token used to create new VM expired
2. I notice there is a resource cleanup right after process start, but when i restart resalloc, the STARTING resources just switch to DELETING because of 1
3. when we fix the situation 1, things go back on track

[praiskup]

the reason why a resource stay STARTING is because the token used to create new VM expired

Can't you modify the starting script so it fails with non-zero exit status if this happens? That's how we do it elsewhere.

but when i restart resalloc, the STARTING resources just switch to DELETING because of 1

Yes, that's unfortunately needed due to the Thread-based implementation of resalloc. We aren't able to recover the threads after process restart. The safest approach is to DELETE the machines (considering we have rights to delete them), and start over again.

when we fix the situation 1, things go back on track

Right.

At the end of the day, we should consider something like ccff92d for TerminateWorker and AllocWorker, because they might hang for whatever reason, too.