Database lock

[jorgemoralespou]

Describe the bug

@hguerrero reported and issue when running a Workshop:

│ [Tue Jun 16 13:52:22.127384 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/base.py", line 877, in save_base                              │
│ [Tue Jun 16 13:52:22.127387 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     updated = self._save_table(                                                                                                          │
│ [Tue Jun 16 13:52:22.127390 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         raw,                                                                                                                             │
│ [Tue Jun 16 13:52:22.127427 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     ...<4 lines>...                                                                                                                      │
│ [Tue Jun 16 13:52:22.127430 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         update_fields,                                                                                                                   │
│ [Tue Jun 16 13:52:22.127432 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     )                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127435 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/base.py", line 1020, in _save_table                           │
│ [Tue Jun 16 13:52:22.127438 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     results = self._do_insert(                                                                                                           │
│ [Tue Jun 16 13:52:22.127441 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         cls._base_manager, using, fields, returning_fields, raw                                                                          │
│ [Tue Jun 16 13:52:22.127445 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     )                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127448 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/base.py", line 1061, in _do_insert                            │
│ [Tue Jun 16 13:52:22.127451 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     return manager._insert(                                                                                                              │
│ [Tue Jun 16 13:52:22.127454 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]            ~~~~~~~~~~~~~~~^                                                                                                              │
│ [Tue Jun 16 13:52:22.127457 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         [self],                                                                                                                          │
│ [Tue Jun 16 13:52:22.127460 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         ^^^^^^^                                                                                                                          │
│ [Tue Jun 16 13:52:22.127463 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     ...<3 lines>...                                                                                                                      │
│ [Tue Jun 16 13:52:22.127466 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         raw=raw,                                                                                                                         │
│ [Tue Jun 16 13:52:22.127469 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         ^^^^^^^^                                                                                                                         │
│ [Tue Jun 16 13:52:22.127472 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     )                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127474 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     ^                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127477 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/manager.py", line 87, in manager_method                       │
│ [Tue Jun 16 13:52:22.127480 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     return getattr(self.get_queryset(), name)(*args, **kwargs)                                                                           │
│ [Tue Jun 16 13:52:22.127483 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^                                                                           │
│ [Tue Jun 16 13:52:22.127486 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/query.py", line 1810, in _insert                              │
│ [Tue Jun 16 13:52:22.127490 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     return query.get_compiler(using=using).execute_sql(returning_fields)                                                                 │
│ [Tue Jun 16 13:52:22.127493 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^                                                                 │
│ [Tue Jun 16 13:52:22.127496 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/models/sql/compiler.py", line 1833, in execute_sql                   │
│ [Tue Jun 16 13:52:22.127499 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     self.connection.ops.fetch_returned_insert_columns(                                                                                   │
│ [Tue Jun 16 13:52:22.127502 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^                                                                                   │
│ [Tue Jun 16 13:52:22.127505 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         cursor,                                                                                                                          │
│ [Tue Jun 16 13:52:22.127508 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         ^^^^^^^                                                                                                                          │
│ [Tue Jun 16 13:52:22.127510 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         self.returning_params,                                                                                                           │
│ [Tue Jun 16 13:52:22.127513 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]         ^^^^^^^^^^^^^^^^^^^^^^                                                                                                           │
│ [Tue Jun 16 13:52:22.127516 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     )                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127518 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     ^                                                                                                                                    │
│ [Tue Jun 16 13:52:22.127521 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/backends/base/operations.py", line 213, in fetch_returned_insert_col │
│ [Tue Jun 16 13:52:22.127524 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     return cursor.fetchone()                                                                                                             │
│ [Tue Jun 16 13:52:22.127527 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]            ~~~~~~~~~~~~~~~^^                                                                                                             │
│ [Tue Jun 16 13:52:22.127530 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/utils.py", line 97, in inner                                         │
│ [Tue Jun 16 13:52:22.127533 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     with self:                                                                                                                           │
│ [Tue Jun 16 13:52:22.127536 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]          ^^^^                                                                                                                            │
│ [Tue Jun 16 13:52:22.127539 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/utils.py", line 91, in __exit__                                      │
│ [Tue Jun 16 13:52:22.127542 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     raise dj_exc_value.with_traceback(traceback) from exc_value                                                                          │
│ [Tue Jun 16 13:52:22.127546 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]   File "/opt/app-root/venv/lib64/python3.13/site-packages/django/db/utils.py", line 98, in inner                                         │
│ [Tue Jun 16 13:52:22.127549 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678]     return func(*args, **kwargs)                                                                                                         │
│ [Tue Jun 16 13:52:22.127552 2026] [wsgi:error] [pid 12:tid 34] [remote 10.1.0.3:49678] django.db.utils.OperationalError: database is locked

Additional information

He's running version 3.7.1 with a workshop with 40+ attendees.

When asked Claude about this specific issue, replied back with:

Diagnosis

  The training portal is a Django app backed by SQLite (/opt/app-root/data/db.sqlite3), served by Apache mod_wsgi with multiple worker threads/processes. The DB file lives on a PersistentVolumeClaim (training-portal, mounted at
  /opt/app-root/data).

  database is locked is SQLite's response when a writer can't acquire the write lock within the busy-timeout. The settings give it 15s (settings.py:83), and it still timed out — so this is one of two things, often both:

  1. Write contention burst — a workshop's worth of attendees hitting the portal at the same moment (registration, session start, event/heartbeat writes). SQLite serializes all writes to a single lock. The portal uses the default
  rollback-journal mode (no WAL), so even reads can block writers. Under a burst, one slow write holds the lock past 15s and everyone else errors.
  2. PVC filesystem locking — if that training-portal PVC is backed by NFS / a network filesystem, SQLite's POSIX fcntl locking is unreliable there and throws database is locked even at low concurrency. This is a very common cause
  of this exact error in Educates portals. Worth checking the storage class.

  Right now (live workshop)

  - The error is transient and per-request. The attendee who hit it can just retry/refresh — their next request will usually succeed. No data corruption.
  - If many are stuck, restart the portal pod — it clears any stale lock and re-serializes cleanly:
  kubectl rollout restart deployment/training-portal -n <portal-namespace>
  - Check what's backing the PVC (this tells you if it's the deeper problem):
  kubectl get pvc training-portal -n <portal-namespace> -o jsonpath='{.spec.storageClassName}'
  kubectl get pvc training-portal -n <portal-namespace> -o jsonpath='{.spec.volumeMode}'
  - If the storage class is NFS-backed, that's your culprit — move the portal to a block-storage / local-path storage class.

  Durable fix (code change)

  Two cheap improvements in training-portal/src/project/settings.py. Want me to make these?

  - Enable WAL + a real busy timeout so readers don't block the writer and locks back off properly:
  "OPTIONS": {
      "timeout": 15,
      "init_command": "PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA busy_timeout=15000;",
      "transaction_mode": "IMMEDIATE",  # Django 5.1+; avoids upgrade-lock deadlocks
  },
  - (WAL only helps on a local/block filesystem — it does not fix NFS. If the PVC is NFS-backed, the storage class change is the actual fix.)

  Want me to:
  1. Make the WAL/busy-timeout settings change and open a branch for it, and/or
  2. Dig into where the high-frequency writes come from (the event/ and schedule/ endpoints are already filtered from logs at httpd.conf — they may be the write-heavy path worth throttling)?

[GrahamDumpleton]

For context, this is a known failure condition which we have seen before.

In the past when this has arisen the trigger has been that the Kubernetes control plain has stalled for some reason, which because there is a call to the Kubernetes REST API service under a lock, results in subsequent requests that arrive hitting the time out.

We have seen this caused by two possible triggers.

The first was due to a bad cluster setup where there was a communications issue between some nodes in the cluster. When the user destroyed that cluster and created a new one they didn't see any issue afterwards.

The second was possibly connected to Kubernetes applying rate limiting/throttling, especially when the training portal is configured for a large maximum capacity for sessions and reserved sessions were not configured. When you get a large number of users trying to request a session at same time and they are being created on demand, concern is what happens if Kubernetes starts throttling requests. This is noted in existing issue #656

The solution is to work out how to further disentangle the need to do a Kubernetes REST API call under the lock. Changing how the current database writes data is not likely to help. Changing to Postgres may be another option as part of the issue is around how SQLite serialises some database operations. Changing to Postgres would be a more heavy weight solution as current model for training portals means that each would have its own Postgres instance. Long term, rewriting things to be a golang service/operator may be best solution, but that is a much bigger undertaking and may be issues in maintaining REST API compatibility and interaction with current workshop session instances, causing even more changes and incompatibilities.

Workaround for now if seeing this with certain cluster setups is to use reserved sessions so workshop sessions are pre-created prior to starting a workshop as then when all the users pile on, is much simpler allocation of a session from database and less need to interact with Kubernetes to update in cluster resources.