Skip to content

Synapse v2.1.11 — "hand it back on the way out"

Patch release closing a permanent deadlock in the ingest concurrency cap, found by reading the ingest dispatch path during weekly maintenance. No schema migrations, no new endpoints, no API changes, no new configuration keys.

Like 2.1.8, 2.1.9 and 2.1.10, this did not come from a live failure report. Its consequence is the worst of that run, though: the failure mode is not a degraded result or a crash you can see in a log, it is ingest silently ceasing to work, with no error anywhere, until the container is restarted.

Fixed

A leaked capability slot deadlocked an entire provider mode until restart

run_ingest_pipeline acquires a per-capability concurrency slot right before the provider is called:

await orch.ingest_queue.acquire_capability_slot(caps.mode)

This is the BE-QUEUE-2 cap (1.9.4 W3) that bounds how many CLI/API/Local runs execute their provider calls at once, on top of the flat INGEST_MAX_CONCURRENCY gate the watcher applies earlier. It is an I7 bound: CLI and Local default to 1, because both are heavy and share a single GPU or a single process. API defaults to 3.

The slot was handed back at three hand-written places — the IngestCancelled branch, the except Exception branch, and the end of the success path. That is the whole defect: a resource acquired in one place and released at every exit by hand gets released only on the exits someone remembered. There was no finally.

What a leak costs. acquire_capability_slot() waits on the semaphore with no timeout:

async def acquire_capability_slot(self, mode: str) -> None:
    sem = self._capability_semaphores.get(mode)
    ...
    await sem.acquire()          # no timeout, no deadline

With a cap of 1, a single leaked slot means the next run of that mode blocks there forever. It has already opened its ingest_runs row, so that row sits at status="running" indefinitely. The watcher holds its own INGEST_MAX_CONCURRENCY slot across the entire call, so those stall behind it as well. Nothing raises, nothing logs, no health check goes red — ingest simply stops happening, and only a restart clears it.

The two exits that were reachable.

The ledger write failing. Both terminal branches call _finalize_ingest_run() — an unguarded UPDATE on ingest_runs — before releasing:

await _finalize_ingest_run(...)                       # DB UPDATE, can raise
orch.ingest_queue.finalize(run_id, "failed", ...)
orch.ingest_queue.release_capability_slot(caps.mode)  # never reached if it did

This is not a theoretical ordering nit. A failing database is precisely the condition that lands a run in the failure branch to begin with, so the failure that sends control here is very likely the same failure that then kills the ledger write. The exception propagates to the caller (which handles it correctly), and the slot is gone.

Cancellation. asyncio.CancelledError derives from BaseException, not Exception, so except Exception never sees it. Container shutdown cancels pending tasks; so does anything else that cancels the task running an ingest. The slot is gone there too — and on shutdown that does not matter, but a cancelled task in a live process leaves the mode wedged.

Worth separating from the above: the cooperative cancel path (IngestCancelled, raised when the user cancels a run through the queue's cancel_event) is an ordinary Exception and was always handled correctly. This is only about a hard Task.cancel().

The fix. One idempotent release, driven from the route try-block's finally — the only point in the function that covers every exit:

_slot_released = False

def _release_slot() -> None:
    nonlocal _slot_released
    if _slot_released:
        return
    _slot_released = True
    orch.ingest_queue.release_capability_slot(caps.mode)

The explicit releases on the two terminal branches stay exactly where they were, so on those paths the release still happens after the ledger write and the queue finalize — the ordering is unchanged, and the finally only closes the gap when one of those lines is skipped.

On the success path the release now happens one DB write earlier, when the try-block exits rather than after _finalize_ingest_run(). That is deliberate and is not a weakening of the cap: the provider work is definitively over the moment that block exits, so the bound still admits exactly N concurrent provider calls. The next queued run simply stops waiting on the previous run's bookkeeping before starting its own.

Security

An unbalanced release_capability_slot() silently widened the concurrency cap

asyncio.Semaphore.release() has no ceiling — it increments the counter unconditionally, whether or not anything was ever acquired. So a release with no matching acquire, or a double release, does not raise and does not no-op: it pushes the semaphore above its configured limit and leaves it there for the life of the process.

The I7 bound the cap exists to enforce would quietly stop holding. More concurrent provider calls would be admitted than the operator configured — more contention on the single GPU the CLI/Local caps exist to serialise, or a larger API bill — with nothing in any log, metric, or status endpoint to explain it. It would be found by noticing the symptom, not the cause.

No caller does this today, and the fix above is careful to release exactly once. But the guard was already documented as though it were safe: release_capability_slot()'s docstring said "no-op", and test_release_is_a_noop_for_never_acquired_mode asserted an intent the code did not honour. Both now describe what actually happens. The method counts held slots and ignores any release beyond them, logging a WARNING — so the cap is a real bound rather than a starting value, and a future caller cannot widen it by accident.

Also in this release

Two findings too large for a weekly patch were recorded in docs/reference/ROADMAP-3.0-IDEAS.md with evidence and a sizing instead of being rushed in:

  • A hard cancellation strands the run. The same missing-finally root cause also leaves the ingest_runs row at "running" and the RunHandle in the queue's _active map — a phantom run in the activity panel and an on_drained callback that stops firing. It is cosmetic rather than blocking, and fixing it properly means deciding what can even be written to the DB during shutdown (probably: extend the existing orphan-running-rows sweep to run on close), so it is a design question, not a finally.
  • pollChain refcount vs. restart. stopInternal() zeroes the refcount while subscribers are still attached, so a late unsubscribe from a previous run can stop a chain that has since been restarted by someone else. Latent — not reachable today, because the two shared chains never reach a terminal state and the ones that do have a single subscriber each.

Upgrade notes

Nothing to do. No migration, no configuration change, no API or MCP contract change, no new environment variable.

  • Behaviour change, success path only: the capability slot is released marginally earlier — when the provider work ends rather than after the ingest_runs UPDATE. Under a queue of waiting runs this is a small throughput improvement; the concurrency cap itself is unchanged.
  • Behaviour change, error path: an unbalanced release_capability_slot() is now ignored and logged at WARNING rather than silently raising the cap. If that warning ever appears in your logs, it is a bug worth reporting — nothing in the codebase should produce it.
  • If you have been restarting the backend to "unstick" ingest, that workaround should no longer be needed. A run already stuck at status="running" from before the upgrade is cleared by the existing orphan-running-rows sweep on the next startup.
  • Backend tests: 3184 passing, coverage 83.23% (gate 82%). Frontend: 2538 passing, untouched by this release.