Skip to content

Synapse v2.1.9 — "one attempt per interval"

Patch release fixing two latent defects — one pacing, one concurrency — found by reading the scheduler and Marker code during weekly maintenance. No schema migrations, no new endpoints, no API changes.

Like 2.1.8, neither of these came from a live failure report. Both are the kind of defect that stays invisible while everything works and only shows up on the bad day: one needs a scheduled import to start failing, the other needs two submissions to overlap. Neither is a security issue.

Fixed

A failing scheduled import spun the scheduler loop at zero delay

ImportScheduler._run() decides how long to sleep before the next scan with:

elapsed = (clock_now - self._last_run_at).total_seconds()
sleep_secs = max(0.0, full_interval - elapsed)

_last_run_at is assigned in exactly one place — self._last_run_at = run_at, on the success path, after _scan_fn has returned. That is correct for the job it was written for (R13-4/T4: after a container restart, don't re-scan immediately if the last scan was recent). It is the wrong clock for pacing retries.

When a scan raises instead of returning, _last_run_at stays frozen at the last success. On the next iteration elapsed is already ≥ full_interval, so max(0.0, …) clamps the sleep to 0.0. The scan fires again, raises again, and the reference still does not move. The result is an unbounded zero-delay retry loop (I7) that burns CPU, hammers the DB with upsert_schedule writes, and floods the log — and that nothing clears short of a container restart.

The distinction that matters is raising versus returning an error. run_one_scan() handles most failure modes gracefully and returns a ("dir_missing" | "error", …) tuple; those paths already advanced _last_run_at and paced correctly. The raising paths are narrower but real — most plainly raw_sources.mkdir(parents=True, exist_ok=True), which sits above the try/except (PermissionError, OSError) that guards the directory scan. A vault volume that goes read-only or fills up (an ordinary enough NAS event) raises OSError there, straight out of the scan and into the loop's except.

Fixed by pacing on the later of last success and last attempt:

reference = _later_of(self._last_run_at, self._last_attempt_at)

_last_attempt_at is stamped in the finally on every exit — success, failure, or cancellation — and measured from completion, matching how _last_run_at is stamped on the success path, so "one attempt per interval" is the ceiling either way. It is deliberately never persisted: a failed attempt must not look like a completed run across a restart, so the R13-4/T4 catch-up behaviour is untouched, and a genuinely overdue run still fires immediately on the first tick.

Regression test T-SP-010 drives a scan_fn that raises on every tick and asserts the sleeps stay at the full interval. Without the fix it records [0.0, 0.0, 0.0, 0.0].

Two concurrent Marker submissions could both start a batch

POST /ingest/convert-marker is single-flight by design — Marker is a single-GPU service that returns 429 under concurrent load, so the driver serialises every call. The guard was:

if is_running():
    raise HTTPException(status_code=409, ...)

But is_running() is _current_batch is not None and _current_batch.running, and _current_batch is only assigned by start_marker_batch() — which the handler calls after reading the entire upload. That read accepts up to MARKER_MAX_UPLOAD_BYTES (a deliberately large cap, since big PDFs are chunked by page range) and can take minutes. For that whole window the endpoint reports running=False while a submission is very much in progress: textbook check-then-act across a long await.

Two submissions overlapping in that window both passed the guard and both reached start_marker_batch(). The second overwrote both module globals:

  • _current_batch — the first batch vanished from GET /ingest/convert-marker/status, so its progress became unobservable;
  • _current_task — the first task lost its only strong reference. asyncio.create_task() keeps just a weak one, so the first conversion became eligible for garbage collection mid-run. This is precisely the BE-BUG-1 hazard the codebase guards against elsewhere with _bg_tasks strong-reference sets.

And both conversions ran against the single GPU the guard exists to protect. A double-click on "Convert" is enough to reach this.

Fixed with a synchronous claim, the same shape ImportScheduler.run_now already uses (the B9 fix): try_claim() reads and sets the flag with no await in between, so two callers can never both succeed. The slot is taken before the upload read and handed over to is_running() when the batch is armed. The handler body now runs inside a try/finally that calls release_claim(), so a 413/415/500 or a client disconnect frees the slot instead of wedging the endpoint at a permanent 409 — releasing after a successful arm is a harmless no-op, since ownership has already passed to _current_batch.

The regression test parks request A inside UploadFile.read on an asyncio.Event, issues request B in full, and asserts B gets 409 and exactly one batch is armed. Without the fix B gets 202 and two batches are armed.

Not fixed — deliberately

  • pypdf advisories are now 40, up from 24 a week ago. Still no fix available in any 5.x (the lowest fixed version is 6.0.0), so closing this means a major bump past the pypdf>=4.2,<6 pin — out of scope for a patch. The backlog entry in docs/reference/ROADMAP-3.0-IDEAS.md has been updated with the new count.
  • 11 npm advisories in frontend build/test tooling (vitest, vite, and transitives). npm audit --omit=dev reports 0 — none of this ships in the bundle. Fixing it requires major bumps to vite and vitest together, touching the build config, 123 test files, and the E2E job. Recorded in the 3.0 backlog instead.

Upgrade notes

None. This release is a drop-in replacement for 2.1.8:

  • No database migrations — alembic upgrade head is a no-op relative to 2.1.8.
  • No configuration changes, no new environment variables.
  • No API surface changes: docs/api/openapi.json differs from 2.1.8 only in the version string.
  • Both fixes are internal to the backend. _last_attempt_at is in-memory only and starts as None, so the first tick after upgrading behaves exactly as before.

Pull the new image and restart as usual:

docker compose pull && docker compose up -d

Desktop users: the macOS and Windows bundles for 2.1.9 are published as release assets and the Tauri updater will offer them automatically.