Case study · Published on PyPI · v0.6.0
Python · Redis · Celery · Hypothesis · GitHub Actions · PyPI Trusted Publishing
Overview
fencekit is a Python library on PyPI for background jobs that use Redis and Celery. It stops duplicate starts, issues monotonic fencing tokens with locks, and gives Django/Postgres a one-statement way to reject stale writes. I extracted it from analysis pipelines on ChessMate after worker crashes and lock expiry produced duplicate Stockfish runs and ambiguous progress rows.
Problem
Celery with late acknowledgements redelivers work when a worker dies. That is the right reliability default. The same batch could start twice and burn CPU. Worse: a paused worker could resume after its lock TTL and overwrite progress a newer worker had already written. Celery tells you to make tasks idempotent. Redis gives SET NX and locks. Storage-side fencing is still your job.
What I built
IdempotencyGuard with owner-checked completion and optional JSON result memoization on redelivery.DistributedLock that increments a fencing token in the same Lua script as acquire.FenceGate.set_if_fresh for atomic Redis writes and fenced_update for Django rows with a fence_token column.idempotency_key from JSON-canonicalized payloads.try_begin_or_reclaim to take over a stale pending key after a worker crash when the lock is free.idempotent_task Celery decorator (fencekit[celery]).Flow
Celery task (redelivered)
IdempotencyGuard.try_begin_or_reclaim
SET NX · reclaim when lock free
DistributedLock.acquire
Lua · lock + fencing token
FenceGate
Redis progress strings
fenced_update
Postgres / Django rows
mark_done + get_result
memo on redelivery
The fencing token is what separates this from celery-once or redis-py Lock. Those tools dedupe or exclude concurrent runs. They do not give Postgres a monotonic number to compare on every write.
What we do not claim
Exactly-once delivery. Redlock or HA mutual exclusion across Redis failover. Safety when application code writes without presenting the token. Those limits are spelled out in DESIGN.md. I wrote them down on purpose: the guarantees are narrow and testable.
Comparison
celery-once and celery-singleton dedupe task scheduling. fencekit targets stale writers after lease expiry and ships a Postgres helper. Full table with sources: docs/COMPARISON.md.
Where it stands