Wetask's next update brings durable, fenced external workers, atomic batch settlement, and first-party Go, TypeScript, and Python clients.

Wetask started with a simple goal: combine the pieces commonly assembled around
a background job system into one Go-based runtime.

That means a task queue, scheduler, cache, consensus, operational APIs, and
observability without requiring a separate broker, result backend, and schedule
service for every deployment.

The next Wetask update takes that idea beyond embedded handlers. Applications
will be able to run their own workers in another process, container, language,
or execution environment while Wetask remains responsible for durable task
state.

Bring your own workers

External workers are useful when execution cannot or should not happen inside
the Wetask server:

  • Python workers running machine-learning jobs
  • TypeScript workers calling application services
  • Go workers handling high-throughput backend work
  • GPU or memory-specialized worker pools
  • independently deployed workers with their own release cycle
  • workers isolated from the queue and control-plane process

The first-party Go, TypeScript, and Python clients expose the worker API, while
HTTP and gRPC remain available for custom integrations.

Workers can request batches by queue and task type, renew long-running work,
report success, or classify a failure as transient, permanent, or cancelled.
Credentials can be limited to specific queues and task types.

Fencing matters more than another Ack endpoint

At-least-once systems must assume a task can be delivered again. A worker can
pause, lose its connection, exceed its lease, and continue running after
another worker has received the same task.

Wetask gives every claimed attempt a one-time fencing token. Completion is
accepted only when the task, claim, token, principal, attempt, and coordinator
still match. Once an attempt expires or is superseded, the old worker cannot
complete it.

The token itself is returned only to the worker. Wetask stores its SHA-256 hash.

This does not make external side effects exactly once. A worker that charges a
card and crashes before acknowledging the task can still cause a retry.
Handlers must use the task ID and attempt to make downstream operations
idempotent.

What fencing does guarantee is that a stale worker cannot overwrite the state
of the current attempt.

Durable and idempotent settlement

Successful and failed settlements are persisted before Wetask returns their
receipts. Retrying the same acknowledgment returns the original receipt instead
of completing or retrying the task twice.

Failure handling is task-aware:

  • transient schedules another attempt when retries remain
  • permanent ends the task and records it in the dead-letter queue
  • cancelled ends execution without retrying
  • lease expiration applies the configured retry and DLQ policy automatically

Workers can acknowledge or reject a complete batch atomically. Wetask validates
every fence first. If one item is stale or invalid, none of the unsettled items
in that batch are changed.

Batch settlement also reduces durable storage overhead by recording the batch
with one WAL synchronization.

Benchmarks

We benchmarked the complete in-process lifecycle:

  1. submit tasks
  2. claim them as an external worker
  3. acknowledge them

The benchmark does not execute application work, and it does not include HTTP
or gRPC network overhead. It measures the task runtime itself.

Environment:

  • Apple M3
  • macOS darwin/arm64
  • Go benchmark process reporting 8-way parallelism
  • batch sizes of 1, 8, and 32
  • three samples per scenario
  • three-second benchmark windows

Command:

go test ./internal/task \
  -run '^$' \
  -bench '^BenchmarkWorkerClaimAck($|Parallel$)' \
  -benchmem \
  -benchtime=3s \
  -count=3

Enter fullscreen mode Exit fullscreen mode

In-memory lifecycle

The table reports median tasks per second:

Scenario Earlier baseline Current result Difference
Batch 1 74,933 77,123 +2.9%
Batch 8 84,434 86,393 +2.3%
Batch 32 84,661 86,783 +2.5%
Parallel, single-task operations 64,429 63,443 -1.5%

Batch 32 completes the full lifecycle in about 11.52 microseconds per task.

The in-memory increase is intentionally modest. There is no disk barrier to
amortize, and every task still needs an ID, claim token, state transitions,
queue bookkeeping, result retention, metrics, and lifecycle events. The
parallel result is effectively flat and is included rather than hidden.

Durable lifecycle

Durable mode includes WAL synchronization for accepted submission, claim, and
settlement:

Scenario Median throughput Relative to earlier 86.34 tasks/s baseline
Batch 1 80.08 tasks/s 0.93x
Batch 8 645.7 tasks/s 7.5x
Batch 32 2,402 tasks/s 27.8x
Parallel, single-task operations 136.4 tasks/s 1.58x

At batch 32, durable processing costs approximately 416 microseconds per task.

The theoretical batching ceiling is 32x because the same durable barriers are
shared by 32 tasks. The measured 27.8x improvement reaches about 87% of that
ceiling. The remaining time covers WAL serialization, task state, metrics, and
the disk synchronization itself.

These are Wetask-only measurements, not claims that Wetask is faster than every
broker. A fair competitive benchmark must use equivalent payloads, transports,
batch sizes, persistence policies, replication, and acknowledgment semantics.

How Wetask compares

RabbitMQ quorum queues

RabbitMQ quorum queues are mature, Raft-replicated queues designed for data
safety and leader failover. Publisher confirms and manual consumer
acknowledgments provide a strong production foundation.

RabbitMQ remains ahead of Wetask in replicated queue durability, operational
history, protocol support, and ecosystem size.

Wetask aims at a more task-specific experience: results, retries, deadlines,
cancellation, dead letters, scheduling, cache facilities, and worker fencing
are available through one runtime and API.

Reference:
RabbitMQ quorum queues

NATS JetStream

JetStream is the closest protocol-level comparison. It offers pull and push
consumers, explicit Ack, Nak, delayed Nak, Term, in-progress acknowledgments,
AckWait, backoff, and maximum delivery counts.

JetStream is significantly more mature in clustered stream storage, pub/sub,
replication, and high-throughput messaging.

Wetask's distinction is strict attempt fencing. JetStream documents that a late
acknowledgment can be accepted after a message has already been redelivered to
another subscriber. Wetask rejects settlement from a superseded attempt.

Reference:
NATS JetStream consumers

AWS SQS and Google Cloud Pub/Sub

Managed queues remove most broker operations and provide elastic,
multi-availability-zone infrastructure.

SQS uses visibility timeouts and receipt handles. Standard queues remain
at-least-once and consumers must tolerate duplicate delivery.

Google Cloud Pub/Sub's exactly-once pull subscriptions provide a particularly
interesting comparison: only the latest acknowledgment ID remains valid after
redelivery. That is conceptually close to Wetask's fencing model.

Wetask is for teams that need self-hosting, portability, local deployment, and
task-specific state. Cloud queues are the stronger choice when a managed
service and provider-operated availability are more important.

References:

Apache Kafka

Kafka is an ordered, replayable event log. It is the better fit for event
streaming, large retained histories, partition ordering, and many independent
consumer groups.

Traditional Kafka consumers commit partition offsets rather than independently
settling arbitrary jobs. Retries for one record can affect partition progress,
and task results, per-task leases, and dead-letter behavior normally require
application conventions or additional topics.

Wetask is the more direct model for background jobs. Kafka is the stronger
model for durable event streams.

Reference:
Kafka delivery semantics

Celery with RabbitMQ

Celery remains a mature choice for Python applications. It has a large
ecosystem, familiar task decorators, retries, Canvas workflows, and years of
operational knowledge.

Its typical production shape combines Celery workers, a broker such as
RabbitMQ, a result backend, and Celery Beat for schedules.

Wetask's proposition is a smaller operational surface: one Go runtime provides
the queue, task state, scheduler, cache, APIs, and administration, while
external workers can still be written in Python, TypeScript, or Go.

Reference:
Celery documentation

Temporal

Temporal solves a larger problem than a task queue. It is designed for durable,
multi-step workflows that resume across process failures and infrastructure
outages.

Temporal is the stronger choice for long-running business processes, sagas,
durable timers, and workflow histories. Wetask is intentionally simpler for
ordinary asynchronous jobs, scheduled work, and application caching.

Reference:
Temporal documentation

Where Wetask fits

Wetask is not trying to replace every messaging system.

Choose Kafka when the log is the product. Choose Temporal when durable workflow
orchestration is the product. Choose a managed cloud queue when outsourcing
operations is the priority. Choose RabbitMQ or JetStream when a mature,
replicated messaging foundation matters more than an integrated application
runtime.

Wetask is for teams that want:

  • a compact self-hosted task platform
  • durable background jobs
  • workers outside the server process
  • strict stale-worker rejection
  • task results, retries, deadlines, and DLQ management
  • scheduling and cache capabilities without another service stack
  • HTTP, gRPC, Go, TypeScript, and Python integration

Availability

External workers are planned for the next Wetask alpha release. The first
release will focus on durable single-node use and controlled production pilots.
Multi-node coordination exists, but task queue WAL ownership is currently
per-node; replicated task ownership and permanent-node-loss recovery remain
future hardening work.

That boundary is deliberate. The goal of the alpha is to put the complete
worker experience into real projects, publish reproducible evidence, and let
production feedback shape the next stage.

If you are evaluating background job infrastructure, the most useful feedback
is not "which benchmark number is largest?" It is: which failure semantics,
operational model, and worker experience match the system you actually need to
run?