Documentation navigation

Background Jobs with Django Q2

Generated projects use Django Q2 for asynchronous work.

If you are still choosing a queue, read Django Background Tasks: Choose a Queue for a workload-based comparison of Django 6 Tasks, Django Q2, and Celery.

If the earlier decision is whether to keep work in the HTTP request, use Django Async Views vs Background Tasks to compare request lifetime, transaction timing, retries, and user-visible state.

Why Django Q2 is used

Django Q2 is simple to operate inside a Django-first stack and works well for tasks like:

  • project artifact generation,
  • analytics/event processing,
  • webhook side effects,
  • email and non-blocking integrations.

For a concrete queue boundary, the Django Stripe subscriptions guide keeps signature verification and entitlement updates near the webhook request while moving slow, repeatable side effects to Q2.

Find the worker in your generated app

Run these steps inside the repository you downloaded from Djass, not inside Djass's hosted generator. Your app's queue settings live in <project_slug>/settings.py under Q_CLUSTER; replace <project_slug> with the module name you chose during generation. Put your application's task functions in apps/core/tasks.py or another importable task module.

The hosted Djass service queues repository generation from apps.core.views.create_project. That handler belongs to the generator; it is not a task you need to copy or run in your downloaded app.

Running workers locally

Start the local stack with make serve as described in the local development guide. The Compose service is named workers and runs uv run --no-sync python manage.py qcluster, separately from the web server.

Check that service and read its recent logs:

docker compose -f docker-compose-local.yml ps workers
docker compose -f docker-compose-local.yml logs --tail=100 workers

A running worker is a starting point, not proof that a task finished. After changing local worker configuration, recreate that service with:

make restart-worker

These commands are for your local stack. Do not restart a production worker or replay payment, email, or other business tasks just to test a diagnosis. Redact credentials and private task data before sharing logs.

Adding a new background task

  1. Create a pure function in apps/core/tasks.py (or a focused task module).
  2. Keep inputs serializable (ids, strings, dicts).
  3. Queue with async_task("path.to.function", ...) from views/services.
  4. Log key context (entity id, action, error path). The production Django logging guide shows how to carry an explicit correlation id from the request into the worker without logging payloads or credentials.
  5. Record user-facing status when task state matters.

Worked example: a private CSV export

A customer should be able to request a report, leave the page, and return to download the finished file without keeping an HTTP request open. Build that workflow around a durable export record, not a task-result ID. This is a feature you add to your downloaded repository; Djass does not generate a report model, CSV endpoint, or private-download policy.

Choose the request boundary

A small, bounded report may need only a normal response. Django 6.0 documents CSV responses and streaming, but streaming does not turn the request into a background job. Under WSGI, a streaming response occupies a worker for its duration. Choose queued generation when the report must survive the browser closing or needs a separate execution budget. Measure representative queries and concurrent requests before choosing limits; there is no universal safe row count.

Give the agent a bounded build brief

Start with one report and an explicit list of allowed columns. Exclude arbitrary SQL, user-selected storage keys, and unrestricted model-field exports. Give your coding agent this contract in the generated app, not the hosted generator:

Add one owner-scoped CSV report. Create an export record containing its owner, validated filters, column selection, state, expiry, and server-generated file key. Authenticate the request and derive ownership on the server. Queue the record ID after commit; never serialize the request, a QuerySet, or a whole dataset into the message. Recheck access when work starts and on every status or application download/link-issuance request. Bound concurrent exports and reconcile stale jobs. Publish one private file only after generation completes; interrupted attempts must not expose partial files or overwrite a later successful attempt. Define retries, expiry, cleanup, and spreadsheet-cell handling. Show allowed and denied paths before calling the feature complete.

Implement the record and task in apps/core, using this page's existing worker setup. Follow the feature-development workflow for owner-scoped lookups. The starter's account profile is not a report model or a workspace-role system. If you need team reports, define membership and revocation rules before widening this owner-only example.

Make the handoffs explicit

  • Save, then dispatch: Django 6.0's transaction.on_commit() prevents dispatch before a successful commit. It does not make the database commit and broker enqueue atomic. Keep failed or missed dispatch discoverable for reconciliation; the starter's Redis broker does not provide receipts for recovering an in-flight job after a worker crash.
  • Freeze the request, not imaginary data: keep validated filters and columns immutable after queueing; changed requests create another export record. Saved filters do not preserve historical row values. State whether the report reads data at execution time or requires a separately implemented snapshot. Recheck authorization before querying, and use deterministic ordering.
  • Bound memory and publication: Django 6.0's QuerySet.iterator() avoids QuerySet-level result caching, not every driver, relation, or file-buffer allocation. Check the database/pool configuration and measure worker memory. Write to an attempt-specific private location, then publish the completed result through a guarded state transition. Do not label a failed file ready.
  • Protect the download: apply the generated-media access checklist before using S3. Its public-read baseline is unsuitable unchanged for private reports. Use an authenticated download or deliberately short-lived signed link, with private storage underneath. Authorize link issuance in the app; direct use of an issued link does not repeat that application check. Export expiry or access revocation must deny new app downloads/link issuance and trigger the appropriate cleanup. An issued link may still work until its own expiry or effective storage-side invalidation. Keep report contents and signed URLs out of logs and analytics.

Prove the customer outcome

Test two accounts: A can request, inspect, and download A's completed report; B cannot access it by changing an ID. Also cover access revoked while queued, double-clicked requests, duplicate worker attempts, worker death before and after file publication, empty results, and expiry during download authorization. Reconciliation must not resurrect an expired or revoked export.

Check column order, Unicode, delimiters, and spreadsheet behavior with hostile cell values in the intended client. CSV quoting alone is not protection against spreadsheet formulas; use the OWASP CSV injection guidance to define and test an output policy. Run storage, memory, concurrency, and proxy checks against the actual deployment separately from unit tests. This brief is an acceptance plan, not an implemented export feature or a security assessment.

Reliability guidelines

  • Make tasks idempotent where possible.
  • Validate records exist before processing.
  • Store failure details on model fields when users need visibility.
  • Prefer retryable operations and defensive exception handling.
  • Monitor useful-work freshness through the queue rather than treating a green web process or Redis check as proof that workers are consuming jobs. The Django health check guide defines the separate readiness and heartbeat contracts.
  • For deployments, verify the intended web and worker release as well as readiness. The release-verification checklist explains the generated starter's revision-scoped marker and why it does not replace a completed-task smoke test.
  • For recurring work, define catch-up, intended-slot, overlap, idempotency, and completion policy with the Django scheduled-tasks guide before creating a Q2 schedule.

Diagnose Django Q2 tasks that are not running

Start with the symptom instead of repeatedly enqueueing the same task:

Symptom Check next What the check tells you
No worker service, or repeated exits Inspect ps workers and recent worker logs using the commands above Whether a worker starts and stays running; a healthy web server does not establish this
Worker runs but does not consume the expected task Compare the effective broker, queue name, and environment in web and worker processes Whether both processes use the same queue, rather than two healthy but separate queues
Task fails when imported or executed Verify the dotted function path, installed code, serialized arguments, and worker error Whether the worker can load and run that task; a returned task ID only proves enqueueing progressed
No result appears where expected Check completion/failure logs and your configured result retention or cache Whether execution failed, is still pending, or its result is stored elsewhere; absence alone does not prove non-execution
Work runs twice, or disappears after a crash Identify the actual broker and its receipt behavior before changing retry settings Which failure modes need task-level idempotency and recovery rather than a larger retry number

Check the same effective configuration on both sides

The bundled starter loads .env for both local backend and workers services. Check the values each running process actually uses, not just whether the files look identical:

  • REDIS_URL selects the Redis endpoint when nonempty. Without it, the starter builds a connection from its Redis component settings. Check the database number too; two Redis databases are not the same queue.
  • Q_CLUSTER_NAME defaults to LOCAL_INSTANCE_ID plus -q. Different instance or cluster names can separate otherwise similar environments.
  • Upstream Q2's cluster requirements require a shared broker, cluster name, and Django SECRET_KEY for processes sharing a queue. Compare secrets privately; never paste their values or a credential-bearing connection URL into a bug report.
  • The starter's DJANGO_TEST_USE_EXISTING_DATABASE branch replaces the Redis broker with orm: "default". A process using that test override will not consume the normal Redis queue; compare the selected database as well.

Use the environment configuration guide for URL overrides and container-versus-host addresses. Inspect the settings in your generated revision before assuming that a later starter default applies.

Treat retries as broker-specific behavior

The bundled configuration sets four workers, a 3,600-second timeout, a 4,800-second retry value, and two maximum attempts. Those numbers are not a completion guarantee. Q2's default Redis broker has no message receipts; its retry setting does not make an in-flight task survive a worker or cluster crash. See the upstream broker reference before choosing a recovery policy.

For receipt-capable brokers, the retry interval also includes time waiting in the cluster's internal queue. Keep it above the execution timeout with room for queueing; do not tune it from execution time alone. Design tasks so a repeated attempt cannot repeat an irreversible business effect.

Once configuration agrees, run one deliberately safe task in local development and verify its intended result. Then apply the worker freshness checks to ongoing operation. If one-off work succeeds but recurring work does not, continue with the scheduled-task checklist: a scheduler trigger and a completed business operation are different events.