Celery Questions and Answers

 

Q1: What is a message broker?

A system that passes messages between producer and worker (Redis, RabbitMQ).

Q2: Difference between synchronous and asynchronous?

Synchronous blocks execution. Asynchronous runs in background.

Q3: How does Celery scale?

By adding more worker processes or containers.

Q4: What happens if worker crashes?

Task remains in broker queue and can be retried.

Q5: Difference between Celery and cron?

Cron schedules OS-level jobs. Celery manages distributed background jobs.

1️⃣ What is Celery?

Celery is a distributed task queue used to run asynchronous and background tasks in Python applications.

It helps execute time-consuming operations outside the request-response cycle.

2️⃣ Why do we need Celery in Django?

In Django:

Without Celery:

  • API waits until email/report/API call finishes

  • Slow response time

With Celery:

  • Tasks run in background

  • Faster API response

  • Better scalability

3️⃣ What is a Message Broker?

A message broker is an intermediary that sends tasks from the application to worker processes.

Common brokers:

  • Redis

  • RabbitMQ

4️⃣ What is a Worker?

A worker is a separate process that:

  • Listens to the broker

  • Picks up tasks

  • Executes them

Run worker:

celery -A project worker --loglevel=info

5️⃣ What is the difference between synchronous and asynchronous tasks?
Synchronous            Asynchronous
Blocks execution            Runs in background
User waits            User gets instant response
Slower APIs            Faster APIs
6️⃣ Explain Celery architecture.
Flow:
Client → Broker → Worker → Result Backend
  1. App sends task
  2. Broker stores task
  3. Worker consumes task
  4. Result stored (optional)
7️⃣ What is a Result Backend?
Stores task results and states.
Example:
CELERY_RESULT_BACKEND = "redis://localhost:6379/0"
Without result backend → task runs but result not stored.
8️⃣ What are Celery task states?
  • PENDING
  • STARTED
  • SUCCESS
  • FAILURE
  • RETRY
9️⃣ How do you retry a failed task?
from celery import shared_task

@shared_task(bind=True, max_retries=3)
def call_api(self):
    try:
        risky_operation()
    except Exception as exc:
        raise self.retry(exc=exc, countdown=5)
Retries 3 times with 5 seconds delay.
🔟 What is Celery Beat?
Celery Beat is a scheduler that runs periodic tasks like cron jobs.
Example:
  • Daily reports
  • Cleanup jobs
  • Auto notifications
1️⃣1️⃣ What is the difference between Celery and Cron?
CeleryCron
Distributed            OS-level
ScalableRuns on single machine
Works with queuesSimple scheduling
1️⃣2️⃣ What are chains in Celery?
Used to run tasks sequentially.
from celery import chain

chain(task1.s(), task2.s())()
task2 runs after task1.
1️⃣3️⃣ What are groups?
Used to execute tasks in parallel.
from celery import group

group(task.s(i) for i in range(10))()
1️⃣4️⃣ What is a chord?
Group of tasks + callback after all complete.
from celery import chord

chord([task1.s(), task2.s()])(callback.s())
1️⃣5️⃣ How does Celery scale?
By:
  • Adding more workers
  • Increasing concurrency
  • Using multiple queues
  • Running in containers (Docker/Kubernetes)
1️⃣6️⃣ What happens if a worker crashes?
  • Task remains in broker
  • Another worker can pick it
  • If acknowledged before crash → may be lost (depends on acks_late setting)
1️⃣7️⃣ What is acks_late=True?
If enabled:
  • Task acknowledged only after completion
  • Prevents data loss on worker crash
  • Ensures reliability
1️⃣8️⃣ What is task idempotency and why important?
Idempotent task → Running multiple times gives same result.
Important because:
  • Tasks may retry
  • Worker may crash
  • Network failures occur
Example:
Instead of:
balance += 100
Use:
if not already_processed:
    balance += 100
1️⃣9️⃣ How do you handle large tasks?
Best practices:
  • Break into smaller subtasks
  • Use task chaining
  • Avoid passing large objects
  • Store files in S3 and pass reference
2️⃣0️⃣ How do you monitor Celery?
Common tools:
  • Flower (UI dashboard)
  • Logs
  • Prometheus + Grafana
  • Redis monitoring
🔹 REAL-TIME SCENARIO QUESTIONS
Scenario 1:
Emails are sending twice. Why?
Possible reasons:
  • Task retried
  • Not idempotent
  • Worker restarted
  • acks_late misconfigured
Scenario 2:
Tasks stuck in PENDING state. Why?
  • Worker not running
  • Broker connection issue
  • Wrong queue
  • Task not registered
Scenario 3:
High memory usage in Celery workers. Why?
  • Large objects in tasks
  • Not restarting workers
  • Memory leak in code
Solution:
celery -A project worker --max-tasks-per-child=100

❓ How do you deploy Celery in production?
  • Separate containers
  • Use Redis/RabbitMQ cluster
  • Supervisor or systemd
  • Auto-scaling
  • Health checks
❓ How do you ensure reliability?
  • acks_late=True
  • Retry with exponential backoff
  • Idempotent tasks
  • Dead letter queues
❓ Can Celery work without Django?
Yes. It works with:
  • Flask
  • FastAPI
  • Pure Python apps
    Concept                Key Point
       Broker                Redis / RabbitMQ
       Worker                Executes tasks
       Beat                Scheduler
       Chain                Sequential
       Group                            Parallel
       Chord                Group + callback
       Retry                Handles failure
       Idempotency                Prevent duplicates

?would you design a scalable background task system?
  1. Use Celery with Redis broker
  2. Separate worker containers
  3. Multiple queues (email, reports, payments)
  4. Idempotent tasks
  5. Monitoring (Flower)
  6. Retry + exponential backoff
  7. Horizontal scaling

Comments

Post a Comment

Popular posts from this blog

Database Integration in FastAPI (SQLAlchemy CRUD)

  Introduction So far, we’ve built APIs without storing data.But real-world applications need a database to: Store user data. Manage products. Persist application state. In this blog, we’ll connect FastAPI with a database using SQLAlchemy and perform CRUD operations . What is SQLAlchemy? SQLAlchemy is a powerful Python ORM (Object Relational Mapper) that allows you to: Interact with databases using Python code. Avoid writing raw SQL queries. Work with models instead of tables. Step 1: Install Dependencies pip install sqlalchemy (For SQLite, no extra driver needed) Project Structure Update app/ ├── main.py ├── database/ │ ├── connection.py ├── models/ │ ├── item.py ├── schemas/ │ ├── item.py ├── routes/ │ ├── item.py Step 2: Database Connection app/database/connection.py from sqlalchemy import create_engine from sqlalchemy . orm import sessionmaker , declarative_base DATABASE_URL = "sqlite:///./test.db" engine = create_engine ( DAT...
Read more

Middleware & CORS in FastAPI

  Introduction As your API grows, you may need to: Log requests Modify responses Handle cross-origin requests This is where Middleware and CORS (Cross-Origin Resource Sharing) come into play.They help control how requests and responses flow through your application. What is Middleware? Middleware is a function that runs: Before a request reaches your route After a response is returned Think of it as a pipeline layer between client and server. Request Flow with Middleware Client → Middleware → Route → Middleware → Response Creating Custom Middleware from fastapi import FastAPI , Request import time app = FastAPI () @ app . middleware ( "http" ) async def log_requests ( request : Request , call_next ): start_time = time . time() response = await call_next ( request ) process_time = time . time() - start_time print ( f"Request: { request . url} | Time: { process_time } " ) return response What This Middle...
Read more

Python Data Handling

Python Data Handling for Beginners In Python, data handling means storing, changing, and using data in different ways. In this post, you will learn: Lists Tuples Dictionaries Sets String Methods Type Conversion 1. Python Lists :Lists store many values in one variable.                fruits = ["apple", "banana", "mango"]                                                                       Method                          Use                                            append()          ...
Read more