Redis Advanced Architecture

 

Redis Streams vs Pub/Sub (Advanced Architecture)

When designing scalable systems using Redis, one of the most common architectural decisions is:

Should I use Pub/Sub or Streams?

1️⃣ Redis Pub/Sub – Fire & Forget Messaging

How It Works

  • Publisher sends message to channel

  • Redis pushes message instantly to subscribers

  • No storage

  • No acknowledgment

SUBSCRIBE notifications
PUBLISH notifications "New Order Created"

Characteristics

  • Real-time delivery

  • No persistence

  • No retry

  • No message history

  • No consumer tracking

Best For

✔ Live chat
✔ Real-time dashboards
✔ WebSocket notifications
✔ Lightweight microservice communication

2️⃣ Redis Streams – Persistent Event Log

Redis Streams (introduced in Redis 5) behave like a lightweight event streaming platform.

XADD orders * user_id 101 product "Laptop"
XREAD STREAMS orders 0

Key Features

  • Persistent message log

  • Consumer groups

  • Acknowledgment support

  • Message replay

  • Message ID tracking

Core Architectural Differences

FeaturePub/SubStreams
Persistence        No            Yes
Replay        No            Yes
Consumer Groups        No            Yes
Message Acknowledgment        No            Yes
Offline Consumers        Lose messages            Can catch up
Ordering        Per subscriber            Global stream order

Q: How would you design a reliable background job system using Redis?

Use Redis Streams, not Pub/Sub.

Why?

  • You need message durability

  • Worker crash recovery

  • Acknowledgment support

  • Retry mechanism

When to Use What?

Use Pub/Sub If:

  • You don’t care about message loss

  • You need ultra-low latency

  • It's purely real-time

Use Streams If:

  • You need durability

  • You need retries

  • You need multiple consumers

Scenario 1: Rate Limiter (API Protection)

Problem: Limit users to 100 requests/minute.

Solution:

Use Redis atomic counters:

INCR user:101:count
EXPIRE user:101:count 60

Why Redis?

  • Atomic operations

  • Fast lookups

  • TTL support

  • You are building a queue system

If reliability matters → Streams win.

If speed + simplicity matters → Pub/Sub is enough.

Scenario 2: Real-Time Chat System

Option A – Pub/Sub

  • Real-time message broadcasting

  • Good for active users

Option B – Streams

  • Store chat history

  • Allow offline replay

  • Better architecture for production

In real systems, combine both:

  • Pub/Sub for live

  • Streams for storage

Scenario 3: Background Job Queue

Instead of basic Pub/Sub:

Use:

  • Redis Streams

  • Or Celery (which often uses Redis as broker)

Why?

  • Acknowledgment

  • Retry

  • Worker coordination

Scenario 4: Leaderboard System

Use Sorted Sets:

ZADD leaderboard 500 user1
ZADD leaderboard 800 user2
ZRANGE leaderboard 0 -1 WITHSCORES

Why Redis?

  • O(log n) ranking

  • Instant score updates

  • Perfect for gaming systems

Scenario 5: Distributed Locking

Use:

SET resource_lock "1" NX EX 30

Prevents:

  • Duplicate payments

  • Concurrent cron execution

  • Double order processing

Scenario 6: Session Store for Django

With:Django

Why Redis?

  • Fast session lookup

  • Centralized session storage

  • Works well in multi-server deployments

In large-scale production systems:

LayerTool
Cache               Redis
Queue               Redis Streams / Celery
Real-time               Pub/Sub
Long-term events               Apache Kafka
Primary DB               PostgreSQL

Redis often acts as a performance accelerator, not primary storage.

“Is Redis Streams a replacement for Kafka?”

Answer:No.

Redis Streams:

  • Great for moderate-scale event systems

  • Simpler to operate

  • Less operational overhead

Kafka:

  • Designed for massive event pipelines

  • Long retention

  • Partition scaling

  • Strong durability guarantees

If You NeedUse
Real-time only          Pub/Sub
Reliable queue          Streams
Background jobs          Streams / Celery
Huge event platform          Kafka
Simple caching          Redis Strings


Comments

Post a Comment

Popular posts from this blog

Database Integration in FastAPI (SQLAlchemy CRUD)

  Introduction So far, we’ve built APIs without storing data.But real-world applications need a database to: Store user data. Manage products. Persist application state. In this blog, we’ll connect FastAPI with a database using SQLAlchemy and perform CRUD operations . What is SQLAlchemy? SQLAlchemy is a powerful Python ORM (Object Relational Mapper) that allows you to: Interact with databases using Python code. Avoid writing raw SQL queries. Work with models instead of tables. Step 1: Install Dependencies pip install sqlalchemy (For SQLite, no extra driver needed) Project Structure Update app/ ├── main.py ├── database/ │ ├── connection.py ├── models/ │ ├── item.py ├── schemas/ │ ├── item.py ├── routes/ │ ├── item.py Step 2: Database Connection app/database/connection.py from sqlalchemy import create_engine from sqlalchemy . orm import sessionmaker , declarative_base DATABASE_URL = "sqlite:///./test.db" engine = create_engine ( DAT...
Read more

Middleware & CORS in FastAPI

  Introduction As your API grows, you may need to: Log requests Modify responses Handle cross-origin requests This is where Middleware and CORS (Cross-Origin Resource Sharing) come into play.They help control how requests and responses flow through your application. What is Middleware? Middleware is a function that runs: Before a request reaches your route After a response is returned Think of it as a pipeline layer between client and server. Request Flow with Middleware Client → Middleware → Route → Middleware → Response Creating Custom Middleware from fastapi import FastAPI , Request import time app = FastAPI () @ app . middleware ( "http" ) async def log_requests ( request : Request , call_next ): start_time = time . time() response = await call_next ( request ) process_time = time . time() - start_time print ( f"Request: { request . url} | Time: { process_time } " ) return response What This Middle...
Read more

Python Data Handling

Python Data Handling for Beginners In Python, data handling means storing, changing, and using data in different ways. In this post, you will learn: Lists Tuples Dictionaries Sets String Methods Type Conversion 1. Python Lists :Lists store many values in one variable.                fruits = ["apple", "banana", "mango"]                                                                       Method                          Use                                            append()          ...
Read more