Redis Beyond Caching: Pub/Sub, Streams, Sorted Sets, and Distributed Locks

Redis is often introduced as a caching solution, but its rich data structures make it a multi-purpose in-memory data store suitable for messaging, rate limiting, session storage, job queues, real-time leaderboards, and distributed coordination.

Pub/Sub for Real-Time Messaging

import redis
r = redis.Redis()

# Publisher
r.publish("article_events", json.dumps({"type": "published", "slug": "my-article"}))

# Subscriber (runs in separate thread/process)
pubsub = r.pubsub()
pubsub.subscribe("article_events")
for message in pubsub.listen():
    if message["type"] == "message":
        event = json.loads(message["data"])

Redis Streams for Durable Event Log

Unlike Pub/Sub (fire-and-forget), Redis Streams persist messages and allow consumer groups to process events reliably with acknowledgment—similar to Kafka but embedded in Redis. Ideal for audit logs and lightweight event sourcing.

Sorted Sets for Leaderboards

# Add user score
r.zadd("article_views", {"my-article-slug": 1500})

# Get top 10 most-viewed articles
top_articles = r.zrevrange("article_views", 0, 9, withscores=True)

Distributed Locks with Redlock

When multiple workers race to perform the same operation (e.g., sending a single welcome email), use Redlock to acquire a distributed lock before proceeding. The lock automatically expires if the worker crashes, preventing deadlocks.

from redis import Redis
from redlock import Redlock

dlm = Redlock([{"host": "localhost"}])
lock = dlm.lock("payment_processing:order_123", 10000)  # 10s TTL
if lock:
    try:
        process_payment()
    finally:
        dlm.unlock(lock)

Production-Grade Python Implementation Example

To demonstrate these concepts, here is a complete, production-grade Python block showing proper error boundary management, type safety annotations, and context lifecycle handling:

import logging
import time
from typing import Generator, Any, Dict, Optional
from functools import wraps

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("MirahLabs.ProductionTelemetry")

class ProductionServiceException(Exception):
    """Custom domain exception for pipeline operations."""
    pass

def with_telemetry(operation_name: str):
    """Decorator to log latency, parameters, and handle exception boundaries."""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            start_time = time.perf_counter()
            logger.info(f"Starting execution of {operation_name} with params: {args}, {kwargs}")
            try:
                result = func(*args, **kwargs)
                elapsed = time.perf_counter() - start_time
                logger.info(f"Successfully completed {operation_name} in {elapsed:.4f} seconds.")
                return result
            except Exception as e:
                elapsed = time.perf_counter() - start_time
                logger.error(f"Failed execution of {operation_name} after {elapsed:.4f}s: {str(e)}")
                raise ProductionServiceException(f"Pipeline error in {operation_name}") from e
        return wrapper
    return decorator

class DataPipelineProcessor:
    def __init__(self, config: Dict[str, Any]) -> None:
        self.config = config
        self.is_active = True

    @with_telemetry("process_data_payload")
    def process_payload(self, payload: Dict[str, Any]) -> Dict[str, Any]:
        if not self.is_active:
            raise ProductionServiceException("Processor is deactivated.")
        if "id" not in payload:
            raise ValueError("Payload missing mandatory key: 'id'")
        
        # Simulating domain-specific calculations
        processed_data = {**payload, "status": "processed", "timestamp": time.time()}
        return processed_data

# Example Usage
if __name__ == "__main__":
    pipeline = DataPipelineProcessor(config={"mode": "production"})
    try:
        pipeline.process_payload({"id": "evt_10928a", "value": 42.0})
    except ProductionServiceException:
        pass

Production Trade-offs & Implementation Decisions

Deploying this solution in production environments requires a careful analysis of the trade-offs involved. For instance, focusing purely on consistency (such as ACID compliance) can limit network throughput and horizontal scalability. On the other hand, adopting an eventual consistency model can lead to dirty reads and requires complex conflict resolution strategies in the application layer.

At MirahLabs, our engineering teams balance these architectural constraints by separating critical transaction paths from analytics workloads. We apply message-driven architectures with idempotent consumer systems to guarantee that network failures or retries do not result in double processing or state contamination.

Real-World Benchmarks & Resource Planning

Below is a typical performance comparison profile compiled by our engineering team in staging environments under simulated loads (10k concurrent virtual users):

When capacity planning, we recommend scaling out horizontally using containerized workloads rather than vertically upgrading underlying instance models. This maximizes uptime and provides cost efficiency through dynamic scaling policies.

Security Considerations & Vulnerability Mitigations

No production blueprint is complete without addressing security. Ensure that all data paths utilize encryption in transit (TLS 1.3) and at rest (using AES-256). Furthermore, implement strict Role-Based Access Control (RBAC) to limit operations. For APIs, always enforce rate limits (e.g. using token bucket algorithms in Redis) and run continuous static application security testing (SAST) in your CI pipeline.

How MirahLabs Applies This in Practice

Our experience building high-volume solutions like MirahCare.ai and Ayurveda.ai has taught us that early optimization is often a trap, but ignoring structural security and data design early leads to fatal development blocks. We design all client products from day one to support modular extensions, robust query indexing, and standard schema definitions, ensuring rapid iteration without technical debt growth.

Production-Grade Python Implementation Example

To demonstrate these concepts, here is a complete, production-grade Python block showing proper error boundary management, type safety annotations, and context lifecycle handling:

import logging
import time
from typing import Generator, Any, Dict, Optional
from functools import wraps

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("MirahLabs.ProductionTelemetry")

class ProductionServiceException(Exception):
    """Custom domain exception for pipeline operations."""
    pass

def with_telemetry(operation_name: str):
    """Decorator to log latency, parameters, and handle exception boundaries."""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            start_time = time.perf_counter()
            logger.info(f"Starting execution of {operation_name} with params: {args}, {kwargs}")
            try:
                result = func(*args, **kwargs)
                elapsed = time.perf_counter() - start_time
                logger.info(f"Successfully completed {operation_name} in {elapsed:.4f} seconds.")
                return result
            except Exception as e:
                elapsed = time.perf_counter() - start_time
                logger.error(f"Failed execution of {operation_name} after {elapsed:.4f}s: {str(e)}")
                raise ProductionServiceException(f"Pipeline error in {operation_name}") from e
        return wrapper
    return decorator

class DataPipelineProcessor:
    def __init__(self, config: Dict[str, Any]) -> None:
        self.config = config
        self.is_active = True

    @with_telemetry("process_data_payload")
    def process_payload(self, payload: Dict[str, Any]) -> Dict[str, Any]:
        if not self.is_active:
            raise ProductionServiceException("Processor is deactivated.")
        if "id" not in payload:
            raise ValueError("Payload missing mandatory key: 'id'")
        
        # Simulating domain-specific calculations
        processed_data = {**payload, "status": "processed", "timestamp": time.time()}
        return processed_data

# Example Usage
if __name__ == "__main__":
    pipeline = DataPipelineProcessor(config={"mode": "production"})
    try:
        pipeline.process_payload({"id": "evt_10928a", "value": 42.0})
    except ProductionServiceException:
        pass

Production Trade-offs & Implementation Decisions

Deploying this solution in production environments requires a careful analysis of the trade-offs involved. For instance, focusing purely on consistency (such as ACID compliance) can limit network throughput and horizontal scalability. On the other hand, adopting an eventual consistency model can lead to dirty reads and requires complex conflict resolution strategies in the application layer.

At MirahLabs, our engineering teams balance these architectural constraints by separating critical transaction paths from analytics workloads. We apply message-driven architectures with idempotent consumer systems to guarantee that network failures or retries do not result in double processing or state contamination.

Real-World Benchmarks & Resource Planning

Below is a typical performance comparison profile compiled by our engineering team in staging environments under simulated loads (10k concurrent virtual users):

When capacity planning, we recommend scaling out horizontally using containerized workloads rather than vertically upgrading underlying instance models. This maximizes uptime and provides cost efficiency through dynamic scaling policies.

Security Considerations & Vulnerability Mitigations

No production blueprint is complete without addressing security. Ensure that all data paths utilize encryption in transit (TLS 1.3) and at rest (using AES-256). Furthermore, implement strict Role-Based Access Control (RBAC) to limit operations. For APIs, always enforce rate limits (e.g. using token bucket algorithms in Redis) and run continuous static application security testing (SAST) in your CI pipeline.

How MirahLabs Applies This in Practice

Our experience building high-volume solutions like MirahCare.ai and Ayurveda.ai has taught us that early optimization is often a trap, but ignoring structural security and data design early leads to fatal development blocks. We design all client products from day one to support modular extensions, robust query indexing, and standard schema definitions, ensuring rapid iteration without technical debt growth.