System Design for Backend Engineers: A Practical Guide

Most system design decisions don't matter until they suddenly do — the day traffic doubles and something that worked fine at 10 users falls over. These are the building blocks I actually reach for to avoid that rewrite.

Horizontal vs. Vertical Scaling

• Vertical scaling — add more CPU and RAM to a single server. Simple but has a ceiling.
• Horizontal scaling — add more servers behind a load balancer. Unlimited ceiling but requires stateless application design.

The rule: design for horizontal scaling from day one. Store sessions in Redis, not in process memory. Store uploads in S3, not on local disk. This makes every instance interchangeable.

Load Balancing

A load balancer distributes incoming traffic across multiple application instances. Common algorithms:

In production, I use AWS Application Load Balancer (ALB) with health checks:

Target Group Health Check:
  Path:     /api/health
  Interval: 30 seconds
  Timeout:  5 seconds
  Healthy:  2 consecutive successes
  Unhealthy: 3 consecutive failures

The health endpoint is trivial but critical:

app.get('/api/health', (req, res) => {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
});

Database Replication

A single database server is a single point of failure. Use primary-replica replication:

• Primary (Master) — handles all writes.
• Replicas (Read Replicas) — handle read queries, reducing primary load.

       Writes
Client ───────> Primary DB
                  │
              Replication
              ┌───┴───┐
           Replica 1  Replica 2
              │          │
           Reads       Reads

In Node.js with Mongoose, route reads to replicas:

await mongoose.connect(MONGODB_URI, {
  readPreference: 'secondaryPreferred',
});

For critical reads that require the latest data, override per-query:

const user = await User.findById(userId).read('primary');

Caching Layers

Caching reduces database load and speeds up responses. I use a multi-layer caching strategy:

1. Application-level cache — in-process cache (e.g., node-cache) for config and rarely-changing data.
2. Distributed cache — Redis for data shared across instances (sessions, computed results).
3. CDN cache — Cloudflare or CloudFront for static assets and API responses with proper Cache-Control headers.

// Application-level cache for config that rarely changes
import NodeCache from 'node-cache';
const appCache = new NodeCache({ stdTTL: 300 }); // 5 minutes

function getFeatureFlags(): FeatureFlags {
  const cached = appCache.get<FeatureFlags>('featureFlags');
  if (cached) return cached;

  const flags = loadFeatureFlagsFromDB();
  appCache.set('featureFlags', flags);
  return flags;
}

Message Queues for Async Processing

Not every operation needs to complete before the API responds. Offload heavy work to background queues:

Client ──> API ──> Response (immediate)
                │
                └──> Message Queue ──> Worker
                                        │
                                    Heavy Work
                                  (emails, PDFs,
                                   analytics, etc.)

This pattern keeps P99 latencies low because the API only does the minimum synchronous work.

Rate Limiting

Protect your APIs from abuse with layered rate limiting:

• Global rate limit — 1000 requests/minute per IP.
• Endpoint-specific limits — 5 login attempts per minute per IP.
• User-specific limits — 100 API calls per minute per API key.

import rateLimit from 'express-rate-limit';

const loginLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 5,
  message: { error: 'Too many login attempts. Try again in 1 minute.' },
});

router.post('/auth/login', loginLimiter, authController.login);

Circuit Breaker Pattern

When a downstream service fails, stop sending it traffic. The circuit breaker pattern prevents cascade failures:

• Closed — requests flow normally.
• Open — requests are rejected immediately (fast fail).
• Half-Open — a limited number of test requests are allowed through.

// Simplified circuit breaker
class CircuitBreaker {
  private failures = 0;
  private readonly threshold = 5;
  private state: 'closed' | 'open' | 'half-open' = 'closed';
  private nextAttempt = 0;

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() < this.nextAttempt) throw new Error('Circuit is open');
      this.state = 'half-open';
    }
    try {
      const result = await fn();
      this.reset();
      return result;
    } catch (err) {
      this.recordFailure();
      throw err;
    }
  }

  private recordFailure() {
    this.failures++;
    if (this.failures >= this.threshold) {
      this.state = 'open';
      this.nextAttempt = Date.now() + 30000; // 30 second cooldown
    }
  }

  private reset() {
    this.failures = 0;
    this.state = 'closed';
  }
}

Wrapping Up

If there's one habit worth building early, it's designing stateless from day one — keep session and upload state out of process memory, and everything else here (load balancing, replicas, queues) gets easier to add later.