The eight fallacies — and what actually happens

• 1. The network is reliable — Reality: packets get dropped, connections time out, routers fail. Every network call can fail. Design for it with retries, timeouts, and circuit breakers.
• 2. Latency is zero — Reality: cross-AZ = ~1ms, cross-region = 30–150ms. N+1 queries that take 1ms locally take 150ms per call cross-region. Batch calls and cache aggressively.
• 3. Bandwidth is infinite — Reality: data transfer costs money and has throughput limits. Sending 1MB per API response at 10k RPS = 10GB/s bandwidth. Paginate, compress, and cache.
• 4. The network is secure — Reality: traffic can be intercepted, DNS can be spoofed. Encrypt everything in transit with TLS. Authenticate every service-to-service call.
• 5. Topology does not change — Reality: servers are added/removed constantly (autoscaling), IPs change, services move. Use service discovery (DNS, Consul, Kubernetes services), not hardcoded IPs.
• 6. There is one administrator — Reality: distributed systems span teams, vendors, and services. Any component can be changed independently. Plan for interface contracts and backward compatibility.
• 7. Transport cost is zero — Reality: serialization/deserialization, network round-trips, and protocol overhead all add latency. gRPC + Protobuf is 5–10× faster than REST + JSON for internal services.
• 8. The network is homogeneous — Reality: different services use different protocols, encodings, and retry policies. Use an API gateway or service mesh to normalize communication patterns.

The three replication strategies and when to use each

• Single-leader (master-replica) — One node accepts all writes and replicates to read replicas. Simple, well-understood. Read scale: excellent (route reads to replicas). Write scale: limited by leader throughput. Used by: PostgreSQL streaming replication, MySQL binary log replication, DynamoDB global tables (within a region). Good for: read-heavy workloads, strong consistency for writes.
• Multi-leader (active-active) — Multiple nodes accept writes. Conflict resolution required. Complex. Used for: multi-region active-active (each region writes locally, syncs globally). Good for: low-write-latency globally. Hard problem: what happens when two users update the same row in different regions simultaneously? Requires conflict resolution (last-write-wins, CRDTs, or application-level merge).
• Leaderless (Dynamo-style) — Any node accepts writes. Quorum-based reads and writes (write to W of N nodes, read from R of N nodes, where W + R > N guarantees overlap). Used by: Cassandra, Amazon DynamoDB, Riak. Good for: high availability, no single point of failure, geographic distribution. Tunable consistency: lower W and R = higher availability but more staleness.

Patterns for resilient distributed communication

• Circuit breaker — Three states: Closed (requests pass through, track error rate), Open (fast-fail all requests without calling downstream — gives it time to recover), Half-Open (let one probe request through, if it succeeds, close the circuit). Libraries: Resilience4j (Java), polly (.NET), opossum (Node.js), Hystrix (Java, Netflix).
• Exponential backoff with jitter — First retry after 1s, then 2s, 4s, 8s... with random jitter (±50%) to prevent thundering herds. Never retry with fixed intervals — 1,000 clients all retrying at t=1s creates a synchronized spike.
• Bulkhead isolation — Separate thread pools (or connection pools) per downstream service. If Service A becomes slow and exhausts its connection pool, it does not affect traffic to Service B. Named after watertight compartments in ships — one flooded compartment does not sink the ship.
• Timeout (always set one) — Every network call must have an explicit timeout. "No timeout" means a slow downstream can hold your thread/connection forever. Rule: timeout < downstream SLA. If the payment service SLA is 2s, your timeout should be 2.5s (with buffer).