Centralized logging collects logs from all services into a single searchable system — replacing the chaos of SSH-ing into individual servers with structured, queryable, correlated log data that powers incident response and root cause analysis.
Centralized logging collects logs from all services into a single searchable system — replacing the chaos of SSH-ing into individual servers with structured, queryable, correlated log data that powers incident response and root cause analysis.
Lesson outline
Without centralized logging, investigating a production incident means SSH-ing into each server and grepping through log files. For a single service on one server, this is inconvenient. For a microservices architecture with dozens of services across hundreds of instances, it is impossible.
What "Flying Blind" Looks Like
At 3 AM, your monitoring alerts that 20% of API requests are failing. Without centralized logging: you SSH into Server 1 (nothing obvious), SSH into Server 2 (different error message), try to figure out if the errors are correlated, check if it is the same user, realize you need to grep 8 different servers and correlate timestamps manually. By the time you find the root cause, an hour has passed. With centralized logging: one query in Kibana or Grafana, filter by error code, group by service, correlate the trace ID. Root cause in 5 minutes.
Centralized logging ships log output from every service and instance to a single system where it can be searched, filtered, and correlated. The key components are: log shippers (Fluentd, Fluent Bit, Logstash, Vector) that collect and forward logs, and storage/query backends (Elasticsearch, Loki, CloudWatch, Datadog) that index and serve them.
What you gain from centralized logging
Log shippers are the agents that collect logs from their sources (application stdout, log files, system journals) and forward them to your storage backend. Choosing the right shipper affects resource consumption, pipeline complexity, and reliability.
| Shipper | Resource usage | Best for | Deployment model |
|---|---|---|---|
| Fluent Bit | Very low (~45MB RAM) | Kubernetes pods, edge devices, resource-constrained environments | DaemonSet in Kubernetes; sidecar for specific pods |
| Fluentd | Moderate (~40-300MB RAM) | Complex log routing, transformation, fan-out to multiple destinations | DaemonSet or standalone agent; aggregator layer |
| Logstash | High (~500MB+ RAM) | Complex ETL pipelines with Elasticsearch; filter and transform heavy workloads | Standalone service; often behind a message queue (Kafka) |
| Vector | Low (~30MB RAM) | High-performance log processing; modern alternative to Fluentd/Logstash | DaemonSet or standalone; Rust-based, very fast |
| CloudWatch Agent | Low | AWS workloads; native CloudWatch Logs integration | EC2 agent; ECS sidecar; Lambda auto-instrumentation |
Use Fluent Bit in Kubernetes
In Kubernetes, deploy Fluent Bit as a DaemonSet (one pod per node). It reads container logs from /var/log/containers/, adds Kubernetes metadata (namespace, pod name, labels), and forwards to your backend. It is lightweight enough to run on every node without impacting application performance. Fluentd can serve as an aggregator layer if you need complex transformation before the backend.
Common Log Shipping Architecture in Kubernetes
Application → stdout/stderr → container runtime → /var/log/containers/ Fluent Bit DaemonSet reads /var/log/containers/ Fluent Bit adds K8s metadata (pod, namespace, labels) Fluent Bit forwards to Loki / Elasticsearch / CloudWatch Grafana / Kibana / CloudWatch Logs Insights queries the backend
Unstructured logs are plain text: "2024-01-15 14:23:01 ERROR PaymentService failed to process order 12345 for user john@example.com". They are human-readable but machine-unfriendly — querying requires regex, filtering is slow, and extending the schema requires changing grep patterns across all your monitoring.
Log in JSON — Always
Structured logs (JSON) make every field a first-class query target: {"timestamp": "2024-01-15T14:23:01Z", "level": "error", "service": "payments", "message": "payment processing failed", "order_id": "12345", "user_id": "usr_456", "error_code": "INSUFFICIENT_FUNDS", "duration_ms": 234, "trace_id": "abc-def-789"} Now you can: filter by error_code, aggregate by service, correlate by trace_id, alert on level:error AND service:payments, build dashboards on duration_ms percentiles. None of this requires parsing text.
Essential fields for every log line
Never Log PII or Secrets
Logs are often retained for 30-90 days and accessible to many engineers. Logging passwords, credit card numbers, SSNs, or full JWT tokens creates a compliance and security liability. Log user IDs and order IDs, not personal data. Use token scrubbing middleware if your framework does not handle this automatically.
Choosing a log aggregation backend is a decision that will be hard to reverse — log schemas, query patterns, and alerting all depend on the backend. The four dominant options each have different trade-offs.
| System | Index model | Query language | Cost model | Best for |
|---|---|---|---|---|
| ELK Stack (Elasticsearch + Logstash + Kibana) | Full-text indexes all fields | Lucene/KQL — powerful | High storage cost; self-managed is complex | Teams needing full-text search, complex aggregations, existing Elastic expertise |
| Grafana Loki | Labels only (like Prometheus for logs) | LogQL — similar to PromQL | Very low storage cost (only indexes labels, not full text) | Teams already using Grafana/Prometheus; cost-sensitive; Kubernetes-native |
| AWS CloudWatch Logs | Managed; full text | CloudWatch Logs Insights (SQL-like) | Per-GB ingestion + storage; expensive at scale | AWS-native teams; simple use cases; no infrastructure to manage |
| Datadog Logs | Fully managed; full text | Datadog query language | Most expensive at scale; $0.10/GB + retention costs | Teams wanting unified metrics + logs + traces + APM in one platform |
Loki is the Cost-Efficient Choice for Kubernetes
Loki stores logs as compressed chunks, indexing only labels (like Prometheus). Query performance is lower than Elasticsearch for ad-hoc full-text search, but for structured JSON logs where you filter by label (service=payments, level=error), it is extremely fast and dramatically cheaper. At 10GB/day ingestion, Loki on S3 costs ~$3/month in storage vs ~$50 for equivalent Elasticsearch.
Log storage costs scale linearly with log volume and retention period. A busy production system can generate gigabytes of logs per hour. Without cost management, log storage becomes one of the largest line items in your infrastructure bill.
Log retention best practices
CloudWatch Logs: Set Retention or Pay Forever
CloudWatch Logs default retention is "never expire." A single EC2 instance logging at moderate volume with 10 years of logs can cost $500+/month in storage. Set retention on every log group: 7 days for dev, 30 days for staging, 90 days for production. Do this from day one — it is painful to clean up years of accumulated logs later.
Log aggregation appears in observability system design questions, SRE interview scenarios about incident response, and DevOps maturity assessments. "How would you investigate a production issue?" almost always involves logging architecture.
Common questions:
Strong answer: Explaining trace IDs for cross-service correlation. Advocating for structured JSON logs with specific field examples. Having a cost management strategy (retention tiers, sampling, filtering). Knowing at least two log aggregation tools and their trade-offs.
Red flags: Describing SSH-ing into servers as a log investigation strategy. Not knowing what structured logging is. No mention of trace/correlation IDs. Not considering log retention costs. Treating logging as optional.
Quick check · Log Aggregation
1 / 3
Key takeaways
💡 Analogy
Centralized logging is like having a court reporter present in every meeting in your organization simultaneously. Without them, if you need to know what was said in a meeting from last Tuesday at 2 PM, you have to track down every attendee, hope they remember, and try to reconcile conflicting accounts. With the court reporter, you search the transcript system: "show me everything said about the budget in the last 30 days, sorted by speaker." Trace IDs are like case numbers that link all documents across different meetings related to the same matter.
⚡ Core Idea
Ship structured JSON logs from every service to a central system. Filter by trace ID to correlate a request across services. Filter by error level to find problems. All without SSH-ing into a single server.
🎯 Why It Matters
Centralized logging is table stakes for operating any system with more than one service or more than one server. The mean time to resolution (MTTR) in incidents is dominated by time to diagnose. With centralized structured logs and trace IDs, diagnosis drops from hours to minutes.
Ready to see how this works in the cloud?
Switch to Career Paths for structured paths (e.g. Developer, DevOps) and provider-specific lessons.
View role-based pathsSign in to track your progress and mark lessons complete.
Questions? Discuss in the community or start a thread below.
Join DiscordSign in to start or join a thread.