How would you design and implement a scalable logging architecture for a distributed system?

Logging in a distributed microservice system requires centralizing logs to avoid logging into individual servers. The architecture is typically implemented using the PLG (Promtail-Loki-Grafana) or EFK (Elasticsearch-Fluentd-Kibana) stack:

1. Structured JSON Logs: Enforce applications to write logs in structured JSON format (containing timestamp, severity level, service name, message, trace/correlation IDs, and request metadata).
2. Correlation IDs (Tracing): Inject a unique Correlation ID (or traceparent header) at the API Gateway. Pass this ID to every downstream service in HTTP/gRPC request headers so logs from different services can be grouped and queried together.
3. Log Forwarding (Collectors): Run a lightweight log collector agent (e.g., Fluent Bit, Promtail, or Logstash) on every node/host. These agents watch container log files (e.g., /var/log/containers/*), append Kubernetes labels, and forward them.
4. Log Indexing & Storage (Aggregator): Forward logs to a central indexing database (e.g., Grafana Loki or Elasticsearch).
5. Visualization & Querying: Use a dashboard (e.g., Grafana or Kibana) to query and analyze log streams during troubleshooting.
6. Retention Policies: Define log rotation and lifecycle policies (e.g., keeping logs in hot index for 7 days, then archiving to cheap cold storage like S3 for 90 days before deletion).