Answer:

• Monitoring: Focuses on known failure modes. It tells you when a system is broken by gathering predefined metrics (e.g., CPU > 90%, disk space full).
• Observability: Focuses on unknown failure modes. It provides the context and raw telemetry (metrics, logs, and distributed traces) to help you understand why a system is broken, especially in complex, distributed microservice architectures.

Answer:

1. Latency: The time it takes to service a request (e.g., HTTP request duration).
2. Traffic: A measure of how much demand is being placed on your system (e.g., HTTP requests per second, network I/O).
3. Errors: The rate of requests that fail (e.g., HTTP 5xx responses).
4. Saturation: How "full" your service is (e.g., database connection pool utilization, CPU run queue length).

Answer: The RED Method is designed for monitoring request-driven microservices:

• Rate: The number of requests per second.
• Errors: The number of failed requests.
• Duration: The time taken to process requests (latency).

Answer: The USE Method is designed for monitoring hardware resources (CPU, Memory, Storage):

• Utilization: The average time that the resource was busy (e.g., RAM usage %).
• Saturation: The volume of extra work queued because the resource is busy.
• Errors: The count of error events.

Answer:

• SLI (Service Level Indicator): A quantitative measure of service performance (e.g., "The latency of successful HTTP requests is < 200ms").
• SLO (Service Level Objective): The target reliability target defined for the SLI (e.g., "99% of HTTP requests will meet the SLI over a rolling 30-day window").
• SLA (Service Level Agreement): The legal/commercial agreement with users defining the consequences if the SLO is not met (e.g., refunding fees if uptime drops below 99.9%).

Answer: An Error Budget is the allowable downtime or failure rate of an application, calculated as: $$\text{Error Budget} = 100% - \text{SLO}$$ For example, a 99.9% availability SLO yields a 0.1% error budget. The team can spend this budget on shipping updates, running experiments, or executing maintenance. If the budget is depleted, deployments are halted, and the team redirects focus entirely to stabilization.

Answer: Alert Fatigue occurs when engineers are overwhelmed by a high volume of frequent, non-actionable, or low-priority alerts, leading to delayed responses or ignoring critical incidents.

Prevention:#

• Alert only on symptoms that impact customers directly (SLO-based alerting), not internal resource causes.
• Ensure every alert has a clear, documented runbook.
• Route non-critical alerts to email/tickets instead of waking up engineers.

Answer: Alertmanager is a Prometheus component that handles alerts sent by client applications or Prometheus servers. It deduplicates, groups, and routes alerts to the correct receiver integration (PagerDuty, Slack, Email) and handles silencing and inhibition rules.

Answer: Prometheus is a time-series database and monitoring tool:

• Pull Model: Prometheus actively scrapes metrics from client endpoints over HTTP at configured intervals.
• TSDB (Time Series Database): Stores metrics locally on disk or in remote storage.
• PromQL: A powerful query language used to extract time-series datasets.
• Alerting: Evaluates alert rules against active metrics and pushes triggers to Alertmanager.

Answer: Allows Prometheus to dynamically discover and scrape target endpoints (like EC2 instances or Kubernetes pods) as they scale, integrating with cloud APIs (AWS, GCP) or Kubernetes API servers, eliminating manual config modifications.

Answer: Exporters act as translation proxies. They collect metrics from systems that do not native export Prometheus format (e.g., MySQL, Redis, HAProxy) and translate them into standard Prometheus metrics format over HTTP.

Answer: Federation allows a Prometheus server to scrape selected metrics from other Prometheus servers, facilitating hierarchical scraping structures (e.g., local cluster-level servers aggregated into a global central monitoring server).

Answer: Remote Write allows Prometheus to forward its scraped metrics data to external long-term storage platforms (like Thanos, Cortex, or Grafana Mimir) for durable, historical metric preservation.

Answer: OpenTelemetry is a vendor-neutral, open-source observability framework under the CNCF that provides a standardized set of APIs, SDKs, and tooling to generate, collect, and export telemetry data (metrics, logs, and traces) to any back-end platform.

Answer: Jaeger is an open-source, distributed tracing system used for monitoring and troubleshooting transactions in complex microservice architectures. It visualizes trace dependency graphs and latency bottlenecks.

Answer: Distributed Tracing tracks the lifecycle of a single request as it propagates through multiple downstream microservices, databases, and message queues. It groups individual service tasks (spans) into a single, unified transaction (trace) showing execution durations and call paths.

Answer: A Correlation ID is a unique identifier attached to an incoming HTTP request at the API Gateway. As the request calls downstream services, it passes the Correlation ID in HTTP headers, allowing log aggregation tools to group logs from different services to reconstruct the entire execution flow.

Answer: The practice of gathering log files from thousands of container instances and servers, parsing them, and indexing them in a centralized database (like Elasticsearch, Loki, or Splunk) for structured searching, analytics, and alerting.

Answer: Continuous Monitoring is the practice of continuously collecting, analyzing, and alerting on real-time operational telemetry (metrics, logs, traces, system events) across the entire software development and deployment lifecycle.

Why it is Important:#

• Rapid Incident Detection: Detects application errors, performance bottlenecks, and infrastructure outages in real-time, reducing Mean Time to Detection (MTTD).
• Operational Visibility: Provides feedback loops to development teams regarding application performance and user patterns in production.
• Validates Deployments: Instantly shows if a newly deployed code version is causing CPU spikes or increased HTTP 5xx error rates, triggering automated rollbacks.
• Capacity Planning: Identifies long-term storage and compute trends, helping optimize infrastructure spend.

Answer: Monitoring a Kubernetes cluster requires gathering metrics and logs at three distinct layers:

1. Cluster/Node Layer (Infrastructure)#

• Node Resource Metrics: CPU, Memory, Disk, and Network utilization for each virtual machine host (gathered using Prometheus Node Exporter).
• Node Status: Checking if nodes are in a Ready or NotReady state.
• Kubelet Metrics: Monitoring node runtime performance and API heartbeats.

2. Kubernetes Control Plane (Control Layer)#

• API Server Health: Latency of API requests and request rate.
• Scheduler Latency: Time required to place pending pods onto nodes.
• etcd Performance: Disk write times and leader elections.

3. Pod/Application Layer (Workload Layer)#

• Container Metrics: CPU limits/throttling, memory limits (tracking OOMKilled events), and restarts (gathered via cAdvisor).
• State Metrics: Pod phase status (Running, Pending, Failed) and deployment replica status (using kube-state-metrics).
• Application Health: Scraping application-specific HTTP/gRPC /healthz endpoints.
• Log Aggregation: Gathering container stdout/stderr log streams using agents like Fluentbit or Promtail and indexing them in Grafana Loki or Elasticsearch.

Answer: Prometheus is an open-source system monitoring and alerting toolkit. It is the de-facto standard for Kubernetes monitoring due to its active scraping model and native support for dynamic container environments.

How it Works:#

1. Metric Scraping (Pull Model): Instead of applications pushing metrics, the Prometheus server actively scrapes metrics from configured HTTP endpoints (e.g., http://my-service:8080/metrics) at configured intervals (e.g., every 15 seconds).
2. Service Discovery: Prometheus queries the cloud provider (AWS/GCP) or Kubernetes API to dynamically discover target endpoints as containers scale out.
3. Data Storage: Metrics are stored locally in a highly optimized custom Time Series Database (TSDB).
4. Data Querying (PromQL): Users query the metrics database using Prometheus Query Language (PromQL) to generate dashboards in Grafana.
5. Alerting: Prometheus evaluates alert rules against metrics; if a rule matches (e.g., up == 0), it pushes an alert to Alertmanager for routing and notifications.

Answer: Logging in a distributed microservice system requires centralizing logs to avoid logging into individual servers. The architecture is typically implemented using the PLG (Promtail-Loki-Grafana) or EFK (Elasticsearch-Fluentd-Kibana) stack:

1. Structured JSON Logs: Enforce applications to write logs in structured JSON format (containing timestamp, severity level, service name, message, trace/correlation IDs, and request metadata).
2. Correlation IDs (Tracing): Inject a unique Correlation ID (or traceparent header) at the API Gateway. Pass this ID to every downstream service in HTTP/gRPC request headers so logs from different services can be grouped and queried together.
3. Log Forwarding (Collectors): Run a lightweight log collector agent (e.g., Fluent Bit, Promtail, or Logstash) on every node/host. These agents watch container log files (e.g., /var/log/containers/*), append Kubernetes labels, and forward them.
4. Log Indexing & Storage (Aggregator): Forward logs to a central indexing database (e.g., Grafana Loki or Elasticsearch).
5. Visualization & Querying: Use a dashboard (e.g., Grafana or Kibana) to query and analyze log streams during troubleshooting.
6. Retention Policies: Define log rotation and lifecycle policies (e.g., keeping logs in hot index for 7 days, then archiving to cheap cold storage like S3 for 90 days before deletion).

Answer: To measure the speed, efficiency, and reliability of your CI/CD pipelines, monitor the following key metrics:

• Build Success Rate: The percentage of pipeline runs that complete successfully without errors. A low success rate indicates flaky tests or unstable build configurations.
• Pipeline Cycle Time (Duration): The average time taken for a commit to traverse from the initial push to successful deployment. Faster pipelines improve developer velocity.
• Queue Time: The time a job spends waiting for an available build runner. High queue times indicate runner resource constraints.
• Deployment Frequency: How often the team successfully deploys code to production.
• Change Failure Rate: The percentage of production deployments that result in service degradation, errors, or require immediate hotfixes/rollbacks.
• Mean Time to Recovery (MTTR): The average time required to restore service or deploy a fix after a pipeline failure or production incident occurs.