Answer:

1. Identify Bottleneck Process: Run top or htop to identify the processes consuming the most CPU.
2. Investigate Thread Details: Press H in top to view individual threads.
3. Trace System Calls: Run strace -p <PID> to see system calls made by the process.
4. Identify Core Source: Match the process ID against logs. If it is Java, run jstack <PID> to dump threads and identify code bottlenecks.
5. Mitigate: If the process is not critical, kill it (kill -9 <PID>). Otherwise, scale up or restart the service.

Answer: A memory leak occurs when an application allocates memory but fails to release it, causing RAM consumption to increase over time.

1. Monitor usage: Track memory trends using free -m and look for steady upward curves.
2. Trace OOM events: Check kernel logs for Out-Of-Memory termination signals: dmesg -T | grep -i oom or grep -i kill /var/log/messages.
3. Profile Heap Memory:
   • For Node.js: Generate a heap snapshot using --inspect flag.
   • For Java: Take a heap dump using jmap -dump:format=b,file=heap.bin <PID> and analyze it using Eclipse Memory Analyzer (MAT).

Answer:

1. Find largest files: Run du -sh /* 2>/dev/null | sort -rh to isolate directories.
2. Safely Truncate Active Logs: If a log file is held open by a process, deleting it with rm will not release disk space. Instead, truncate the file: > /var/log/nginx/access.log.
3. Purge Cache: Run package cleanups (e.g., apt-get clean or docker system prune -af).
4. Configure Logrotate: Ensure long-term rotation is set up.

Answer:

• Symptoms: Application returns 500 Internal Server Error with Too many connections messages.
• Immediate Fix:
  • Connect to DB as admin and terminate idle, hung connections.
  • Temporarily increase maximum connection parameters (e.g., max_connections in Postgres/MySQL).
• Permanent Fix:
  • Implement connection pooling in the application layer (e.g., HikariCP, PgBouncer).
  • Scale database instance size or route read traffic to read-replicas.

Answer:

• Run kubectl logs <pod-name> to check stdout/stderr outputs.
• Check previous failed instance logs: kubectl logs <pod-name> --previous.
• Run kubectl describe pod <pod-name> and look at the Events section to check for OOMKilled errors, failed probes, or volume mount errors.
• Check if configuration maps or secrets referenced in the pod spec actually exist in the namespace.

Answer: A NotReady status means the node is not responding to control plane heartbeats.

1. SSH into Node: Log in directly to the affected node.
2. Check Kubelet daemon: Run systemctl status kubelet to see if the agent is running.
3. Check system resources: Verify if the node is experiencing disk pressure, memory pressure, or PID exhaustion using df -h and free -m.
4. Audit Logs: Inspect kubelet logs: journalctl -u kubelet -n 100 --no-pager.
5. Check Docker/Containerd: Ensure the container runtime is active (systemctl status containerd).

Answer: A Pending status means the scheduler cannot place the pod on any node.

• Run kubectl describe pod <pod-name> and read the latest events.
• Common Causes:
  • Insufficient Resources: Nodes do not have enough unallocated CPU/RAM requests to satisfy the pod's requests.
  • Taints & Tolerations: The pod does not tolerate node taints.
  • Unattached Volumes: The pod is requesting a Persistent Volume (PV) that is located in a different Availability Zone.

Answer:

1. Analyze HTTP Metrics: Check latency durations at the load balancer (ALB) and separate connection time from target response time.
2. Distributed Tracing: Query Jaeger or AWS X-Ray using the request Correlation ID to trace spans across microservices.
3. Database query check: Check for slow database queries in DB engine logs.
4. Resource Bottlenecks: Verify if downstream containers are experiencing CPU throttling or memory swapping.

Answer:

1. Verify resolve configurations: cat /etc/resolv.conf.
2. Test using direct nameservers: dig google.com @8.8.8.8 (isolates local DNS server issues from network routing issues).
3. Check if local DNS service is running (e.g., systemd-resolved, CoreDNS in K8s).
4. Verify security group rules permit outbound UDP port 53 traffic.

Answer:

1. Verify the backend application port matches the health check target.
2. Run curl -Iv http://localhost:<port>/healthz directly on the server to check if the health check endpoint returns 200 OK.
3. Audit security groups to ensure the load balancer IP is allowed to access the backend instance port.
4. Check application log files for runtime startup errors.

Answer:

1. Retrieve backup: Remote backends like S3 keep history/versioning. Locate the previous version of the state file.
2. Force state unlock: If the state is locked by a failed run, run terraform force-unlock <lock-id>.
3. Run state commands: Use terraform state pull > state.json to inspect the JSON schema and correct entries manually, then restore using terraform state push.

Answer:

1. Review build runner logs.
2. Isolate the failed step (e.g., dependency check, unit tests, code linting).
3. Check runner resources (e.g., runner container running out of disk space).
4. Verify access secrets/credentials used in the pipeline are active.

Answer: A 502 means the reverse proxy (Nginx, Ingress) received an invalid response from the backend application.

1. Check if the backend process is running: ps aux or systemctl status.
2. Verify the backend is listening on the expected port: ss -tulnp.
3. Review proxy error log files (/var/log/nginx/error.log).
4. Confirm backend host name resolves correctly within the internal network.

Answer: A 504 means the proxy gateway waited too long for the upstream application server to return data.

1. Audit database lock tables or slow queries that might halt request processing.
2. Check for network packet loss or routing issues between proxy and backend.
3. Increase proxy connection timeouts (e.g., Nginx proxy_read_timeout parameters).
4. Verify external dependency APIs are not hanging.

Answer:

1. Renew the certificate immediately from CA (e.g., Let's Encrypt, DigiCert).
2. Deploy the new .crt and .key files to the target load balancer, CDN, or web server.
3. Reload the web server daemon configuration: systemctl reload nginx.
4. Validate using openssl s_client -connect domain.com:443 | grep -i expiry.

Answer:

• GitOps: Revert the bad commit in Git (git revert <hash>) and push to main, triggering auto-sync.
• Kubernetes: Run kubectl rollout undo deployment/<deployment-name> to revert to the previous ReplicaSet snapshot.
• Cloud (Blue-Green): Toggle the ALB listener rules to point 100% of user traffic back to the green target group.

Answer:

1. Isolate database: Revoke write permissions to block further corruptions.
2. Identify Point-in-Time: Find the exact timestamp before corruption occurred.
3. Restore Backup: Initiate AWS RDS Point-in-Time Recovery (PITR) to restore a backup instance to the selected timestamp.
4. Verify Integrity: Run integrity checks.
5. DNS Switch: Swap connections to point applications to the restored database instance.

Answer:

1. Identify the region loss (e.g., AWS us-east-1 outage).
2. Trigger failover routing via Route 53 DNS (or let it failover automatically via health check latency routing).
3. Verify the hot/warm standby region cluster has scaled up.
4. Promote RDS cross-region read-replicas to primary database instances.
5. Validate application health.

Answer:

1. Isolation: Terminate network connectivity on affected systems immediately.
2. Audit: Identify compromised databases and access keys.
3. Rotate credentials: Update all root and service account passwords.
4. Re-provision: Build clean infrastructure using Terraform.
5. Restore: Mount clean data backups from offline, immutable, or write-once-read-many (WORM) storage.

Answer:

1. Revoke immediately: Treat the secret as compromised. Revoke the API key/password in the target system.
2. Generate new secret: Create new credentials and update application secret managers.
3. Purge Git history: Use Git-filter-repo or BFG Repo-Cleaner to delete the file from local repository history and push-force.
4. Review Audit Logs: Audit system access logs (e.g., CloudTrail) to verify if the leaked key was abused.

Answer: Database schema migrations present a high risk of application downtime during CD pipeline runs. The process requires careful coordination to maintain compatibility:

1. Enforce Backward Compatibility: Design schema changes so the old application version can run on the new schema. (e.g., if adding a new column, configure it to accept null values initially).
2. Use Migration Tools: Manage schema versions using migration tools like Flyway or Liquibase to run version-controlled, repeatable SQL scripts.
3. Execute in Phased Stages:
   • Phase 1 (Pre-deployment): Run migrations to add new tables, columns, or indexes. Old application containers continue to read from and write to the database.
   • Phase 2 (Deployment): Deploy the new application version that utilizes the new schema columns.
   • Phase 3 (Post-deployment): Once the new release is verified, run a cleanup script to drop or alter deprecated columns or tables.
4. Idempotency: Enforce migration scripts to be idempotent (able to run multiple times without causing errors or data corruption).
5. Scheduled Backups: Always run automated database backups immediately before executing migration scripts to facilitate quick recovery.