Dhan V Sagar

Distributed Infrastructure & Telemetry at Scale

• Designed and deployed Grafana Tempo distributed tracing across 80+ microservices, enabling end-to-end request stitching and reducing Mean Time to Resolution (MTTR).

• Architected and scaled a multi-cluster Grafana Mimir infrastructure to ingest and process 500M+ active series, ensuring high availability and low-latency cross-cluster querying.

• Engineered a hybrid, multi-tenant logging ecosystem utilizing Loki, OpenSearch, Elasticsearch, and Splunk, standardizing log ingestion pipelines via Logstash and OpenTelemetry.

• Developed a Hub-and-Spoke telemetry architecture using custom-built OpenTelemetry Collectors to dynamically route high-volume signals to multiple destinations (SignalFx, Splunk HEC, Loki, Mimir).

• Deployed distributed Splunk clusters on Kubernetes using the Splunk Operator, configuring indexers, heavy forwarders, and search heads for enterprise-grade log and event analysis.

• Conducted a deep-dive evaluation and PoC for VictoriaMetrics, comparing performance, cardinality handling, and resource utilization against Mimir for high-throughput use cases.

Platform Engineering & Developer Self-Service

• Built a developer self-service portal using FastAPI and Next.js, empowering engineers to autonomously provision monitoring backends and customized OTel collectors, reducing onboarding from days to minutes.

• Developed a high-performance cardinality analysis tool in Go that consumed Kafka/MSK streams via Strimzi, decoded Avro/Influx Line Protocol payloads, and aggregated metrics to Splunk for real-time monitoring.

• Championed Dashboard and Alerting-as-Code (DaC/AaC) via GitOps workflows, standardizing Grafana visualizations and external Alertmanager routing topologies across the organization.

• Automated end-to-end CI/CD pipelines and infrastructure provisioning using Terraform, Kubernetes, Helm, ArgoCD, Drone.io, and Jenkins for automated image builds, chart publishing, and autoscaling.

Strategic Migrations & Data Architecture

• Spearheaded large-scale observability migration initiatives, moving multiple engineering teams and legacy workloads from expensive vendor platforms (Datadog, Circonus) to cost-effective LGTM and Splunk ecosystems.

• Designed a long-term data lake storage strategy using Apache Iceberg, AWS Glue, and S3 Tables, offloading historical observability data from expensive hot storage while preserving query capabilities.

• Authored comprehensive architectural RFCs and technical documentation establishing corporate standards for telemetry boundaries and instrumentation patterns.

Incident Response & AI-Ops Innovation

• Conducted a proof-of-concept integrating a Tempo MCP server with LLMs to enable natural-language trace exploration and accelerate root cause analysis during incidents.

• Developed a Retrieval-Augmented Generation (RAG) AI tool ingested with historical on-call playbooks, drastically lowering engineering cognitive load during critical incidents.

• Participated in a Tier-1 follow-the-sun on-call rotation, leading cross-functional Root Cause Analysis (RCA) investigations and implementing long-term incident prevention safeguards.

• Developed automated testing frameworks and containerized test environments using Docker and Kubernetes.

• Containerized multimedia applications using Docker and performed end-to-end validation on cloud-based systems.

• Automated performance analysis of video/audio traffic with Python scripting.

• Automated CI/CD pipelines using GitLab CI, Docker, and Python.

• Led network configuration and VPN integrations for production environments.

• Enhanced service reliability through functional and performance testing on Linux platforms.

• End to End integration and validation of IPTV/OTT Products

Graduate Coursework:
Machine Learning; Agent based intelligent systems; Statistics; Soft Computing, Computational theory.\

UnderGraduate Coursework:
Datastructures & Algorithms; Operating Systems; Computer Networks; Software Engineering Methodology.

• Best Paper Award IEEE-ICCIC (2014): Author of the research paper Titled: “Random Forest and Change Point Detection for Root Cause LocalizaUon in Large Scale Systems.”
• Research Paper in SPRINGER (2015): Co- author of the paper Titled, “ForecasUng the stability of Data Center based on Real Time data of batch workload using Time Series models.”