Observability
Telemetry, root cause and incident response
Real-time observability dashboards and telemetry analysis across production systems — where I learned that a system you cannot measure is a system you cannot improve.
- Where
- Infosys
- When
- 2021 — 2023
- Role
- Senior System Engineer
- Stack
- Azure Monitor · Log Analytics · Application Insights · Power BI · Power Apps
Problem
Production incidents were being diagnosed by people reading logs after the fact. The time-to-understanding, not the time-to-fix, was the real cost — and nobody could point at a number that said whether last quarter was better than this one.
Constraints
- 01Enterprise environment — instrumentation had to fit the existing Azure estate
- 02Non-engineers needed to read the same dashboards as engineers
- 03Improvements had to be demonstrable against a baseline, not asserted
- 04Manual workflows were the incumbent, and they mostly worked
Architecture
Key decisions & trade-offs
One source of truth, two audiences
Engineers and stakeholders were reading different numbers from different places, which meant arguments about the data rather than about the system. Building the Power BI layer directly on the same telemetry the engineers debugged from ended that.
Trade-offDashboards intended for non-engineers constrain how raw the underlying metrics can be, and there is ongoing work keeping one view legible to both. Worth it to stop relitigating the numbers.
Root cause analysis before more alerting
The instinct when incidents are slow is to add alerts. Going the other way — investing in diagnosing why downtime happened — cut downtime 25%, which more alerting on the same unexamined failures would not have.
Trade-offRoot cause work is slow, unglamorous and hard to show progress on week to week, compared to shipping a dashboard. The compounding is much better.