How Can On-Call Monitoring Spot Anomalies at a Glance Across 200 K8s Pods?
An Incident Response After Release
At 10:40 a.m. on release day, a business contact posts a message in the group chat: "Users are reporting slow responses. Can you check whether the backend is having problems?"
On-call engineer Xiao Zhou opens the monitoring page and sees the status of 200 Pods.
He only wants to do one thing: see which ones are unhealthy.
But those 200 rows of Pod status require scrolling through three, four, then five screens. What he is doing is no longer troubleshooting. It is "scrolling to find anomalies."
People in the group chat start asking, "Which service is slow?" Xiao Zhou returns to the monitoring page and continues scrolling to the sixth screen.
By the time he reaches the sixth screen, Xiao Zhou realizes that the problem is not "the list is bad." The problem is that the list is being used in the wrong place: a density scenario.