Why Do Nightly Checks and Cleanup So Often Break After Shift Handover?
On the first morning of month-end, the most unsettling sentence in the operations channel is usually not, “Did anything alert last night?” It is this one:
“Who actually ran that nightly inspection round, and who can clearly explain the result now?”
The main character here is Lao Zhao, a platform operations engineer. Before the handover the previous night, he had already posted a reminder in the chat: run one round of disk inspection overnight, clean old logs on several business servers, and check the status of a few critical services. Right after that, a new alert came in. Once an emergency troubleshooting task cut in, this round of work that everyone thought was “easy” and “something we can do in a moment” kept getting pushed back.
By the next day, what really turned the scene upside down was not that nobody knew how to write the commands, nor that the scripts did not exist at all. It was that suddenly nobody could explain the whole round of actions from start to finish in one pass.
Who actually took over and ran it? Which batch of machines did last night’s inspection and cleanup really hit? After it ran, did it finish normally, or had some nodes already failed in the middle?The channel is not quiet. One person says, “I think I may have run that last night.” Another says, “The cleanup probably ran, we just never replied with the result.” But the more the scene sounds like everyone did part of it, the easier it is for the whole thing to drag on. Because very quickly, people stop arguing about “whether they know how to do it” and start arguing about “whether that round of work was actually carried through completely”.
Many teams first realize that routine server maintenance can spin out of control not when the script cannot be written, but at exactly this moment, when the action obviously should have happened and yet nobody can confirm the result.