Mayday Mayday

She, Amanda, Sr. Administrator, walks into the server room and notices that the report server is down. She has not even put her purse up or sat at her desk. Her first order of business was to troubleshoot the major fire. Which had shut down the report server over the weekend.

The error message log is visible on her desk. Ever dismayed she perused the reports, trying to make sense of them. The switch error is labeled as “Mayday Mayday”. Wherever it shows that name it means the report server is crashed.

There is a backup server: “report_can_2”. As the backup server attempts to run, it notifies the administrator of a warning, and the processor is set to run the backup for 29 reports. If the problem is not fixed the server crashes and renders error_msg: Mayday Mayday.

She goes back through her logs to see why she didn't catch this before leaving on Friday at COB (close of business). There was one report that kept running, yet not clear why. In the midst of her troubleshooting, she gets pulled away to put out other fires and bring them to containment.

Because she’s been in hyperdrive looking for a resolution. She can't keep her eyes solely on the main fire. Which caused a gaping hole in their Usability Reporting Infrastructure. If she just had more hands to troubleshoot, was her thought.

It's now the middle of the day and she's still not clear why she's getting the error message “Mayday Mayday”. One thing she begins to see is the repetition of the warning logs for the backup server. It shows the reporting cap of 29 then it shuts off. This setting was too low, for the whole server room as well as the entire campus that it supports.

The administrator is starting to come unglued and frantic, with all of the incidents. At least by increasing the reporting cap to 111 the bandwidth is increased. It's a strategic configuration for supporting the entire campus. In looking through her reports she concluded that there has to be another threshold setting added into the code. Ensuring coverage over the weekend, so the backup server would not shut down, going forward. That's still not a remedy for the root problem, but a good catch.

However, it is an assurance that the backup will not shut down in the future. Since she was slammed from the time she walked in, her boss wanted Amanda to know she appreciated her attention to detail. Especially since multiple fires necessitated her focus. Her boss wanted her to be forewarned, she needed her fully engaged and expected a full resolution by EOB (end of business).

To ensure the priority was on high alert and to keep her attention, she increased the ticket level to III. As she continued to look at the main problem, it seemed like there was no resolution in sight. The more she looked, the more it was still a problem. The server was seemingly tied in knots, as though there was a server demon that prohibited her from figuring it out.

At this point, she had rubbed her eyes to the point of strain. Amanda, decides to put someone else's lens on it. Teddy, was the chosen one. To her chagrin, he looks at it for all of five minutes. His finding was entirely different. He spots the coding error and if that wasn't enough. He also determines who made the coding error.

This was a find that spoke volumes. It also warranted a mercy dinner, owed to Teddy. That was an unspoken rule in their department. To buy lunch or dinner for anyone, who worked out a long-standing problem. Before he said anything he took a mental note, to tread lightly. After all, she was definitely his boss. He says, boss I have figured it out. Amanda looks back at him, you figured out what? I know what caused the reporting server shut down. He begins to stutter, “ I, I, I know what the problem is”. She was quiet and reserved in her reply. Ok, what is it? He was hesitant because her boss was standing in the midst. Teddy says, can we step outside? Amanda says sure!

All these things were floating through her mind. She’s thinking to herself, I know I did this, I did that, it couldn't be me? While the two of them are standing in the hallway. He talks in a low voice, so she wouldn't be embarrassed. After he gets it all out, she is stunned. Forty seconds go by and she is still speechless.

The two proceed back into the office. Amanda first apologizes to Teddy. Then she asked him to explain to everyone, what his findings were. She is still in her mind trying to figure out how she made such a rookie mistake. As she mulled over the weight of what she had done. She could hear Teddy explain and it made her feel like an ill repute. You see, that's what she called him once or twice before, when he made a similar mistake.

As she pondered this colossal error. She began to understand why no one caught this or maybe even any other errors she had made previously. Who, would have called her out? No one, it had to be Teddy since she had spoken worst to him, for the same reason. The moral of this story, although there are several. The most obvious one is that it's prudent to have dual sets of eyes on all projects, before implementation. Indeed it may be time-consuming, but it will be worth it in the end.