Throwback Thursday: Well, trial and error IS a mechanism
Credit to Author: Sharky| Date: Thu, 06 Sep 2018 03:00:00 -0700
New regulations go into effect requiring more physical and electronic security at this health insurance company, so the company hires a chief security officer to oversee the efforts, says a pilot fish there.
“I was involved in the original security implementation on most of the systems and offered to help, but the new CSO refused our input,” fish says. “He put keycard access on the computer room and UPS room and confiscated any physical keys he could find.
“When asked what would happen if the keycard system went down, he responded that ‘mechanisms are in place,'” fish recalls.
Soon, only three people have physical keys: the CSO, chief financial officer and facilities manager.
Fast forward two weeks: A large thunderstorm at 1 a.m. Sunday morning knocks out power to the building.
Automated alerts notify the systems administrators, who arrive promptly but can’t get in through the new keycard-access system. In fact, the card reader doesn’t appear to have power.
Phone calls to the CSO, CFO and facilities manager go unanswered. Without access to the building, sysadmins can’t gracefully power down the systems, so a few hours later, everything in the computer room goes down hard.
It’s not until Sunday night that the facilities manager returns from his hunting weekend and returns the sysadmins’ calls. He meets the sysadmins at the door and lets them in.
But something is odd. Building power has been restored, but keycard access still isn’t working. Doors have to be propped open so the sysadmins can get the systems running again and clean up the mess from the hard crash in the middle of the updates and backup jobs.
Turns out that the security system’s battery was never hooked up. It wasn’t on the UPS, either. And it hadn’t been backed up when it lost its configuration after the power went out.
So on Monday, when the CSO shows up, the first thing he does is to set the security doors to be always unlocked until he can rebuild the keycard-access configuration.
Then he reports to the CEO that the mechanisms in place worked and that the system is back up for Monday morning’s business despite the emergency over the weekend.
“I guess ‘mechanisms in place’ was short for ‘I don’t have a clue, but someone else will take care of it for me,'” grumbles fish.
“Within a week, the keycard system was hooked up to the UPS with a secondary backup battery and was tested repeatedly. And the sysadmins were allowed keys ‘in case of emergency.'”
In case of a true tale of IT life, you know where to send it: sharky@computerworld.com. You can also comment on today’s tale at Sharky’s Google+ community, and read thousands of great old tales in the Sharkives.
Get Sharky’s outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.