I believe at some point in every sysadmins career, they all eventually inherit what I like to term “the mystery machine.” This machine is typically a production server that is running an OS years out of date (since I’ve worked with Linux flavored machines, we’ll go with that for the rest of this analogy). The mystery server is usually introduced to you by someone else on the team as “that box running important custom created software with no documentation, shutdown or startup notes, etc.” This is a machine where you take a peek at top/htop and notice it has an uptime of 2314 days 9 hours. This machine has faithfully been running a program in htop called “accounting_conversion_6b”
You do a quick search on the box and find the folder with this file and some bin/dat files in the folder, but lo’ and behold not a sign or trace of even a readme. This is the machine that, for whatever reason, your boss asks you to update and then reboot.
“No sir, I’d strongly advise against updating right now — we should get more informa..”
“NO! It has to be updated. I want the latest security patches installed!”
You look at the uptime again, the folder with the cryptic sounding filenames and not a trace of any documentation on what this program even does.
“Sir, could you tell me what this machine is responsib …”
“It does conversions for accounting. A guy named Greg 8 years ago wrote a program to convert files from <insert obscure piece of accounting software that is now unsupported because the company is no longer in business> and formats the data so that <insert another obscure piece of accounting software here> can generate the accounting files for payroll.
And then, at the insistence of a boss who doesn’t understand how the IT gods work, you apply an update and reboot the machine. The machine reboots and then you log in and fire up that trusty piece of code — except it immediately crashes. Sweat starts to form on your forehead as you nervously check log files to piece together this puzzle. An hour goes by and no progress has been made whatsoever.
And then, the phone rings. Peggy from accounting says that the file they need to run payroll isn’t in the shared drive where it has dutifully been placed for the last 243 payroll cycles.
“Hi this is Peggy in accounting. We need that file right now. I started payroll late today and I need to have it into the system by 5:45 or else I can’t run payroll.”
“Sure Peggy, I’ll get on this imme ..” *phone clicks*
You look up at the clock on the wall — it reads 5:03.
Welcome to the fun and fascinating world of “the mystery server.”