When I took over on Friday, the girl who had been occupying the position assured me that it had been a quiet week, and all of the critical reports that we monitor had been working smoothly.
Saturday went fine, then I woke up on Sunday, logged in and noticed that only the first batch of output had been produced. I took a sip of my coffee and fired up Skype to find out if the support people had an ETA on completion. I dialled into a hornets' nest. It ended up being a four-hour phone call, ripe with hand wringing, accusations and counter-accusations. There were a lot of people very angsty about missing high-visibility deadlines.
"Oh well," I thought, "There has to be at least one day that does not pass by on cruise control."
This morning, one of my machines stopped responding to external connections, leaving me to operate with only a single desktop. No problem. As long as everything was running properly, I could just call somebody at the office in a couple of hours and have them power-cycle the machine.
Then I logged in and discovered that nothing had run this morning. There were not even status messages, or alerts advising that things would be running late. Uh oh.
I called the help desk, and asked them to give me the trunk line number for the hand-wringing session so that I could find out how long it might take for things to start working. I dialled in, and it was surprisingly quiet. I asked about the reports, and they said, "Oh, this is the other disaster line for the other set of critical reports that did not work today. You want to call into the other conference line."
I did not like the sound of that. I was right not to like the sound of that. I had no idea when I called the other conference line that I would be spending the next seven and a half hours on the phone, dealing with a data crisis.
The worst of it is that nobody knows why our systems were having problems. Individually, everything was working exactly like it should. Things that should have pinged were pinging. Things that should have traced were tracing. Reads and writes were happening. Sorts were sorting. It was like a beautiful ballet where all of the dancers were performing perfectly, and it would have been a lovely show if not for the whirling blades.
Anyway, it's over now. One more day of this and I am off for a couple weeks.