Work has been a combination of comedy, tragedy, stupidity, and a variety of other adjectives ending in "y". The interesting times started when we threw the switch on a new report that has been months (over a year, actually) in the making. Without going into great detail, it can best be described as a measure of our end-point service levels. After a couple of weeks of reliable performance, we finally rolled it out to the AVPs and senior managers amid much fanfare.
The next day, the server on which it was hosted got slammed by a sudden spike in workload which brought it to its knees. I fought with it for days to ensure that the numbers got out, though they were often as much as 5 hours late. Finally the server came up long enough for me to transfer all of the critical files and databases to another, slower and less-reliable server which - for all that - was at least running. I spent another day rewiring the nebulous web of data feeds that generated our scorecard and dashboard, and after a few days I finally got everything working on a reliable schedule. It was reliable, but it was also publishing about 5-15 minutes after the deadline every day. The business was happy to be getting the numbers, but unhappy that they were being published too late for their morning strategy and planning sessions.
The business has been pushing back pretty hard on this, demanding to know how we are going to address the timing issue. I have been stewing over this, because none of the others involved in this project will let me push back at them over the reason why everything is so late. The bulk of the reports reside and run on a server out in Markham Ontario. Our connection to this server is slow (at best), and irritatingly unreliable. This selfsame server used to reside in our Winnipeg office, where the connection was blazingly fast and reliable. The people who are raising a fuss about our tardy and unreliable service are the same ones who gave EDS permission to move the servers to Markham in the first place. They granted this permission when EDS approached them about it the day after our local IT specialist left town on vacation. The timing was not a coincidence; EDS knew they could save money by moving the server, and they knew that the local IT guy would never approve the move.
In any event, since I can't control the connection, my limited options are based around how to best optimize my use of a slow and unreliable server. As I was drifting off to sleep Monday night I was struck by an idea that I thought might resolve the issue. I have been limited in how much I can move around the execution times of some of our scorecards because there are cascading dependencies, but it occurred to me that I could split one of the earlier ones. I could run the queries for which there were downstream dependencies, but defer publishing it until after the more time-sensitive dashboards had run. I made the changes on Tuesday, and this morning everything ran flawlessly. It was like the gates of Heaven opened up and granted me a glimpse of perfection.
My glee was somewhat tempered by the fact that twenty minutes before I was due to head home this afternoon, my remote desktop machine out in Calgary that performs all of the scheduling and management of these reports suffered a hardware failure and died.
Tomorrow is going to be interesting.