I have been working a lot in Microsoft’s Azure platform, and for the most part it has been a relatively smooth transition. Lots of new things to learn, a fair bit of code refactoring and adapting to the new cloud environment.
Recently, I ran into a odd problem at a client of mine that runs on Azure. My client came to me regarding a January email that was supposed to go out at the beginning of the month, and the January email showed December data.
So I dug into this, and verified the report was scheduled to run on the First Wednesday of every month, and verified that the emails were inserted into our outgoing mailbox a minute after midnight on Wednesday January 1st. I spot checked a few of the emails, and indeed they did have (incorrectly) December information. I looked thru the code, and the sql server stored procedure and still could not find a reason.
Finally, I looked at the job run log and found the problem:
The job that was supposed to run at midnight was kicked off at 11:59:57 by Azure’s job trigger. When the report runs, if checks to see the current month year of the run to choose its data, and in this case, it was December.
It’s curious to me that the scheduler is that imprecise. For years networks have had a common time synchronization service which pretty much guaranteed times across servers, but apparently there is a discrepancy between Azure services. I will have to get use to the idea that these cloud services provided by Azure my be loosely coupled, and these services may be more autonomous than my previous windows network world.
My work around for this is simple. I can delay the run of this job a couple hours, so the job will run at 2:00am instead of midnight. My assumption at this point is 2 hours should be safe. But in this brand new cloud world, there is always something to surprise you.