And with the start of September comes the summer heatwave that never happened. Temperatures have suddenly jumped into the 30s and with humidity around 90% it feels around 40. It’s the sort of weather that makes me instantly grateful for air conditioning to give me a fighting chance at sleep.
But also September brings maybe the end of the supermarket project. It’s been a bit of a rough one - and on many fronts. The project, the science, the staffing. Even when I thought I was clear of this thing, it finds a way to throw some last minute surprises at me.
With literal single digits to delivery and walking away for good a couple of last minute sanity and safety checks revealed that the solution was only producing 1/16 the volume of output it was a few months ago.
Was this expected? Who knows? Was the quality of the stuff we were producing reduced? Again who knows? And when did all of this happen? Beats me!
It just goes to show that the monitoring of the ongoing solution was subpar at best. And it highlights to me a woefully shortcoming in the amount of routine, end-to-end testing that we put in place.
As a former president said “the buck stops with me”. Which meant that I spent a large majority of my long weekend going through every single pull request over the last 6 months - line by line - and evaluating every change that could have made a difference to the model output. A thankless, dumpster diving activity trying to cross reference work against tickets and an absence of documentation. You would think at this point I would’ve learnt the lesson around writing documentation and keeping an audit trail for this exact reason.
But it’s the same on ever project. Every time I think this time the team will be different and people are grown ups and I won’t need to baby everyone into writing stuff down. And time and time again I get halfway through the project and realize that we’re stuffed and need a hard pivot. Perhaps the rational move would be to set high expectations early and just keep them there.