2023-09-05

And with the start of September comes the summer heatwave that never happened. Temperatures have suddenly jumped into the 30s and with humidity around 90% it feels around 40. It’s the sort of weather that makes me instantly grateful for air conditioning to give me a fighting chance at sleep.

But also September brings maybe the end of the supermarket project. It’s been a bit of a rough one - and on many fronts. The project, the science, the staffing. Even when I thought I was clear of this thing, it finds a way to throw some last minute surprises at me.

With literal single digits to delivery and walking away for good a couple of last minute sanity and safety checks revealed that the solution was only producing 1/16 the volume of output it was a few months ago.

Was this expected? Who knows? Was the quality of the stuff we were producing reduced? Again who knows? And when did all of this happen? Beats me!

It just goes to show that the monitoring of the ongoing solution was subpar at best. And it highlights to me a woefully shortcoming in the amount of routine, end-to-end testing that we put in place.

As a former president said “the buck stops with me”. Which meant that I spent a large majority of my long weekend going through every single pull request over the last 6 months - line by line - and evaluating every change that could have made a difference to the model output. A thankless, dumpster diving activity trying to cross reference work against tickets and an absence of documentation. You would think at this point I would’ve learnt the lesson around writing documentation and keeping an audit trail for this exact reason.

But it’s the same on ever project. Every time I think this time the team will be different and people are grown ups and I won’t need to baby everyone into writing stuff down. And time and time again I get halfway through the project and realize that we’re stuffed and need a hard pivot. Perhaps the rational move would be to set high expectations early and just keep them there.

Older post

2023-09-03

Newer post

2023-09-06

What distinguishes you from other developers?

I've built data pipelines across 3 continents at petabyte scales, for over 15 years. But the data doesn't matter if we don't solve the human problems first - an AI solution that nobody uses is worthless.

Are the robots going to kill us all?

Not any time soon. At least not in the way that you've got imagined thanks to the Terminator movies. Sure somebody with a DARPA grant is always going to strap a knife/gun/flamethrower on the side of a robot - but just like in Dr.Who - right now, that robot will struggle to even get out of the room, let alone up some stairs.

But AI is going to steal my job, right?

A year ago, the whole world was convinced that AI was going to steal their job. Now, the reality is that most people are thinking 'I wish this POC at work would go a bit faster to scan these PDFs'.

When am I going to get my self-driving car?

Humans are complicated. If we invented driving today - there's NO WAY IN HELL we'd let humans do it. They get distracted. They text their friends. They drink. They make mistakes. But the reality is, all of our streets, cities (and even legal systems) have been built around these limitations. It would be surprisingly easy to build self-driving cars if there were no humans on the road. But today no one wants to take liability. If a self-driving company kills someone, who's responsible? The manufacturer? The insurance company? The software developer?