AI Skepticism is Reasonable (And Recommended)
I’ve been excited about AI for years. Even studying the field back in college, when there was only so much you could do with the lack of processing power, tooling and data, the possibilities were inspiring.
Generative AI has unlocked a new peak of excitement and with that should come an ever increasing level of skepticism. It’s easy to fall for the hype and take at face value the headlines surrounding every new model, demo or paper, but that excitement should be soon followed up by some level of diligence. I’m starting to see a bit more thoughtful push back on the unrestrained hype, which is a good sign that this AI cycle is maturing.
Let’s highlight a few recent examples.
OpenAI Sora
Soon after Sora was announced, I wrote an article breaking down a series of promotional videos that were (partially) created by Sora. Here’s what I said at the time:
Let me reiterate that Sora is an amazing accomplishment and looks like it really is a major step forward for foundational video generation models. With that being said, I think it’s clear that between the inherent drawbacks of the model by the way of its architecture and the lack of powerful and fine grained controls, Sora will not replace the vast majority of video generation tasks. At least not right away and not just by itself.
Since then, shy kids shared more information about their process of working with Sora to create their video “Air Head.” From a short video posted to their Twitter account, you can see example after example of raw Sora output that they had to edit:
An in depth interview with The Hollywood Reporter features some additional perspective:
It has the same pitfalls as a lot of generative AI in that consistency is difficult. Control is difficult, which as creatives, I mean, you want control. That was our general feedback to the researchers: More control would be great, because we want to try to tell a story using a consistent character from shot to shot and from generation to generation. Part of that is the reason why we ended up going with the guy with the balloon for a head, because it’s a little bit easier to tell that that’s the same character from shot to shot compared to someone whose face is changing.
It’s not something that can replace people. I feel like if more people have their hands on it, they’d realize that for themselves pretty quickly, too. It’s not a magic box that will suddenly make you Stanley Kubrick.
Devin AI
In my article Can We Prevent LLMs From Hallucinating? I briefly wrote about Devin AI, the “first AI software engineer”:
The technology has promise and I am sure it will improve over time. But the actual benchmarks today when you look at solving tasks end to end speak for themselves…Would you hire a software engineer that solves problems correctly 13.86% of the time? If you have to review every task for accuracy, how much value does this provide today?
Carl aka @internetOfBugs , a 35 year software developer, has since done an amazing breakdown of Devin AI:
“I’m not anti AI, but I am anti Hype” - this is a great perspective.
Amazon’s Just Walk Out
This video published yesterday by ColdFusion caught my attention. The highlight is the realization that Amazon’s Just Walk Out in person shopping experience was not powered by AI or technology but actually manually reviewed by 1,000 people in India: