Loading…
4
Pro tip: I keep seeing folks treat a new AI model's first demo as its final, locked-in ability, which is a huge mistake because the training data and fine-tuning over the next 6 months will completely change its performance.
I watched a project team in Austin base their whole quarterly plan on a single demo's output, only to have the model's reasoning on the same task shift dramatically after the next data batch was processed.
3 comments
Log in to join the discussion
Log In3 Comments
derekcarr5mo ago
The Austin team story is a perfect example of why you can't treat a demo like a finished product. I actually get where @troy977 is coming from with that Seattle setback, but calling it just a planning problem feels short-sighted. When the core tool you're building with can change its whole approach overnight, what does good planning even look like? You're basically trying to hit a moving target that nobody has full control over.
8
troy9775mo ago
Our design lead in Seattle did the same thing last spring, betting our whole prototyping schedule on the reasoning style from a single GPT-4 demo. When the next version rolled out, it started structuring its code answers totally differently, which set us back almost a month. It's so easy to forget these models are still moving targets. That kind of planning whiplash is brutal to actually live through.
6
jana_hayes315mo ago
Honestly, that sounds like a planning problem, not a model problem. You can't build a schedule around a demo.
10