Your AI Workaround Has an Expiration Date
TL;DR: As models improve, they can often solve problems that previously required explicit rules or product workarounds. For product teams, that raises a practical question: how much should you invest in fixing limitations that better models may soon solve?
When building an AI product, one of the most important questions is whether a model limitation is worth solving yourself, or as models improve, does the problem disapper?
That question connects closely to Rich Sutton’s The Bitter Lesson. His argument is that, over time, methods that use more computation to search and learn have tended to outperform approaches built around hand-crafted human knowledge. Chess is a simple example. You could try to make a computer better by encoding detailed instructions from expert players. Or you could give it the rules, define the objective, and let it explore possible moves and learn from the results.
The second approach shifts where the model work happens. Instead of encoding the strategies we already know, you build a system that allows the model discover better ones. The “bitter” part is that people can spend years building expert rules only to watch a more general approach eventually outperform them.
AI product teams face a similar risk. A team might spend significant time designing around something a model cannot do today, then continue carrying that complexity even after the model improves. Imagine a customer support product that struggles with requests containing multiple issues. The team might ask customers to choose a category, split their questions, and submit each one separately. That may be a reasonable solution for the model available at the time. If a later model can understand the full request, those steps become unnecessary. The product should change with the model.
I think about fine-tuning the same way. Before investing in additional training for a specific task, I would first test what current models can do with good instructions, context, and tooling. If the remaining gap is not critical, it may be better to invest elsewhere and revisit the problem as models improve. Fine-tuning can still be the right choice. It may improve consistency or allow a smaller, cheaper model to perform well on a narrow task. The question is whether those benefits justify the cost of building, evaluating, and maintaining it.
This being said, there is also a cost to waiting. If a limitation prevents customers from getting value today, solving it now may be worthwhile even if the solution later becomes obsolete. And there is no guarantee that the next model will fix the exact problem you care about.
The practical lesson is simple: know why every workaround exists. Document the failure it addresses and keep examples of those failures. When models improve, test them again. If the workaround is no longer necessary, remove it. That must be a normal part of maintaining an AI product.