Maybe the LLM Shouldn’t Be the Core of the System
I want to propose a thought experiment for you.
Imagine treating a computer processor the way we currently treat many AI systems.
Would you ever say:
Here is access to our payroll database, our accounting tools and our company policies. Please understand the general objective and run payroll.
The processor is incredibly powerful precisely because we do not ask it to independently become the entire application. Early software was often pretty close to that: huge, monolithic programs written in close-to-the-metal assembly.
There’s a reason we don’t do that anymore.
An LLM may make more sense viewed the same way. As a new kind of computational component that performs certain kinds of fuzzy, linguistic or interpretive work inside a larger deterministic system.
You can still test the system, if you genuinely build a system
I talked about testing and AI (LLM) systems recently link to test LLM article. And pointed out that “benchmarks” aren’t really a substitute for “software tests” in business software.
But suppose an LLM is being used to interpret a customer request. The interpretation itself may still be probabilistic.
But the rest of the process doesn’t have to be. The fuzzy component can stay fuzzy without making the entire application fuzzy with it.
Instead of asking an LLM to calculate a price, let it identify missing information or interpret something ordinary software can’t easily parse. Then store that result, validate it, and let deterministic software apply the pricing rule.
Don’t make the model rediscover the same fact every time just because inference happens to be cheap enough today. The DRY principle still applies. Don’t repeat yourself.
Instead of letting an LLM freely modify a customer record, let it extract the requested change by quickly building a set of fuzzy search parameters (and have a separate context verify the similarity of the request and the result).
THEN ordinary software validates and applies it.
Instead of asking a model to run an entire process from beginning to end, give it only the parts where linguistic interpretation is actually useful.
That smaller role is also much easier to evaluate.
If “the LLM fails this task 4% of the time,” that may be a sign that the task still contains too many hidden subtasks.
Break it down. Hand whatever you can to regular software. Keep narrowing the probabilistic part until its failures are measurable, detectable, and cheap enough to tolerate.
Now the rest of the system can go back to being tested “normally”.
With reliable, repeatable, boring software tests. But of course, this also means not everything can be made better with AI. If you have a well-optimized, smoothly running system that involves no fuzzy language processes… why are you trying to add “AI” to it?
Capability and predictability are different things
It is easy to watch an impressive model demonstration and conclude that an entire workflow can now be automated.
And yes, a model may be able to read the document. Understand the request. Choose an action. Call a tool. Produce a response.
So why not just connect everything together? Because capability and predictability are different properties. A person can do all of those things too.
Businesses still surround people with procedures, approval structures, accounting controls, checklists, and software systems. Not because people are useless, but because complex organizations cannot operate on the assumption that every person will interpret every situation “correctly” every time.
Yet with AI, we sometimes seem tempted to remove those structures and replace them with an especially detailed paragraph telling the model to behave itself.
Then, when that proves unreliable, we add validation, permissions, monitoring, review, and deterministic controls back on afterward.
That gets expensive quickly. And more importantly, it’s backwards.
Good systems reduce unnecessary variability from the start and assign each part of a process to something well suited to doing it. The same should be true here.
Especially because your new “expert” isn’t actually an expert.
It’s a language processor.
Maybe this is what mature AI software looks like
The lesson isn’t that LLMs are too unreliable for business.
They’re remarkably capable systems. Their flexibility is exactly what makes them useful.
But maybe we’re asking them to operate at the wrong level of abstraction.
We keep taking a probabilistic language system, bolting it onto an entire workflow, and asking it to behave like conventional software and a wildly overqualified person (and a bit of a mind reader) all at the same time.
But maybe mature AI systems look more like:
Give the language processor a bounded operation inside the job system.
Let it interpret language where interpretation is useful. Let it generate where variation is useful. Let it classify where fuzzy boundaries are unavoidable, but give it the classifications instead of asking it to invent your business structure on the fly.
Let it bridge messy human communication and structured software.
Then let ordinary software do what ordinary software is actually good at, guaranteeing that certain things happen exactly the way we intended.
If the model is one bounded processor inside a larger system, the model can remain probabilistic while the system becomes testable again.
Which may be a much more productive way to build with AI than continually trying to persuade a probabilistic language model to become an application, a workflow, or a person.
This is not just theory
I want to leave you with this. This is not theoretical. This approach is what powers SkillSaige’s Logic Lattice, and every way we use AI (and perhaps more importantly, don’t use AI) at the company.
In our internal bench tests, this approach to systems means a roughly fifteen-minute training scene can run with as much as a 97% token cost reduction compared to trying to get an LLM-centered chatbot to run the same scene end-to-end.
More importantly, it gives us something the LLM-centric approach doesn’t.
A system with behavior, state and business logic we can actually test.
The LLM can stay a “magical” (it’s really not, but we’ll just call it that for now) probability box.
The system that leverages it doesn’t have to be.