Route
Send each request
to the model it needs.
Frontier as the exception. Route on the content of the payload, not its label. Cheap work to cheap models, specialist work to specialists.
Where you are today
Everything goes to the same place.
Routing decisions get made on labels, headers and tags. None of them say what is actually in the request, and reading the payload to find out costs more than the routing would save. So the safe default is one endpoint, the frontier model, doing work a smaller one could have done.
What changes
Routed on the content, not the label.
NOL8 reads the payload while it is still moving and routes on what it finds. The frontier model gets the requests that need it, and nothing else.
The read is already happening for inspection, so the NOL8 routing decision costs nothing extra: no extra pass and no extra hardware.
One read, not two
Routing on content normally costs a read of its own. Inside NOL8 it costs nothing, because the payload is being read anyway.
The usual arrangement
A routing layer reads the content, then an inspection layer reads it again. Two reads, two hops, two things to keep in step.
NOL8
One read. Routing is a verdict on a payload that was going to be inspected regardless, so the decision costs nothing in latency and nothing in hardware.
Common questions
Uncertainty sucks.
Stop estimating. Your rules, your payloads, your P99.