Hacker News

> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.

I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.

I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.

If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.

Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:

> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."

I'm still working on mine: https://github.com/rush86999/atom/blob/main/docs/architectur...

I think fail-over for a production critical service is an equally important responsibility of the router.

Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.

Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.

Routing belongs on the client side

[flagged]

[dead]