One endpoint, many model makers
· Nicrron Engineering
The Nicrron API puts every major model behind one OpenAI-compatible endpoint. Routing names such as nicrron/auto turn into ordered fallback chains, each model's endpoints are balanced by price and health, and repeat requests can be answered from a per-account cache at no charge.
Routing names
Besides real model names, the API accepts routing names. nicrron/auto sends long prompts to the most capable models and short ones to fast, light models. nicrron/reasoning, nicrron/coding and nicrron/fastest each name a short ordered list, and nicrron/cheapest orders every chat model by price. If the first model fails, the next one in the list is tried.
Choosing an endpoint
A model can be served by more than one endpoint. By default, healthy endpoints are shuffled with a weight of one over price squared, so cheaper ones are picked more often without starving the rest, and endpoints with recent errors go to the back of the line. Callers can instead ask to sort by price, latency or throughput, pin an order, or turn fallbacks off.
Caching
A request with temperature 0, or one that asks for it with a header, can be answered from cache. The cache key covers everything that changes what the model says, ignores fields that only change framing, and is scoped to the account, so one account is never served another's answer. A cache hit is recorded in the usage log and is not charged.
What surprised us
- 01The cache key hashes every field except a short list that doesn't change the answer. An allowlist would silently drop parameters the API passes through and serve an answer made under different settings.
- 02Responses that include tool calls are never cached, because a replay returns the text only and would quietly drop the tool calls.