Coming soon
Model Serving
Where model capability enters the enterprise: bought by the token from managed inference APIs, or run on rented compute through hosting platforms and accelerator clouds. Model serving is how platform teams make that choice once, for every team that builds agents.
What is
Model Serving
Model serving is the part of the model layer that turns model weights into inference endpoints the enterprise can call. Platform teams source it from two markets: managed inference APIs, which sell tokens at a metered price from a catalog the vendor chooses, and model hosting platforms and accelerator clouds, which sell compute time to run whatever weights the buyer brings. In an enterprise that only consumes APIs, model serving is a provider-portfolio function. In one that hosts its own models for sovereignty, latency, cost or fine-tuning, it becomes a full engineering discipline covering GPU capacity, inference serving and model lifecycle.
Why model serving is platform work, not procurement
Every agent a platform team ships runs on inference it has to buy or run. Left unowned, every application team picks its own provider, negotiates its own rate limits and builds its own cost attribution, and the enterprise ends up with dozens of integrations nobody can govern.
The choice is rarely clean. Inference API vendors now sell dedicated deployments, and hosting platforms ship per-token APIs, so one buyer decision spans both markets. Pricing power sits upstream with model owners; volume and margin pressure sit with both. Platform teams that make it deliberately, once, give every team a fast default and keep the option to move workloads as prices, models and volumes change.
Tokens or compute hours
Where model capability enters the enterprise: bought by the token from managed inference APIs, or run on rented compute through hosting platforms and accelerator clouds. Model serving is how platform teams make that choice once, for every team that builds agents.
Know where the cost curve crosses
Where model capability enters the enterprise: bought by the token from managed inference APIs, or run on rented compute through hosting platforms and accelerator clouds. Model serving is how platform teams make that choice once, for every team that builds agents.
Hosting posture sets team size
Where model capability enters the enterprise: bought by the token from managed inference APIs, or run on rented compute through hosting platforms and accelerator clouds. Model serving is how platform teams make that choice once, for every team that builds agents.
23%
State of AI in Platform Engineering, Volume 2 (Weave Intelligence, 2026), n=350
At Uber, model access started unowned: application teams built about sixty separate model integrations, each solving routing, rate limits and cost attribution on its own. That is the pattern a platform-owned serving layer prevents.
State of AI in Platform Engineering 2026
State of AI in Platform Engineering Vol. 2: why 38% of orgs ship more with AI but only 8% see real returns, and what separates the two.
Download
FinOps Trends in Platform Engineering
Cost ownership has moved out of finance and into engineering, and the platform teams that run the IDP now answer for spend that is committed long before anyone sees the bill. As storage, data platforms, SaaS and AI turn more of that spend into something consumption-based and unpredictable, after-the-fact reporting can’t keep up. This market guide from Weave Intelligence defines FinOps as a platform capability, and what it takes to make economics a first-class part of the platform.
Download
Every week, a new, better model comes out. And we're all using the same models. Why are some organizations 10x faster than other ones? … It's because they have the platform around the model.
![](data:image/svg+xml,<svg display="block" id="3044341856" role="presentation" viewBox="0 0 122 105" xmlns="http://www.w3.org/2000/svg"><path d="M 72.296 105 L 72.296 71.346 C 72.296 52.799 76.514 37.094 84.948 24.231 C 93.684 11.068 106.035 2.99 122 0 L 122 19.295 C 113.264 20.49 106.486 24.829 101.667 32.308 C 97.148 39.488 94.889 49.658 94.889 62.821 L 79.074 58.782 L 119.289 58.782 L 119.289 105 Z M 0 105 L 0 71.346 C 0 52.8 4.217 37.094 12.652 24.231 C 21.388 11.068 33.738 2.99 49.704 0 L 49.704 19.295 C 40.968 20.49 34.19 24.829 29.37 32.308 C 24.852 39.488 22.593 49.658 22.593 62.821 L 6.778 58.782 L 46.993 58.782 L 46.993 105 Z" fill="rgb(0, 76, 255)" height="105px" id="hkcc3mE4d" width="121.9999976196289px"/></svg>)
Practice area
Model serving trends to watch in 2026
Model access is no longer the differentiator
Only about 1 in 10 organizations say model capability holds back AI at scale. Platform readiness (27%) and token and infrastructure cost (23%) top the list. Everyone can buy the same models; how they are served is where teams pull ahead.
The two markets are converging
Inference API vendors now sell dedicated deployments, and hosting platforms and neoclouds ship per-token APIs. One buyer decision now spans both, and the same vendor can sit on either side of it.
Token spend outruns ownership
35% of organizations manage LLM token cost through central billing or FinOps, 22% through rate limits or quotas and 17% through team budgets. 13% do not track it at all and 14% do not know how it is managed.
Cost per token gives way to cost per outcome
Compressing tokens per action can trigger more agent turns and retries, so total cost rises. Leading teams measure cost per successful, quality-adjusted outcome instead.
Source: State of AI in Platform Engineering, Volume 2 (Weave Intelligence, 2026), 350 respondents; FinOps Trends in Platform Engineering (Weave Intelligence, October 2026).
Frequently asked questions
What is the difference between a managed inference API and a model hosting platform?
A managed inference API sells tokens at a metered price, from a catalog of models the vendor chooses. A model hosting platform sells compute time to run weights you bring, including fine-tuned or in-house models. The first is faster to start; the second gives more control over models, tenancy and cost at volume.
When should an enterprise self-host models instead of using an API?
When sovereignty, latency, cost at sustained volume or fine-tuning demand it. Per-token pricing usually wins at low or spiky volume. Dedicated capacity wins once utilization is high and steady, provided the team can carry the operating burden that comes with it.
What is a neocloud?
A cloud provider built around GPU capacity for AI training and inference, such as CoreWeave, Nebius, Crusoe, nscale or Lambda. Neoclouds sell accelerator hours, and increasingly managed inference on top, competing with hyperscalers on GPU availability and price.
Who should own model serving in the enterprise?
The AI platform team, through a Models & Serving function. Left unowned, every application team solves provider choice, rate limits and cost attribution on its own. How big that function gets depends on hosting posture, not company size.
How should platform teams measure inference cost?
Attribute every model call to a team, a path and a model, and watch spend velocity, not just totals. Judge optimizations by cost per successful, quality-adjusted outcome rather than cost per token, because cutting tokens per action can raise total cost through retries.
