An AI platform team is the central team that owns an enterprise's agent infrastructure: the engine room of any agentic platform. It encompasses the models (accessed, routed, and, where the enterprise hosts its own, served), alongside the agent harness and agent governance. The team builds and runs that infrastructure as a product, so every vertical can build and operate its own agents on one governed substrate.
Platform teams have become established in IT organizations over the past decade, tasked with building Internal Developer Platforms (IDPs) that automate the repetitive workflows for software engineers. From that lineage, many AI platform teams are now being repurposed, applying the methodology of platform engineering to the development of agentic platforms.
Over the same decade, ML platform teams industrialized model training which includes serving and feature infrastructure for data scientists, creating a second lineage. Where such a team exists, it often becomes the AI platform team or merges into it.
In this article we explore why agentic platforms are a prerequisite for becoming an agentic enterprise and the role AI platform teams play in that transition. We look at what these teams do and how they are structured before discussing how to build one.
Why the models need a platform, and why one team should own it
Most organizations are entering a race to become agentic. Not enough realize that handing out licenses for frontier models is insufficient if the goal is to treat AI as more than a chatbot or autocomplete functionality.
For agents to work autonomously and actually yield productivity gains, they have to be managed in an environment optimized for them. Take the analogy of hiring an intern eager to learn. Imagine spending zero time onboarding them and throwing them into a work environment that isn’t optimized for their specific characteristics. Would you be overly surprised if they hallucinated and were unproductive? Probably not.
Here are a few things you’d want to give them on day one: a handbook, a badge, a toolkit, and a reviewer.
For agents, it’s quite similar. An environment is ready for agents when it is legible, known, capable, and checked.
A handbook: the environment is legible. Context is written down and machine-readable (docs, ontology, service graph), rather than held in people's heads
A badge: the agent is known. It has an identity, a scope, and a budget, so what it may touch and spend are decided before it acts.
A toolkit: the agent can act. Tools and systems are exposed through interfaces it can call, with clear contracts.
A reviewer: the agent is checked. Its output is evaluated against a definition of done, and failures loop back so it can iterate safely until the gate passes.
This “environment for agents” is called an agentic platform. Now we could just delegate the job of creating that platform to each individual business vertical, which is something we’ve done in the past with similar transformation projects (think the business RPA movement, for example). For several reasons, however, that is not a good idea in this case:
Agent infrastructure is technically challenging to get right. At enterprise scale, it has to be secure and reliable as well as traceable. Vertical units rarely have the technical capabilities to build this layer themselves.
Agent infrastructure is essentially the same regardless of what action an agent is performing. Leaving responsibility with individual business units would mean losing the benefits of scale.
Interoperability is key. An agent in one vertical unit should be able to communicate with an agent in another without unnecessary restrictions.
Uber ran this experiment for us. By 2024, the company counted more than 60 distinct LLM use cases, with each team wiring its own integration: its own provider access, security review, logging, and billing. After seeing the complexity this created, Uber quickly realized that the answer was not sixty better integrations.
The Michelangelo team, which had run Uber's central ML platform since 2016, built a single GenAI gateway: one interface in front of OpenAI, Vertex AI, and Uber-hosted models, with authentication, authorization, audit logging, and PII redaction built in once, for everyone (Uber Engineering, July 2024).
The pain arrived at the gateway first. The fix was central, and the team that built it was the existing platform team with an expanded scope.
So if we follow this example rather than leaving it to the individual business unit, the cleaner approach is to establish a central team to own it. That team is the AI platform team.
What the team owns: the agent infrastructure layer
In general, agentic platforms have three layers:
Tooling: digital tools that store, manipulate, or display data. These tools are usually specific to the vertical. Examples include a CRM in sales, a Git repository in software engineering, or an ERP in finance.
Paths: the documentation or configuration of repeatable flows of work from intent to outcome. Think of these as the map through the platform: the .md files and skills that agents can load and deterministic workflows, along with workflows that combine probabilistic and deterministic elements.
The agent infrastructure: the fabric that runs the agents including the models, harness, and governance frameworks.
Figure 1: Reference architecture for agentic platforms, including the tooling layer, path specifications, and the agent infrastructure.
While the AI platform team contributes to harmonization on the path specification layer and provides recommendations for discoverability and accessibility at the tooling layer, it usually focuses most of its efforts on the agent infrastructure.
If we zoom into the Agent Infrastructure, we see the primary capabilities the AI platform team needs to provide. The agent infrastructure comprises 3 key areas:
The harness: how a single agent does the work. The harness is the operating system around one agent. Execution covers orchestration. It also includes gateway and model routing, and the runtime and its sandbox. Context covers what the agent knows: the service graph and ontology, retrieval, memory, and state. Capability covers what it can touch: protocols, tool gateways, contracts, and registries. Evaluation covers how we know it did the job: evals, goldens and rubrics, replay, and the change-safety gates a path must pass before anything ships.
Three of the four day-one requirements live here. The harness is what makes the environment legible and capable, then checked.
Governance: who is allowed to do what across the whole fleet. While the harness serves one agent, governance serves all of them at once. Identity gives every agent a workload identity, its secrets and its budget. Agent security puts the output guardrails, compliance checks, and data-loss prevention in one place. Agent observability provides the traces. It also provides the audit trail and the cost attribution needed to determine, after the fact, which agent did what and what it cost.
This layer makes agents known, and it is built once for the fleet rather than once per agent, because governance assembled agent by agent produces an estate nobody can audit.
Models: what intelligence is being consumed. Beneath both sits the model layer: the provider portfolio, the inference endpoints, and self-hosting where sovereignty, latency, or cost demand it. In an enterprise that consumes models purely via API, this is a provider-portfolio function. In one that hosts its own models, for sovereignty, latency, cost, or fine-tuning, it becomes a full engineering discipline covering GPU capacity, inference serving, model lifecycle, and evaluation of the models themselves.
Either way, it is platform work, not procurement. Leaving model access unowned is exactly how every application team ends up solving routing, rate limits, and cost attribution on its own. Uber's sixty integrations started here. Whether this area is a thin provider-portfolio function or a full serving team depends on the enterprise's hosting posture, not its size.
Everything the AI platform team owns answers one of these three questions: How does an agent do the work? Who is allowed to do what? What intelligence is being consumed?
How an AI platform team is structured
If we zoom in on the team, there are some cross-functional roles alongside specialized functions. In smaller teams, these responsibilities can of course be shared across fewer people and don’t require a dedicated FTE for every role. Let’s start with the cross-functional roles.
The Head of AI Platform is responsible for coordination and delivery. This role usually reports either to the CTO/CIO or the data lineage, under a Chief Digital and Artificial Intelligence Officer (CDAO) or a newly established role such as the Chief AI Officer (CAIO). We believe reporting into engineering or IT can be more effective because these functions are accustomed to building and operating products and internal platforms, rather than focusing primarily on data
Next is the AI Platform Product Manager, a role we’ve found to be make-or-break in many platforming initiatives over the years when building IDPs. The job is straightforward: talking to users, understanding their needs, translating those into requirements, and overseeing the roadmap. Finally, there is a function focused on enablement and paths. These people act as change agents, helping business units create the paths and repeatable flows, calibrate the agents that run on the agent infrastructure, and connect their tooling.
Figure 2: The organizational structure of an AI platform team, including cross-functional and specialized roles.
The specialized roles focus on the different components of the agent infrastructure:
Gateway & Execution owns the model gateway and routing, the orchestration, and the runtime sandbox in which agents operate, along with the associated token FinOps.
Models & Serving manages the provider portfolio, inference endpoints, and self-hosting or fine-tuning capacity. This component is where the ML infrastructure, inference, and MLOps engineers from the ML platform lineage sit. The function can scale from a small team to a full engineering function depending on hosting posture.
Context & Knowledge builds the service graph and ontology. It also delivers the retrieval and RAG pipelines, and the memory and state agents draw on.
Capability & Tools exposes the enterprise's systems to agents through tool gateways, MCP servers, registries, and contracts, keeping them interoperable with the existing estate.
Evaluation & Quality runs the evaluation infrastructure, including goldens and rubrics, replay, change-safety gates, and the agent observability that shows whether an agent did the job.
Governance provides workload identity and secrets, budget enforcement, output guardrails, DLP, and audit capabilities. These are built once for the fleet and jointly staffed by the AI platform team and the security organization.
When you need one and how to start it
So, do you need an AI platform team? At this point, for most organizations pursuing agentic AI at scale, the answer is a clear yes. We’re in a global race for productivity, and those that become agentic first will have an advantage. That requires a platform, and the platform must be built and operated by people with the right expertise.
Some organizations are installing a center of excellence and consider the job done. But there’s a significant difference between advising teams on how to adopt and use AI, which is typically the role of a Center of Excellence (CoE), and building the technical foundation for it, which is the role of the AI platform team.
Another school of thought places the responsibility with the MLOps teams. Yet these teams have traditionally focused on building for data teams and data scientists, rather than providing the shared internal fabric on which an agentic business runs.
If you’re thinking about establishing an AI platform team, the key is not to start too big. Follow the established patterns learned from a decade of platform engineering in software development: start with a small team of people who have built internal platforms before, and treat the work as a product with users rather than a project with an end date.
Start with a Minimum Viable Platform (MVP) and make the gateway its foundation: Model access is where problems around spend, credentials, and provider access emerge first, and where one central piece of infrastructure can pay off fastest. Build in identity, budgets, and evaluation from the start, because they are far harder to retrofit later. Then grow path by path. Let the verticals author their own flows, with the enablement function reviewing and promoting the good ones. And resist the temptation to build the paths for them. One model across all verticals, one team behind it.
Platform engineering spent a decade learning how to turn shared capability into a product that people actually use. The AI platform team is where that decade gets applied to the most important platform most enterprises will ever build.
What a time to be in this job.
