AI clusters have gotten a shared infrastructure. Neoclouds, enterprise AI platform groups, monetary providers organizations, life sciences groups, and analysis teams must share GPU capability. This shared infrastructure can undergo from decrease monetization, elevated operational complexity, and restricted management and visibility throughout tenants, workloads, hosts, and the community cloth.
EVPN/VXLAN is the sensible community basis. It offers tenant-scoped overlay segmentation utilizing VRFs, VNIs, route distinguishers, and route targets. Nevertheless, tenant-aware segmentation shouldn’t be job-aware segmentation. The scheduler understands jobs; the community usually understands routes, interfaces, queues, drops, and flows.
Why AI clusters want multitenancy
Devoted GPU clusters are easy to isolate, however they’re inefficient to function at scale. As GPU estates develop, organizations desire a shared useful resource pool that may serve a number of groups, prospects, and workload lessons with out forcing each group into its personal bodily cluster. In any other case, one group can have stranded GPUs in a devoted island whereas one other waits in queue.
The requirement seems in a number of patterns:
A GPU-as-a-Service supplier maps every tenant to an exterior buyer with its personal handle and coverage area (per-customer isolation whereas preserving the GPU pool shareable).
An enterprise platform workforce maps tenants to growth, testing, manufacturing fine-tuning, mannequin analysis, or regulated analytics (constant surroundings boundaries with out constructing separate clusters).
A monetary service division separates fraud analytics, threat modeling, and analysis workloads on one coaching cluster (stronger management boundaries and auditability with out duplicating GPU islands).
A analysis group assigns shared GPU capability to impartial analysis teams (clearer quota, utilization, and troubleshooting accountability throughout competing initiatives).
For this reason multitenancy can not cease at compute allocation. Distributed coaching is determined by east-west GPU communication, usually over Ethernet materials, so the community turns into an integral a part of the isolation and efficiency boundary.
How business solves it right now
Present AI multitenancy is normally carried out throughout three layers:
Orchestration and scheduler layer. Kubernetes-based platforms, GPU cloud orchestration programs, and Slurm schedulers outline the logical possession mannequin for the cluster. They observe tenants or initiatives, customers, queues or namespaces, job requests, node placement, and GPU allocation. For instance, Tenant A would possibly submit Job 100 requesting eight GPUs throughout two servers, whereas Tenant B submits Job 200 requesting 4 GPUs on a special set of nodes. For example, in an orchestration platform like Rafay, the platform can personal tenant onboarding and infrastructure intent, whereas the precise job scheduling could occur in Kubernetes, Slurm, or a tenant-operated scheduler.
Host isolation layer. The host enforces the native gadget boundary for every workload. If a tenant receives entire servers, isolation is easier as a result of the server, GPU set, and NIC set could be handled as one tenant-owned unit. If a number of tenants or jobs share GPUs throughout the similar server, the runtime should expose solely the assigned GPU units and bind the workload’s communication libraries, similar to NCCL or UCX, to the meant NICs. This host-side mapping issues as a result of a GPU server could have a number of NICs linked to totally different switches or tenant-facing community segments. Material segmentation can isolate site visitors as soon as it enters the community, nevertheless it can not right an incorrect native task the place the workload is allowed to make use of the incorrect GPU or NIC.
Community segmentation layer. EVPN/VXLAN offers scalable tenant segmentation throughout the material. VXLAN encapsulates tenant site visitors and makes use of VNIs to establish the overlay section or routing area. EVPN makes use of BGP to promote endpoint and prefix reachability and to manage which VTEPs import a tenant’s routes by way of route targets. In a routed AI cloth, a tenant generally maps to a VRF and a number of VNIs, with route distinguishers preserving tenant routes distinctive and route targets controlling import-export coverage. This enables overlapping tenant handle house and scoped reachability throughout a shared underlay.
ACLs or safety group ACLs can add supply and vacation spot coverage, particularly in brownfield L3 designs or the place the material can not but devour richer workload identification. Their limitation is operational scale: static or manually up to date ACL and VRF insurance policies don’t naturally comply with fast-changing AI job placement.
Collectively, these layers present a workable tenant-level mannequin. The remaining hole is job context: the community can normally see tenant context, interfaces, routes, queues, and counters, however not the particular scheduler job working inside a tenant. Tenant segmentation itself doesn’t robotically isolate Job 100 from Job 101 inside the identical tenant until job identification can be carried, derived, or programmed into community coverage.
Cisco Nexus One integration with AI iorchestration platforms
Cisco Nexus One is nicely positioned because the broader basis for making tenant-aware AI materials extra deterministic. On this structure, Nexus One is the entire cloth automation, integration, and visibility floor for your entire cloth.
Nexus One can present cloth topology context to an AI infrastructure orchestration platform similar to Rafay by way of integration workflows or APIs. That lets groups map tenant VRFs, VLANs, and port assignments on to a tenant, relatively than managing them solely as an summary tenant label.
As well as, Nexus One extends the mannequin past provisioning. Tenant-level visibility can present the tenant’s cloth path and related well being indicators similar to congestion, drops, and so forth. This enhances AI job observability: job-aware views can correlate scheduler, topology, optics, GPU telemetry, analytics, and anomalies, whereas tenant-specific Job-ID enforcement stays a separate future-facing coverage functionality.
Tenant-aware shouldn’t be job-aware
Tenant segmentation solutions the query, “Which buyer or group owns this site visitors?” AI operations usually want, “Which coaching job is creating or experiencing this site visitors inside a tenant?”
This distinction issues for segmentation in addition to throughout troubleshooting. A scheduler can establish the job, allotted nodes, GPUs, and runtime state. The community can establish interfaces, routes, queues, drops, ECN marks, PFC occasions, optics well being, and paths. With out correlation, operators should manually join these two views.
The result’s a standard operational drawback: the material exhibits a scorching uplink or lossy interface, whereas the platform workforce sees a sluggish coaching job. The lacking hyperlink is the workload identification within the community working mannequin.
Future route: AI Job-ID-aware segmentation
Job-ID-aware segmentation route—patent-pending know-how from Cisco—is the logical subsequent step. (Observe that this describes our architectural route, not a delivery function.) The objective is for infrastructure orchestrator (similar to Rafay) and scheduler (similar to Slurm) intent to hold each tenant identification and job identification into the material management and data-plane mannequin.
In that mannequin, the material controller interprets job intent into coverage. The swap information airplane carries or derives a job ID, for instance by way of VXLAN GPO bits, and enforces that solely endpoints in the identical licensed tenant and job can alternate RoCEv2 site visitors.
The anticipated advantages are operationally vital:
Less complicated operations, as a result of groups can cause in tenants and jobs as a substitute of translating each grow to be static community objects—essential for NetOps and cloth operations groups.
Deeper visibility, as a result of drops, congestion, paths, and optics could be correlated to workload context relatively than solely to interfaces or tenant VRFs—helpful for platform engineering and SRE groups.
Extra granular segmentation, as a result of coverage can comply with the lifecycle of a job relatively than stopping on the tenant boundary—vital for safety, compliance, and tenant governance groups.
This strategy is constructed on open requirements—not a proprietary overlay. EVPN/VXLAN is IETF-defined, and the Group Coverage Choice (GPO) follows the identical path: an IETF-defined mechanism that encodes a gaggle/coverage identifier within the VXLAN header alongside the VNI, which Cisco NX-OS implements in alignment with the open specification. Tenant scope (VNI) and workload/job scope (GPO) are due to this fact expressed in constructs a standards-compliant cloth can interpret—letting operators evolve from tenant-aware to job-aware enforcement with out a cloth forklift.
Technical instance: tenant and job boundaries
Contemplate a GPU-as-a-Service surroundings with two prospects, Tenant A and Tenant B. Every tenant is mapped to its personal VRF/VNI within the EVPN/VXLAN cloth. Tenant-level segmentation prevents Tenant B from reaching Tenant A.
Now assume Tenant A runs two concurrent coaching jobs. Job 100 makes use of GPUs on servers 1 and a couple of. Job 101 makes use of totally different GPUs on the identical shared cloth. Tenant-level EVPN/VXLAN nonetheless treats each jobs as Tenant A site visitors. Job-ID-aware segmentation would add one other enforcement dimension: Job 100 endpoints may alternate RoCEv2 site visitors with different Job 100 endpoints, however not with Job 101 endpoints, even inside the identical tenant.
That’s the architectural shift: EVPN/VXLAN stays the tenant basis, whereas Job ID turns into the long run workload-level coverage and observability attribute.
Advancing safety from tenant-level to job-level segmentation
AI information middle multitenancy begins with EVPN/VXLAN tenant segmentation, nevertheless it doesn’t finish there. The stronger working mannequin combines topology-aware provisioning, tenant-level enforcement, and end-to-end visibility right now, then evolves towards Job-ID-aware segmentation as scheduler and orchestrator integration matures.
In case you are designing a shared AI cluster right now, tenant-aware EVPN/VXLAN is the inspiration. Job-aware enforcement and observability are the following frontier.
*Particular due to Ramesh Ponnapalli and his workforce, whose engineering management has been instrumental in bringing this know-how to life.
Further assets:
