Federated AI Architectures: A Careful Balancing Act

Enterprises deploying AI agents across multiple domain platforms face a critical governance challenge: Domain-specific systems handle local agents well, but cross-domain agents move beyond assumed trust and governance frameworks. This creates risks that traditional security approaches often don't cover. A dual-track AI platform strategy resolves this tension.

• Leverage domain-specific platforms for domain-specific, low-risk agents: Many major platforms (e.g., ServiceNow, Salesforce, SAP) now feature mature, embedded agentic AI functionality. In a federated architecture, agents that operate strictly within the boundaries of such a platform and rely solely on that platform’s data should be built there. This empowers domains to innovate rapidly, avoiding unnecessary custom development.
• Deploy a centralised AI platform for high-risk and cross-domain agents: For domains without such a platform, high-risk agents, or agents that span multiple systems, a central domain-agnostic AI platform is necessary. This typically means a foundational cloud AI hub (e.g., MS Azure AI or AWS Bedrock), an enterprise AI orchestration platform (e.g., Dataiku or Databricks Mosaic AI), and/or a low-code environment (e.g., MS Copilot Studio or UiPath). This platform should provide a convenient environment where agents can be built across domains, yet managed and monitored by central risk, legal, and IT teams. 

When scaling agentic AI, a lack of visible reusable assets often drives teams to rebuild the same foundational components from scratch (pipelines, integration tools, etc.). These duplications waste resources, introduce inconsistencies, and create unnecessary risk. By establishing a centrally-managed library of reusable agentic AI building blocks, organisations can significantly accelerate the development of new agents withindomains while reducing risks and lowering costs.

• Identify high-value centralised assets: A central team should identify and provide standardised components, as shown in this graphic
• Define the sourcing strategy (build vs buy): The primary role of the central team is to curate this ecosystem, rather than build every component in-house: for many building blocks – especially the more complex ones – integrating a third-party solution is faster, safer, and more cost-effective
• Provide an internal developer marketplace: Ideally, the building blocks will be easily discoverable via the company’s existing service catalogue (or internal developer portal, API marketplace, or similar), or on the AI platform(s) depending on their type. Seamless discovery is the key to ensuring these reusable assets are actually adopted by domain developers instead of being bypassed.

AI agents present significant security, privacy, and governance risks including autonomous task execution, prompt injection, and credential exposure. For example, earlier this year, widespread usage of the new OpenClaw agent framework, highlighted both the potential of agents to act autonomously as well as significant security, privacy, and governance threats (such as taking undesired actions, prompt injection, credential exposure, etc.). Addressing these risks and ensuring compliance with regulations, such as the EU AI Act, requires centralised visibility over federated agents and updated risk frameworks.

• Modernise identity & access management (IAM) for agents: Traditional IAM frameworks, built around human users and static service accounts, aren’t adequate for autonomous agents that can escalate privileges or delegate tasks. Organisations should implement a first-class "agent identity" construct with scoped, short-lived credentials tied to specific task contexts. This enforces least-privilege at the action level, requires re-authorisation at trust boundaries, and mandates high-impact actions pass through enterprise-approved guardrails or “human-in-the-loop” gateways. When an agent acts on behalf of a user, audit trails must preserve both the user's and agent's identity.
• Extend enterprise catalogues with AI metadata: Central teams should integrate AI asset tracking into existing service catalogues to manage risks across federated landscapes without bloating the architecture with isolated “AI registries”. 
• Implement observability mechanisms: Mandating the usage of central gateways for all LLM / tool invocations is usually unrealistic when AI agents reside inside robust domain platforms, such as SAP Joule. Instead, organisations should monitor agent usage and behaviour through:
  • routing tool invocations via existing API gateways, 
  • establishing AI-specific telemetry trace header standards and federating telemetry to enterprise observability stacks for tracking execution flows end-to-end, 
  • deploying a mandatory centralised LLM gateway for central AI platforms.

To successfully support cross-domain AI use cases. However, if agents deployed within domains cannot easily communicate with each other, their value is capped within their silo. It is important to standardise how agents connect, communicate, and discover oneanother across the organisation.

• Enable dynamic agent discovery via a dedicated machine catalogue: A separate, centralised agent catalogue allows agents to dynamically find and delegate tasks to authorised peers (e.g., "which agent is able to process an invoice?"). Unlike traditional IT service catalogues, this catalogue should include rich metadata (capabilities, data access rights, trust levels, ownership, SLAs) and enable impact analysis for changes.
• Avoid creating an AI multiverse: Emerging AI protocols (such as MCP, A2A, ACP and ANP) are gaining popularity. Rather than creating parallel “shadow” networking and API universes, enterprise architects should integrate them into established integration backbones. This aligns with the core architecture principle to reuse, which translates into a balanced approach to: 
  • Adapting over rebuilding for systems-of-record: Instead of introducing a parallel MCP layer, provide a generic OpenAPI-to-MCP adapter to build on existing REST APIs and OpenAPI specifications. 
  • Relying on established standards for cross-domain communication: Agents should use enterprise API standards when communicating outside of their own domain boundaries, keeping A2A protocols confined to intra-domain use. New protocols should be treated strictly as application-layer payloads rather than new transport mechanisms, so that they remain fully subject to centralised traffic management, security, and observability controls.
  • Ensuring existing APIs are AI-ready: To reuse existing APIs, their documentation must be suitable for LLMs. An effective way to achieve this is by updating API linting tools to require verbose, highly detailed descriptions of endpoints, payloads and parameters.
• Define flexible and controlled orchestration patterns: Multiple orchestration models (supervisor/hierarchical, decentralised, hybrid reactive–deliberative, graph-based, etc.) can coexist. The choice is often solution-specific, depending on whether guardrails are essential or fault tolerance is prioritised. Enterprise standards should guide orchestration selection to ensure AI solutions are secure and trustworthy. For example, base orchestration could be used for low-risk use cases and supervisory orchestration for high-risk scenarios.