Open Source Summit India 2026

Current evaluation methods for AI agents, which depend on static benchmarks, are in adequate for real-world systems. This session reveals why accuracy-centric evaluations create a dangerous illusion of readiness and how real systems—shifting APIs, incomplete signals, tool failures, cascading dependencies, and environment drift—expose brittleness that benchmarks never measure. We introduce and demonstrate a chaos engineering driven AgentCert approach as an open source project, developed through a collaborative effort of Infosys and Microsoft, that evaluates agents the way real infrastructure tests resilience: by injecting controlled faults, observing recovery behavior, and measuring adaptation under stress. Participants will learn how scenario-based fault models, workload-coupled tests, and observability-rich metrics uncover latent weaknesses and emergent behaviors. The session presents a practical, open, and repeatable framework for certifying agents on robustness, not just correctness—empowering teams to build AI agents that remain stable, safe, and dependable when reality gets messy. See https://github.com/AgentCert for more details.