How to Diagnose Failures in Multi-Agent AI Systems: A Step-by-Step Guide

Introduction

Large Language Model (LLM) multi-agent systems are increasingly used to tackle complex problems by having multiple specialized agents collaborate. However, these systems often fail due to miscommunication, errors by individual agents, or cascading mistakes across long interaction chains. Debugging such failures manually—by sifting through extensive logs—is like hunting for a needle in a haystack. To streamline this process, researchers from Penn State University, Duke University, Google DeepMind, and other institutions have introduced automated failure attribution, along with the first dedicated benchmark dataset Who&When. This guide will walk you through the steps to effectively diagnose and attribute failures in your multi-agent systems using their open-source methods.

What You Need

• A multi-agent system powered by LLMs (e.g., using frameworks like LangChain, AutoGen, or similar).
• Interaction logs from your system, ideally timestamped and agent-labeled.
• Basic knowledge of Python and machine learning concepts.
• Access to the Who&When dataset (available on Hugging Face) and the open-source code repository.
• Computing resources (GPU recommended for model training).
• Development environment with Python 3.8+, PyTorch, and required libraries (see repository for details).

Step-by-Step Guide

Step 1: Understand Your Multi-Agent System and Failure Scenarios

Before diving into attribution, clearly define your system architecture. Identify each agent’s role (e.g., planner, executor, critic), their communication channels, and the overall task objective. Note common failure patterns: a single agent outputting wrong information, agents misinterpreting messages, or information being lost in long chains. The original research emphasizes that failures can be attributed to a specific agent (Who) and a specific time (When). For example, if a planning agent suggests an incorrect action, the failure likely originates there. Document these scenarios to build a mental model of where failures typically occur.

Step 2: Collect and Structure Interaction Logs

Log every agent-to-agent and agent-to-environment interaction with timestamps, agent IDs, message content, and any intermediate outputs. A well-structured log format—such as JSON or CSV—makes subsequent analysis easier. Ensure logs include both successful and failed task executions. The Who&When dataset uses a structured log format that you can adapt. If your system already logs verbosely, parse it into this structured format. This step is crucial because the success of automated attribution depends on the quality and granularity of your logs.

Step 3: Define and Label Failure Instances

For each task run, determine whether the final outcome was a success or failure. If it failed, label it with the ground truth: which agent caused the failure and at which step. This labeling is necessary to train or evaluate attribution models. You can involve a domain expert to manually annotate a subset of logs. The Who&When benchmark provides such labels for generic multi-agent scenarios, but you may need to create a custom set for your domain. Consistent labeling helps in identifying patterns and training accurate attributors.

Step 4: Set Up the Automated Attribution Pipeline

Clone the open-source code repository. Install dependencies as specified in the requirements.txt. The repository includes several attribution methods: causal tracing (backtracking from failure to likely cause), agent-sensitivity analysis (perturbing agent outputs and measuring impact), and language model classifiers that directly read logs. Each method can be run on the Who&When dataset or your own data. Start by running the provided scripts to understand how they work. Adjust hyperparameters (e.g., number of test cases, threshold for attribution) to match your system’s complexity.

Step 5: Apply Attribution to Your System

Feed your own interaction logs into the pipeline. The tools will output a ranked list of potential failure sources (agent + timestamp). Compare these predictions against your ground truth labels (if available) to measure accuracy, precision, and recall. If you don’t have labels, start by manually inspecting the top predictions to verify plausibility. The paper recommends evaluating with metrics like Top-1 accuracy and Mean Reciprocal Rank (MRR). Iterate: if results are poor, consider adding more logging detail, adjusting the attribution method, or training a domain-specific classifier using the provided codebase.

Step 6: Iterate and Optimize Your System

Once you identify recurring failure sources, take corrective actions: re-train a specific agent, improve inter-agent communication protocols, or add error-checking mechanisms. Re-run the attribution pipeline after modifications to confirm that failures are reduced or shifted. The goal is to create a feedback loop where failure attribution informs system improvement, making your multi-agent system more robust. The Who&When dataset itself can be expanded by adding your own scenarios and sharing them with the community, as the researchers have open-sourced everything.

Tips for Success

• Start small: Apply attribution to a single, well-understood failure mode before scaling to all possible errors.
• Automate logging: Build logging directly into your multi-agent framework to avoid manual data extraction.
• Use the benchmark: The Who&When dataset provides a controlled environment to test your pipeline before deploying on your own system.
• Collaborate: Since this is cutting-edge research (accepted at ICML 2025), engage with the community via the GitHub repository for tips and updates.
• Document everything: Keep records of your attribution runs and system changes to track progress over time.

Conclusion

Automated failure attribution is a powerful new approach to debugging LLM multi-agent systems. By following these steps—understanding your system, collecting structured logs, labeling failures, setting up the open-source attribution pipeline, and iteratively improving—you can drastically reduce the time spent on manual log archaeology. The research from Penn State University, Duke University, and collaborators provides both the theoretical foundation and practical tools to get started. Embrace this methodology to build more reliable and intelligent multi-agent collaborations.