PandaProbe is an open-source agent engineering platform created by Chirpz AI designed to provide deep observability into AI agent applications. It serves as a unified solution for tracing, evaluating, monitoring, and debugging agents throughout their development and production lifecycles. By capturing the full lifecycle of an agent—including LLM calls, tool invocations, sub-agent handoffs, and custom logic—PandaProbe allows developers to maintain complete oversight of complex agentic workflows.

The platform operates by generating structured traces and spans, which are then aggregated into sessions. This session-level context is critical for understanding long-running agent trajectories. Beyond basic logging, the platform features sophisticated evaluation capabilities, utilizing state-of-the-art metrics to score agent behavior, detect uncertainty, and pinpoint drifts across the entire lifespan of a session. It provides both trace-level metrics, such as task completion and tool correctness, and session-level metrics, such as reliability and consistency, to ensure agents remain safe and performant before reaching users.

Some of the key features are:

• Tracing: Capture comprehensive trajectories including LLM calls, tool usage, and custom logic via zero-code wrappers or framework-specific integrations.
• SOTA Metrics: Utilize research-grounded evaluation metrics purpose-built for long-running agents to detect uncertainty and score behavior.
• Monitoring: Schedule recurring eval runs against production traffic to detect behavioral drift and performance regressions automatically.
• Agent Native Integration: Utilize ready-made skills for coding agents, enabling natural language management of traces and evals through terminal-based CLI tools.
• Framework Support: Leverage native instrumentation for leading agent frameworks like LangGraph, LangChain, CrewAI, and major LLM providers.
• Session Management: Aggregate related traces under a single session to understand the entire decision-making lifecycle of an agent.
• Flexible Deployment: Choose between managed Cloud hosting or self-hosted options under an Apache 2.0 license.

PandaProbe is used by installing the Python SDK with specific extras tailored to the agent framework or LLM provider in use. It functions by instrumenting the application code, which then streams telemetry data to the PandaProbe platform. Once configured, developers can run evaluations either through the dashboard for manual analysis or via APIs and CLI commands for CI/CD integration. The platform's automated monitoring capability allows teams to set cadence-based evals, ensuring continuous quality assurance for agent versions in production.

Some common use cases include:

• Debugging Agent Drifts: Pinpoint the exact step in a long trajectory where an agent began to drift from its task or plan.
• Production Monitoring: Automatically detect regressions in agent logic by scheduling periodic evaluations against production traffic.
• Validating Tool Selection: Assess whether an agent is correctly choosing, over-selecting, or mis-selecting tools based on user input.
• Improving Agent Reliability: Utilize session-level metrics to identify high-risk traces and improve the consistency of multi-step agent workflows.
• Automated CI/CD Testing: Script evaluation workflows in deployment pipelines to catch issues before new agent versions reach end users.