# AI-Assisted Integration Source: https://docs.atla-ai.com/ai-integration This guide helps AI coding assistants like Claude Code / Cursor agent automatically integrate the Atla Insights SDK into your Python codebase. For Python users, we provide a [onboarding.txt](https://github.com/atla-ai/atla-insights-sdk/blob/main/onboarding.txt) file and a [CLAUDE.md](https://github.com/atla-ai/atla-insights-sdk/blob/main/CLAUDE.md) file you can use to automatically hook up your agent to Atla. Just copy, paste, and run. For live assistance with onboarding, you can [book an onboarding slot](https://calendly.com/d/csym-9kk-bbk/insights-ui-onboarding) with our team. # Data Model Source: https://docs.atla-ai.com/api-reference/data-model The core data model for developers consuming AI observability data through Atla's SDK and APIs. ## Overview This guide explains the core data model for developers consuming AI observability data through Atla's SDK and APIs. Our data model is built on OpenTelemetry standards and captures the complete lifecycle of AI applications from traces to custom metrics. ## Traces A **Trace** represents a complete AI interaction or workflow - typically a full conversation or agent execution. ### Trace Structure ```jsonc icon="code" lines theme={null} { "id": "string", // Unique trace identifier "environment": "string", // Deployment environment (PROD | DEV) // Execution metadata "isSuccess": "boolean | null", // Overall success status "isCompleted": "boolean", // Whether trace finished // Timing information "startedAt": "string", // Trace start time (ISO 8601) "endedAt": "string", // Trace completion time (ISO 8601) // Custom metadata "metadata": "{ [key: string]: string } | null", // Custom trace metadata // Related data "spans?": "Span[]", // Individual AI operations "customMetricValues?": "CustomMetricValue[]" // Quality metrics } ``` ## Spans A **Span** represents an individual operation within a trace - typically a single AI model call, tool execution, or processing step. ### Span Structure ```jsonc icon="code" lines theme={null} { "id": "string", // Unique span identifier "traceId": "string", // Parent trace reference // Span hierarchy "parentSpanId": "string | null", // Parent span in call stack // Span metadata "spanName": "string | null", // Operation name (e.g., "openai.chat.completions") // Timing "startTimestamp": "string", // Operation start (ISO 8601) "endTimestamp": "string", // Operation completion (ISO 8601) // OpenTelemetry data "otelEvents": "any[]", // GenAI events (messages, choices) // Analysis data "isException": "boolean | null", // Whether span had errors // Related data (when included) "annotations?": "SpanAnnotation[]" // Issue analysis } ``` ## Span Annotations **Span Annotations** provide AI-powered analysis of potential issues, failures, or interesting patterns within individual spans. ### Annotation Structure ```jsonc icon="code" lines theme={null} { "id": "string", // Unique annotation ID "spanId": "string", // Target span reference // Issue classification "failureMode": "string", // Type of issue detected // Atla's AI analysis "atlaCritique": "string" // AI-generated explanation } ``` ## Custom Metrics **Custom Metrics** capture quantitative quality measurements across traces, such as user satisfaction scores, factual accuracy ratings, or business-specific KPIs. ### Custom Metric Definition ```jsonc icon="code" lines theme={null} { "id": "string", // Unique metric ID "name": "string", // Metric name (unique per org) // Metric configuration "dataType": "string" // Value format (BOOLEAN | LIKERT_1_TO_5) } ``` ### Custom Metric Values ```jsonc icon="code" lines theme={null} { "id": "string", // Unique value ID "customMetricId": "string", // Parent metric reference "traceId": "string", // Target trace // Metric result "value": "string", // The measured value ("true", "3", etc.) // Related data (when included) "customMetric?": "CustomMetric" // The metric definition } ``` # Get Trace Source: https://docs.atla-ai.com/api-reference/get-trace api-reference/openapi.yaml get /api/sdk/v1/traces/{traceId} Get a single trace by ID Retrieve a specific trace by its unique identifier. Returns complete trace data including all spans, annotations, trace summary, and custom metric values. # Get Traces Source: https://docs.atla-ai.com/api-reference/get-traces api-reference/openapi.yaml get /api/sdk/v1/traces Get traces Retrieve specific traces by providing an array of trace IDs. Returns complete trace data including spans, summaries, and custom metrics for all found traces. # Get Multiple Traces by IDs Source: https://docs.atla-ai.com/api-reference/get-traces-by-ids api-reference/openapi.yaml get /api/sdk/v1/traces/ids Get multiple traces by IDs Retrieve specific traces by providing an array of trace IDs. Returns complete trace data including spans, summaries, and custom metrics for all found traces. # OpenTelemetry GenAI Events Source: https://docs.atla-ai.com/api-reference/otel-events How to interpret and work with the GenAI event data available through Atla Insights. ## Overview Atla Insights captures generative AI interactions using the [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), an emerging standard for observing AI systems. This guide explains how to interpret and work with the GenAI event data available through our platform. ## What are OpenTelemetry GenAI Events? OpenTelemetry GenAI events are structured telemetry data that capture the inputs, outputs, and metadata of generative AI operations. They provide a standardized way to observe AI system behavior across different providers (OpenAI, Anthropic, etc.) while maintaining consistent data structures. Each event represents a discrete piece of an AI interaction: * **Message events** capture individual messages in a conversation * **Choice events** capture the AI's response options and selections ## Standard Background The OpenTelemetry GenAI semantic conventions are actively developed by major AI and observability companies including OpenAI, Google, Microsoft, and Amazon. The standard aims to: * Provide consistent telemetry across AI providers * Enable privacy-conscious AI observability * Support both simple chat and complex agent workflows * Allow vendor-specific extensions while maintaining interoperability ## Event Types and Structure ### Message Events Message events follow the pattern `gen_ai.{role}.message` and capture conversation flow: #### `gen_ai.system.message` System prompts and instructions that set AI behavior. ```jsonc icon="code" lines theme={null} { "event_name": "gen_ai.system.message", "body": { "content": "string", "name?": "string" // Optional system message identifier } } ``` #### `gen_ai.user.message` User inputs to the AI system. ```jsonc icon="code" lines theme={null} { "event_name": "gen_ai.user.message", "body": { "content": "string | null", // null for multimodal requests "name?": "string" // Optional user identifier } } ``` #### `gen_ai.assistant.message` AI responses, including tool/function calls. ```jsonc icon="code" lines theme={null} { "event_name": "gen_ai.assistant.message", "body": { "content?": "string | null", // Response text "tool_calls?": "ToolCall[]", // Function calls made "name?": "string", // Assistant identifier "refusal?": "string" // Refusal reason } } ``` #### `gen_ai.tool.message` Results from tool/function executions. ```jsonc icon="code" lines theme={null} { "event_name": "gen_ai.tool.message", "body": { "content": "string", // Tool execution result "id": "string | null" // Tool call identifier } } ``` ### Choice Events #### `gen_ai.choice` Captures AI response selection and metadata. ```jsonc icon="code" lines theme={null} { "event_name": "gen_ai.choice", "body": { "index": "number", // Choice index (0-based) "finish_reason": "string | null", // Why generation stopped "message": { // The selected response "content?": "string", "tool_calls?": "ToolCall[]", "annotations?": "Annotation[]" }, "tools?": "ChatCompletionToolParam[]" // Available tools (Atla extension) } } ``` ### Common Metadata All events include: * **`event_timestamp`**: ISO 8601 timestamp * **`attributes`**: Contains `gen_ai.system` (required) and other metadata * **`source`**: Optional span ID reference ## Relationship to OpenAI's Data Model OpenTelemetry GenAI events map closely to OpenAI's chat completion structure: ### OpenAI → OpenTelemetry Mapping **OpenAI Messages Array**: ```jsonc icon="code" lines theme={null} [ { "role": "system", "content": "..." }, { "role": "user", "content": "..." }, { "role": "assistant", "content": "...", "tool_calls": ["..."] } ] ``` **Becomes OpenTelemetry Events**: ```jsonc icon="code" lines theme={null} [ { "event_name": "gen_ai.system.message", "body": { "content": "..." } }, { "event_name": "gen_ai.user.message", "body": { "content": "..." } }, { "event_name": "gen_ai.assistant.message", "body": { "content": "...", "tool_calls": ["..."] } } ] ``` **OpenAI Choices** → `gen_ai.choice` events with finish reasons, tool calls, and response metadata. ## Interpreting Event Data ### Conversation Flow Reconstruct conversations by ordering message events by timestamp: 1. System messages establish context 2. User/assistant messages alternate in conversation 3. Tool messages provide function call results 4. Choice events show response selection process ### Tool/Function Usage Track AI function calling through the event sequence: 1. `gen_ai.assistant.message` with `tool_calls` → AI requests function execution 2. `gen_ai.tool.message` with matching `id` → Function execution result 3. Next `gen_ai.assistant.message` → AI processes function result # Overview Source: https://docs.atla-ai.com/api-reference/overview Overview of the Atla Insights Data API The Atla Insights Data API provides programmatic access to project data in Atla's platform for monitoring and improving AI agents. You can interact with the Data API through HTTP requests from any programming language or through our [Python SDK](https://github.com/atla-ai/atla-insights-sdk) or [TypeScript SDK](https://github.com/atla-ai/atla-insights-sdk-js). For details on the data model, see the [documentation](/api-reference/data-model). ## Authentication All API requests must include an API key for authorization. You can get an API key from your Insights dashboard. For HTTP requests, include an `Authorization` header as follows: ```http theme={null} Authorization: Bearer $ATLA_INSIGHTS_TOKEN ``` Or use our SDK for authentication. ## Content types The Atla API accepts and returns JSON in request and response bodies. Include the following header in your API requests: ```http theme={null} Content-Type: application/json ``` As with authorization, the Python SDK handles content type headers automatically. # Platform Comparison Source: https://docs.atla-ai.com/comparison Compare Atla, Langfuse, and LangSmith for AI agent monitoring and evaluation ## Introduction This page provides a comparison of Atla, Langfuse, and LangSmith, three platforms used to monitor and evaluate AI agents. Atla focuses on proactively detecting recurring failure patterns and surfacing insights, while Langfuse and LangSmith emphasize observability and dataset management. ## Platform Overview **Atla**: Evaluation platform for agentic systems. Detects recurring failure patterns from prompts, tools, and user interactions, with trace summaries and step-level annotations. Supports custom LLM-as-a-judge metrics, surfaces problems proactively before customers notice, and reduces debugging time by up to 5×. **Langfuse**: Open-source observability with trace logging, cost/latency metrics, prompt & dataset management, and LLM-as-a-judge evaluations. Available as self-hosted or managed cloud. **LangSmith**: Closed-source observability platform tightly integrated with LangChain. Provides tracing, cost/latency metrics, prompt & dataset management, and LLM-as-a-judge evaluations in a polished SaaS. Atla can run alongside Langfuse or LangSmith to deliver deeper insights and accelerate teams getting agents production-ready. ## Feature Comparison | **Feature** | **Atla** | **Langfuse** | **LangSmith** | | ------------------------------------- | --------------------------------------------------------- | --------------------------------------------------- | --------------------------------------------- | | **LLM Trace Logging & Visualisation** | ✅ Trace logging with summaries and error annotations | ✅ Trace inspection & dashboards | ✅ Trace inspection & dashboards | | **Automatic Failure Mode Detection** | ✅ Auto-clusters recurring failure patterns | ❌ Manual inspection | ❌ Manual inspection | | **Automated Fix Suggestions** | ✅ Actionable recommendations to improve reliability | ❌ Developer-led | ❌ Developer-led | | **LLM Evaluation** | ✅ Includes LLM-as-a-judge | ✅ Includes LLM-as-a-judge | ✅ Includes LLM-as-a-judge | | **Token/Cost & Latency Tracking** | ➖ Latency tracked; focus on failure analysis | ✅ Detailed per call | ✅ Detailed per call | | **Prompt Management** | ➖ In development. Playground in beta | ✅ Versioning & playground | ✅ Versioning & playground | | **Dataset Management** | ❌ In development | ✅ Datasets for evals | ✅ Datasets for evals | | **SDKs / API for Integration** | ✅ Python & TS SDKs; integrates with other tools | ✅ Python/JS, OpenTelemetry, API | ✅ LangChain callbacks, Python/TS, API | | **Multi-user Collaboration & RBAC** | ➖ Supports multiple organisations; users invited per org | ✅ Collaboration; RBAC in enterprise | ✅ Team management & SSO/SAML on enterprise | | **Security & Compliance** | SOC 2 Type I, HIPAA, GDPR; DPA available | ISO 27001 & SOC 2 Type II; GDPR; EU/US hosting | SOC 2 Type II, HIPAA, GDPR; EU data residency | | **Deployment Option** | SaaS by default; self-hosting for enterprise upon request | Open-source (MIT core) + self-host or managed cloud | SaaS by default; self-hosting for Enterprise | ## Conclusion Atla is designed to go beyond observability by automatically surfacing recurring failure patterns and suggesting actionable fixes, helping teams reduce debugging time and improve reliability. Unlike Langfuse or LangSmith, it focuses less on dataset and prompt management, and more on ensuring agents perform consistently in production. Teams can use Atla on its own as a dedicated reliability layer or seamlessly alongside Langfuse and LangSmith to complement their existing observability setup. This flexibility makes Atla well-suited for teams that want to move faster without compromising on quality. # Coding agents Source: https://docs.atla-ai.com/integrations/coding-agents Integration for coding agents to use Atla's insights to auto-improve your AI application # MCP Server for Coding Agents We're building an **MCP server** that enables your coding agent to connect directly to Atla's insights and automatically improve your AI application. ## How it works Our MCP server allows your coding agent to: * **Data-driven improvements**: Atla identifies systematic issues in your AI agent and provides this context to your coding agent, enabling well-reasoned fixes based on real usage data. * **Early issue detection**: Through Atla's error detection capabilities, your coding agent can spot problems in your AI application and take action before they escalate. * **Continuous quick fixes**: Atla recommends targeted improvements for your coding agent to implement in your AI application, enabling rapid iteration cycles. ## Register your interest Interested in joining our private beta for this feature? [Sign up here](https://form.typeform.com/to/hbKf89Mk) to get early access! # Python Integrations Source: https://docs.atla-ai.com/integrations/python Supported frameworks and LLM providers for Atla Insights in Python Atla Insights supports a wide range of LLM providers and AI agent frameworks. All instrumentation methods share a common interface for easy integration. ## LLM Providers We currently support the following LLM providers: | Provider | Instrumentation Function | Notes | | ---------------- | ------------------------- | ------------------------------------------------------ | | **Anthropic** | `instrument_anthropic` | Also supports `AnthropicBedrock` client from Anthropic | | **Bedrock** | `instrument_bedrock` | | | **Google GenAI** | `instrument_google_genai` | E.g., Gemini | | **LiteLLM** | `instrument_litellm` | Supports all available models in the LiteLLM framework | | **OpenAI** | `instrument_openai` | Includes Azure OpenAI | ### Basic Provider Usage ```python theme={null} from atla_insights import configure, instrument_openai from openai import OpenAI configure(token="") instrument_openai() client = OpenAI() # All OpenAI calls are now instrumented ``` By default, instrumented LLM calls will be treated independently from one another. To logically group LLM calls into a trace, use the `@instrument` decorator. ## Agent Frameworks We currently support the following frameworks: | Framework | Instrumentation Function | Notes | | -------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | **Agno** | `instrument_agno` | Supported with `openai`, `google-genai`, `litellm` and/or `anthropic` models | | **BAML** | `instrument_baml` | Supported with `openai`, `anthropic` or `bedrock` models | | **Claude Agent SDK** | `instrument_claude_agent_sdk` | | | **Claude Code SDK** | `instrument_claude_code_sdk` | | | **CrewAI** | `instrument_crewai` | | | **Google AI SDK** | `instrument_google_adk` | | | **LangChain** | `instrument_langchain` | This includes e.g., LangGraph as well | | **MCP** | `instrument_mcp` | Only includes context propagation. You will need to instrument the model calling the MCP server separately. | | **OpenAI Agents** | `instrument_openai_agents` | Supported with `openai`, `google-genai`, `litellm` and/or `anthropic` models | | **Pydantic AI** | `instrument_pydantic_ai` | | | **Smolagents** | `instrument_smolagents` | Supported with `openai`, `google-genai`, `litellm` and/or `anthropic` models | ### Framework + Provider Instrumentation For Agno, BAML, OpenAI Agents and Smolagents in Python, you will need to instrument both the framework *and* the underlying LLM provider(s). ```python theme={null} from atla_insights import configure, instrument_agno configure(token="") # If you are using a single LLM provider (e.g., via `OpenAIChat`) instrument_agno("openai") # If you are using multiple LLM providers (e.g., `OpenAIChat` and `Claude`) instrument_agno(["anthropic", "openai"]) ``` ## Can't find your framework or LLM provider? If you are using a framework or LLM provider without native support, you can manually record LLM generations via our lower-level SDK. ```python icon="python" lines theme={null} from atla_insights.span import start_as_current_span with start_as_current_span("my-llm-generation") as span: # Run my LLM generation via an unsupported framework. input_messages = [{"role": "user", "content": "What is the capital of France?"}] tools = [ { "type": "function", "function": { "name": "get_capital", "parameters": {"type": "object", "properties": {"country": {"type": "string"}}}, }, } ] result = my_client.chat.completions.create(messages=input_messages, tools=tools) # Manually record LLM generation. span.record_generation( input_messages=input_messages, output_messages=[choice.message for choice in result.choices], tools=tools, ) ``` Note that all arguments are expected to be in an OpenAI-compatible format. See the relevant OpenAI documentation for more details: * [input\_messages](https://platform.openai.com/docs/api-reference/chat/create#chat_create-messages) * [output\_messages](https://platform.openai.com/docs/api-reference/chat/object#chat/object-choices-message) * [tools](https://platform.openai.com/docs/api-reference/chat/create#chat_create-tools) (optional but recommended) Feel free to let us know which frameworks and LLM providers you would like to see supported! [Schedule a call with the Atla team](https://calendly.com/d/csym-9kk-bbk/insights-ui-onboarding) # TypeScript Integrations Source: https://docs.atla-ai.com/integrations/typescript Supported frameworks and LLM providers for Atla Insights in TypeScript Atla Insights supports a wide range of LLM providers and AI agent frameworks. All instrumentation methods share a common interface for easy integration. ## LLM Providers We currently support the following LLM providers: | Provider | Instrumentation Function | Notes | | ---------- | ------------------------ | --------------------- | | **OpenAI** | `instrumentOpenAI` | Includes Azure OpenAI | ## Agent Frameworks We currently support the following frameworks: | Framework | Instrumentation Function | Notes | | ----------------- | ------------------------ | ------------------------------------- | | **LangChain** | `instrumentLangChain` | This includes e.g., LangGraph as well | | **OpenAI Agents** | `instrumentOpenAIAgents` | | ## Can't find your framework or LLM provider? If you are using a framework or LLM provider without native support, you can manually record LLM generations via our lower-level SDK. ```typescript icon="square-js" lines theme={null} import { startAsCurrentSpan } from "@atla-ai/insights-sdk-js"; const { span, endSpan } = startAsCurrentSpan("my-llm-generation"); try { // Run my LLM generation via an unsupported framework. const inputMessages = [{ role: "user", content: "What is the capital of France?" }]; const tools = [ { type: "function", function: { name: "get_capital", parameters: { type: "object", properties: { country: { type: "string" } } }, }, }, ]; const result = await myClient.chat.completions.create({ messages: inputMessages, tools }); // Manually record LLM generation. span.recordGeneration({ inputMessages, outputMessages: result.choices.map(choice => choice.message), tools, }); } finally { endSpan(); } ``` Note that all arguments are expected to be in an OpenAI-compatible format. See the relevant OpenAI documentation for more details: * [input\_messages](https://platform.openai.com/docs/api-reference/chat/create#chat_create-messages) * [output\_messages](https://platform.openai.com/docs/api-reference/chat/object#chat/object-choices-message) * [tools](https://platform.openai.com/docs/api-reference/chat/create#chat_create-tools) (optional but recommended) Feel free to let us know which frameworks and LLM providers you would like to see supported! [Schedule a call with the Atla team](https://calendly.com/d/csym-9kk-bbk/insights-ui-onboarding) # Overview Source: https://docs.atla-ai.com/overview Learn about Atla

ProductHunt Badge

Atla is designed to clarify the messiness of agent behaviour. Automatically detect errors and get insights to continually improve agent performance. ## Why Atla? LLMs only reach their full potential through rapid iteration and continuous improvement. With a few lines of code, you can identify issues, track performance, and understand failure modes - enabling you to iterate faster and build better AI. If you are building agents, having a tight feedback loop between deployment and improvement is one of the most impactful things you can do. ## Product Walkthrough