Evaluation Components

Inputs

Base Input Variables

Every evaluation requires the following variables:

Input Variable	Description
`model_id`	The Atla evaluator model to use. See our models page for details.
`model_input`	The user input to your GenAI app (e.g., a question, instruction, or chat dialogue).
`model_output`	The response from your LLM.

Evaluation Metrics vs Evaluation Criteria

After selecting an evaluator model and specifying the LLM interaction, you need to define what to evaluate. Choose one of these approaches:

Evaluation metrics

Use a prompt that captures a specific metric (e.g., logical_coherence) to evaluate the LLM interaction. You can use our default metrics or create your own custom metrics:

Input Variable	Description
`metric_name`	The name of the metric to use. See our metrics page for details.

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
)

Evaluation criteria

For rapid experimentation or to use an existing evaluation prompt, provide evaluation criteria directly:

Input Variable	Description
`evaluation_criteria`	A prompt instruction defining how to evaluate the LLM interaction.

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    evaluation_criteria="Rate the response's relevance to the user's query on a scale of 1 to 5 where 1 is not relevant and 5 is highly relevant.",
)

Use either an evaluation metric or evaluation criteria, not both.

Additional Inputs

Depending on your evaluation, you may need to provide additional inputs.

RAG Contexts

For RAG evaluations, provide the context available to the model:

Input Variable	Description
`model_context`	The context provided to the LLM for grounding.

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_faithfulness",
    model_context="France is a country in Western Europe. Paris is the capital and largest city of France. Located on the Seine River, Paris is known for landmarks like the Eiffel Tower and Louvre Museum.",
)

Reference Answers

Using reference answers is recommended for evaluation when available:

Input Variable	Description
`expected_model_output`	A reference “ground truth” that meets the evaluation criteria.

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_correctness",
    expected_model_output="The capital of France is Paris.",
)

Few-Shot Examples

Providing few-shot examples is one of the best ways to align your evaluation, regardless of your use case:

Input Variable	Description
`few_shot_examples`	A list of examples with known evaluation scores.

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
    few_shot_examples=[
        {
            "model_input": "What is the capital of Germany?",
            "model_output": "Berlin",
            "score": "5",
        },
        {
            "model_input": "What is the capital of Italy?",
            "model_output": "Italy is known for its rich history and culture.",
            "score": "1",
        },
    ],
)

Evaluator Output

Each evaluation produces two outputs:

Output Variable	Description
`score`	A numerical score indicating how well the LLM interaction meets the criteria.
`critique`	A brief explanation justifying the score.

See our API reference for complete details.

Atla models generate the critique before deciding on a score.

Get Started

Build with Atla

Integrations

Self-deployment

Evaluation Components

Inputs

Base Input Variables

Evaluation Metrics vs Evaluation Criteria

Additional Inputs

RAG Contexts

Reference Answers

Few-Shot Examples

Evaluator Output

Get Started

Build with Atla

Integrations

Self-deployment

​Inputs

​Base Input Variables

​Evaluation Metrics vs Evaluation Criteria

​Additional Inputs

​RAG Contexts

​Reference Answers

​Few-Shot Examples

​Evaluator Output

Inputs

Base Input Variables

Evaluation Metrics vs Evaluation Criteria

Additional Inputs

RAG Contexts

Reference Answers

Few-Shot Examples

Evaluator Output