Overview - Atla

Inputs

Base Input Variables

Every evaluation requires the following variables:

Input Variable	Description
`model_id`	The Atla evaluator model to use. See our models page for details.
`model_input`	The user input to your GenAI app (e.g., a question, instruction, or chat dialogue).
`model_output`	The response from your LLM.

Evaluation Metrics vs Evaluation Criteria

After selecting an evaluator model and specifying the LLM interaction, you need to define what to evaluate.

Metrics or evaluation critieria that you create and refine for one model are optimized for that model. We advise against using models interchangeably on the same criteria without testing.

Use either an evaluation metric or evaluation criteria, not both.

Choose one of these approaches:

Evaluation metrics

Use a prompt that captures a specific metric (e.g., logical_coherence) to evaluate the LLM interaction. You can use our default metrics or create your own custom metrics.

Input Variable Description
metric_name The name of the metric to use. See our metrics page for details.

Input Variable	Description
`metric_name`	The name of the metric to use. See our metrics page for details.

Example with evaluation metrics

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
)

Evaluation criteria

For rapid experimentation or to use an existing evaluation prompt, provide evaluation criteria directly:

Input Variable Description
evaluation_criteria A prompt instruction defining how to evaluate the LLM interaction.

Input Variable	Description
`evaluation_criteria`	A prompt instruction defining how to evaluate the LLM interaction.

Example with evaluation criteria

from atla import Atla

client = Atla()

client.evaluation.create(
      model_id="atla-selene",
      model_input="What is the capital of France?",
      model_output="Paris",
      evaluation_criteria="Rate the response's relevance to the user's query on a scale of 1 to 5 where 1 is not relevant and 5 is highly relevant.",
)

While using atla-selene-mini as your evaluation model, we strongly advise that you use the recommended template for best results.

The evaluation_criteria template for Selene Mini has the following 3 components:

Description of the evaluation

List of scores and their corresponding criteria

A sentence that specifies constraints on the score. This sentence should contain the string Your score should be followed by the corresponding criteria for the binary or the Likert type.

Example evaluation_criteria for binary metrics

from atla import Atla

client = Atla()
evaluation_criteria = """Is the response correct?
0: It is not correct.
1: It is correct.

Your score should be 0 or 1.
"""

client.evaluation.create(
  model_id="atla-selene-mini",
  model_input="What is the capital of France?",
  model_output="Paris",
  evaluation_criteria=evaluation_criteria,
)

Example evaluation_criteria template for Likert metrics

from atla import Atla

client = Atla()
evaluation_criteria = """How concise is the response?
Score 1: Very verbose and has redundant information.
Score 2: Quite verbose, still has redundant information.
Score 3: Slightly verbose, has room for removing redundant information.
Score 4: Almost concise, but has at least one redundant information.
Score 5: Very concise and is to the point.

Your score should be an integer between 1 and 5.
"""

client.evaluation.create(
  model_id="atla-selene-mini",
  model_input="Tell me about the the capital of France.",
  model_output="Paris is the capital of France.",
  evaluation_criteria=evaluation_criteria,
)

Additional Inputs

Depending on your evaluation, you may need to provide additional inputs.

RAG Contexts

For RAG evaluations, provide the context available to the model:

Input Variable	Description
`model_context`	The context provided to the LLM for grounding.

Example with model context

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_faithfulness",
    model_context="France is a country in Western Europe. Paris is the capital and largest city of France. Located on the Seine River, Paris is known for landmarks like the Eiffel Tower and Louvre Museum.",
)

Reference Answers

Using reference answers is recommended for evaluation when available:

Input Variable	Description
`expected_model_output`	A reference “ground truth” that meets the evaluation criteria.

Example with expected model output

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_correctness",
    expected_model_output="The capital of France is Paris.",
)

Few-Shot Examples

Providing few-shot examples is one of the best ways to align your evaluation, regardless of your use case:

Input Variable	Description
`few_shot_examples`	A list of examples with known evaluation scores.

Example with few-shot examples

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
    few_shot_examples=[
        {
            "model_input": "What is the capital of Germany?",
            "model_output": "Berlin",
            "score": "5",
        },
        {
            "model_input": "What is the capital of Italy?",
            "model_output": "Italy is known for its rich history and culture.",
            "score": "1",
        },
    ],
)

Evaluator Output

Each evaluation produces two outputs:

Output Variable	Description
`score`	A numerical score indicating how well the LLM interaction meets the criteria.
`critique`	A brief explanation justifying the score.

Atla models generate the critique before deciding on a score.

On this page

Inputs
Base Input Variables
Evaluation Metrics vs Evaluation Criteria
Additional Inputs
RAG Contexts
Reference Answers
Few-Shot Examples
Evaluator Output

Inputs

Base Input Variables

Every evaluation requires the following variables:

Input Variable	Description
`model_id`	The Atla evaluator model to use. See our models page for details.
`model_input`	The user input to your GenAI app (e.g., a question, instruction, or chat dialogue).
`model_output`	The response from your LLM.

Evaluation Metrics vs Evaluation Criteria

After selecting an evaluator model and specifying the LLM interaction, you need to define what to evaluate.

Metrics or evaluation critieria that you create and refine for one model are optimized for that model. We advise against using models interchangeably on the same criteria without testing.

Use either an evaluation metric or evaluation criteria, not both.

Choose one of these approaches:

Evaluation metrics

Use a prompt that captures a specific metric (e.g., logical_coherence) to evaluate the LLM interaction. You can use our default metrics or create your own custom metrics.

Input Variable Description
metric_name The name of the metric to use. See our metrics page for details.

Input Variable	Description
`metric_name`	The name of the metric to use. See our metrics page for details.

Example with evaluation metrics

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
)

Evaluation criteria

For rapid experimentation or to use an existing evaluation prompt, provide evaluation criteria directly:

Input Variable Description
evaluation_criteria A prompt instruction defining how to evaluate the LLM interaction.

Input Variable	Description
`evaluation_criteria`	A prompt instruction defining how to evaluate the LLM interaction.

Example with evaluation criteria

from atla import Atla

client = Atla()

client.evaluation.create(
      model_id="atla-selene",
      model_input="What is the capital of France?",
      model_output="Paris",
      evaluation_criteria="Rate the response's relevance to the user's query on a scale of 1 to 5 where 1 is not relevant and 5 is highly relevant.",
)

While using atla-selene-mini as your evaluation model, we strongly advise that you use the recommended template for best results.

The evaluation_criteria template for Selene Mini has the following 3 components:

Description of the evaluation

List of scores and their corresponding criteria

A sentence that specifies constraints on the score. This sentence should contain the string Your score should be followed by the corresponding criteria for the binary or the Likert type.

Example evaluation_criteria for binary metrics

from atla import Atla

client = Atla()
evaluation_criteria = """Is the response correct?
0: It is not correct.
1: It is correct.

Your score should be 0 or 1.
"""

client.evaluation.create(
  model_id="atla-selene-mini",
  model_input="What is the capital of France?",
  model_output="Paris",
  evaluation_criteria=evaluation_criteria,
)

Example evaluation_criteria template for Likert metrics

from atla import Atla

client = Atla()
evaluation_criteria = """How concise is the response?
Score 1: Very verbose and has redundant information.
Score 2: Quite verbose, still has redundant information.
Score 3: Slightly verbose, has room for removing redundant information.
Score 4: Almost concise, but has at least one redundant information.
Score 5: Very concise and is to the point.

Your score should be an integer between 1 and 5.
"""

client.evaluation.create(
  model_id="atla-selene-mini",
  model_input="Tell me about the the capital of France.",
  model_output="Paris is the capital of France.",
  evaluation_criteria=evaluation_criteria,
)

Additional Inputs

Depending on your evaluation, you may need to provide additional inputs.

RAG Contexts

For RAG evaluations, provide the context available to the model:

Input Variable	Description
`model_context`	The context provided to the LLM for grounding.

Example with model context

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_faithfulness",
    model_context="France is a country in Western Europe. Paris is the capital and largest city of France. Located on the Seine River, Paris is known for landmarks like the Eiffel Tower and Louvre Museum.",
)

Reference Answers

Using reference answers is recommended for evaluation when available:

Input Variable	Description
`expected_model_output`	A reference “ground truth” that meets the evaluation criteria.

Example with expected model output

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_correctness",
    expected_model_output="The capital of France is Paris.",
)

Few-Shot Examples

Providing few-shot examples is one of the best ways to align your evaluation, regardless of your use case:

Input Variable	Description
`few_shot_examples`	A list of examples with known evaluation scores.

Example with few-shot examples

from atla import Atla

client = Atla()

client.evaluation.create(
    model_id="atla-selene",
    model_input="What is the capital of France?",
    model_output="Paris",
    metric_name="atla_default_relevance",
    few_shot_examples=[
        {
            "model_input": "What is the capital of Germany?",
            "model_output": "Berlin",
            "score": "5",
        },
        {
            "model_input": "What is the capital of Italy?",
            "model_output": "Italy is known for its rich history and culture.",
            "score": "1",
        },
    ],
)

Evaluator Output

Each evaluation produces two outputs:

Output Variable	Description
`score`	A numerical score indicating how well the LLM interaction meets the criteria.
`critique`	A brief explanation justifying the score.

Atla models generate the critique before deciding on a score.

On this page

Inputs
Base Input Variables
Evaluation Metrics vs Evaluation Criteria
Additional Inputs
RAG Contexts
Reference Answers
Few-Shot Examples
Evaluator Output

​Inputs

​Base Input Variables

​Evaluation Metrics vs Evaluation Criteria

​Additional Inputs

​RAG Contexts

​Reference Answers

​Few-Shot Examples

​Evaluator Output

API Documentation

​Inputs

​Base Input Variables

​Evaluation Metrics vs Evaluation Criteria

​Additional Inputs

​RAG Contexts

​Reference Answers

​Few-Shot Examples

​Evaluator Output

Inputs

Base Input Variables

Evaluation Metrics vs Evaluation Criteria

Additional Inputs

RAG Contexts

Reference Answers

Few-Shot Examples

Evaluator Output

Inputs

Base Input Variables

Evaluation Metrics vs Evaluation Criteria

Additional Inputs

RAG Contexts

Reference Answers

Few-Shot Examples

Evaluator Output