Atla provides simple out of the box metrics that can be used to perform base evaluations of your LLM application. Atla will score results on a Likert scale (1-5).
You can pass any or all of these metrics to the evaluate function of the Atla client and it will perform independent evaluations over the input and response provided.
Metric Name
Description
recall
Measures how completely the response captures the key facts and details.
precision
Assesses the relevance of all the information in the response.
logical_coherence
Measures the logical flow, consistency, and rationality of the response.
The recall score measures how well a response covers the requested input. It is calculated by the proportion of required content included. A high score indicates a comprehensive response, while a low score indicates an incomplete response. This score reflects the thoroughness of the information provided.
Recall=∣Total number of distinct, relevant claims that could be made to directly answer the user query∣∣Number of claims in the AI answer that address the user query∣
Evaluates the specificity and directness of the information provided in the response. The precision score is based on the ratio of relevant claims to the total content in the response. A high score signifies that the content is specifically relevant to the input with minimal extraneous information. A low score indicates the presence of superfluous or less relevant details. This score focuses on the accuracy and relevance of the details within the answer, not the volume of information.
Precision=∣Total number of claims in the AI answer∣∣Number of claims in the AI answer that directly address the user query∣
Determines the logical flow, consistency, and rationality of the response. A high score means that the content is highly logical and coherent, with a clear, consistent, and rational progression of ideas from beginning to end, free of any significant contradictions or illogical statements. A low score signifies that the content demonstrates significant logical inconsistencies or contradictions, lacking a coherent flow of ideas.
