API Documentation
Parallel Evaluations
The Atla REST API is capable of handling multiple requests in an asynchronous manner as well as in a parallel manner, useful for when you want to run evaluations over a large batch of data.
Async Usage
import asyncio
from atla import AsyncAtla
client = AsyncAtla()
async def main() -> None:
evals = await client.evaluation.create(
input="If 5 machines take 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?",
response="Total processing time is 5 widgets * 5 minutes = 25 minutes. Then 25 mins / 100 machines = 15 seconds.",
metrics=["logical_coherence"]
)
print(f"Score: {evals.evaluations['logical_coherence'].score}")
asyncio.run(main())
The functionality between the synchronous and asynchronous clients is otherwise identical.
Parallel Evaluations
Using the asynchronous capability of the API, you can send multiple evaluations in parallel and gather the results together, significantly speeding up evaluations across a large dataset.
import asyncio
import pandas as pd
from atla import AsyncAtla
client = AsyncAtla()
# loading a sample dataset of n = 11 evals
data_url = "https://raw.githubusercontent.com/atla-ai/eval_demo_data/main/yc_launch_demo_data.csv"
data = pd.read_csv(data_url)
async def send_eval(index, input, response, reference):
return index, await client.evaluation.create(input=input, response=response, reference=reference, metrics=["logical_coherence"])
async def main():
tasks = []
for index, row in data.iterrows():
tasks.append(send_eval(index, row["Question"], row["AI's Response"], row["Reference Response"]))
results = await asyncio.gather(*tasks)
for index, evals in results:
print(f"Index: {index}, Score: {evals.evaluations['logical_coherence'].score}")
asyncio.run(main())
When making large, repeated batch requests to the API, you may encounter internal rate limits. More information about our rate limiting plans are available here.