# Error Codes
Source: https://docs.genai.scale.com/api-reference/error-codes
Common error codes returned by the API.
Our API uses standard HTTP response codes to indicate the success or failure of requests.
| Code | Description |
| ---- | ------------------------------------------------------------------------------------------------ |
| 200 | OK -- Everything worked as expected |
| 400 | Bad Request -- The request was unacceptable, often due to missing or invalid parameter |
| 401 | Unauthorized - Authentication failed |
| 402 | Not enabled -- Please contact Scale before using this endpoint |
| 404 | Not Found -- The requested resource doesn't exist |
| 409 | Conflict -- The provided idempotency key or unique\_id is already in use for a different request |
| 429 | Too Many Requests -- Too many requests hit the API too quickly |
| 500 | Internal Server Error -- We had a problem with our server. Contact Scale. |
## Response Code Categories
### 2xx - Success
Indicates that your request was successfully received, understood, and processed.
### 4xx - Client Errors
Indicates that the request contains invalid data or is improperly formed. These errors occur when:
* Required parameters are missing
* Authentication credentials are invalid
* Resource access is unauthorized
* A resource that is trying to be created already exists
### 5xx - Server Errors
Indicates issues with our servers. These errors are rare and typically temporary.
Please visit [status.scale.com](https://status.scale.com) for more information about our API status.
# Expanding Entities
Source: https://docs.genai.scale.com/api-reference/expanding-entities
Expand certain entities within the API response.
Certain entities are returned as a unique ID for brevity. These entities can be expanded into the underlying object when specified via the `expand` request parameter.
```bash Regular theme={null}
curl 'https://api.scale.com/v2/task'
```
```bash Expanded theme={null}
curl 'https://api.scale.com/v2/task?expand=project'
```
```json Regular theme={null}
{
"project": "abc123",
"batch": "efg456",
"delivery": "dce789",
...
}
```
```json Expanded theme={null}
{
"project": {
"id": "abc123",
"name": "My Project",
...
},
"batch": {
"id": "efg456",
"name": "my_batch_01",
},
"delivery": {
"id": "dce789",
"name": "delivery-math-01",
"task_count": 25,
"delivered_at": "2023-11-07T05:31:56Z",
"project": "abc123",
},
...
}
```
# Idempotent Requests
Source: https://docs.genai.scale.com/api-reference/idempotency
Idempotency for task creation endpoints.
Scale supports idempotency for task creation endpoints. This allows you to resubmit requests that may have failed in transit without risk of creating duplicate tasks.
To use this feature, a `unique_id` attribute is expected to be provided into the task creation payload. The client is responsible for ensuring the uniqueness of your chosen keys. We recommend using V4 UUIDs.
If we later receive a matching request with the same idempotency key, no additional task will be created and the endpoint will return a 409 error.
# Introduction
Source: https://docs.genai.scale.com/api-reference/introduction
Scale's information hierarchy is comprised of [Projects](/core-resources/project), [Batches](/core-resources/batch), and [Tasks](/core-resources/task). A Project consists of Batches and Tasks. Batches are an optional entity used to group Tasks within a Project. Tasks are a unit of work that needs to be completed by a contributor.
Within a Task, labeled data is represented in a hierarchy, where each level has relevant [Annotations](/core-resources/annotation) for that level. A Task consists of [Threads](/core-resources/thread), a Thread consists of [Turns](/core-resources/turn), and a Turn consists of [Messages](/core-resources/message).
More details about Scale's information hierarchy can be found under [Core Resources](/core-resources).
# Start an evaluation job
Source: https://docs.genai.scale.com/auto-evaluation/autoevals
POST /v2/autoevals
Submits an evaluation job for a model version config.
# Retrieve evaluation jobs for a Model Version Config
Source: https://docs.genai.scale.com/auto-evaluation/model-statuses
GET /v2/autoevals/model_version_configs/{id}/statuses
# Retrieve evaluation results
Source: https://docs.genai.scale.com/auto-evaluation/results
GET /v2/autoevals/{jobId}/results
Fetches the results of a completed or in-progress evaluation job. Accuracy, histogram, and errored response count values are only available for completed jobs.
# Retry an evaluation job
Source: https://docs.genai.scale.com/auto-evaluation/retry
POST /v2/autoevals/retry
Retries an evaluation job.
# Retrieve evaluation job details
Source: https://docs.genai.scale.com/auto-evaluation/status
GET /v2/autoevals/{jobId}/status
Fetches the details of a specific evaluation job.
# Retrieve statuses of all evaluation jobs
Source: https://docs.genai.scale.com/auto-evaluation/statuses
GET /v2/autoevals/statuses
Fetches the status of all evaluation jobs.
# Terminate an evaluation job
Source: https://docs.genai.scale.com/auto-evaluation/terminate
POST /v2/autoevals/terminate
Terminates an evaluation job.
# Annotation
Source: https://docs.genai.scale.com/core-resources/annotation
An Annotation represent a piece of labeled data.
Annotations exist within a Task and are the end-result from contributors. An Annotation includes a unique identifier for the annotation, a human-readable key, the value of the annotation, all possible values for the annotation, and any other information provided to the contributor (question title, description, etc).
# Batch
Source: https://docs.genai.scale.com/core-resources/batch
Batches organize [Tasks](/core-resources/task) within a [Project](/core-resources/project).
For high-volume projects, batches can optionally be used to further divide work inside a project. When organizing tasks into batches you can consider one of the following approaches:
* Specific datasets: You can organize batches according to the source datasets you use. This approach enables precise control over the number of tasks processed from each dataset.
* Model versions: You can organize batches according to the model versions you use internally. This approach enables precise control over the number of tasks processed from each model version (or pairs in case of an eval prokect)
* Language: You can organize batches according to the language of the tasks.
* Time: If your data structure will be the same throughout the lifespan of the project, you can organize batches according to the time they were created.
# Chunk
Source: https://docs.genai.scale.com/core-resources/chunk
A Chunk is a piece of content and [Annotations](/core-resources/annotation) that are relevant to that Chunk.
A Chunk is a subset of a [Message](/core-resources/message). [Annotations](/core-resources/annotation) at the Chunk level directly apply to that part of the [Message](/core-resources/message). For example, a single sentence in a paragraph could be a Chunk with its own Annotations.
# Dataset
Source: https://docs.genai.scale.com/core-resources/dataset
Pre-built collections of data created by subject matter experts to accelerate model development.
A Dataset is a pre-built collection of data created by subject matter experts to accelerate model development.
# Dataset delivery
Source: https://docs.genai.scale.com/core-resources/dataset-delivery
Deliveries represent a group of delivered [Dataset Tasks](/core-resources/dataset-task) within a [Dataset](/core-resources/dataset).
A Dataset Delivery is a collection of dataset tasks that are delivered together. Each delivery:
* Has a unique name within its dataset
* Has a globally unique delivery ID
* Contains one or more related dataset tasks that are delivered together
* Is always associated with exactly one dataset
Deliveries are available in Scale Dashboard, under [Datasets](https://console.scale.com/data-gallery/data-streams) tab then click on a dataset to see the deliveries.
Scale API offers the following endpoints to retrieve deliveries and related tasks:
Check out the [List all deliveries by dataset](/v2/datasets/deliveries) docs on how to list all deliveries for a dataset.
Check out the [Download dataset tasks in a delivery](/v2/datasets/delivery) docs to download tasks in a specific delivery.
# Dataset task
Source: https://docs.genai.scale.com/core-resources/dataset-task
A Dataset Task represents an individual unit of data in a dataset.
Scale API offers following endpoints to download dataset tasks:
Check out the [Retrieve a Dataset Task](/v2/datasets/task) docs on how to get a single dataset task by its identifiers.
Check out the [Get Multiple Dataset Tasks](/v2/datasets/tasks) docs to learn more about how to download multiple dataset tasks by different parameters like dataset, delivery, or delivery date.
Check out the [Download dataset tasks in a delivery](/v2/datasets/delivery) docs to download tasks in a specific delivery.
```json Dataset Task theme={null}
{
"task_id": "task_123",
"delivery": "delivery_123",
"dataset": "dataset_123",
"response": {
"key_1": "value_1",
"key_2": "value_2",
}
}
```
# Delivery
Source: https://docs.genai.scale.com/core-resources/delivery
Deliveries represent a group of delivered [Tasks](/core-resources/task) within a [Project](/core-resources/project).
A Delivery is a collection of tasks that are delivered together as a single unit within a project. Each delivery:
* Has a unique name within its project
* Has a globally unique delivery ID
* Contains one or more related tasks that are delivered together
* Is always associated with exactly one project
Deliveries are available in Scale Dashboard, under [Deliveries](https://console.scale.com/projects?view=deliveries) tab and in your project's page as well.
Scale API offers the following endpoints to retrieve deliveries and related tasks:
Check out the [List all deliveries by project](/v2/deliveries) docs on how to list all deliveries for a project.
Check out the [Download tasks in a delivery](/v2/delivery) docs to download tasks in a specific delivery.
# Message
Source: https://docs.genai.scale.com/core-resources/message
A Message is a piece of content and [Annotations](/core-resources/annotation) that are relevant to that Message.
Messages represent the content that is sent to / returned from LLMs. They contain the message text, any associated files with the content, the author of the content, and the role in which the content applies.
[Annotations](/core-resources/annotation) at the Message level directly apply to this specific message. For example, the annotation for "response formatting" will be at this level.
# Project
Source: https://docs.genai.scale.com/core-resources/project
Projects are the root nodes of Scale's information hierarchy. [Tasks](/core-resources/task) belong to a Project.
Projects define the instructions and task parameters for all tasks in the project. Projects control which contributors are allowed to work on certain classes of tasks.
# Task
Source: https://docs.genai.scale.com/core-resources/task
A Task represents an individual unit of work done by a contributor.
The work to be done is defined in the Task parameters. For example, a single Task might ask a contributor to have a conversation with a chat model and label all of the responses from the model. These labels are called [Annotations](/core-resources/annotation).
Scale API offers following endpoints to download tasks:
Check out the [Retrieve a Task](/v2/task) docs on how to get a single task by its identifiers.
Check out the [Get Multiple Tasks](/v2/tasks) docs to learn more about how to download multiple tasks by different parameters like project, batch, status, or completion date.
Check out the [Download tasks in a delivery](/v2/delivery) docs to download tasks in a specific delivery.
```json Completed Task theme={null}
{
"task_id": "task_abc123",
"project": "project_abc123",
"batch": "batch_abc123",
"status": "completed",
"created_at": "2023-11-07T05:31:56Z",
"completed_at": "2023-11-07T05:31:56Z",
"metadata": {},
"threads": [
...omitted...
]
}
```
```json Pending Task theme={null}
{
"task_id": "task_abc123",
"project": "project_abc123",
"batch": "batch_abc123",
"status": "pending",
"created_at": "2023-11-07T05:31:56Z",
"metadata": {}
}
```
```json Errored Task theme={null}
{
"task_id": "task_abc123",
"project": "project_abc123",
"batch": "batch_abc123",
"status": "error",
"created_at": "2023-11-07T05:31:56Z",
"metadata": {},
"errors": [
{
"type": "UNSUPPORTED_LANGUAGE",
"message": "The specified language or locale is not supported"
}
]
}
```
```json Reported Task theme={null}
{
"task_id": "task_abc123",
"project": "project_abc123",
"batch": "batch_abc123",
"status": "error",
"created_at": "2023-11-07T05:31:56Z",
"metadata": {},
"sensitive_content_reports": [
{
"type": "violence",
"message": "The prompt in the task promotes violence."
}
]
}
```
# Thread
Source: https://docs.genai.scale.com/core-resources/thread
A Thread is a series of [Turns](/core-resources/turn) and [Annotations](/core-resources/annotation) that are relevant to the entire Thread.
A Thread can be thought of as a conversation. Each Thread is broken down into a series of [Turns](/core-resources/turn), where each Turn contains a user [Message](/core-resources/message) and any LLM response message(s).
[Annotations](/core-resources/annotation) at the Thread level apply to the entire thread as a whole, across all [Turns](/core-resources/turn).
# Turn
Source: https://docs.genai.scale.com/core-resources/turn
A Turn is a series of [Messages](/core-resources/message) and [Annotations](/core-resources/annotation) that are relevant to the Turn.
Turns are distinct breaks between messages in a [Thread](/core-resources/thread). Turns contain a user message and any LLM response(s) to that user message.
[Annotations](/core-resources/annotation) at the Turn level apply to group of [Messages](/core-resources/message) in that turn. For example, the annotation for message preference is at the turn level.
# Authentication
Source: https://docs.genai.scale.com/get-started/authentication
Scale uses HTTP Bearer Authentication to authenticate all API calls to our platform. Provide your **API Key** as the Bearer Authorization header.
```python ScaleSDK theme={null}
import scaleapi
API_KEY = 'live_...'
client = scaleapi.ScaleClient(API_KEY)
```
```bash cURL {4} theme={null}
API_KEY='live_...'
curl --request GET \
--url 'https://api.scale.com/v2/tasks?project_name=My+Test+Project&limit=1' \
--header "Authorization: Bearer $API_KEY"
```
```python Python {10} theme={null}
import requests
API_KEY = 'live_...'
response = requests.request(
"GET",
url="https://api.scale.com/v2/tasks",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
params={"project_name": "My Test Project", "limit": "1"},
)
print(response.json())
```
```javascript JavaScript {8} theme={null}
const API_KEY = 'live_...';
const params = new URLSearchParams({ task_id: 'abc123' });
const response = await fetch('https://api.scale.com/v2/task?' + params.toString(), {
method: 'GET',
headers: {
Accept: 'application/json',
Authorization: `Bearer ${API_KEY}`,
},
});
console.log(await response.json());
```
```go Go {9} theme={null}
func main() {
apiKey := "live_..."
taskId := "abc123"
url := fmt.Sprintf("https://api.scale.com/v2/task?task_id=%s", taskId)
req, _ := http.NewRequest("GET", url, nil)
req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", apiKey))
res, _ := http.DefaultClient.Do(req)
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
fmt.Println(string(body))
}
```
```java Java {9} theme={null}
public class App {
public static void main(String[] args) {
String apiKey = "live_...";
String taskId = "abc123";
String url = String.format("https://api.scale.com/v2/task?task_id=%s", taskId);
HttpResponse response = Unirest.get(url)
.header("Authorization", String.format("Bearer %s", apiKey))
.asString();
System.out.println(response.getBody());
}
}
```
### Getting your API key
Your API keys are conveniently located in your [dashboard](https://dashboard.scale.com/settings/apikey). Access to your dashboard is gained by [logging in](https://dashboard.scale.com/login) to your existing account.
Once you're in, navigate to your profile and select 'API Key' from the menu.
Your "Live API Key" should start with `live_`
### Don't see the "API Keys" section?
Only Admins and Managers have access to API Keys for your security. You can either ask your admin head over to [Team Page](https://dashboard.scale.com/settings/team) and update your [role](https://scale.com/docs/team-getting-started) to be a "Manager", or share the API key with you.
# Quickstart
Source: https://docs.genai.scale.com/get-started/quickstart
How to use Scale APIs
## Your First API Call
Scale API Key can be retrieved by logging into the Scale dashboard. In the customer [dashboard](https://dashboard.scale.com), click your profile and visit the "[API Key](https://dashboard.scale.com/settings/apikey)" section to get your "Live API Key", which should start with `live_`.
You can browse your projects and batches in the [Scale Dashboard](https://dashboard.scale.com).
API supports searching tasks by name or ID of the associated projects, batches or deliveries.
Scale API offers multiple ways to search tasks that you want to download:
* You can download all completed tasks from a project or a batch
* You can download tasks directly from a specific delivery
* You can continuously download recently completed tasks from a project
Below you can find an example of how to download completed tasks from a project: [Downloading Tasks](#downloading-tasks)
You can also explore other ways to download tasks from Scale API:
* [Download multiple tasks](/v2/tasks) from a project or batch with different parameters
* [Download tasks from a given delivery](/v2/delivery) from a project
## Initializing [Scale Python SDK](https://github.com/scaleapi/scaleapi-python-client)
First install the Python SDK.
```bash theme={null}
pip install --upgrade scaleapi
```
Then use your API key to initialize the client.
```python theme={null}
import scaleapi
API_KEY = "live_...."
client = scaleapi.ScaleClient(API_KEY)
```
## Downloading Tasks
See our API Reference to [retrieve a single task](/v2/task) or [query for tasks](/v2/tasks) in a project or batch.
```python theme={null}
# Downloads all completed tasks from a project and saves them to a jsonl file
import json
PROJECT_NAME = "My Test Project"
def get_tasks_by_project_name(project_name: str):
tasks_generator = client.v2_get_tasks(
project_name=project_name,
status="completed",
limit=100,
)
tasks = list(tasks_generator)
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_project_name(PROJECT_NAME)
with open(f"{PROJECT_NAME}.jsonl", "w+") as f:
for task in tasks:
json.dump(task.model_dump(mode='json'), f)
f.write("\n")
```
## Uploading Tasks to Scale
### Creating a Batch
First you are expected to create a **batch** in the project.
```python theme={null}
from scaleapi.exceptions import (
ScaleException,
ScaleResourceNotFound,
)
def get_or_create_batch(client: scaleapi.ScaleClient, project_name, batch_name):
try:
batch = client.get_batch(batch_name)
return batch
except ScaleResourceNotFound:
batch = client.create_batch(project_name, batch_name, "")
return batch
except ScaleException as err:
raise Exception(f"❌ ERROR Batch Creation: {err.message}") from err
batch = get_or_create_batch(client, project.name, "test_api_batch_20240708")
print(f"Batch: {batch.name}")
```
### Creating Tasks
Once a batch is created, then the tasks can be created within that batch. Each task is expected to have certain information sent in the request payload that corresponds to the project.
```python theme={null}
# Those variables are specific to each project and need to be aligned with Scale before sending them
template_variables = {
"id": "0",
"prompt": "Hello world!",
"base_model_response": "..."
"test_modeL_response": "..."
"languageCode": "en_US",
}
# Task Payload
payload = {
"project": project.name,
"batch": batch.name,
"template_variables": template_variables,
"unique_id":"unique_id_12345"
}
def create_task_request(client: scaleapi.ScaleClient, payload):
try:
task = client.create_task(TaskType.Chat, **payload)
return task.id
except ScaleDuplicateResource as dup_err: # Existing Task
print(f"Task already exists: {dup_err.message}")
except ScaleException as err:
print(f"❌ ERROR Task Creation: {err.message}")
return None
task_id = create_task_request(client, payload)
```
# Retrieve all models
Source: https://docs.genai.scale.com/model-management/model-endpoints
GET /v2/model_endpoints
Fetches a list of all available models.
# Search Model Version Configs
Source: https://docs.genai.scale.com/model-management/model-version-config-get
GET /v2/model_version_configs
Fetches a list of all registered model version configs for a given Model Endpoint.
# Upload a new Model Version Config
Source: https://docs.genai.scale.com/model-management/model-version-config-post
POST /v2/model_version_configs
Create a new Model Version Config to be evaluated.
# Retrieve a Model Version Config
Source: https://docs.genai.scale.com/model-management/model-version-configs
GET /v2/model_version_configs/{id}
Fetches details of a specific Model Version Config.
# Evals
Source: https://docs.genai.scale.com/project-archetypes/evals
Model Evaluation
Evaluation tasks are structured assessments used to measure the performance and capabilities of Large Language Models (LLMs). These tasks provide quantitative and qualitative insights into how well a model performs across various benchmarks.
By using Eval tasks, researchers can test model responses against specific datasets and scoring criteria, ensuring that their model's capabilities align with intended use cases.
## Overview
Each evaluation task is designed to test specific aspects of model performance, from basic comprehension to complex reasoning abilities, across various dimensions such as accuracy, reasoning, safety, and domain-specific knowledge.
## Task Structure
The generic structure of a Task is defined under [Core Resources](../core-resources/task). An Eval Task specifically consists of three core components:
* A prompt (question)
* A reference (base) model's response
* A test model's response
Additional context can be included when necessary for proper evaluation of the
responses.
This structure enables direct comparison between different models' outputs, allowing for systematic evaluation of model performance improvements or regressions.
![]()
![]()
### Messages
Each turn consists of sequential [messages](../core-resources/message) that represent a user prompt, base model response and test model response.
#### User message
Contains the initial prompt or question (role: `user`).
```json Sample Prompt theme={null}
{
"content": {
"text": "This is an example prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
}
```
```python get_eval_user_prompts.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to user prompt for every turn
def get_eval_user_prompts(task):
thread = task['threads'][0] # evals have single threads
user_prompts = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "user"]
assert len(responses) == 1, "Turn must contain a user prompt. Turn ID: {}".format(turn['id'])
user_prompts[turn['id']] = responses[0]
return user_prompts
```
#### Base model response
Contains the reference model's answer (role: `assistant`)
```json Sample Base Model Response theme={null}
{
"content": {
"text": "This is the base model response"
},
"role": "assistant",
"source_id": "base_model",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 2 },
{ "key": "conciseness", "value": 3 },
{ "key": "format", "value": 3 },
{ "key": "safety", "value": 3 },
{ "key": "overall", "value": 5 }
]
}
```
```python get_eval_base_model_response.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to base model response for every turn
def get_eval_base_model_response(task):
thread = task['threads'][0] # evals have single threads
base_model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "base_model"]
assert len(responses) == 1, "Turn must contain a base model response. Turn ID: {}".format(turn['id'])
base_model_responses[turn['id']] = responses[0]
return base_model_responses
```
#### Test model response
Contains the candidate model's answer (role: `assistant`)
```json Sample Test Model Response theme={null}
{
"content": {
"text": "This is test model response"
},
"role": "assistant",
"source_id": "test_model",
"annotations": [
{ "key": "instruction_following", "value": 2 },
{ "key": "truthfulness", "value": 1 },
{
"key": "truthfulness_justification",
"value": "The response incorrectly classifies..."
},
{ "key": "conciseness", "value": 2 },
{ "key": "format", "value": 3 },
{ "key": "safety", "value": 3 },
{ "key": "overall", "value": 3 }
]
}
```
```python get_eval_test_model_response.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to test model response for every turn
def get_eval_test_model_response(task):
thread = task['threads'][0] # evals have single threads
test_model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "test_model"]
assert len(responses) == 1, "Turn must contain a test model response. Turn ID: {}".format(turn['id'])
test_model_responses[turn['id']] = responses[0]
return test_model_responses
```
Each message includes a `source_id` that uniquely identifies the source that
generated the response. Possible sources are: `user`, `base_model`,
`test_model`.
#### Message Annotations
Both base model and test model responses are evaluated across multiple dimensions, such as:
* Instruction following
* Truthfulness
* Conciseness
* Format adherence
* Safety
* Overall quality
Message's [`annotations`](../core-resources/annotation) include the evaluation results for each dimension.
The evaluation dimensions are flexible and can be customized based on project
requirements and evaluation objectives.
### Turn-Level Annotations
The `annotations` at the **turn** level, specifies preference related or aggregated information. Some common examples are:
* Selected model identifier: `selected_model_id`
* Likert scale rating: `likert_value`
* Detailed justification for the selection: `justification`
* Any other comparative analysis between model responses
```json Sample Turn-Level Annotations theme={null}
[
{
"key": "selected_model_id",
"value": "base_model"
},
{
"key": "likert_value",
"value": 2
},
{
"key": "justification",
"value": "@Response 1 is better than @Response 2. @Response 2 has an issue in Truthfulness ..."
}
]
```
```python get_turn_annotations.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to turn-level annotations for every turn
def get_turn_annotations(task):
thread = task['threads'][0]
turn_annotations = {}
for turn in thread['turns']:
turn_annotations[turn['id']] = turn['annotations']
return turn_annotations
```
### Expanded Eval Task Output
This is a sample expanded sample Eval Task output returned by [`/v2/task`](../v2/task).
```json Sample Eval Task Output theme={null}
{
"task_id": "task_123",
"project": "project_123",
"batch": "batch_123",
"status": "completed",
"created_at": "2025-01-01T08:31:03.169Z",
"completed_at": "2025-01-02T04:00:39.923Z",
"threads": [
{
"id": "thread_0",
"turns": [
{
"id": "turn_0",
"messages": [
{
"content": {
"text": "This is an example prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
},
{
"content": {
"text": "This is the base model response"
},
"role": "assistant",
"source_id": "base_model",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 2 },
{ "key": "conciseness", "value": 3 },
{ "key": "format", "value": 3 },
{ "key": "safety", "value": 3 },
{ "key": "overall", "value": 5 }
]
},
{
"content": {
"text": "This is test model response"
},
"role": "assistant",
"source_id": "test_model",
"annotations": [
{ "key": "instruction_following", "value": 2 },
{ "key": "truthfulness", "value": 1 },
{ "key": "truthfulness_justification", "value": "The response incorrectly classifies..." },
{ "key": "conciseness", "value": 2 },
{ "key": "format", "value": 3 },
{ "key": "safety", "value": 3 },
{ "key": "overall", "value": 3 }
]
}
],
"annotations": [
{
"key": "selected_model_id",
"value": "base_model"
},
{
"key": "likert_value",
"value": 2
},
{
"key": "justification",
"value": "@Response 1 is better than @Response 2. @Response 2 has an issue in Truthfulness ..."
}
]
}
],
"annotations": []
}
]
}
```
# Introduction
Source: https://docs.genai.scale.com/project-archetypes/introduction
Project archetypes identify the type of flow contributors will experience when tasking. These archetypes allow standardization of the instructions, task parameters and response formats while allowing a customizable task flow.
It is possible for projects to have multiple archetypes. For example, it is possible to construct a project that does RLHF on Process Supervision data.
#### Supported Archetypes
The following archetypes are currently supported in the [v2 API](/v2).
* [**Evals**](evals) - Contributors can rate model responses on a pre-defined numeric scale.
* [**RLHF**](rlhf) - Contributors can indicate preferred model responses.
* [**Rubrics**](rubrics) - Contributors can create annotations on a pre-defined set of criteria determined during project creation.
* [**SFT**](sft) - Contributors can write and rewrite model responses.
#### Future Archetypes
The following archetypes are planned for the v2 API.
* **Multi-modal** - Contributors can interact with multimodal data (e.g. images, videos, audio).
* **Process Supervision** - Contributors can annotate the chain of thought of a model.
* **Browser Trajectories** - Contributors can annotate browser usage of a model.
The following sections provide more detail on the response structure for each of the supported archetypes.
# RLHF
Source: https://docs.genai.scale.com/project-archetypes/rlhf
Reinforcement Learning from Human Feedback
RLHF tasks are structured datasets used to train and fine-tune Large Language Models (LLMs) using human feedback. These tasks help align model behavior with human preferences by providing paired responses with clear preference indicators.
## Overview
RLHF tasks consist of prompts and multiple model-generated responses, where human annotators provide explicit preferences between responses. This feedback helps models learn which outputs are more desirable according to human judgment across various dimensions such as Helpfulness, Accuracy, Safety, Writing quality, and Task completion.
## Task Structure
An RLHF task contains:
* A prompt (question or instruction)
* Multiple model generated responses
* Human preference annotations
* Detailed feedback and justification
RLHF tasks can be followed by an SFT stage where the preferred model's
response is rewritten
![]()
![]()
### Messages
Each turn consists of sequential [messages](../core-resources/message) that represent a user prompt, multiple model responses, and corresponding human preference annotations.
#### User message
Contains the initial prompt or instruction (role: `user`).
```json Sample Instruction theme={null}
{
"content": {
"text": "This is a test prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
}
```
```python get_rlhf_user_prompts.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to user prompt for every turn
def get_rlhf_user_prompts(task):
thread = task['threads'][0]
user_prompts = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "user"]
assert len(responses) == 1, "Turn must contain a user prompt. Turn ID: {}".format(turn['id'])
user_prompts[turn['id']] = responses[0]
return user_prompts
```
#### Model Responses
Contains responses from different models (role: `assistant`). Optionally includes `rewrite` annotations to store human rewritten responses.
```json Sample Model Response theme={null}
[
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{ "key": "instruction_following", "value": 1 },
{ "key": "truthfulness", "value": 1 },
{
"key": "truthfulnes_justiifcation",
"value": "Justification for the rating"
},
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 }
]
},
{
"content": {
"text": "This is model 2 response"
},
"role": "assistant",
"source_id": "model_2",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 3 },
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 },
{
"key": "rewrite",
"value": "Model 2 rewritten response"
}
]
}
]
```
```python get_rlhf_model_responses.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to model responses for every turn
def get_rlhf_model_responses(task):
thread = task['threads'][0]
model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['role'].lower() == "assistant"]
assert len(responses) > 0, "Turn must contain at least one model response. Turn ID: {}".format(turn['id'])
model_responses[turn['id']] = responses
return model_responses
```
Each model response includes a `source_id` that uniquely identifies the model
that generated the response.
#### Message Annotations
Each model response is evaluated across multiple dimensions, which may include:
* Instruction following
* Truthfulness
* Factuality
* Tone
If the model response is rewritten, the `rewrite` annotation is added to the list.
Message's [`annotations`](../core-resources/annotation) include the ratings for each dimension.
The rating dimensions are flexible and can be customized based on project
requirements and objectives.
### Turn-Level Annotations
The `annotations` at the **turn** level, specifies preference related or aggregated information. Some common examples are:
* Selected model identifier: `selected_model_id`
* Likert scale rating: `likert_value`
* Detailed justification for the selection: `justification`
* Any other comparative analysis between model responses
```json Sample Turn-Level Annotations theme={null}
[
{
"key": "selected_model_id",
"value": "base_model"
},
{
"key": "likert_value",
"value": 2
},
{
"key": "justification",
"value": "@Response 1 is better than @Response 2. @Response 2 has an issue in Truthfulness ..."
}
]
```
```python get_turn_annotations.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to turn-level annotations for every turn
def get_turn_annotations(task):
thread = task['threads'][0]
turn_annotations = {}
for turn in thread['turns']:
turn_annotations[turn['id']] = turn['annotations']
return turn_annotations
```
### Expanded RLHF Task Output
This is a sample expanded sample RLHF Task output returned by [`/v2/task`](../v2/task).
```json Sample RLHF Task Output theme={null}
{
"task_id": "task_123",
"project": "project_123",
"batch": "batch_123",
"status": "completed",
"created_at": "2025-01-01T08:31:03.169Z",
"completed_at": "2025-01-02T04:00:39.923Z",
"threads": [
{
"id": "thread_0",
"turns": [
{
"id": "turn_0",
"messages": [
{
"content": {
"text": "This is a test prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
},
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{ "key": "instruction_following", "value": 1 },
{ "key": "truthfulness", "value": 1 },
{
"key": "truthfulnes_justiifcation",
"value": "Justification for the rating"
},
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 }
]
},
{
"content": {
"text": "This is model 2 response"
},
"role": "assistant",
"source_id": "model_2",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 3 },
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 },
{
"key": "rewrite",
"value": "Model 2 rewritten response"
}
]
}
],
"annotations": [
{
"key": "selected_model_id",
"value": "model_2"
},
{
"key": "likert_value",
"value": 6
},
{
"key": "justification",
"value": "@Response 2 is much better than @Response 1 as @Response 1 made a mistake in ..."
}
]
}
],
"annotations": []
}
]
}
```
# Rubrics
Source: https://docs.genai.scale.com/project-archetypes/rubrics
Contributor defined rating criteria
Rubrics tasks are a specific type of Evaluation task where models are assessed based on a dynamic rubric, which is defined by the contributor during the task. These tasks provide quantitative and qualitative insight into how well a model can follow arbitrary criteria.
## Overview
Each rubrics task is designed to test specific aspects of model performance, from basic comprehension to complex reasoning abilities, across natural human defined criteria.
## Task Structure
A Rubrics task consists of:
* A prompt (question or instruction)
* A list of model evaluation criteria (contributor defined based on the prompt)
* Multiple model generated responses
* Human evaluation of the model responses against the criteria
Additional annotations can be included when necessary for standard evaluation
of the responses.
![]()
![]()
### Messages
Each turn consists of sequential [messages](../core-resources/message) that represent a user prompt, multiple model responses, and corresponding human evaluation of the model responses against the criteria.
#### User message
Contains the initial prompt or instruction (role: `user`).
```json Sample Instruction theme={null}
{
"content": {
"text": "Put the following data into a table and sort by name:\n\nSue,$20\nJared,$40\nMike,$60"
},
"role": "user",
"source_id": "user",
"annotations": []
}
```
```python get_rubric_user_prompts.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to user prompt for every turn
def get_rubric_user_prompts(task):
thread = task['threads'][0]
user_prompts = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "user"]
assert len(responses) == 1, "Turn must contain a user prompt. Turn ID: {}".format(turn['id'])
user_prompts[turn['id']] = responses[0]
return user_prompts
```
#### Model Responses
Contains responses from different models (role: `assistant`).
```json Sample Model Response theme={null}
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{
"id": "rubric_0_criteria_0_rating",
"title": "The model must respond with a formatted table.",
"value": "no_issues",
"metadata": { "criteria": "rubric_0_criteria_0" }
},
{
"id": "rubric_0_criteria_1_rating",
"title": "The response must sort the names in alphabetical order.",
"value": "major_issues",
"metadata": { "criteria": "rubric_0_criteria_1" }
},
{
"id": "rubric_0_criteria_2_rating",
"title": "The response must include bold headers.",
"value": "minor_issues",
"metadata": { "criteria": "rubric_0_criteria_2" }
}
]
}
```
```python get_rubrics_model_responses.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to model responses for every turn
def get_rubrics_model_responses(task):
thread = task['threads'][0]
model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['role'].lower() == "assistant"]
assert len(responses) > 0, "Turn must contain at least one model response. Turn ID: {}".format(turn['id'])
model_responses[turn['id']] = responses
return model_responses
```
Each model response includes a `source_id` that uniquely identifies the model
that generated the response.
#### Message Annotations
Each model response is evaluated across the contributor-defined criteria. For example:
* "The response must display information in a table."
* "The response must sort the names in alphabetical order."
* "The response must use metric units."
Message's [`annotations`](../core-resources/annotation) include the evaluation results for each dimension.
```json Sample Message Annotations theme={null}
[
{
"id": "rubric_0_criteria_0_rating",
"title": "The model must respond with a formatted table.",
"value": "no_issues",
"metadata": {
"criteria": "rubric_0_criteria_0" // rubric criteria defined at turn-level annotation
}
}
]
```
The rubric criteria is unique to the prompt and is different per task.
Additional evaluation dimensions can be included based on project requirements
and evaluation objectives.
### Turn-Level Annotations
The `annotations` at the **turn** level, specifies contributor-defined rubrics, preference related or aggregated information. Some common examples are:
* Selected model identifier: `selected_model_id`
* Likert scale rating: `likert_value`
* Detailed justification for the selection: `justification`
* Any other comparative analysis between model responses
```json Sample Turn-Level Annotations theme={null}
[
{
"key": "selected_model_id",
"value": "model_2"
},
{
"key": "rubric_0_criteria_0",
"title": "The model must respond with a formatted table.",
"value": "objective",
},
{
"key": "rubric_0_criteria_1",
"title": "The response must sort the names in alphabetical order.",
"value": "objective",
},
{
"key": "rubric_0_criteria_2",
"title": "The response must include bold headers.",
"value": "implicit",
}
]
```
```python get_turn_annotations.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to turn-level annotations for every turn
def get_turn_annotations(task):
thread = task['threads'][0]
turn_annotations = {}
for turn in thread['turns']:
turn_annotations[turn['id']] = turn['annotations']
return turn_annotations
```
### Expanded Rubrics Task Output
This is a sample expanded sample Rubrics Task output returned by [`/v2/task`](../v2/task).
```json Sample Rubrics Task Output theme={null}
{
"task_id": "task_123",
"project": "project_123",
"batch": "batch_123",
"status": "completed",
"created_at": "2025-01-01T08:31:03.169Z",
"completed_at": "2025-01-02T04:00:39.923Z",
"threads": [
{
"id": "thread_0",
"turns": [
{
"id": "turn_0",
"messages": [
{
"content": {
"text": "Put the following data into a table and sort by name:\n\nSue,$20\nJared,$40\nMike,$60"
},
"role": "user",
"source_id": "user",
"annotations": []
},
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{
"id": "rubric_0_criteria_0_rating",
"title": "The model must respond with a formatted table.",
"value": "no_issues",
"metadata": { "criteria": "rubric_0_criteria_0" }
},
{
"id": "rubric_0_criteria_1_rating",
"title": "The response must sort the names in alphabetical order.",
"value": "major_issues",
"metadata": { "criteria": "rubric_0_criteria_1" }
},
{
"id": "rubric_0_criteria_2_rating",
"title": "The response must include bold headers.",
"value": "minor_issues",
"metadata": { "criteria": "rubric_0_criteria_2" }
}
]
},
{
"content": {
"text": "This is model 2 response"
},
"role": "assistant",
"source_id": "model_2",
"annotations": [
{
"id": "rubric_0_criteria_0_rating",
"title": "The model must respond with a formatted table.",
"value": "no_issues",
"metadata": { "criteria": "rubric_0_criteria_0" }
},
{
"id": "rubric_0_criteria_1_rating",
"title": "The response must sort the names in alphabetical order.",
"value": "no_issues",
"metadata": { "criteria": "rubric_0_criteria_1" }
},
{
"id": "rubric_0_criteria_2_rating",
"title": "The response must include bold headers.",
"value": "major_issues",
"metadata": { "criteria": "rubric_0_criteria_2" }
}
]
}
],
"annotations": [
{
"key": "selected_model_id",
"value": "model_2"
},
{
"key": "rubric_0_criteria_0",
"title": "The model must respond with a formatted table.",
"value": "objective",
},
{
"key": "rubric_0_criteria_1",
"title": "The response must sort the names in alphabetical order.",
"value": "objective",
},
{
"key": "rubric_0_criteria_2",
"title": "The response must include bold headers.",
"value": "implicit",
}
]
}
],
"annotations": []
}
]
}
```
# SFT
Source: https://docs.genai.scale.com/project-archetypes/sft
Supervised Fine-Tuning
SFT tasks are structured datasets used to train Large Language Models (LLMs) with human created data.
## Overview
Each SFT task is designed to improve a model's capabilities with source data or improvements to a model's response.
## Task Structure
A SFT task consists of:
* A prompt (question or instruction)
* A model response
* A human written response (optimal response)
Additional annotations can be included when necessary for classification of
the prompt & response.
![]()
![]()
### Messages
Each turn consists of sequential [messages](../core-resources/message) that represent a user prompt, model response and human written response.
#### User message
Contains the initial prompt or instruction (role: `user`).
```json Sample Prompt theme={null}
{
"content": {
"text": "How can I bake a raspberry pie?"
},
"role": "user",
"source_id": "user",
"annotations": []
}
```
```python get_sft_user_prompts.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to user prompt for every turn
def get_sft_user_prompts(task):
thread = task['threads'][0]
user_prompts = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "user"]
assert len(responses) == 1, "Turn must contain a user prompt. Turn ID: {}".format(turn['id'])
user_prompts[turn['id']] = responses[0]
return user_prompts
```
#### Model Response
Contains the reference model's answer (role: `assistant`).
```json Sample Model Response theme={null}
{
"content": {
"text": "I'm sorry, I can't help you with that."
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{
"key": "rewrite",
"value": "To bake a raspberry pie, let's start by gathering ingredients. ..."
}
]
}
```
```python get_sft_model_responses.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to model responses for every turn
def get_sft_model_responses(task):
thread = task['threads'][0]
model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['role'].lower() == "assistant"]
assert len(responses) > 0, "Turn must contain at least one model response. Turn ID: {}".format(turn['id'])
model_responses[turn['id']] = responses
return model_responses
```
The model response includes a `source_id` that uniquely identifies the model
that generated the response.
#### Message Annotations
The model response can be evaluated across multiple dimensions, which may include:
* Instruction following
* Truthfulness
* Factuality
* Tone
If the model response is rewritten, the `rewrite` annotation is added to the list.
Message's [`annotations`](../core-resources/annotation) include the ratings for each dimension.
The rating dimensions are flexible and can be customized based on project
requirements and objectives.
### Turn-Level Annotations
The `annotations` at the **turn** level, specifies preference related or aggregated information. Some common examples are:
* Detailed `justification` for why a certain response is better
* Any comparative analysis between model responses
```json Sample Turn-Level Annotations theme={null}
[
{
"key": "justification",
"value": "The model did not response to the prompt."
}
]
```
```python get_turn_annotations.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to turn-level annotations for every turn
def get_turn_annotations(task):
thread = task['threads'][0]
turn_annotations = {}
for turn in thread['turns']:
turn_annotations[turn['id']] = turn['annotations']
return turn_annotations
```
### Expanded SFT Task Output
This is a sample expanded sample SFT Task output returned by [`/v2/task`](../v2/task).
```json Sample SFT Task Output theme={null}
{
"task_id": "task_123",
"project": "project_123",
"batch": "batch_123",
"status": "completed",
"created_at": "2025-01-01T08:31:03.169Z",
"completed_at": "2025-01-02T04:00:39.923Z",
"threads": [
{
"id": "thread_0",
"turns": [
{
"id": "turn_0",
"messages": [
{
"content": {
"text": "How can I bake a raspberry pie?"
},
"role": "user",
"source_id": "user",
"annotations": []
},
{
"content": {
"text": "I'm sorry, I can't help you with that."
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{
"key": "rewrite",
"value": "To bake a raspberry pie, let's start by gathering ingredients. ...",
}
]
}
],
"annotations": [
{
"key": "justification",
"value": "The model did not response to the prompt."
}
]
}
],
"annotations": []
}
]
}
```
# Get Task (legacy)
Source: https://docs.genai.scale.com/v1/task
GET /v1/task/{task_id}
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get an Annotation
Source: https://docs.genai.scale.com/v2/annotation
GET /v2/annotation
Retrieve an [Annotation](/core-resources/annotation) from its `id`.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get a Batch
Source: https://docs.genai.scale.com/v2/batch
GET /v2/batch
Retrieve a [Batch](/core-resources/batch) from its `batch_id` or `batch_name`.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You are expected to provide one of the following:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Cancel a Batch
Source: https://docs.genai.scale.com/v2/batch-cancel
POST /v2/batch/cancel
Cancel processing of a [Batch](/core-resources/batch). Unstarted tasks will be cancelled. Completed batches cannot be cancelled.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Batch Status Requirements
You can cancel batches in any status except **completed** or **cancelled** (already cancelled).
Once cancelled, the batch status will change to **cancelled** and all unstarted tasks will be cancelled. Cancelled batches cannot be resumed.
### Required request parameters
You are expected to provide one of the following in the request body:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Create a Batch
Source: https://docs.genai.scale.com/v2/batch-create
POST /v2/batch
Create a new [Batch](/core-resources/batch) for organizing tasks.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required request parameters
You must provide:
* **Name** (`name`) - The name for the batch
* **Project** (`project`) - Project ID or project name to associate with the batch
# Finalize a Batch
Source: https://docs.genai.scale.com/v2/batch-finalize
POST /v2/batch/finalize
Move a [Batch](/core-resources/batch) from staging to in-progress status to begin processing tasks.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Batch Status Requirement
You can only finalize batches that are currently in **staging** status. Batches in other statuses (in\_progress, completed, paused) cannot be finalized.
Once finalized, the batch status will change to **in\_progress** and task processing will begin.
### Required request parameters
You are expected to provide one of the following in the request body:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Pause a Batch
Source: https://docs.genai.scale.com/v2/batch-pause
POST /v2/batch/pause
Pause processing of an in-progress [Batch](/core-resources/batch). Only unstarted tasks will be paused.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Batch Status Requirement
You can only pause batches that are currently **in\_progress**. Batches in other statuses (staging, completed, paused) cannot be paused.
Once paused, the batch status will change to **paused** and all unstarted tasks will be paused.
### Required request parameters
You are expected to provide one of the following in the request body:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Resume a Batch
Source: https://docs.genai.scale.com/v2/batch-resume
POST /v2/batch/resume
Resume processing of a paused [Batch](/core-resources/batch). Previously paused tasks will be resumed.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Batch Status Requirement
You can only resume batches that are currently **paused**. Batches in other statuses (staging, in\_progress, completed) cannot be resumed.
Once resumed, the batch status will change to **in\_progress** and paused tasks will begin processing.
### Required request parameters
You are expected to provide one of the following in the request body:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Set Batch Metadata
Source: https://docs.genai.scale.com/v2/batch-set-metadata
POST /v2/batch/metadata
Update the metadata for a [Batch](/core-resources/batch).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You are expected to provide one of the following:
* **Batch ID** (`batch_id`)
* **Batch Name** (`batch_name`)
# Get Multiple Batches
Source: https://docs.genai.scale.com/v2/batches
GET /v2/batches
Retrieve multiple [Batches](/core-resources/batch) from a [Project](/core-resources/project).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# List Datasets
Source: https://docs.genai.scale.com/v2/datasets
GET /v2/datasets
Retrieve a list of delivered [Datasets](/core-resources/dataset) with their IDs and names.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# List All Dataset Deliveries
Source: https://docs.genai.scale.com/v2/datasets/deliveries
GET /v2/datasets/deliveries
Lists of [Deliveries](/core-resources/dataset-delivery) from datasets.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You are expected to provide the following to find deliveries:
* `dataset_id` - The ID of the Scale dataset.
# Get Dataset Tasks in a Delivery
Source: https://docs.genai.scale.com/v2/datasets/delivery
GET /v2/datasets/delivery
Retrieve multiple [Dataset Tasks](/core-resources/dataset-task) from a [Delivery](/core-resources/dataset-delivery).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
Remember to handle pagination when downloading large sets of tasks.
The API returns a maximum of 100 tasks per request (or less if you specify a `limit`).
If your request returns more tasks than your specified `limit`, API response will also contain a `next_token` until you reach to the last page.
You can set the `next_token` in your next request to continue downloading tasks from the next page.
### Required query parameters
You are expected to provide the following to find a delivery:
* `delivery_id` Delivery IDs are globally unique and can be used to retrieve a specific delivery.
### Example Code
```python theme={null}
# Downloads all tasks from a delivery
import json
import requests
API_KEY = 'live_...'
delivery_id = 'delivery_123'
def get_tasks_by_delivery(delivery_id: str):
tasks = []
params = {
"delivery_id": delivery_id,
}
should_fetch = True
while should_fetch:
response = requests.request(
"GET",
url="https://api.scale.com/v2/datasets/delivery",
params=params,
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
json_resp = response.json()
next_token = json_resp.get('next_token')
if next_token:
params['next_token'] = next_token
else:
should_fetch = False
tasks.extend(json_resp['tasks'])
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_delivery(delivery_id)
with open(f'{delivery_id}.jsonl', 'w+') as f:
for task in tasks:
json.dump(task, f)
f.write('\n')
```
# Get Response Attachment
Source: https://docs.genai.scale.com/v2/datasets/response_url
GET /v2/datasets/task/{task_id}/response_url/{attachment_id}
Retrieve a [Dataset Task](/core-resources/dataset-task) response attachment. Response attachment URLs can be found in the `response` object of a [Dataset Task](/core-resources/dataset-task) and only exist in some datasets.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get a Dataset Task
Source: https://docs.genai.scale.com/v2/datasets/task
GET /v2/datasets/task
Retrieve a [Dataset Task](/core-resources/dataset-task) from its `task_id`.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get Multiple Dataset Tasks
Source: https://docs.genai.scale.com/v2/datasets/tasks
GET /v2/datasets/tasks
Retrieve multiple [Dataset Tasks](/core-resources/dataset-task) from a [Dataset](/core-resources/dataset) or [Delivery](/core-resources/dataset-delivery).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
Remember to handle pagination when downloading large sets of tasks.
The API returns a maximum of 100 tasks per request (or less if you specify a `limit`).
If your request returns more tasks than your specified `limit`, API response will also contain a `next_token` until you reach to the last page.
You can set the `next_token` in your next request to continue downloading tasks from the next page.
### Required query parameters
You are expected to provide one of the following to start downloading tasks:
* **Dataset** (`dataset_id`)
* **Delivery** (`delivery_id`)
### Example Code
```python theme={null}
# Downloads all tasks from a dataset
import json
import requests
API_KEY = 'live_...'
DATASET_ID = 'MY DATASET ID'
def get_tasks_by_dataset(dataset_id: str):
tasks = []
params = {
"dataset_id": dataset_id,
}
should_fetch = True
while should_fetch:
response = requests.request(
"GET",
url="https://api.scale.com/v2/datasets/tasks",
params=params,
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
json_resp = response.json()
next_token = json_resp.get('next_token')
if next_token:
params['next_token'] = next_token
else:
should_fetch = False
tasks.extend(json_resp['tasks'])
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_dataset(DATASET_ID)
with open(f'dataset_{DATASET_ID}_tasks.jsonl', 'w+') as f:
for task in tasks:
json.dump(task, f)
f.write('\n')
```
# List All Deliveries
Source: https://docs.genai.scale.com/v2/deliveries
GET /v2/deliveries
Lists of [Deliveries](/core-resources/delivery) from a [Project](/core-resources/project).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You are expected to provide one of the following to find deliveries:
* `project_id` - The ID of the Scale project.
* `project_name` - The name of the Scale project.
# Get Tasks in a Delivery
Source: https://docs.genai.scale.com/v2/delivery
GET /v2/delivery
Retrieve multiple [Tasks](/core-resources/task) from a [Delivery](/core-resources/delivery).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
Remember to handle pagination when downloading large sets of tasks.
The API returns a maximum of 100 tasks per request (or less if you specify a `limit`).
If your request returns more tasks than your specified `limit`, API response will also contain a `next_token` until you reach to the last page.
You can set the `next_token` in your next request to continue downloading tasks from the next page.
### Required query parameters
You are expected to provide one of the following to find a delivery:
* `delivery_id` Delivery IDs are globally unique and can be used to retrieve a specific delivery.
* `delivery_name` with a project identifier (`project_id` or `project_name`)
### Example Code
```python theme={null}
# Downloads all tasks from a delivery
import json
import requests
API_KEY = 'live_...'
PROJECT_NAME = 'My Test Project'
DELIVERY_NAME = 'My Test Delivery'
def get_tasks_by_delivery(project_name: str, delivery_name: str):
tasks = []
params = {
"project_name": project_name,
"delivery_name": delivery_name,
}
should_fetch = True
while should_fetch:
response = requests.request(
"GET",
url="https://api.scale.com/v2/delivery",
params=params,
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
json_resp = response.json()
next_token = json_resp.get('next_token')
if next_token:
params['next_token'] = next_token
else:
should_fetch = False
tasks.extend(json_resp['tasks'])
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_delivery(PROJECT_NAME, DELIVERY_NAME)
with open(f'{DELIVERY_NAME}.jsonl', 'w+') as f:
for task in tasks:
json.dump(task, f)
f.write('\n')
```
# Get Delivered Task Responses
Source: https://docs.genai.scale.com/v2/delivery-tasks
GET /v2/delivery/tasks
Retrieve delivered [Task](/core-resources/task) response data from a [Project](/core-resources/project), optionally filtered by [Delivery](/core-resources/delivery).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
Remember to handle pagination when downloading large sets of tasks.
The API returns a maximum of 100 tasks per request (or less if you specify a `limit`).
If your request returns more tasks than your specified `limit`, API response will also contain a `next_token` until you reach to the last page.
You can set the `next_token` in your next request to continue downloading tasks from the next page.
### Required query parameters
You must provide one of the following:
* `project_id` or `project_name` - Get all delivered tasks from a project
### Optional filters
* `delivery_id` - Filter to tasks from a specific delivery
* `status` - Filter by task status
* `completed_after`, `completed_before` - Filter by completion time range
## Overview
Returns delivered task responses from a project. For delivered tasks, includes:
* Complete response data (threads, turns, messages, annotations, etc.)
* Delivery timestamp and ID
* Task metadata
Tasks without deliveries have `null` for delivery-related fields.
**Compared to `/v2/delivery`:** Returns full task response content, not just task metadata.
# Get a Project
Source: https://docs.genai.scale.com/v2/project
GET /v2/project
Retrieve a [Project](/core-resources/project) from its `project_id` or `project_name`.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You are expected to provide one of the following:
* **Project ID** (`project_id`)
* **Project Name** (`project_name`)
# Get Multiple Projects
Source: https://docs.genai.scale.com/v2/projects
GET /v2/projects
Retrieve multiple [Projects](/core-resources/project).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get a Delivery JSON Schema
Source: https://docs.genai.scale.com/v2/schema
GET /v2/schema
Retrieve a delivery JSON schema by its unique identifier. Delivery JSON schemas define the structure and validation rules for task responses.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required query parameters
You must provide:
* `schema_id` - Schema IDs are globally unique and used to validate delivery data structures.
## Overview
Retrieves a delivery JSON schema by its unique identifier. Delivery JSON schemas define the structure and validation rules for task responses in customer applications.
### Use Cases
* **Validate delivery data**: Ensure your delivery data conforms to the expected structure
* **Data integration**: Understand the structure of delivery data for integration purposes
### Response
Returns a JSON object containing the complete schema definition including:
* `id` - Unique MongoDB ObjectId identifier for the schema
* `name` - Human-readable name of the schema
* `createdBy` / `updatedBy` - User IDs of creators/updaters
* `createdAt` / `updatedAt` - Timestamps
* `json` - Stringified JSON Schema definition containing validation rules and structure
The `json` field contains a **stringified JSON Schema** that needs to be parsed before use. Parse it with `JSON.parse()` in JavaScript or `json.loads()` in Python.
### Error Responses
* **400 Bad Request**:
* Missing `schema_id` parameter
* Malformed `schema_id` (invalid MongoDB ObjectId format)
* **404 Not Found**: Schema with the specified ID does not exist
* **401 Unauthorized**: Invalid or missing authentication token
### Example Code
```python theme={null}
import json
import requests
API_KEY = 'live_...'
SCHEMA_ID = '69320cc6425a4bf55273a32b'
def get_schema(schema_id: str):
response = requests.get(
url="https://api.scale.com/v2/schema",
params={"schema_id": schema_id},
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
if response.status_code == 200:
data = response.json()
schema_metadata = data['schema']
# Parse the JSON Schema string
json_schema = json.loads(schema_metadata['json'])
return schema_metadata, json_schema
else:
print(f"Error: {response.status_code}")
print(response.json())
return None, None
if __name__ == "__main__":
metadata, json_schema = get_schema(SCHEMA_ID)
if metadata:
print(f"Schema ID: {metadata['id']}")
print(f"Schema Name: {metadata['name']}")
print(f"Created At: {metadata['createdAt']}")
print(f"\nRequired fields: {json_schema.get('required', [])}")
print(f"Properties: {list(json_schema.get('properties', {}).keys())}")
```
```javascript theme={null}
const API_KEY = 'live_...';
const SCHEMA_ID = '69320cc6425a4bf55273a32b';
async function getSchema(schemaId) {
try {
const response = await fetch(
`https://api.scale.com/v2/schema?schema_id=${schemaId}`,
{
method: 'GET',
headers: {
'Accept': 'application/json',
'Authorization': `Bearer ${API_KEY}`,
},
}
);
if (response.ok) {
const data = await response.json();
const schemaMetadata = data.schema;
// Parse the JSON Schema string
const jsonSchema = JSON.parse(schemaMetadata.json);
return { metadata: schemaMetadata, jsonSchema };
} else {
const error = await response.json();
console.error(`Error: ${response.status}`, error);
return null;
}
} catch (error) {
console.error('Request failed:', error);
return null;
}
}
// Usage
getSchema(SCHEMA_ID).then(result => {
if (result) {
const { metadata, jsonSchema } = result;
console.log('Schema ID:', metadata.id);
console.log('Schema Name:', metadata.name);
console.log('Created At:', metadata.createdAt);
console.log('\nRequired fields:', jsonSchema.required);
console.log('Properties:', Object.keys(jsonSchema.properties));
}
});
```
```bash theme={null}
curl --request GET \
--url 'https://api.scale.com/v2/schema?schema_id=69320cc6425a4bf55273a32b' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer live_...'
```
### Validating Data with the Schema
Once you retrieve a schema, you can use it to validate task data:
```python theme={null}
import json
import requests
from jsonschema import validate, ValidationError
API_KEY = 'live_...'
SCHEMA_ID = '69320cc6425a4bf55273a32b'
# Get the schema
response = requests.get(
url="https://api.scale.com/v2/schema",
params={"schema_id": SCHEMA_ID},
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
schema_metadata = response.json()['schema']
json_schema = json.loads(schema_metadata['json'])
# Example task data to validate
task_data = {
"task_id": "task_123",
"project": "my_project",
"status": "completed",
"created_at": "2025-01-13T10:00:00Z",
"batch": "batch_456",
"prompt_audio_url": "https://example.com/audio.mp3",
"natural_response_url": "https://example.com/response.mp3",
"natural_response_transcript": "Hello world",
"response_reading_url": "https://example.com/reading.mp3",
"metadata": {}
}
# Validate the data
try:
validate(instance=task_data, schema=json_schema)
print("✓ Task data is valid!")
except ValidationError as e:
print(f"✗ Validation failed: {e.message}")
```
### Integration with Deliveries
Schemas are referenced in delivery objects and can be used to validate the structure of task responses. When working with deliveries, you can:
1. Retrieve the delivery using `/v2/delivery`
2. Extract the `schema_id` from the delivery metadata
3. Fetch the full schema definition using `/v2/schema`
4. Validate task responses against the schema
Schema IDs are immutable once created. If you need to modify a schema, you must create a new schema version with a new ID.
# Get a Task
Source: https://docs.genai.scale.com/v2/task
GET /v2/task
Retrieve a [Task](/core-resources/task) from its `task_id`.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Create a Chat Task
Source: https://docs.genai.scale.com/v2/task-chat-create
POST /v2/task/chat
Create a new [Task](/core-resources/task) for GenAI projects. Uses structured task templates, not free-form instructions.
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
### Required parameters
You are expected to provide one of the following:
* **Project ID** (`project_id`)
* **Project Name** (`project_name`)
### Template Variables
Template variables are key-value pairs that populate your project's task template. The available variables depend on your specific project configuration.
**Common template variables:**
* `prompt` - The main instruction or question for the task
* `context` - Additional context or background information
* `examples` - Sample inputs or outputs to guide the task
* `constraints` - Specific requirements or limitations
# Set Task Metadata
Source: https://docs.genai.scale.com/v2/task-set-metadata
POST /v2/task/metadata
Update the metadata for a [Task](/core-resources/task).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
# Get Multiple Tasks
Source: https://docs.genai.scale.com/v2/tasks
GET /v2/tasks
Retrieve multiple [Tasks](/core-resources/task) from a [Project](/core-resources/project) or [Batch](/core-resources/batch).
Every request sent to Scale's API requires authentication. In short, your API Key is the Bearer token. See the [Authentication](/get-started/authentication) section for more details.
Remember to handle pagination when downloading large sets of tasks.
The API returns a maximum of 100 tasks per request (or less if you specify a `limit`).
If your request returns more tasks than your specified `limit`, API response will also contain a `next_token` until you reach to the last page.
You can set the `next_token` in your next request to continue downloading tasks from the next page.
### Required query parameters
You are expected to provide one of the following to start downloading tasks:
* **Project** (`project_id` or `project_name`)
* **Batch** (`batch_id` or `batch_name`)
### Example Code
```python theme={null}
# Downloads all completed tasks from a batch
import json
import requests
API_KEY = 'live_...'
PROJECT_NAME = 'My Test Project'
BATCH_NAME = 'My Test Batch'
def get_tasks_by_batch(project_name: str, batch_name: str):
tasks = []
params = {
"project_name": project_name,
"batch_name": batch_name,
"status": "completed"
}
should_fetch = True
while should_fetch:
response = requests.request(
"GET",
url="https://api.scale.com/v2/tasks",
params=params,
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
json_resp = response.json()
next_token = json_resp.get('next_token')
if next_token:
params['next_token'] = next_token
else:
should_fetch = False
tasks.extend(json_resp['tasks'])
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_batch(PROJECT_NAME, BATCH_NAME)
with open(f'{BATCH_NAME}.jsonl', 'w+') as f:
for task in tasks:
json.dump(task, f)
f.write('\n')
```