curl --request GET \
--url https://api.scale.com/v2/delivery \
--header 'Authorization: Bearer <token>'{
"tasks": [
{
"task_id": "task_abc123",
"project": "project_abc123",
"status": "pending",
"created_at": "2023-11-07T05:31:56Z",
"batch": "batch_abc123",
"completed_at": "2023-11-07T05:31:56Z",
"metadata": {},
"threads": [
{
"id": "thread_abc123",
"turns": [
{
"id": "turn_abc123",
"messages": [
{
"role": "system",
"content": {
"text": "<string>",
"reference_texts": [
{
"content": "<string>",
"category": "<string>",
"url": "<string>"
}
],
"attachments": [
{
"content": "aSDinaTvuI8gbWludGxpZnk=",
"mime_type": "text/plain",
"scale_url": "<string>",
"url": "<string>",
"name": "<string>"
}
],
"chunks": [
{
"type": "<string>",
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
],
"text": "<string>"
}
],
"reasoning": [
{
"content": "<string>"
}
]
},
"source_id": "source_abc123",
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
],
"model_parameters": {
"model": "my-model-123",
"temperature": 0.9,
"max_completion_tokens": 1000,
"top_p": 0.9,
"top_k": 4
}
}
],
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
]
}
],
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
]
}
],
"errors": [
{
"type": "UNSUPPORTED_LANGUAGE",
"message": "The specified language or locale is not supported"
}
],
"sensitive_content_reports": [
{
"type": "violence",
"message": "The prompt in the task promotes violence."
}
]
}
],
"delivery": {
"id": "delivery_abc123",
"name": "My Scale Delivery",
"task_count": 1000,
"delivered_at": "2023-11-07T05:31:56Z",
"project": "project_abc123"
},
"next_token": "<string>"
}{
"status_code": 500,
"error": "An error has occurred."
}Deliveries
Get Tasks in a Delivery
GET
/
v2
/
delivery
curl --request GET \
--url https://api.scale.com/v2/delivery \
--header 'Authorization: Bearer <token>'{
"tasks": [
{
"task_id": "task_abc123",
"project": "project_abc123",
"status": "pending",
"created_at": "2023-11-07T05:31:56Z",
"batch": "batch_abc123",
"completed_at": "2023-11-07T05:31:56Z",
"metadata": {},
"threads": [
{
"id": "thread_abc123",
"turns": [
{
"id": "turn_abc123",
"messages": [
{
"role": "system",
"content": {
"text": "<string>",
"reference_texts": [
{
"content": "<string>",
"category": "<string>",
"url": "<string>"
}
],
"attachments": [
{
"content": "aSDinaTvuI8gbWludGxpZnk=",
"mime_type": "text/plain",
"scale_url": "<string>",
"url": "<string>",
"name": "<string>"
}
],
"chunks": [
{
"type": "<string>",
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
],
"text": "<string>"
}
],
"reasoning": [
{
"content": "<string>"
}
]
},
"source_id": "source_abc123",
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
],
"model_parameters": {
"model": "my-model-123",
"temperature": 0.9,
"max_completion_tokens": 1000,
"top_p": 0.9,
"top_k": 4
}
}
],
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
]
}
],
"annotations": [
{
"id": "an_abc123efg456",
"key": "formatting",
"type": "<string>",
"title": "Response Formatting",
"description": "Does the response contain issues with formatting?",
"labels": [
"Major Issues",
"Minor Issues",
"No Issues"
],
"metadata": {
"criteria": "overall_quality"
},
"value": 3,
"possible_values": [
[
1,
2,
3
]
]
}
]
}
],
"errors": [
{
"type": "UNSUPPORTED_LANGUAGE",
"message": "The specified language or locale is not supported"
}
],
"sensitive_content_reports": [
{
"type": "violence",
"message": "The prompt in the task promotes violence."
}
]
}
],
"delivery": {
"id": "delivery_abc123",
"name": "My Scale Delivery",
"task_count": 1000,
"delivered_at": "2023-11-07T05:31:56Z",
"project": "project_abc123"
},
"next_token": "<string>"
}{
"status_code": 500,
"error": "An error has occurred."
}Authentication
Authentication
Every request sent to Scale’s API requires authentication. In short, your API Key is the Bearer token. See the Authentication section for more details.
Pagination
Pagination
Remember to handle pagination when downloading large sets of tasks.
limit).
If your request returns more tasks than your specified limit, API response will also contain a next_token until you reach to the last page.You can set the next_token in your next request to continue downloading tasks from the next page.Required query parameters
You are expected to provide one of the following to find a delivery:delivery_idDelivery IDs are globally unique and can be used to retrieve a specific delivery.delivery_namewith a project identifier (project_idorproject_name)
Example Code
Download All Tasks From A Delivery as JSONL
Download All Tasks From A Delivery as JSONL
# Downloads all tasks from a delivery
import json
import requests
API_KEY = 'live_...'
PROJECT_NAME = 'My Test Project'
DELIVERY_NAME = 'My Test Delivery'
def get_tasks_by_delivery(project_name: str, delivery_name: str):
tasks = []
params = {
"project_name": project_name,
"delivery_name": delivery_name,
}
should_fetch = True
while should_fetch:
response = requests.request(
"GET",
url="https://api.scale.com/v2/delivery",
params=params,
headers={
"Accept": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
)
json_resp = response.json()
next_token = json_resp.get('next_token')
if next_token:
params['next_token'] = next_token
else:
should_fetch = False
tasks.extend(json_resp['tasks'])
return tasks
if __name__ == "__main__":
tasks = get_tasks_by_delivery(PROJECT_NAME, DELIVERY_NAME)
with open(f'{DELIVERY_NAME}.jsonl', 'w+') as f:
for task in tasks:
json.dump(task, f)
f.write('\n')
Authorizations
bearerAuthbasicAuth
Your API Key is the Bearer token. See the Authentication section to learn how to access your key.
Query Parameters
Scale's unique identifier for the delivery. A unique identifier for the delivery.
Example:
"delivery_abc123"
The name of the delivery.
Example:
"My Scale Delivery"
Scale's unique identifier for the project. A unique identifier for the project.
Example:
"project_abc123"
The name of the project.
Example:
"My Scale Project"
Limit the number of entities returned.
Required range:
1 <= x <= 100A token used to retrieve the next page of results if there are more. You can find the next_token in your last request
List of properties to include in the task response. Additional properties that can be included in the annotations
Property that can be included in the annotations
Available options:
annotation_details, attachment_details, model_parameters Response
List of delivered tasks.