> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genai.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.
# RLHF
> Reinforcement Learning from Human Feedback
RLHF tasks are structured datasets used to train and fine-tune Large Language Models (LLMs) using human feedback. These tasks help align model behavior with human preferences by providing paired responses with clear preference indicators.
## Overview
RLHF tasks consist of prompts and multiple model-generated responses, where human annotators provide explicit preferences between responses. This feedback helps models learn which outputs are more desirable according to human judgment across various dimensions such as Helpfulness, Accuracy, Safety, Writing quality, and Task completion.
## Task Structure
An RLHF task contains:
* A prompt (question or instruction)
* Multiple model generated responses
* Human preference annotations
* Detailed feedback and justification
RLHF tasks can be followed by an SFT stage where the preferred model's
response is rewritten
### Messages
Each turn consists of sequential [messages](../core-resources/message) that represent a user prompt, multiple model responses, and corresponding human preference annotations.
#### User message
Contains the initial prompt or instruction (role: `user`).
```json Sample Instruction theme={null}
{
"content": {
"text": "This is a test prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
}
```
```python get_rlhf_user_prompts.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to user prompt for every turn
def get_rlhf_user_prompts(task):
thread = task['threads'][0]
user_prompts = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['source_id'].lower() == "user"]
assert len(responses) == 1, "Turn must contain a user prompt. Turn ID: {}".format(turn['id'])
user_prompts[turn['id']] = responses[0]
return user_prompts
```
#### Model Responses
Contains responses from different models (role: `assistant`). Optionally includes `rewrite` annotations to store human rewritten responses.
```json Sample Model Response theme={null}
[
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{ "key": "instruction_following", "value": 1 },
{ "key": "truthfulness", "value": 1 },
{
"key": "truthfulnes_justiifcation",
"value": "Justification for the rating"
},
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 }
]
},
{
"content": {
"text": "This is model 2 response"
},
"role": "assistant",
"source_id": "model_2",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 3 },
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 },
{
"key": "rewrite",
"value": "Model 2 rewritten response"
}
]
}
]
```
```python get_rlhf_model_responses.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to model responses for every turn
def get_rlhf_model_responses(task):
thread = task['threads'][0]
model_responses = {}
for turn in thread['turns']:
responses = [msg for msg in turn['messages'] if msg['role'].lower() == "assistant"]
assert len(responses) > 0, "Turn must contain at least one model response. Turn ID: {}".format(turn['id'])
model_responses[turn['id']] = responses
return model_responses
```
Each model response includes a `source_id` that uniquely identifies the model
that generated the response.
#### Message Annotations
Each model response is evaluated across multiple dimensions, which may include:
* Instruction following
* Truthfulness
* Factuality
* Tone
If the model response is rewritten, the `rewrite` annotation is added to the list.
Message's [`annotations`](../core-resources/annotation) include the ratings for each dimension.
The rating dimensions are flexible and can be customized based on project
requirements and objectives.
### Turn-Level Annotations
The `annotations` at the **turn** level, specifies preference related or aggregated information. Some common examples are:
* Selected model identifier: `selected_model_id`
* Likert scale rating: `likert_value`
* Detailed justification for the selection: `justification`
* Any other comparative analysis between model responses
```json Sample Turn-Level Annotations theme={null}
[
{
"key": "selected_model_id",
"value": "base_model"
},
{
"key": "likert_value",
"value": 2
},
{
"key": "justification",
"value": "@Response 1 is better than @Response 2. @Response 2 has an issue in Truthfulness ..."
}
]
```
```python get_turn_annotations.py theme={null}
# task: output of `/v2/task`
# returns a map of turn_id to turn-level annotations for every turn
def get_turn_annotations(task):
thread = task['threads'][0]
turn_annotations = {}
for turn in thread['turns']:
turn_annotations[turn['id']] = turn['annotations']
return turn_annotations
```
### Expanded RLHF Task Output
This is a sample expanded sample RLHF Task output returned by [`/v2/task`](../v2/task).
```json Sample RLHF Task Output theme={null}
{
"task_id": "task_123",
"project": "project_123",
"batch": "batch_123",
"status": "completed",
"created_at": "2025-01-01T08:31:03.169Z",
"completed_at": "2025-01-02T04:00:39.923Z",
"threads": [
{
"id": "thread_0",
"turns": [
{
"id": "turn_0",
"messages": [
{
"content": {
"text": "This is a test prompt"
},
"role": "user",
"source_id": "user",
"annotations": []
},
{
"content": {
"text": "This is model 1 response"
},
"role": "assistant",
"source_id": "model_1",
"annotations": [
{ "key": "instruction_following", "value": 1 },
{ "key": "truthfulness", "value": 1 },
{
"key": "truthfulnes_justiifcation",
"value": "Justification for the rating"
},
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 }
]
},
{
"content": {
"text": "This is model 2 response"
},
"role": "assistant",
"source_id": "model_2",
"annotations": [
{ "key": "instruction_following", "value": 3 },
{ "key": "truthfulness", "value": 3 },
{ "key": "factuality", "value": 2 },
{ "key": "tone", "value": 2 },
{
"key": "rewrite",
"value": "Model 2 rewritten response"
}
]
}
],
"annotations": [
{
"key": "selected_model_id",
"value": "model_2"
},
{
"key": "likert_value",
"value": 6
},
{
"key": "justification",
"value": "@Response 2 is much better than @Response 1 as @Response 1 made a mistake in ..."
}
]
}
],
"annotations": []
}
]
}
```