|
| 1 | +# Label View |
| 2 | + |
| 3 | +## What is a Label View? |
| 4 | + |
| 5 | +A **label view** is a Feast primitive for storing **judgments about entities** — reward signals, safety scores, human reviews, ground-truth answers — separately from the **immutable observations** stored in [feature views](feature-view.md). |
| 6 | + |
| 7 | +| | Feature View | Label View | |
| 8 | +|---|---|---| |
| 9 | +| **Stores** | What was observed | What was judged | |
| 10 | +| **Example** | Agent prompt and response | `response_quality: "poor"`, `is_safe: 0` | |
| 11 | +| **Writers** | Usually one source | Multiple labelers (human, LLM judge, code) | |
| 12 | +| **Changes over time** | Append-only | Updated by new label writes | |
| 13 | + |
| 14 | +Use a label view when you need **governed, training-ready labels** that multiple sources can write, disagree on, and resolve — not when you need append-only feature data. |
| 15 | + |
| 16 | +## Prerequisites |
| 17 | + |
| 18 | +Before using label views, you need: |
| 19 | + |
| 20 | +* Feast with the LabelView primitive |
| 21 | +* A feature repo with at least one `Entity` |
| 22 | +* A `PushSource` (or batch `DataSource`) for label ingestion |
| 23 | +* `feast apply` run after defining label views |
| 24 | + |
| 25 | +## Why use Label Views? |
| 26 | + |
| 27 | +**Separate features from judgments.** Mixing reward labels into feature views blurs two different lifecycles — observations are append-only; labels are overwritten, corrected, and debated. |
| 28 | + |
| 29 | +**Support multiple labelers.** Human reviewers, safety scanners, and LLM judges can all label the same entity. Feast tracks who wrote what via `labeler_field` and resolves conflicts via `ConflictPolicy`. |
| 30 | + |
| 31 | +**Generate training datasets.** Compose label views with feature views in a `FeatureService` and retrieve features + labels together with point-in-time correctness. |
| 32 | + |
| 33 | +**Annotate in the UI.** Configure annotation profiles so data scientists can label data directly in the Feast UI — entity forms, document spans, bulk review, or active learning. |
| 34 | + |
| 35 | +## Feedback vs Expectations |
| 36 | + |
| 37 | +Not all labels serve the same purpose. Feast distinguishes two common types — both stored in a label view, modeled with field names and tags: |
| 38 | + |
| 39 | +| | **Feedback** | **Expectation** | |
| 40 | +|---|---|---| |
| 41 | +| **Question** | How good was the actual output? | What is the correct answer? | |
| 42 | +| **Example** | `response_quality: "poor"`, `relevance: "irrelevant"` | `ground_truth: "relevant"`, `is_default: 1` | |
| 43 | +| **Typical writers** | Human, LLM judge, automated code | Human experts (gold standard) | |
| 44 | +| **Training use** | Reward signal, quality filter, active-learning queue | Supervised target column | |
| 45 | +| **Conflict handling** | Common — use `ConflictPolicy` | Rare — usually one authoritative source | |
| 46 | + |
| 47 | +Feast does not require separate primitives for feedback and expectations. Model them with **field names** and **tags**: |
| 48 | + |
| 49 | +```python |
| 50 | +tags={ |
| 51 | + "feast.io/field-role:response_quality": "feedback", # judgment about output |
| 52 | + "feast.io/field-role:ground_truth": "expectation", # correct answer |
| 53 | +} |
| 54 | +``` |
| 55 | + |
| 56 | +**Practical pattern:** |
| 57 | + |
| 58 | +* **Feedback label view** — multi-labeler, `ConflictPolicy`, fields like `response_quality`, `safety_score`, `relevance` |
| 59 | +* **Expectation fields or view** — stable ground truth, fields like `ground_truth`, `expected_answer`, `is_default` |
| 60 | +* **Mixed view** — one label view with both (e.g. RAG: `relevance` = feedback, `ground_truth` = expectation) |
| 61 | + |
| 62 | +## Sources of Labels |
| 63 | + |
| 64 | +Label views accept labels from any source. Track the writer in `labeler_field`: |
| 65 | + |
| 66 | +| Source | `labeler` example | Typical role | |
| 67 | +|---|---|---| |
| 68 | +| **Human reviewer** | `human-reviewer@company.com` | Feedback or expectation | |
| 69 | +| **LLM judge** | `gpt-4-evaluator` | Feedback (quality scores) | |
| 70 | +| **Automated scanner** | `nemo-guardrails` | Feedback (safety signals) | |
| 71 | +| **Batch import** | `risk-ops-team` | Expectation (historical outcomes) | |
| 72 | + |
| 73 | +```python |
| 74 | +labels_df = pd.DataFrame({ |
| 75 | + "user_id": ["user-001"], |
| 76 | + "response_quality": ["poor"], |
| 77 | + "is_safe": [0], |
| 78 | + "labeler": ["human-reviewer@company.com"], |
| 79 | + "event_timestamp": [pd.Timestamp.now()], |
| 80 | +}) |
| 81 | +store.push("agent_feedback_labels_push_source", labels_df) |
| 82 | +``` |
| 83 | + |
| 84 | +## When to use Label Views |
| 85 | + |
| 86 | +| Use a **FeatureView** when… | Use a **LabelView** when… | |
| 87 | +|---|---| |
| 88 | +| Data is observational and append-only | Data is a judgment or annotation | |
| 89 | +| One source writes the data | Multiple labelers may disagree | |
| 90 | +| No conflict resolution needed | You need governed conflict resolution | |
| 91 | +| No labeling UI needed | You want structured annotation workflows | |
| 92 | + |
| 93 | +## How Label Views Work |
| 94 | + |
| 95 | +### Step 1: Define a label view |
| 96 | + |
| 97 | +```python |
| 98 | +from datetime import timedelta |
| 99 | + |
| 100 | +from feast import Entity, FeatureService, Field, FileSource, PushSource |
| 101 | +from feast.labeling import ConflictPolicy, LabelView |
| 102 | +from feast.types import Float32, Int64, String |
| 103 | + |
| 104 | +user = Entity(name="user_id", join_keys=["user_id"]) |
| 105 | + |
| 106 | +agent_feedback_labels = LabelView( |
| 107 | + name="agent_feedback_labels", |
| 108 | + entities=[user], |
| 109 | + schema=[ |
| 110 | + Field(name="response_quality", dtype=String), |
| 111 | + Field(name="is_safe", dtype=Int64), |
| 112 | + Field(name="reviewer_notes", dtype=String), |
| 113 | + Field(name="labeler", dtype=String), |
| 114 | + ], |
| 115 | + source=PushSource( |
| 116 | + name="agent_feedback_labels_push_source", |
| 117 | + batch_source=FileSource( |
| 118 | + name="agent_feedback_labels_batch", |
| 119 | + path="data/agent_feedback.parquet", |
| 120 | + timestamp_field="event_timestamp", |
| 121 | + ), |
| 122 | + ), |
| 123 | + labeler_field="labeler", |
| 124 | + conflict_policy=ConflictPolicy.LAST_WRITE_WINS, |
| 125 | + reference_feature_view="user_profile", |
| 126 | + description="Human and automated feedback on agent responses.", |
| 127 | + tags={ |
| 128 | + "feast.io/labeling-method": "entity-form", |
| 129 | + "feast.io/field-role:response_quality": "feedback", |
| 130 | + "feast.io/field-role:is_safe": "feedback", |
| 131 | + "feast.io/field-role:reviewer_notes": "metadata", |
| 132 | + "feast.io/label-values:response_quality": "excellent,good,acceptable,poor,harmful", |
| 133 | + "feast.io/label-values:is_safe": "1,0", |
| 134 | + "feast.io/label-widget:response_quality": "enum", |
| 135 | + "feast.io/label-widget:is_safe": "binary", |
| 136 | + "feast.io/label-widget:reviewer_notes": "text", |
| 137 | + }, |
| 138 | +) |
| 139 | +``` |
| 140 | + |
| 141 | +Run `feast apply` to register the label view. |
| 142 | + |
| 143 | +### Step 2: Push labels |
| 144 | + |
| 145 | +Labels are written with `FeatureStore.push()`: |
| 146 | + |
| 147 | +```python |
| 148 | +import pandas as pd |
| 149 | +from feast import FeatureStore |
| 150 | + |
| 151 | +store = FeatureStore(repo_path="feature_repo/") |
| 152 | + |
| 153 | +labels_df = pd.DataFrame({ |
| 154 | + "user_id": ["user-001", "user-002"], |
| 155 | + "response_quality": ["good", "harmful"], |
| 156 | + "is_safe": [1, 0], |
| 157 | + "reviewer_notes": ["Accurate summary", "Unsafe medical advice"], |
| 158 | + "labeler": ["human-reviewer", "nemo-guardrails"], |
| 159 | + "event_timestamp": pd.to_datetime(["2025-01-15", "2025-01-15"]), |
| 160 | +}) |
| 161 | + |
| 162 | +store.push("agent_feedback_labels_push_source", labels_df) |
| 163 | +``` |
| 164 | + |
| 165 | +Each push appends to the offline store (full history retained) and updates the online store (latest value per key). |
| 166 | + |
| 167 | +### Step 3: Annotate in the Feast UI |
| 168 | + |
| 169 | +Open the label view in the Feast UI **Annotate** tab. The UI reads annotation tags and shows the right workflow: |
| 170 | + |
| 171 | +1. Open **Label Views** in the sidebar |
| 172 | +2. Select a label view (check the **Annotation** badge on the list page) |
| 173 | +3. Go to the **Annotate** tab |
| 174 | +4. Choose a method (Entity Form, Document Span, Review & Edit, or Active Learning) |
| 175 | +5. Submit labels — they are pushed to the label view's `PushSource` |
| 176 | + |
| 177 | +### Step 4: Join labels with features for training |
| 178 | + |
| 179 | +```python |
| 180 | +training_service = FeatureService( |
| 181 | + name="agent_training_service", |
| 182 | + features=[ |
| 183 | + user_profile_fv, # immutable features |
| 184 | + agent_feedback_labels, # mutable labels |
| 185 | + ], |
| 186 | +) |
| 187 | + |
| 188 | +training_df = store.get_historical_features( |
| 189 | + entity_df=entities_df, |
| 190 | + features=training_service, |
| 191 | +).to_df() |
| 192 | +``` |
| 193 | + |
| 194 | +Training pipelines get features and resolved labels in one retrieval call. |
| 195 | + |
| 196 | +## Annotation Profiles |
| 197 | + |
| 198 | +Annotation profiles configure **how** labels are created in the UI. Set them via `tags` — no schema changes required. |
| 199 | + |
| 200 | +### Supported profiles |
| 201 | + |
| 202 | +| Profile | Best for | UI experience | |
| 203 | +|---|---|---| |
| 204 | +| `entity-form` | RLHF, safety review, per-entity feedback | Form — one entity at a time | |
| 205 | +| `document-span` | RAG chunk labeling, span annotation | Load document, label chunks | |
| 206 | +| `table` | Bulk review, correcting existing labels | Editable table with dropdowns | |
| 207 | +| `active-learning` | Label high-value unlabeled entities | Queue from a reference feature view | |
| 208 | + |
| 209 | +### Choosing a profile |
| 210 | + |
| 211 | +Answer one question: |
| 212 | + |
| 213 | +1. **"I need to review agent responses one at a time"** → `entity-form` |
| 214 | +2. **"I need to label document chunks for RAG"** → `document-span` |
| 215 | +3. **"I need to correct labels in bulk"** → `table` |
| 216 | +4. **"I want to label only the most valuable unlabeled items"** → `active-learning` (requires `reference_feature_view`) |
| 217 | + |
| 218 | +The **Annotate** tab shows only relevant methods per profile: |
| 219 | + |
| 220 | +| Profile | Methods shown | |
| 221 | +|---|---| |
| 222 | +| `document-span` | Document Span, Review & Edit | |
| 223 | +| `entity-form` | Entity Form, Review & Edit, Active Learning | |
| 224 | +| `table` | Review & Edit, Active Learning, Entity Form | |
| 225 | +| `active-learning` | Active Learning, Entity Form, Review & Edit | |
| 226 | + |
| 227 | +### Tag reference |
| 228 | + |
| 229 | +| Tag | Purpose | Example values | |
| 230 | +|---|---|---| |
| 231 | +| `feast.io/labeling-method` | Primary UI workflow | `entity-form`, `document-span`, `table` | |
| 232 | +| `feast.io/field-role:<field>` | Semantic role of a field | `feedback`, `expectation`, `label`, `metadata`, `content`, `span_start`, `span_end` | |
| 233 | +| `feast.io/label-values:<field>` | Allowed label values | `relevant,irrelevant` | |
| 234 | +| `feast.io/label-widget:<field>` | Input widget type | `enum`, `binary`, `text`, `number` | |
| 235 | + |
| 236 | +## Examples by use case |
| 237 | + |
| 238 | +### Agent feedback (RLHF / safety) |
| 239 | + |
| 240 | +Feedback from human reviewers and automated safety layers on agent responses. |
| 241 | + |
| 242 | +```python |
| 243 | +agent_feedback_labels = LabelView( |
| 244 | + name="agent_feedback_labels", |
| 245 | + entities=[user], |
| 246 | + schema=[ |
| 247 | + Field(name="response_quality", dtype=String), |
| 248 | + Field(name="is_safe", dtype=Int64), |
| 249 | + Field(name="reviewer_notes", dtype=String), |
| 250 | + Field(name="labeler", dtype=String), |
| 251 | + ], |
| 252 | + source=feedback_push_source, |
| 253 | + labeler_field="labeler", |
| 254 | + conflict_policy=ConflictPolicy.LAST_WRITE_WINS, |
| 255 | + reference_feature_view="user_profile", |
| 256 | + tags={"feast.io/labeling-method": "entity-form", ...}, |
| 257 | +) |
| 258 | +``` |
| 259 | + |
| 260 | +### RAG chunk labeling (feedback + expectation) |
| 261 | + |
| 262 | +One view can hold both a retrieval judgment and ground truth: |
| 263 | + |
| 264 | +```python |
| 265 | +rag_chunk_labels = LabelView( |
| 266 | + name="rag_chunk_labels", |
| 267 | + entities=[chunk], |
| 268 | + schema=[ |
| 269 | + Field(name="source_document", dtype=String), |
| 270 | + Field(name="chunk_text", dtype=String), |
| 271 | + Field(name="relevance", dtype=String), # feedback |
| 272 | + Field(name="ground_truth", dtype=String), # expectation |
| 273 | + Field(name="chunk_start", dtype=Int64), |
| 274 | + Field(name="chunk_end", dtype=Int64), |
| 275 | + Field(name="labeler", dtype=String), |
| 276 | + ], |
| 277 | + source=rag_push_source, |
| 278 | + labeler_field="labeler", |
| 279 | + conflict_policy=ConflictPolicy.MAJORITY_VOTE, |
| 280 | + tags={ |
| 281 | + "feast.io/labeling-method": "document-span", |
| 282 | + "feast.io/field-role:relevance": "feedback", |
| 283 | + "feast.io/field-role:ground_truth": "expectation", |
| 284 | + "feast.io/field-role:chunk_text": "content", |
| 285 | + "feast.io/label-values:relevance": "relevant,irrelevant", |
| 286 | + }, |
| 287 | +) |
| 288 | +``` |
| 289 | + |
| 290 | +### Historical ground truth (batch labels) |
| 291 | + |
| 292 | +Pre-existing label tables loaded via `feast materialize`: |
| 293 | + |
| 294 | +```python |
| 295 | +loan_default_labels = LabelView( |
| 296 | + name="loan_default_labels", |
| 297 | + entities=[loan], |
| 298 | + schema=[ |
| 299 | + Field(name="is_default", dtype=Int64), |
| 300 | + Field(name="delinquency_days", dtype=Int64), |
| 301 | + Field(name="loss_severity", dtype=Float32), |
| 302 | + Field(name="labeler", dtype=String), |
| 303 | + ], |
| 304 | + source=PushSource( |
| 305 | + name="loan_default_labels_push_source", |
| 306 | + batch_source=FileSource( |
| 307 | + path="data/loan_defaults.parquet", |
| 308 | + timestamp_field="event_timestamp", |
| 309 | + ), |
| 310 | + ), |
| 311 | + labeler_field="labeler", |
| 312 | + conflict_policy=ConflictPolicy.LABELER_PRIORITY, |
| 313 | + tags={ |
| 314 | + "feast.io/labeling-method": "table", |
| 315 | + "feast.io/field-role:is_default": "expectation", |
| 316 | + }, |
| 317 | +) |
| 318 | +``` |
| 319 | + |
| 320 | +## Conflict policies |
| 321 | + |
| 322 | +When multiple labelers write different values for the same entity, `ConflictPolicy` picks one value for **offline store reads** (training, UI browse): |
| 323 | + |
| 324 | +| Policy | When to use | |
| 325 | +|---|---| |
| 326 | +| `LAST_WRITE_WINS` | Default. Most recent write wins. | |
| 327 | +| `LABELER_PRIORITY` | Trusted labelers override others (e.g. human over LLM judge). | |
| 328 | +| `MAJORITY_VOTE` | Consensus labeling (e.g. multiple annotators on RAG chunks). | |
| 329 | + |
| 330 | +{% hint style="info" %} |
| 331 | +Conflict policies apply to the **offline store** (training). The **online store** always uses last-write-wins. Full label history is always retained in the offline store. |
| 332 | +{% endhint %} |
| 333 | + |
| 334 | +## Best practices |
| 335 | + |
| 336 | +**Name fields by intent.** Use `response_quality` (feedback) and `ground_truth` (expectation) — not generic `score` or `label`. |
| 337 | + |
| 338 | +**Tag field roles.** Set `feast.io/field-role:<field>` to `feedback` or `expectation` so your team and UI know what each field means. |
| 339 | + |
| 340 | +**Match conflict policy to label type.** Use `LABELER_PRIORITY` when humans correct automated judges. Use `MAJORITY_VOTE` for multi-annotator consensus. Use `LAST_WRITE_WINS` for simple feedback streams. |
| 341 | + |
| 342 | +**Link to features.** Set `reference_feature_view` so the UI and documentation show which feature view the labels annotate. |
| 343 | + |
| 344 | +**Separate noisy feedback from stable ground truth.** When possible, put expectations in dedicated fields or views with stricter writer conventions (human-only). |
| 345 | + |
| 346 | +## Limitations |
| 347 | + |
| 348 | +* Conflict policies are enforced on offline reads only; online store is always last-write-wins. |
| 349 | +* `LABELER_PRIORITY` requires explicit labeler ordering configuration. |
| 350 | +* Annotation profiles are UI configuration via tags — not enforced at the SDK write path. |
| 351 | + |
| 352 | +## Next steps |
| 353 | + |
| 354 | +* [Feature view](feature-view.md) — immutable features that label views annotate |
| 355 | +* [Feature retrieval](feature-retrieval.md) — point-in-time joins for training |
| 356 | +* [ADR-0012: LabelView](../../adr/ADR-0012-label-view.md) — full design rationale |
0 commit comments