ft_filepicker_003

• Gemini Endpoint ID: 196166068534771712
• Finetune file: gemini-tune-data-002.jsonl, available in the GCS bucket
• Finetune samples: 2 (just Venki's data)
• Finetune date: Apr 14, 2025
• Base model: gemini-2.0-flash-001
• Distilled model: claude-3-5-sonnet-20241022
• Notes:
  • Test model, uses only 2 samples from Venki's local database
  • Method: blind distillation from Claude's results
• Est costs:
  • ~$0.1 each on finetuning & relabeling

ft_filepicker_005

• Gemini Endpoint ID: 8493203957034778624
• Finetune date: Apr 23, 2025
• Finetune file: gemini-tune-data-007.jsonl, available in the GCS bucket
• Finetune samples: 1934 messages
• Tokens: 74.6M
• Epochs: 2
• Base model: gemini-2.0-flash-001
• Distilled model: claude-3-5-sonnet-20241022
• Notes:
  • Forced maximum adapter size, but kept all the other Gemini "advanced" finetuning parameters at default values.
  • Has a minor data issue: Non-Obvious vs Key requests were not disambiguated, and thus requests were joined with relabels in an odd manner.
  • Seems okay for a test, but we might be encouraging the model to pick worse files etc.
  • Might also be worth trying out other models to teach (eg: 3.7-reasoning? o3?)
• Est costs:
  • ~$400 for finetuning ($3.00/1M tokens _ 2 epochs _ 74.6M tokens)
  • ~$200 for generating relabel inputs (at $3.00/1M input Claude tokens)

ft_filepicker_007

• Gemini Endpoint ID: 2589952415784501248
• Finetune date: May 29, 2025
• Finetune file: gemini-tune-data-027.jsonl, available in the GCS bucket
• Finetune samples: 1077 messages
• Tokens: 44.7M
• Epochs: 1
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Unlike previous finetunes, this also provided "additional file context" to the teacher model, so it not only used a stronger model, but also more informative context.
  • Forced maximum adapter size, and 1 epoch
  • Only used 'Key' traces AFAIK - TODO: Will this worsen non-obvious, or encourage 'Key' to pick too many?
• Est costs:
  • ~$150 for finetuning ($3.00/1M tokens _ 1 epoch _ 44.7M tokens)
  • over $660 for generating relabel inputs (at $15.00/1M input Claude tokens + some additional file context tokens)

ft_filepicker_topk_001

• Gemini Endpoint ID: 3676445825887633408
• Finetune date: May 30, 2025
• Finetune file: gemini-tune-data-039-top2.jsonl, available in the GCS bucket
• Validation file: gemini-tune-data-039-top2-validation.jsonl, available in the GCS bucket
• Finetune samples: 1432 messages
• Validation samples: 119 messages
• Tokens: 60.5M
• Epochs: 2
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Trained only on the top-2 file predictions from claude-opus-4
    • Otherwise, identical to the ft_filepicker_007 finetune
  • Forced maximum adapter size, and 2 epochs
• Est costs:
  • ~$300 for finetuning ($3.00/1M tokens _ 1 epoch _ 44.7M tokens)
  • Mostly reuses relabel inputs from other Claude runs - no additional cost anticipated.

ft_filepicker_008

• Gemini Endpoint ID: 2672143108984012800
• Finetune date: May 31, 2025
• Finetune file: gemini-tune-data-043.jsonl, available in the GCS bucket
• Validation file: gemini-tune-data-043.jsonl, available in the GCS bucket
• Finetune samples: 2648 messages
• Validation samples: 253 messages
• Tokens: 108M
• Epochs: 2
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Identical to ft_filepicker_007 - except more data and epochs
• Est costs:
  • $648 for finetuning ($3.00/1M tokens _ 2 epoch _ 108M tokens)
  • about $3K for generating relabel inputs (at $15.00/1M input Claude tokens + some additional file context tokens) (about lot of this reused from ft_filepicker_007 though)

ft_filepicker_topk_002

• Gemini Endpoint ID: 1694861989844615168
• Finetune date: May 31, 2025
• Finetune file: gemini-tune-data-045-top2.jsonl, available in the GCS bucket
• Validation file: gemini-tune-data-045-top2-validation.jsonl, available in the GCS bucket
• Finetune samples: 909 messages
• Validation samples: 71 messages
• Tokens: TBD
• Epochs: 2
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Similar to ft_filepicker_topk_001 but each input has three lines of dashes appended to the end - to encourage it to "stop" early
  • Super small training run to test: "does this work at all?" - thus fewer epochs and less data.
• Est costs:
  • TBD for finetuning ($3.00/1M tokens _ 1 epoch _ 44.7M tokens)
  • Mostly reuses relabel inputs from other Claude runs - no additional cost anticipated.

ft_filepicker_010

• Gemini Endpoint ID: 3808739064941641728
• Finetune date: June 2, 2025
• Finetune file: gemini-tune-data-048.jsonl
• Finetune samples: 2648 messages
• Tokens: 109M
• Epochs: 3
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Same training data as ft_filepicker_008 but with assistant messages converted to user messages
  • This preprocessing change aims to improve the model's ability to handle conversation context
  • Uses the BLOBBIFY_MESSAGE_HISTORY approach where assistant messages are wrapped in <previous_assistant_message> tags and converted to user messages
  • 3 epochs instead of 2 to improve results
  • Also try ft_filepicker_010_epoch_2 if you wanna compare what one less epoch does.
• Est costs:
  • ~$980 for finetuning ($3.00/1M tokens _ 3 epochs _ 109M tokens)
  • Reuses relabel inputs from ft_filepicker_008 - no additional labeling cost

ft_filepicker_topk_003

• Gemini Endpoint ID: 1502192368286171136
• Finetune date: Jun 5, 2025
• Finetune file: gemini-tune-data-048-top2.jsonl, available in the GCS bucket
• Finetune samples: 2648 messages
• Tokens: 109M
• Epochs: 2
• Base model: gemini-2.0-flash-001
• Distilled model: claude-opus-4-20250514
• Notes:
  • Combines characteristics of ft_filepicker_topk_002 and ft_filepicker_010, ie: uses top-2, with 3 lines of dashes at the end, but also blobbifies message history.
• Est costs:
  • ~$660 for finetuning ($3.00/1M tokens _ 2 epochs _ 109M tokens)
  • Mostly reuses relabel inputs from other Claude runs - no additional cost anticipated.

Contains a variety of scripts for inspecting and processing finetuning data from BigQuery.

print-recent-traces.ts [--prod] - Prints the most recent traces from BigQuery.

print-recent-relabels.ts [--prod] - Prints the most recent relabels from BigQuery.

relabel-traces.ts [--prod] - Relabels traces in BigQuery, ie: generates new outputs using more powerful models for real production inputs.

relabel-traces-with-context.ts [--prod] - Relabels traces using the uploaded "full file context"

collect-tuning-data.ts <model> [--prod] - Downloads tuning data from BigQuery, using real inputs + relabeled outputs. The model parameter controls which relabeler model is used.

relabel-for-offline-scoring.ts [--relabel|--score] [--prod] - Relabels validation traces for offline evaluation or scores existing relabels using metrics like Jaccard similarity and MRR.