This repository contains the training code for our inverse dynamics model for screencasts. Please refer to our blog post for more details.

We release the trained model checkpoint and corresponding evaluation clips on Hugging Face.

• actions.py: action parsing, normalization, and scoring
• prompts.py: canonical IDM prompt
• data.py: processed dataset and collator used for SFT
• prepare_data.py: raw crowd-cast recordings to training clips
• train.py: Qwen3-VL LoRA training entrypoint
• infer.py: sliding-window evaluation on short clips
• label_agicast.py: label long AGI-CAST videos with a served IDM
• merge_and_save.py: merge a trained LoRA checkpoint into the base model
• tests/: unit tests for action conversion, scoring, and collation

To install the requirements, run:

uv sync
uv pip install flash-attn --no-build-isolation

If you encounter issues with FlashAttention, you can run the training with --attn-implementation auto. You also need ffmpeg available on PATH.

The model expects 10 frames sampled at 5 FPS. Each frame should be preceded by a text anchor such as Frame F00:, Frame F01:, etc. The output is a JSON array:

[
  {"frame": "F02", "type": "MouseMove", "details": "120,45"},
  {"frame": "F03", "type": "MouseClick", "details": "Left"},
  {"frame": "F05", "type": "KeyPress", "details": "Cmd+S"}
]

Download or clone the eval set first:

huggingface-cli download p-doom/idm-eval-set \
  --repo-type dataset \
  --local-dir idm-eval-set

Start an OpenAI-compatible server for the model, for example with sglang, then run:

uv run python infer.py \
  --clips-dir idm-eval-set \
  --api-url http://localhost:30000/v1 \
  --model-id p-doom/idm \
  --fps 5 \
  --interleave-labels \
  --max-pixels 524288 \
  --output results.json

infer.py runs overlapping 10-frame windows with stride 5 and keeps only the non-overlapping scoring zones.

Training data is not part of this repository. train.py expects processed clips created by prepare_data.py, with train/train.jsonl under --data-dir.

We train our model on Qwen3-VL 8B with LoRA on both language and vision modules:

uv run torchrun --nproc_per_node 8 train.py \
  --model-id Qwen/Qwen3-VL-8B-Instruct \
  --data-dir /path/to/processed-data \
  --out-dir runs/idm-8b \
  --video-mode image \
  --interleave-labels \
  --train-vision \
  --lora-r 256 \
  --lora-alpha 512 \
  --batch-size 2 \
  --grad-accum 1 \
  --max-steps 5000 \
  --lr 2e-5 \
  --warmup-steps 500 \
  --wsd-decay-steps 1333 \
  --max-pixels 524288 \
  --max-length 10000

After training, merge the adapter:

uv run python merge_and_save.py \
  --model-id Qwen/Qwen3-VL-8B-Instruct \
  --checkpoint runs/idm-8b/step_5000/checkpoint.pt \
  --output-dir merged/idm-8b \
  --lora-r 256 \
  --lora-alpha 512 \
  --train-vision

As a demo, we also provide a script to label long AGI-CAST videos using our IDM. label_agicast.py labels videos without ground truth actions. It expects an OpenAI-compatible model endpoint:

uv run python label_agicast.py \
  --root /path/to/AGI-CAST-videos \
  --api-url http://localhost:30000/v1 \
  --model-id p-doom/idm \
  --output-dir agicast-idm-actions \
  --fps 5 \
  --chunk-frames 10 \
  --max-pixels 524288

uv run pytest -q

• The action format is intentionally sparse: frames with no user input are omitted.
• Mouse deltas are normalized to a 0-1000 per-axis screen scale.
• The released checkpoint was trained on macOS recordings; OS-specific keyboard shortcuts may not transfer perfectly to Windows or Linux.