This README explains how to generate user tasks for each application using the primitive_operation and primitive_operation_composition pipeline with enhanced control over instruction diversity and rephrasing.

Each application requires a primitive operation JSON file describing its low-level actions.
This file defines:

• Primitive operations (e.g., BingSearchLaunch, InsertImage, etc.)
• Arguments for each operation.
• Instruction templates for human-readable instructions.
• Argument generators to produce realistic values automatically.

The pipeline maps each argument to a generator function that yields realistic values.
Common generators are implemented in argument_value_generator.py, including:

• generate_random_number
• select_from_options
• select_file_path_in_directory
• generate_string

If your application requires custom logic, implement a new generator and register it in ARGUMENT_GENERATORS.

📄 Example: bingsearch_primitive_operation.json

A composition combines primitive operations into a higher-level user task.

• id – unique identifier for the composition.
• steps – ordered list of primitive operations (each may include argument overrides).

• Missing arguments are automatically filled by default argument generators.
• If a step specifies argument values, those override the generated ones.

• If a composition has no explicit instruction, the script automatically composes one from step-level instructions.
• If an instruction is provided in the composition, it is directly used for the user task.

📄 Example: bingsearch_primitive_operation_composition.json

Run the generator to produce user tasks from the compositions.

python user_task_generation/user_task_generator.py   --primitive-operation ./asset/primitive_operation/bingsearch_primitive_operation.json   --composition ./asset/primitive_operation_composition/bingsearch_primitive_operation_composition.json   --app-name bingsearch   --out-dir ./asset/user_task   --num-tasks 10000   --instruction-dropoff-prob-range '[0.1,0.2]'   --llm-rephrase-prob-range '[0.9]'   --launch-app-instruction-dropoff-prob '[0.6]'

During synthesis, each step instruction can be dropped with a probability sampled from the provided range.

• Example:
  • [0.2] → always drop 20% of steps.
  • [0.1, 0.5] → randomly pick a drop probability between 0.1–0.5 for each task.

If a step’s primitive operation ends with "Launch", it has an additional drop probability defined by:

--launch-app-instruction-dropoff-prob '[0.5]'

This avoids redundant phrases like “Launch Bing Search.”

Optionally, the script uses Qwen2.5-VL-7B-Instruct to rephrase the final instruction into a more natural, concise user command.
Triggered when a random value < --llm-rephrase-prob-range.

Example:

--llm-rephrase-prob-range '[0.1, 0.3]'

This gives a 10–30% chance of rephrasing each task instruction.

📘 The model runs locally on CUDA:0 (bf16 with FlashAttention2). If you only need text processing, a lightweight model or stub can be substituted.

The script reports generation progress using tqdm:

Generating user tasks: 73%|███████▎ | 7300/10000 [00:42<00:15, 180.2 task/s] attempts=7450

The generated file contains all tasks keyed by unique IDs:

{
  "bingsearch_basic_000001_4f9a2c91a1b2c3d4": {
    "id": "bingsearch_basic_000001_4f9a2c91a1b2c3d4",
    "instruction": "Search for AI trends and open the first result.",
    "domain": "bingsearch",
    "steps": [
      {
        "primitive_operation": "BingSearchLaunch",
        "instruction": "Open Bing Search.",
        "arguments": {}
      },
      {
        "primitive_operation": "BingSearchQuery",
        "instruction": "Search for AI trends.",
        "arguments": {"query": "AI trends"}
      },
      {
        "primitive_operation": "BingResultClick",
        "instruction": "Open the first result.",
        "arguments": {"rank": 1}
      }
    ]
  }
}

By default, the generated file is saved to:

./asset/user_task/{app_name}_tasks.json

You can override it with:

--out-file custom_tasks.json