Inducing Task Models from Computer-Use Traces

The 30-second take

• What: Task Model Induction discovers latent tasks in unconstrained computer-use traces and, for each, builds a hierarchical objective tree paired with a procedure (control-flow) model.
• Why it matters: Everyday work knowledge is tacit and hard to audit. Structured task models are a step toward reusable how-to knowledge as a default layer for agents and organizations.
• Who should care: Computer-use agent researchers, process-mining and audit teams, and enterprises that need inspectable models of digital work.

What the paper actually did

Passively recorded screenshots plus mouse and keyboard events are a rich log of how work actually happens, but they are low-level and multi-threaded. Existing methods assume a known task or a single workflow and emit step summaries rather than structured task models.

Task Model Induction (TMI) does two jobs. First, it discovers latent tasks in an unconstrained trace and disentangles concurrent activity. Second, for each latent task it induces a model that pairs a hierarchical objective model (recursive goal decomposition) with a procedure model (the control flow that organized execution).

On controlled human and agent trajectories, TMI matches ground-truth task groupings with 0.974 agreement and reconstructs 74.9% of observed execution steps—far above the strongest workflow-induction baseline. Skills derived from TMI’s models improve held-out task accuracy by 30.0% over the strongest baseline.

What makes this disruptive

The scarce capability is turning raw, interleaved computer-use into symbolic, auditable, reusable task structure—not another screen caption. If 0.974 grouping agreement and a 30% agent lift hold up, “how work is done” becomes an extractable object instead of tribal knowledge.

Why it matters (outside the lab)

Abundance lens: expert judgment about messy digital workflows is scarce. If agents and organizations can induce inspectable task models from traces, capable assistance and process memory can become a default software layer rather than a scarce analyst service.

Horizon is near-to-mid if privacy, consent, and reliability are solved. No year attached. Near-term: a methods result on controlled traces.

Limitations & open questions

Intrinsic numbers are on controlled human and agent trajectories with ground-truth groupings; unconstrained office life may be harder. 74.9% step reconstruction still leaves a large unreconstructed fraction. The 30% accuracy lift is versus the strongest baseline they name, on held-out tasks derived from those models. Preprint ≠ enterprise process-mining product. Abundance is not automatic if traces are sensitive or incomplete.