Datasets

Multimodal Desktop Action and Accessibility Dataset

A large-scale dataset of goal-directed desktop interactions, collected from users completing AI-generated tasks across both VMs and personal computers. Each trajectory includes screen recordings, accessibility trees, interaction logs, and the various AI task prompts that guided the human behavior. Spans Linux, Windows, and macOS environments. Designed for training action models & UI parsing models.