Watch Vector Faculty Member Victor Zhong answer the question: how can we build generalist language agents that assist us in the digital world? during his talk "Generalist Language Agents in General Purpose Operating Systems" presented during Vector's monthly Research Day.

Talk abstract:
Real-world computer use involves navigating and using multiple applications over long horizons while reasoning over textual and visual observations. First, we will discuss OSWorld, a new interactive, executable testbed for generalist agents where agents follow natural language instructions to perform long-horizon real-world tasks in virtual machines in real-time. OSWorld is multi-modal, multi-task, and multi-application, and presents significant challenges for state-of-the-art foundation model agents. Second, we will discuss recent and ongoing efforts to train generalist language agents, including scalable collection of demonstrations, and learning from both human and automatic language feedback.