MUSE

The MUSE project introduces a new way of exploring and understanding information by “bringing text to life” through 3D interactive storytelling. Taking as input natural language text like children’s stories or medical patient education materials, MUSE processes the natural language, translates it into formal knowledge that represents the actions, actors, plots and surrounding world, and then renders these as virtual 3D worlds in which the user can explore the text through interaction, re-enactment and guided game play.

To enable such a system, MUSE makes targeted advances in natural language processing that enable the translation of natural language text to the necessary knowledge representations, as well as targeted advances in the action representation and story planning necessary for interactive storytelling. In natural language processing, MUSE develops new techniques for finding explicit action structures in text and combining them with implicit action structures inferred from the context based on probabilistic models of translation and automatic methods for acquiring world knowledge from large corpora. In interactive storytelling, MUSE develops action and object representations that bridge the gap between natural language and virtual worlds, and creates advanced techniques for planning virtual world stories given inconsistent and incomplete information.

The proposed methodology is evaluated and showcased on two scenarios: one for creating immersive children’s stories from text and one for allowing medical patients to interact with patient education materials. MUSE’s long-term goal is to enable the innovative text-to-virtual-world translation system to be used in many applications and a variety of domains. Current information communicated to citizens is often very complex making written text difficult to understand. Comparable to the invention of symbolic writing systems several millennia ago, MUSE contributes to a novel symbolic system of communicating natural language utterances.