Wizard of props

Mixed Reality Prototyping with Physical Props to Design Responsive Environments

ABSTRACT

Driven by the vision of future responsive environments, where everyday surroundings can perceive human behaviors and respond through intelligent robotic actuation, we propose Wizard of Props (WoP): a human-centered design workflow for creating expressive, implicit, and meaningful interactions. This collaborative experience prototyping approach integrates full-scale physical props with Mixed Reality (MR) to support ideation, prototyping, and rapid testing of responsive environments. We present two design explorations that showcase our investigations of diverse design solutions based on varying technology resources, contextual considerations, and target audiences. Design Exploration One focuses on mixed environment building, where we observe fluid prototyping methods. In Design Exploration Two, we explore how novice designers approach WoP, and illustrate their design ideas and behaviors. Our findings reveal that WoP complements conventional design methods, enabling intuitive body-storming, supporting flexible prototyping fidelity, and fostering expressive environment-human interactions through in-situ improvisational performance.

Idea behind the WoP Workflow

TWO DESIGN EXPLORATIONS

Fluidity in the Workflow of Prototyping the Mixed Environment

The WoP method inherently values and fosters the flexible design process, which we define as adaptive fluidity. In creating the physical prototype, designers would either reappropriate objects they found in their environment or build objects from scratch out of cardboard. For the virtual model, they would trace physical objects like furniture with our tool, draw mid-air, find existing 3D models online, or create them in CAD. The figure below shows the fluidity of the different mixed environment building processes. The most intriguing takeaway was how all these approaches would be combined by the designers to remix and invent to support their prototyping and iteration at all stages while also accommodating different skill levels.

Door Interactions Designed by Participants

In Design Exploration Two, eight novice designers were invited and randomly paired into groups of two. Working as collaborative design groups, they were given a speculative scenario: “Envision a future expressive subway door. How would you imagine it to interact with an approaching passenger? What if the passenger also carries a bulky box?”

Here, we illustrate the results of the expressive subway door ideated from our four design groups.

REFLECTIONS

  • Human Scale Matters: WoP eases the sense-making of the human scale, especially for the “animated” layer

  • Flexible Fidelity for Prototyping: WoP eases designers from the trade-off of prototyping fidelities while holding the potential for simulating flexible motion quality

  • Collaborative Design in MR: WoP provokes collaborative and improvisational design. “surprising and accidental” interactions led by reality separation might inspire new possibilities

 

In the final reflection, We view the WoP’s environment as a stage where all participants become actors in a hybrid scene, portraying either a human character or an object prop to complete the improvisational performance. While the role-playing method facilitates a more personal and expressive object-human interaction stemming from interpersonal dynamics, the physicality of the props gives objects greater immediacy and agency, requiring the prop-actuator to act in a way that makes sense to the object’s affordance, which in turn shapes a new type of “body language.”

PUBLICATION

Yuzhen Zhang*, Ruixiang Han*, Ran Zhou*, Peter Gyory, Clement Zheng, Patrick C. Shih, Ellen Yi-Luen Do, Malte F Jung, Wendy Ju, Daniel Leithinger. 2024. Wizard of Props: Mixed Reality Prototyping with Physical Props to Design Responsive Environments. In Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '24). Association for Computing Machinery, New York, NY, USA, Article 47, 1–15. https://doi.org/10.1145/3623509.3633395 (*equal contribution)