Paper to Presence
Participants sketch a birthday cake on paper, then handle it in AR as a 3D model with whatever gestures feel natural to them.
Sketching on paper is where most designers start. AR tools can turn a sketch into 3D, and they struggle with depth perception, control and the accuracy of gestures in mid-air.
Which gestures do non-experts reach for when their sketch becomes a 3D object?
- Note 01 Six participants drew a birthday cake on paper. A sketch-to-3D platform I built on TripoSR turned each sketch into a 3D model.
- Note 02 They moved, turned and scaled the model in AR with one or two hands, in their own way.
- Note 03 Google MediaPipe tracked the hands. We sorted 36 gesture recordings by type of action, number of hands and how each was carried out.
Participants borrowed gestures from the physical world. Rotating and scaling drew similar answers from most of them. Selecting an object did not: one participant drew a circle around it, the way the selection tool works in Photoshop. Confidence depended on the system's response. Clear, immediate feedback and a way to undo mattered as much as the gesture itself. The imperfect system drew the most instinctive reactions. A polished one would have narrowed what people tried.
A full gesture elicitation study with more people, then a prototype that supports those gestures with feedback, undo and cancel. This first study had six participants, and the HoloLens 2 viewer imposed its own pinch gestures.
- Wobbrock, J. O., Morris, M. R., & Wilson, A. D. (2009). User-defined gestures for surface computing. CHI '09, 1083–1092.
- Jacob, R. J. K., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., & Zigelbaum, J. (2008). Reality-based interaction: A framework for post-WIMP interfaces. CHI '08, 201–210.