A novel user-based gesture vocabulary for conceptual design
Graphical abstract
Introduction
Hand gestures are motions performed by humans’ hands. Reviews show that they have been extensively researched since the 80s, as a natural and intuitive means of interaction between humans and computerised systems (RAUTARAY and AGRAWAL, 2015), Milani et al., 2017, Santos¹ et al., 2015, Al-Shamayleh et al., 2018, Pisharady and Saerbeck, 2015). Hand gestures are explored as a means that allows for touchless interaction providing specific benefits such as hygiene (Lopes et al., 2017) undivided attention focused on the main task (Riener et al., 2013), a more natural fit for interaction with household items (Dinh et al., 2014), large displays (Foehrenbach et al., 2009), or interaction with three-dimensional (3D) objects in Virtual Reality (VR)/Augmented Reality (AR) supported spaces (Kim and Lee, 2016, Memo and Zanuttigh, 2018).
Conceptual design is the initial, fundamental outline of a product during which ideas are explored before the design specification is established (Keinonen and Takala, 2010, pg 17). Hand gestures have also been explored in the field of conceptual design. However, in the applications which have resulted in a prototype the gestures employed are either prescribed, where the authors identify the need for further exploration of more appropriate gestures (Huang et al., 2019), or free-form, which allow natural and efficient but limited types of interactions (Vinayak and Ramani, 2015). Where hand gestures have been explored in the context of conceptual design without practical application, but considering user based input, the work performed has been linked to envisaged Computer Aided Design (CAD) application (Khan and Tunçer, 2019). User based research has been established as an effective gesture elicitation method. It results in a vocabulary of gestures for a specific purpose that is likely to be adopted by the users, and can, if needed, be used to eliminate technological limitations from consideration (Wobbrock et al., 2009, Wobbrock et al., 2005, Ruiz et al., 2011, Piumsomboon et al., 2013, Khan and Tunçer, 2019). However, use of CAD has been found to interrupt the designers thinking, as instead of thinking about the design they tend to focus on commands and procedures used in CAD for specific shape creation (Huang, 2007).
As the gestures used in interfaces have typically been defined by designers or driven by technological capability, the aim of this paper is to instead identify a vocabulary of gestures that designers would naturally use if there were no limitations. Hand gestures are thus explored for conceptual design in isolation from the technology enabling their implementation, which has not been done in the past. Established user based research approach is applied, disregarding what is possible to achieve with existing technology, in order to identify the most natural and intuitive gestures. Exploring what the designers would naturally do instead of allowing the gesture definition to be driven by what is technologically possible at any given time could be used to identify the requirements that need to be met by a future technology and potentially even drive future implementations. The goal of the research reported in this paper is to explore gestures that are most suitable for the conceptual design process and most likely to support and enhance the design process instead of interrupting it. It is envisaged these gestures would be utilised in a hypothetical future system akin to CAD currently used for detailed design, although at this stage the system itself is not explored and the focus is instead on gestures that may inspire the development of a more appropriate system to support the conceptual design.
This paper reports a user based study of hand gestures for conceptual design, focusing on identification of natural and intuitive hand gestures, addressing the gap identified in the field. The aim of the study is to create a gesture vocabulary for conceptual design, focusing on the user preferences, in terms of how they would use their hands to create, modify and manipulate 3D objects shown on a screen in front of them. The participants, final year design students, were not given any instructions on how to interact with the objects, other than to use their hands. Instead, they are shown different objects on a screen placed on a wall across from the participants, and asked to emulate how they would interact with the objects using their hands. From that, various gestures for creation, manipulation and modification of 3D objects are identified. The gesture vocabulary is evaluated by a different group of other professionals, in order to explore the suitability of the vocabulary for user groups outside of the niche conceptual design application. Fig. 1 illustrates the elements of the research reported in this paper, and their key outcomes. Green boxes illustrate user based studies. Two boxes on the left illustrate a user centred gesture study (consisting of two parts) during which a vocabulary/consensus set of gestures for conceptual design is identified, which is considered a primary outcome of this work. Then this vocabulary of gesture is evaluated. Set of green boxes on the right illustrates the evaluation of the consensus set, performed for the full consensus set abstractly, and for the 16 implemented gestures using a VR prototype. Outcome of the evaluation is a variation of the vocabulary, taking into consideration usability and implementation limitations, along with recommendations for future work. Outcomes are represented using blue boxes. Yellow boxes indicate pilot studies that were used to test and refine the approach taken in the user centred gesture study.
Section 2 provides a short review of the related work present in the literature. Sections 3 and 4 introduce the methodology used in the user centered gesture study and the analysis approach, respectively. The study results are reported in Section 5, including the key findings and the consensus set of vocabulary of gestures for conceptual design. Section 6 details the evaluation study performed to test the initial findings and refine the consensus set of vocabulary of gestures for conceptual design. Section 7 provides a discussion of the contributions and explores some potential future research directions. Then the conclusion is given in Section 8.
Section snippets
Related work
Gestures are frequently used in novel interaction approaches, primarily because they are habitually used in human communication processes. The use of gestures is even more frequent when the interaction includes physicality of a three-dimensional (3D) object e.g. interaction with a 3D/VR/AR environment (Rautaray and Agrawal, 2015, Koutsabasis and Vogiatzidakis, 2019, (Vuletic et al., 2019).
Developing the user based gesture vocabulary
The key goal of the study was to discover how designers would perform conceptual design if they were not influenced by what was achievable using existing technology and could instead use their hands to create virtual 3D objects in any manner they wished to. Gestures performed were identified, parsed, coded, categorised and analysed for patterns and relations. The most repeated gestures that were not likely to have been repeated by chance then form a gesture vocabulary for conceptual design. The
Data analysis
For both parts of the study, video recordings were reviewed and the hand gestures performed in them were first identified and described (see Supplementary material, Section 1 for details). Then each gesture was sketched on a post-it. Gestures were then parsed by grouping the same gestures on post-its together. When the same gesture was performed in different planes, and where a change of a point of view would lead to the gestures being identified as the same they were grouped together. For
Results of the user centered gesture study
This section reports the results of the data analysis approach outlined in Section 4, for 1785 gestures collected during the were collected during the user centered gesture study.
Evaluation of the consensus set
In order to further generalise the vocabulary and make it easily adoptable by those without specific design training, ten other professionals with varied backgrounds evaluated both the consensus set and the excluded set. The goal was to refine the vocabulary set so that it is natural, intuitive, and does not require extensive training to use. Refinement also aims to reduce bias design students may have introduced during gesture elicitation.
Evaluation consisted of abstract evaluation and VR
Discussion
Research reported in this paper resulted in three contributions which can be summarised as:
- 1
Knowledge about the natural gestures for conceptual design in 3D space, as the key contribution, which provides a basis for future development of CAD like systems for the support of conceptual design and could inform the future technology development requirements. This contribution will be discussed in Section 7.1.
- 2
Knowledge about the implementation of the most repeated gestures using currently readily
Conclusions
A gesture elicitation study was performed including 44 product design engineering students, in order to create a novel user based vocabulary of gestures for conceptual design. The aim of the study was to discover which gestures designers would use during conceptual design if they were not influenced by what was achievable using existing technology. Performed gestures were parsed, sketched and coded based on the adopted taxonomy, following the user based gesture elicitation approach established
CRediT authorship contribution statement
Tijana Vuletic: Conceptualization, Data curation, Funding acquisition, Formal analysis, Investigation, Resources, Writing - original draft, Writing - review & editing. Alex Duffy: Conceptualization, Funding acquisition, Resources, Supervision. Chris McTeague: Formal analysis, Investigation. Laura Hay: Conceptualization, Writing - review & editing. Ross Brisco: Formal analysis. Gerard Campbell: Project administration. Madeleine Grealy: Funding acquisition, Formal analysis, Resources, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This research was supported by the United Kingdom’s Engineering and Physical Sciences Research Council (EPSRC) (grant number EP/M012123/1 – LH, AD, MG;), and two EPSRC/University of Strathclyde Research Studentships (EP/M508159/1 – GC; EP/M506643/1 – CM).The funding for the development of the VR evaluation system was provided by AFRC (Advanced Forming Research Centre) CATAPULT Route to Impact funding awarded in 2018 - AFRC_CATP_1220_RTI.
References (84)
- et al.
An exploratory study on the use of digital sculpting in conceptual product design
Design Studies
(2013) - et al.
AIR-MODELLING: A tool for gesture-based solid modelling in context during early design stages in AR environments
Computers in Industry
(2015) - et al.
Taking the LEAP with the Oculus HMD and CAD-Plucking at thin Air?
Procedia Technology
(2015) - et al.
VR–CAD integration: Multimodal immersive interaction and advanced haptic paradigms for implicit edition of CAD models
Computer-Aided Design
(2010) - et al.
Multi-sensory user interface for a virtual-reality-based computeraided design system
Computer-Aided Design
(1997) - et al.
Creation of concept shape designs via a virtual reality interface
Computer-Aided Design
(1997) - et al.
Hand gesture recognition and interface via a depth imaging sensor for smart home appliances
Energy Procedia
(2014) - et al.
Tactile feedback enhanced hand gesture interaction at large, high-resolution displays
Journal of Visual Languages & Computing
(2009) - et al.
Instant 3D design concept generation and visualization by real-time hand gesture recognition
Computers in Industry
(2013) - et al.
Gesture and speech elicitation for 3D CAD modeling in conceptual design
Automation in Construction
(2019)
The computationalist paradigm in design research. Design studies
Exploring problem decomposition in conceptual design among novice designers
Design studies
On the utility of 3D hand cursors to explore medical volume datasets with a touchless interface
Journal of biomedical informatics
Potential of multimodal and multiuser interaction with virtual holography
Advances in Engineering Software
Recent methods and databases in vision-based hand gesture recognition: A review
Computer Vision and Image Understanding
Finger-based gestural interaction for exploration of 3D heart visualization
Procedia-Social and Behavioral Sciences
System design and user evaluation of Co-Star: An immersive stereoscopic system for cable harness design
Computer-Aided Design
GaFinC: Gaze and Finger Control interface for 3D model manipulation in CAD application
Computer-Aided Design
Shape-It-Up: Hand gesture based creative expression of 3D shapes using intelligent generalized cylinders
Computer-Aided Design
A gesture-free geometric approach for mid-air expression of design intent in 3D virtual pottery
Computer-Aided Design
Systematic literature review of hand gestures used in human computer interaction interfaces
International Journal of Human-Computer Studies
A systematic literature review on vision based gesture recognition techniques
Multimedia Tools and Applications
Using Natural Gesture Interactions Leads to Higher Usability and Presence in a Computer Lesson
Congress of the International Ergonomics Association
Creative stimulation in conceptual design
Music Interfaces for Novice Users: Composing Music on a Public Display with Hand Gestures
NIME
A gesture controlled user interface for inclusive design and evaluative study of its usability
Journal of software engineering and applications
FingARtips: gesture based direct manipulation in Augmented Reality
An approach to functional synthesis of solutions in mechanical conceptual design. Part I: Introduction and knowledge representation
Research in Engineering Design
Gesture interface for 3d cad modeling using kinect
Computer-Aided Design and Applications
Flexible gesture recognition for immersive virtual environments. Information Visualization, 2006. IV 2006
Designing Mid-air TV gestures for blind people using user-and Choice-Based Elicitation Approaches
An Elicitation Study on Gesture Attitudes and Preferences Towards an Interactive Hand-Gesture Vocabulary
Protocol analysis: Verbal reports as data
Beyond QWERTY: augmenting touch screen keyboards with multi-touch gestures for non-alphanumeric input
Conceptual design for engineers
Computer-Based Form Generation in Architectural Design — A Critical Review
International Journal of Architectural Computing
Virtual pottery: a virtual 3D audiovisual interface using natural hand motions
Multimedia Tools and Applications
Defining expertise in the use of constraint-based CAD tools by examining practicing professionals
Engineering Design Graphics Journal
Answering the call for a standard reliability measure for coding data
Communication methods and measures
Data miming: inferring spatial object descriptions from human gesture
Computer-Aided Architectural Design Futures (CAADFutures) 2007
Gesture-based system for next generation natural and intuitive interfaces
AI EDAM
Cited by (8)
Touchable pixels: Examining the affordance effect between an on-screen object and a user-elicited gesture on the touchscreen
2023, Computers in Human BehaviorCitation Excerpt :Under this theoretical view, this study additionally provides a new methodological approach for gesture design. The gestural studies have focused on merely geometric and kinematic characteristics (e.g., body part, pose, flow) of the user-elicited gesture itself (Vogiatzidakis & Koutsabasis, 2018; Vuletic et al., 2019, 2021), neglecting the effect of the digital affordance on the screen. However, our study proposed an empirical method to evaluate the intuitiveness of user-elicited gestures through examining an object-based correspondence—affordance effect.
Investigating user-defined flipping gestures for dual-display phones
2022, International Journal of Human Computer StudiesInteraction and Perception of Interaction with 3D Objects during Design Activities
2022, Proceedings of the Design SocietyEffects of activity time limitation on gesture elicitation for form creation
2023, Journal of Engineering DesignDigital Modeling for Everyone: Exploring How Novices Approach Voice-Based 3D Modeling
2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)