Skip to main content
SpringerLink
Log in

Impact of missing attributes on the novelty metric of Shah et al.

  • Original Paper
  • Published:
Research in Engineering Design Aims and scope Submit manuscript

Abstract

The a posteriori novelty metric proposed in 2003 by Shah and colleagues is still one of the most widely used in design research, constituting the reference for many experiments on creativity and effectiveness of ideation. However, recent research highlights that in the presence of ideas implementing different numbers of attributes, some non-negligible problems arise. The objective of this paper is to estimate the potential errors that can be committed when applying the metric of Shah et al. in these situations. The investigation has been performed by relying on a high number of randomly generated virtual sets of ideas. The results obtained revealed that the problem of “missing attributes”, if not opportunely faced, leads the considered metric to misleading novelty scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Atilola O, Linsey J (2015) Representing analogies to influence fixation and creativity: a study comparing computer-aided design, photographs, and sketches. Artif Intell Eng Des Anal Manuf 29:161–171. https://doi.org/10.1017/S0890060415000049

    Article  Google Scholar 

  • Bao Q, Faas D, Yang M (2018) Interplay of sketching and prototyping in early stage product design. Int J Des Creat Innov 0349:1–23. https://doi.org/10.1080/21650349.2018.1429318

    Article  Google Scholar 

  • Berthelsdorf FA, Arlitt RM, Armstrong JL, et al (2016) Creative personalities: how personality influences creativity of concept ideation methods. In: The fourth international conference on design creativity (4th ICDC). pp 1–8

  • Boden MA (2004) The creative mind: myths and mechanism, 2nd edn. Routledge, London

    Book  Google Scholar 

  • Borgianni Y, Rotini F, Tomassini M (2017) Fostering ideation in the very early design phases: how textual, pictorial and combined stimuli affect creativity. In: ICED17: 21st international conference on engineering design 8:139–148

  • Brown DC (2014) Problems with the calculation of novelty metrics. In: Proceedings of the 6th international conference on design computing and cognition

  • Cascini G, Fiorineschi L, Rotini F (2019) Impact of design representations on creativity of design outcomes. J Integr Des Process Sci. https://doi.org/10.3233/jid180012(in press)

    Article  Google Scholar 

  • Chiu I, Shu LHH (2012) Investigating effects of oppositely related semantic stimuli on design concept creativity. J Eng Des 23:271–296. https://doi.org/10.1080/09544828.2011.603298

    Article  Google Scholar 

  • Choo PK, Lou ZN, Koo B et al (2014) Ideation methods: a first study on measured outcomes with personality type. In: Proceedings of the ASME 2014 international design engineering technical conferences and computers and information in engineering conference IDETC/CIE 2014 August 17–20, 2014, Buffalo, New York, USA, pp 1–12

  • Chulvi V, Mulet E, Chakrabarti A et al (2012) Comparison of the degree of creativity in the design outcomes using different design methods. J Eng Des 23:241–269. https://doi.org/10.1080/09544828.2011.624501

    Article  Google Scholar 

  • Corazza GE, Agnoli S (eds) (2016) Creativity in the twenty first century multidisciplinary contributions to the science of creative thinking. Springer, Singapore

    Google Scholar 

  • Eckert C, Alink T, Ruckpaul A (2011) Different notions of function : results from an experiment on the analysis of an existing product. J Eng Des 22:811–837. https://doi.org/10.1080/09544828.2011.603297

    Article  Google Scholar 

  • Fiorineschi L, Frillici FS, Rotini F (2018a) Issues related to missing attributes in a-posteriori novelty assessments. In: International design conference—design 2018, pp 1067–1078

  • Fiorineschi L, Frillici FS, Rotini F (2018b) A-posteriori novelty assessments for sequential design sessions. In: International design conference—design 2018, pp 1079–1090

  • Fiorineschi L, Frillici FS, Rotini F, Tomassini M (2018c) Exploiting TRIZ tools for enhancing systematic conceptual design activities. J Eng Des 29:259–290. https://doi.org/10.1080/09544828.2018.1473558

    Article  Google Scholar 

  • Jagtap S (2016) Assessing design creativity: refinements to the novelty assessment method. In: International design conference—design 2016, pp 1045–1054

  • Jansson DG, Smith SM (1991) Design fixation. Des Stud 12:3–11. https://doi.org/10.1016/0142-694X(91)90003-F

    Article  Google Scholar 

  • Johnson TA, Caldwell BW, Cheeley A, Green MG (2016) Comparison and extension of novelty metrics for problem-solving tasks. In: Proceedings of the ASME 2016 international design engineering technical conferences and computers and information in engineering conference IDETC/CIE 2016

  • Linsey JS, Clauss EF, Kurtoglu T et al (2011) An experimental study of group idea generation techniques: understanding the roles of idea representation and viewing methods. J Mech Des 133:031008. https://doi.org/10.1115/1.4003498

    Article  Google Scholar 

  • Lopez-Mesa B, Vidal R (2006) Novelty metrics in engineering design experiments. In: 9th International design conference, DESIGN 2006. Dubrovnik—Croatia, pp 557–564

  • López-Mesa B, Mulet E, Vidal R, Thompson G (2011) Effects of additional stimuli on idea-finding in design teams. J Eng Des. https://doi.org/10.1080/09544820902911366

    Article  Google Scholar 

  • Montgomery DC (2012) Design and analysis of experiments, 8th edn. Wiley, Hoboken

    Google Scholar 

  • Nelson BA, Wilson JO, Rosen D, Yen J (2009) Refined metrics for measuring ideation effectiveness. Des Stud 30:737–743. https://doi.org/10.1016/j.destud.2009.07.002

    Article  Google Scholar 

  • Oman SK, Tumer IY, Wood K, Seepersad C (2013) A comparison of creativity and innovation metrics and sample validation through in-class design projects. Res Eng Design 24:65–92. https://doi.org/10.1007/s00163-012-0138-9

    Article  Google Scholar 

  • Pahl G, Beitz W, Feldhusen J, Grote KH (2007) Engineering design, 3rd edn. Springer, London

    Book  Google Scholar 

  • Peeters J, Verhaegen PA, Vandevenne D, Duflou JR (2010) Refined metrics for measuring novelty in ideation. Proc IDMME Virtual Concept 2010:1–4

    Google Scholar 

  • Ranjan BSC, Siddharth L, Chakrabarti A (2018) A systematic approach to assessing novelty, requirement satisfaction, and creativity. Artif Intell Eng Des Anal Manuf 32:390–414. https://doi.org/10.1017/S0890060418000148

    Article  Google Scholar 

  • Sääksjärvi M, Gonçalves M (2018) Creativity and meaning: including meaning as a component of creative solutions. Artif Intell Eng Des Anal Manuf 32:365–379

    Article  Google Scholar 

  • Sarkar P, Chakrabarti A (2011) Assessing design creativity. Des Stud 32:348–383. https://doi.org/10.1016/j.destud.2011.01.002

    Article  Google Scholar 

  • Shah JJ, Kulkarni SV, Vargas-Hernandez N (2000) Evaluation of idea generation methods for conceptual design: effectiveness metrics and design. J Mech Des 122:377–384. https://doi.org/10.1115/1.1315592

    Article  Google Scholar 

  • Shah JJ, Vargas-Hernandez N, Smith SM (2003) Metrics for measuring ideation effectiveness. Des Stud 24:111–134. https://doi.org/10.1016/S0142-694X(02)00034-0

    Article  Google Scholar 

  • Sluis-Thiescheffer W, Bekker T, Eggen B et al (2016) Measuring and comparing novelty for design solutions generated by young children through different design methods. Des Stud 43:48–73. https://doi.org/10.1016/j.destud.2016.01.001

    Article  Google Scholar 

  • Song B, Srinivasan V, Luo J (2017) Patent stimuli search and its influence on ideation outcomes. Des Sci 3:e25. https://doi.org/10.1017/dsj.2017.27

    Article  Google Scholar 

  • Srivathsavai R, Genco N, Katj, Seepersad CC (2010) Study of existing metrics used in measurement if ideation effectiveness. In: Proceedings of the ASME 2010 international design engineering technical conferences and computers and information in engineering conference IDETC/CIE 2010

  • Tsenn J, Atilola O, McAdams DA, Linsey JS (2014) The effects of time and incubation on design concept generation. Des Stud 35:500–526. https://doi.org/10.1016/j.destud.2014.02.003

    Article  Google Scholar 

  • Vargas-Hernandez N, Okudan GE, Schmidt LC (2012a) Effectiveness metrics for ideation: merging genealogy trees and improving novelty metric. In: Proceedings of the ASME 2012 international design engineering technical conferences and computers and information in engineering conference IDETC/CIE 2012

  • Vargas-Hernandez N, Schmidt LC, Okudan GE (2012b) Systematic ideation effectiveness study of TRIZ. In: Proceedings of the ASME 2012 international design engineering technical conferences and computers and information in engineering conference IDETC/CIE 2012, pp 1–10

  • Vargas-Hernandez N, Schmidt LC, Okudan GE (2013) Systematic ideation effectiveness study of TRIZ. J Mech Des 135:101009. https://doi.org/10.1115/1.4024976

    Article  Google Scholar 

  • Vasconcelos LA, Crilly N (2016) Inspiration and fixation: questions, methods, findings, and challenges. Des Stud 42:1–32. https://doi.org/10.1016/j.destud.2015.11.001

    Article  Google Scholar 

  • Vermaas PE, Eckert C (2013) My functional description is better! Artif Intell Eng Des Anal Manuf 27:187–190. https://doi.org/10.1017/S089006041300019X

    Article  Google Scholar 

  • Viswanathan VK, Linsey JS (2018) Role of sunk cost in engineering idea generation: an experimental investigation. J Mech Des 135(121002):1–12. https://doi.org/10.1115/1.4025290

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, University of Florence, Via Santa Marta 3, 50139, Florence, Italy

    Lorenzo Fiorineschi, Francesco Saverio Frillici & Federico Rotini

Authors
  1. Lorenzo Fiorineschi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francesco Saverio Frillici
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Federico Rotini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lorenzo Fiorineschi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fiorineschi, L., Frillici, F.S. & Rotini, F. Impact of missing attributes on the novelty metric of Shah et al.. Res Eng Design 31, 221–234 (2020). https://doi.org/10.1007/s00163-020-00332-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00163-020-00332-x

Keywords

Search

Navigation