Abstract
Gap analysis is a common approach in industry to evaluate the gaps between the implemented software processes and the requirements of process quality frameworks or standard norms. Gap analysis processes are usually executed by means of questionnaires that need to be crafted ad hoc according to specific appraisal goals. The approaches used for developing, compiling and evaluating the answers given to these questionnaires usually do not follow well-defined methodologies or processes, and lack adequate tool support. This paper aims at investigating novel approaches for the execution of questionnaire-based gap analysis (QBGA) processes in industrial practices. We propose the adoption of state-of-the-art software engineering technologies and methodologies like application lifecycle management (ALM) and model-driven engineering (MDE) to support these processes. We perform an industrial survey for understanding the main issues affecting questionnaire-based gap analysis processes in industrial practices. We exploit model-driven engineering for building an ALM-based tool that supports the QBGA process execution and allows us to overcome the emerged process issues. We implement the GADGET tool to apply the MDE approach we use for developing the ALM-based tool. The feasibility of the proposed approach has been evaluated by a case study conducted in the automotive industrial domain. Two different QBGA processes have been configured and implemented in an ALM system with the support of the GADGET tool. The resulting ALM tool was used to perform the gap analysis processes. Semi-structured interviews with the involved industrial personnel were conducted to carry out a qualitative evaluation. The case study results show that the introduction of ALM improves the quality of the questionnaire-based gap analysis processes. Moreover, the adoption of model-driven engineering approach implemented by the GADGET tool provides a viable solution for configuring application lifecycle management systems and supporting the process execution.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Al-Mayahi, I., & Mansoor, S. P. (2012). Iso 27001 gap analysis - case study. In WorldComp 2012 - Proceedings of the world congress in computer science, computer engineering, and applied computing.
Amaral, L., & Faria, J. (2010). A gap analysis methodology for the team software process, In: 2010 7th international conference on the quality of information and communications technology (QUATIC), pp. 424–429. https://doi.org/10.1109/QUATIC.2010.78.
Bolstorff, P., & Rosenbaum, R. (2007). Supply chain excellence: a handbook for dramatic improvement using the scor model, 2nd edn. New York: Amacom.
Bornstein, M. H., Jager, J., Putnick, D. L. (2013). Sampling in developmental science: Situations, shortcomings, solutions, and standards. Developmental Review, 33(4), 357–370. https://doi.org/10.1016/j.dr.2013.08.003. http://www.sciencedirect.com/science/article/pii/S0273229713000385.
Cao, M., Fan, J. J., Lv, H. P., Chen, J. L. (2010). Evaluation of enterprise knowledge management performance based on gap analysis. In: 2010 IEEE international conference on management of innovation and technology (ICMIT), pp. 894–897. https://doi.org/10.1109/ICMIT.2010.5492810.
Ceccarelli, A., & Silva, N. (2015). Computer safety, reliability, and security: SAFECOMP 2015 Workshops, ASSURE, DECSoS. ISSE, ReSA4CI, and SASSUR, Delft, The Netherlands, September 22, 2015, Proceedings, Springer International Publishing, Cham, Ch. Analysis of Companies Gaps in the Application of Standards for Safety-Critical Software, pp. 303–313.
Ciccozzi, F., Ruscio, D. D., Malavolta, I., Pelliccione, P. (2016). Adopting mde for specifying and executing civilian missions of mobile multi-robot systems. IEEE Access, 4, 6451–6466. https://doi.org/10.1109/ACCESS.2016.2613642.
Conrad, M. (2012). Artifact-centric compliance demonstration for ISO 26262 projects using model-based design. In: Informatik 2012, 42. Jahrestagung der Gesellschaft für Informatik e.V. (GI), 16.-21.09.2012, Braunschweig, pp. 807–816. http://subs.emis.de/LNI/Proceedings/Proceedings208/article98.html.
de la Vara, J. L., Ruiz, A., Attwood, K., Espinoza, H., Panesar-Walawege, R. K., López, Á., del Río, I., Kelly, T. (2016). Model-based specification of safety compliance needs for critical systems: A holistic generic metamodel. Information and Software Technology, 72, 16–30. https://doi.org/10.1016/j.infsof.2015.11.008. http://www.sciencedirect.com/science/article/pii/S0950584915002025.
De Lucia, A., Fasolino, A.R., Pompelle, E. (2001). A decisional framework for legacy system management. In: IEEE international conference on software maintenance, 2001. Proceedings, pp. 642–651. https://doi.org/10.1109/ICSM.2001.972781.
De Simone, V., Amalfitano, D., Fasolino, A. R. (2018). Exploiting alm and mde for supporting questionnaire-based gap analysis processes. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 1–8. https://doi.org/10.1109/SEAA.2018.00010.
Dillman, D. A., Smyth, J. D., Christian, L. M. (2008). Internet, mail, and mixed-mode surveys: The tailored design method, 3rd edn., Wiley Publishing, New Jersey.
Dybå, T., Prikladnicki, R., Rönkkö, K., Seaman, C., Sillito, J. (2011). Qualitative research in software engineering. Empirical Softw. Engg., 16(4), 425–429. https://doi.org/10.1007/s10664-011-9163-y.
Falessi, D., Sabetzadeh, M., Briand, L., Turella, E., Coq, T., Panesar-Walawege, R. (2012). Planning for safety standards compliance: A model-based tool-supported approach. IEEE Software, 29(3), 64–70. https://doi.org/10.1109/MS.2011.116.
Ferry, N., Song, H., Rossini, A., Chauvel, F., Solberg, A. (2014). Cloudmf: Applying mde to tame the complexity of managing multi-cloud applications. In: 2014 IEEE/ACM 7th international conference on utility and cloud computing, pp. 269–277. https://doi.org/10.1109/UCC.2014.36.
Fuggetta, A., & Di Nitto, E. (2014). Software process. In Proceedings of the on future of software engineering, FOSE 2014. https://doi.org/10.1145/2593882.2593883(pp. 1–12). New York: ACM.
Gatrell, M. (2016). The value of a single solution for end-to-end alm tool support. IEEE Software, 33(5), 103–105. https://doi.org/10.1109/MS.2016.109.
Hurtado, J. A., Alegría, M. C., Bastarrica, A., Quispe, S., Ochoa, F. (2011). An mde approach to software process tailoring. In Proceedings of the 2011 international conference on software and systems process, ICSSP ’11. https://doi.org/10.1145/1987875.1987885 (pp. 43–52). New York: ACM.
IEC 62304 Medical device software – Software life cycle processes. (2006). Standard, IEC, Geneva Switzerland.
IEC 61513 Nuclear power plants - Instrumentation and control important to safety. (2011). Standard, IEC, Geneva, Switzerland.
ISO 26262:2011 - Road vehicles – Functional safety. (2011). Standard, International Organization for Standardization, Geneva, CH.
Karabacak, B., & Sogukpinar, I. (2006). A quantitative method for iso 17799 gap analysis. Computers & Security, 25(6), 413–419. https://doi.org/10.1016/j.cose.2006.05.001. http://www.sciencedirect.com/science/article/pii/S0167404806000757.
Kent, S. (2002). Model driven engineering. In Proceedings of the 3rd international conference on integrated formal methods, IFM ’02. http://dl.acm.org/citation.cfm?id=647983.743552 (pp. 286–298). London: Springer-Verlag.
Lacheiner, H., & Ramler, R. (2011). Application lifecycle management as infrastructure for software process improvement and evolution: Experience and insights from industry. In: 37th EUROMICRO conference on software engineering and advanced applications, SEAA 2011, Oulu, Finland, August 30 - September 2, 2011, pp. 286–293.
McMahon, P. E. (2010). Integrating CMMI and Agile development: Case studies and proven techniques for faster performance improvement, 1st edn. Boston: Addison-Wesley Professional.
Moratalla, J., de Castro, V., Sanz, M.L., Marcos, E. (2012). A gap-analysis-based framework for evolution and modernization: Modernization of domain management at red.es. In: 2012 Annual SRII Global Conference, pp. 343–352. https://doi.org/10.1109/SRII.2012.45.
Nguyen, D. K., van den Heuvel, W. -J., Papazoglou, M. P., de Castro, V., Marcos, E. (2009). GAMBUSE: A gap analysis methodology For engineering SOA-based applications, (pp. 293–318). Berlin: Springer.
Palmer, N., & Mooney, L. (2007). Building a business case for BPM – a fast path to real result. Tech. rep., OpenText Corporation.
Panesar-Walawege, R., Sabetzadeh, M., Briand, L., Coq, T. (2010). Characterizing the chain of evidence for software safety cases: A conceptual model based on the iec 61508 standard. In: 2010 3rd international conference on software testing, verification and validation (ICST), pp. 335–344. https://doi.org/10.1109/ICST.2010.12.
Panesar-Walawege, R. K., Sabetzadeh, M., Briand, L. (2013). Supporting the verification of compliance to safety standards via model-driven engineering: Approach, tool-support and empirical validation. Information and Software Technology, 55 (5), 836–864. https://doi.org/10.1016/j.infsof.2012.11.009. http://www.sciencedirect.com/science/article/pii/S0950584912002352.
Panesar-Walawege, R.K., Sabetzadeh, M., Briand, L. (2013). Supporting the verification of compliance to safety standards via model-driven engineering: Approach, tool-support and empirical validation. Inf. Softw. Technol., 55 (5), 836–864. https://doi.org/10.1016/j.infsof.2012.11.009.
Parasuraman, L.L.B.A., & Zeithaml, V.A. (1985). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49(4), 41–50. http://www.jstor.org/stable/1251430.
Picha, P., & Brada, P. (2016). Alm tool data usage in software process metamodeling. In: 2016 42th Euromicro conference on software engineering and advanced applications (SEAA), pp. 1–8. https://doi.org/10.1109/SEAA.2016.37.
Postina, M., Sechyn, I., Steffens, U. (2009). Gap analysis of application landscapes. In: 2009 13th enterprise distributed object computing conference workshops, pp. 274–281. https://doi.org/10.1109/EDOCW.2009.5331980.
Runeson, P., Host, M., Rainer, A., Regnell, B. (2012). Case study research in software engineering: Guidelines and examples, 1st edn., Wiley Publishing, New Jersey.
Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation, and analysis of questionnaires for survey research. New Jersey: Wiley Publishing.
Sommerville, I. (2006). Software engineering: (Update) (8th Edition) (International Computer Science). Boston: Addison-Wesley Longman Publishing Co., Inc.
Valdevit, T., & Mayer, N. (2010). A gap analysis tool for smes targeting ISO/IEC 27001 compliance. In: ICEIS 2010 - Proceedings of the 12th international conference on enterprise information systems, vol. 3, ISAS, Funchal, Madeira, Portugal, June 8 - 12, 2010, pp. 413–416.
Wieringa, R., & Daneva, M. (2015). Six strategies for generalizing software engineering theories. Science of Computer Programming, 101, 136–152. towards general theories of software engineering. https://doi.org/10.1016/j.scico.2014.11.013. http://www.sciencedirect.com/science/article/pii/S0167642314005450 http://www.sciencedirect.com/science/article/pii/S0167642314005450.
Wilson, C.E. (2007). Designing useful and usable questionnaires: You can’t just ”throw a questionnaire together”. Interactions, 14(3), 48–ff. https://doi.org/10.1145/1242421.1242453.
Funding
This work was carried out in the context of the research project APPS4SAFETY (Active Preventive Passive Solutions for Safety - PON03PE_00159_3), partially funded by the Italian Ministry for University and Research (MIUR).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix
A - Details on the proposed metamodels
In this Appendix, we report additional details on the metamodels that we introduced to support the MDE approach, i.e., metamodels representing the QBGA process application domain and metamodels abstracting the characteristics of the ALM target platform. To infer this metamodel, we analysed different standards. More precisely, we focused on the ISO/IEC 61508 (Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems in a generic domain), the ISO 26262 (Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems in the automotive domain), the ISO 10303 (Product data representation and exchange for production manufacturing processes), the ISO/IEC 15504 (Software Process Improvement and Capability Determination in any domain) and the Automotive SPICE.
2.1 A.1 The QBGA process metamodel
The QBGA process metamodel comprises the concepts related to a given Document of the Standard that the company is willing to comply with, the Questionnaire used for understanding if the company satisfies the Standard requirements and the Lifecycle of the elements of the process that defines how it is to be carried out. We report the description of these different aspects of the QBGA process metamodel in the following.
2.1.1 A.1.1 The document of the standard
The metamodel reported in Fig. 3 proposes a representation of a document of the standard.
According to this metamodel, the Document of the Standard is composed by several StructuralElements. They represent the different parts in which the document is organised (e.g. headers, chapters, paragraphs) and may contain other StructuralElements.
Each StructuralElement contains one or more Requirement s requested by the Standard. A requirement may report, among other things, constraints that need to be respected or techniques that should be applied.
The Document of the Standard may define several Level s of capability of the company to apply what requested by the standard. These levels influence the applicability of the Requirements reported in the standard. Depending on the level, the applicability for each requirement may be mandatory, optional or not required at all.
To represent this aspect, we introduced the RequiredApplicability class in Fig. 3 having the attribute applicabilityValue. Examples of possible required applicability values may be (1) Mandatory, (2) Optional, or (3) Not applicable.
2.1.2 A.1.2 The QBGA questionnaire
The gap analysis process we are interested in is focused on questionnaires that will be developed, completed and analysed during the QBGA process. The main aspects of any questionnaire for a QBGA process are described by the metamodel shown in Fig. 4.
This metamodel defines how a gap analysis questionnaire can be structured. The main element of this metamodel, the Questionnaire, contains all the Questions defined for each identified requirement of the standard. The questionnaire is defined according to the Goal the company is willing to achieve by the execution of the QBGA process. Different questionnaires could be associated to a Standard Document, according to different goals pursued by the company.
Each Question reports a textual description that is defined for understanding if the company complies with a given requirement. Questions can be either Essay or MultiChoice questions. Each question is linked to the Answer that is provided to it. The Answer reports the given response and a rationale for this choice.
Depending on the type of question, essay or MultiChoice, the answer either may be a statement, or it may assume one of its multiple predefined values, respectively.
Possible predefined values of answers may be Yes, Partially, No or Not Applicable representing the cases in which the company is able to fully, partially, does not achieve, or it is not possible to apply what requested by the question, respectively. The possible AnswerValues usable for answering a MultiChoice question may be defined according well-known appraisal methods (e.g. SCAMPI), such as yes, no, not applicable, or not answered.
Each answer may be characterised by a rationale that is used to explain the reasons behind the given answer. They are needed in case of Standards for which the compliance can be met even if one of its requirements is not satisfied, but a valid rationale is given for that choice.
The questionnaire is carried out by different Employees. Each employee may have different Roles in the execution of the questionnaire, such as a respondent and internal assessor. Each question is assigned to an employee that has the responsibility to answer it.
As another aspect, an answer may be linked to different evidences. An Evidence represents the proof demonstrating the fulfillment of a given requirement. Depending on the specific standard, the evidences may be either statements provided by an actor involved in a specific process, or the produced artefacts.
The metamodel introduces also the elements that are needed to evaluate the gaps, at different degrees of detail, on the basis of the provided answers. The Rating represents an evaluation of the compliance. It is characterised by a complianceValue that reports the defined value of compliance and a rationale attribute. To evaluate if and to what degree the company is able to comply with each requirement, the rating is specialised by the CapabilityRating that is associated to a requirement.
To understand if and to what degree the company is able to comply with all the requirements belonging to a given StructuralElement, the rating is extended by the ElementComplianceRating. It is associated to a StructuralElement and it presents the complianceValue attribute reporting the compliance level for the associated StructuralElement.
The rating is also specialised by the StandardComplianceRating, having the complianceValue to indicate the level of compliance of the company with respect to all the considered parts of the standard.
The values for the capability of each requirement, each StructuralElement and of the Document_of_the_Standard are defined according to the RatingMethod defined for the questionnaire.
For each questionnaire, it is possible to apply a specific RatingMethod that could be of two different types, namely ExpertBased and FormulaBased. It the first case, the evaluation are carried out by an expert. For the FormulaBased, the evaluation is automatically performed by applying the defined RatingFormulas.
When the analysis of the gap analysis questionnaire have been accomplished, the results of this analysis are summarised in different Reports containing information about the values of compliance reached for the considered requirements of the standard and the possible identified gaps.
In addition, if a Gap related to a requirement of the standard is identified, the company may define an Improvement Action in order to reduce it and to reach its defined goal.
2.1.3 A.1.3 The QBGA process elements lifecycle
In order to specify how the QBGA process should be carried out, it is needed to define the lifecycle for the question and rating elements. The proposed metamodel explicitly represents the Dynamic aspects of the process that are necessary to rule the activity of questionnaire completion. These aspects are modelled by the diagram in Fig. 5. Each element of the questionnaire has its own Lifecycle that is described by the State s that the element can assume and all the possible Transition s among these states. Each transition can be triggered only by actors having specific Role and only if peculiar Rule s are met.
An example of a question lifecycle is reported by the states-transitions matrix reported in Table 5 and by the statechart diagram in Fig. 14.
Question lifecycle—statechart diagram
Table 6 reports an example of how the roles involved in the QBGA process may handle the lifecycle of a question.
2.2 A.2 The metamodel of the ALM project
The ALM project metamodel we introduced is reported in Fig. 6.
A Project can be organised in different ProjectSegments, representing specific activities of the project. A peculiar element managed by the Project is the WorkItem. It may be specialised in WorkUnit, representing a task to carry out during the project, or in Artifact produced or needed to carry out a task. Each ProjectSegment may be structured in different WorkUnits, that may require different WorkItems. A WorkItem may have CustomFields, that represent its peculiar properties, such as a filename for an artifact. Moreover, each WorkItem has a Workflow, representing its lifecycle. The workflow is characterised by possible Transitions among its possible States. WorkItems may be related to each other through a LinkRole, that may be used for traceability purposes or to represent logical connections among WorkItems. A LinkRole represents a unidirectional relationship between a startingMemberEnd and an endingMemberEnd WorkItem.
An ALM project may involve different Actors, each having its own Competencies. A competence represent expertise, certifications and skills of a person. An actor may have different Identities on the project, representing the different accounts on the ALM project. An actor may have a specific Identity representing its user account in a particular tool, since a project can use more than one tool for ALM purposes (Picha and Brada 2016). The ALM project allows to define Groups for aggregating different Identities. Moreover, an identity may have several Roles. Each role may have specific permissions for triggering transitions on WorkItems workflow.
Rights and permissions
About this article
Cite this article
Amalfitano, D., De Simone, V., Scala, S. et al. A model-driven engineering approach for supporting questionnaire-based gap analysis processes through application lifecycle management systems. Software Qual J 28, 535–565 (2020). https://doi.org/10.1007/s11219-019-09479-w
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-019-09479-w