On the use of the analytic hierarchy process in the evaluation of domain-specific modeling languages for multi-agent systems

https://doi.org/10.1016/j.cola.2020.101020Get rights and content

Abstract

Software agents and Multi-agent Systems (MAS) composed by these agents are used in the development of the complex intelligent systems. In order to facilitate MAS software development, various domain-specific modeling languages (DSMLs) exist. Unfortunately, the usability evaluation of these languages are mostly not considered or only a few assessments which cover one single MAS DSML are made. A comparative evaluation, which is missing in the existing studies, may help agent software developers to choose the MAS DSML which fits well into the system development requirements. Hence, in this paper, we introduce a comparative MAS DSML evaluation methodology based on the Analytical Hierarchy Process (AHP). A categorized set of MAS DSML criteria which can be used for the multi-criteria decision making is defined. These criteria can be prioritized by the developers according to their modeling language expectations and the application of the methodology allows the evaluation of DSML alternatives based on this prioritization. As the result of the automatic calculation of the importance distributions, the most appropriate DSML is determined. With the voluntarily participation of a group of agent software developers, the proposed methodology was applied for the comparative evaluation of four well-known MAS DSMLs. The conducted evaluation showed that the agent developers prioritized appropriateness, completeness and shortening the development time as the most significant criteria for the MAS DSML assessment while the attractiveness of the notations had a minimum effect on preferring a language. Favorite DSML for each comparison category and criteria was determined within this evaluation.

Introduction

Software agents and Multi-agent Systems (MASs) composed by these agents are used effectively in the design and implementation of various autonomous and complex intelligent systems [1], [2], [3], [4]. However, autonomous, reactive and pro-active behaviors of agents make it more difficult to develop agent-based software systems than other systems. As also indicated in [5], software systems, made up of a possibly millions of agents need to be developed in near future where agents are able to behave in a reliable, intelligent, and trustworthy way at any scale of observation. Modeling both the open nature of MAS and its relation with the outer environment is still challenging. The implementation of the interactions between agents showing autonomous and proactive behavior require a more extensive and complicated program development including verification, data integrity, resource management and concurrency of these agents [6]. While programming agent plans or rules, the mechanisms for selecting and achieving goals as well as selecting and scheduling the agent intentions that are applicable in all situations of agent execution should be constructed properly. Besides, the communication of agents within MAS organizations and the composition of agent behaviors become even more complicated and challenging to implement while the varying needs of different agent environments are considered [7]. Hence, it is important to work at higher levels of abstraction prior to coding agent software during the development of MASs [8].

In order to solve the aforementioned problems of MAS development, the researchers working in the field of Agent-oriented Software Engineering (AOSE) define a variety of meta-models that incorporate the basic components and relationships of agent models [9]. By enriching these meta-models with specific syntactic and semantic definitions, the researchers also propose various domain-specific languages (DSLs) [10], [11] or domain-specific modeling languages (DSMLs) [12] to facilitate model-driven engineering (MDE) of agents (e.g. [8], [13], [14], [15], [16], [17], [18]). These modeling languages, which are shortly referred as MAS DSMLs hereinafter, may leverage the abstraction, expression power and ease of use in software development, since MAS models can be defined visually with these DSMLs first and then the software required for the related MAS can be automatically generated via a series of transformations defined on these models. In addition, most of these languages provide both static and dynamic modeling of agent software through various perspectives such as internal agent behavior, interactions with other agents and the use of other environment sources. Hence, the modeling domain of these DSMLs cover the whole MAS including all of the agent components indicated above. Due to the capacity of identifying the essence of a MAS using visual abstractions which are close to the agent domain elements, it is expected that new DSMLs will also be introduced in the near future to address a variety of concerns in MAS [19].

Although the descriptions of the MAS DSMLs are given in the studies with including some examples of how they can be utilized during MAS development, unfortunately many of these studies do not consider a through evaluation of the proposed language, e.g. evaluating the usability of the language and the efficiency of the generated artifacts. Remaining studies (e.g. [8], [16], [20], [21]) cover some sort of quantitative and/or qualitative language assessment in an idiosyncratic manner, i.e. they only aim at the evaluation of some specific features of one single MAS DSML and hence they lack providing a general evaluation method for MAS DSMLs. More importantly they do not support the comparative evaluation of many MAS DSMLs altogether. Such a comparative evaluation may help agent software developers to choose the MAS DSML which fits well into the requirements of their MAS software development processes.

Multiple-criteria Decision Making (MCDM) techniques help decision-makers to choose the best among many alternatives in environments with different competitors and often conflicting criteria. Over the past few years, the consistent growth has been observed in the literature on MCDM, while the Analytical Hierarchical Process (AHP) [22] keeps the position which is the most frequently used MCDM technique. The AHP has been proposed to build multiple-criteria decision support method via the prioritization of each alternative based on the pairwise comparisons of each object with all the others. The popularity of AHP in the decision-making approach derives from its ability to form MCDM that includes qualitative data. The method uses a decision matrix obtained by making pairwise comparisons of the linguistic or discrete presentation of the information. Generally, it seeks expert opinions to get the information about decision alternatives. The AHP, which is often used in solving complicated decision-making problems in other areas (e.g. [23], [24], [25]) as well as in software engineering (e.g. [26], [27], [28], [29], [30]) may also contribute to the assessment of MAS DSMLs. Hence, in this paper, we introduce an AHP-based methodology which provides the comparative evaluation of MAS DSMLs according to many language criteria prioritized by the agent software developers and leads the selection of the best MAS DSML among the alternatives. Main contributions of the study can be listed as follows:

* A well-defined and categorized set of criteria and sub-criteria is presented first time which can be used to create AHP hierarchy models for the comparative evaluation of various MAS DSMLs.

* With conforming to the AHP steps, a tool assisted evaluation methodology is constructed which consists of determining MAS DSML alternatives, prioritization of both evaluation criteria and MAS DSML alternatives according to these criteria, automatic calculation of the importance distributions and finally the suggestion of the best DSML among the alternatives.

* Based on the voluntarily participation of a group of agent software developers, the proposed methodology was applied for the comparative evaluation of four MAS DSMLs widely used in AOSE.

The rest of the paper is organized as follows: Section 2 gives the related work. Section 3 briefly describes the AHP for the background. Section 4 presents the AHP-based MAS DSML evaluation methodology. Section 5 discusses the experiment for comparing MAS DSML alternatives as well as the results achieved from applying the methodology. Section 6 concludes the paper.

Section snippets

Related work

In the early 2000s, use of UML-based languages, such as AUML [31] and AML [32] became popular in MAS modeling. However, relying too much on UML which is originally designed for object-oriented system specification as well as lacking both formal and/or operational semantics and up-to-date tools led to the decline of agent developers’ interest in these languages over time [9], [33]. Following these first efforts, the researchers have made and are still making significant contribution on the

The analytic hierarchy process

One of the most frequently used approaches to MCDM problems in which factors are organized in a hierarchical structure is the Analytic Hierarchy Process (AHP) [22], [54]. The AHP derives priority scales through pairwise comparisons of preferences [55]. In this section, we briefly introduce the AHP on which our evaluation methodology is based. For readers who are not familiar with AHP, Table 1 describes each fundamental AHP matrix and vector. To solve a decision problem by AHP method, following

The AHP-based evaluation methodology for MAS DSMLs

This section introduces the methodology which we propose for the comparative evaluation of MAS DSMLs. As conforming to the AHP steps described in the previous section, the application of this evaluation methodology consists of defining MAS DSML evaluation criteria and determining MAS DSML alternatives, collecting and preparing data for the evaluation, calculating weights, and finally analyzing the achieved results. Fig. 1 illustrates the overview of this methodology. Definition of the

Experiment and discussion

The methodology was used for the comparative evaluation of four different MAS DSMLs. In the following subsections, determining these MAS DSML alternatives, collecting and preparing data, calculating the weights and finally analyzing the results are discussed.

Conclusion

An AHP-based evaluation methodology for the comparative evaluation of MAS DSMLs has been introduced in this paper. For this purpose, a categorized set of criteria which can be used for the multi-criteria decision making has been defined. These criteria can be prioritized by the agent developers according to their modeling language expectations and the application of the methodology allows the evaluation of MAS DSML alternatives based on this prioritization. As the result of the automatic

CRediT authorship contribution statement

Tansu Zafer Asici: Investigation, Software, Methodology, Visualization, Writing - original draft. Baris Tekin Tezel: Conceptualization, Methodology, Validation, Writing - original draft. Geylani Kardas: Conceptualization, Methodology, Supervision, Writing - original draft, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This study was funded by the Scientific and Technological Research Council of Turkey (TÜBİTAK) under grant 115E591.

References (68)

  • HuangC.-C. et al.

    A novel approach to evaluate software vulnerability prioritization

    J. Syst. Softw.

    (2013)
  • MaH. et al.

    User feature-aware trustworthiness measurement of cloud services via evidence synthesis for potential users

    J. Vis. Lang. Comput.

    (2014)
  • KaurA. et al.

    Investigation on test effort estimation of mobile applications: Systematic literature review and survey

    Inf. Softw. Technol.

    (2019)
  • GascueñaJ.M. et al.

    Model-driven engineering techniques for the development of multi-agent systems

    Eng. Appl. Artif. Intell.

    (2012)
  • Fuentes-FernándezR. et al.

    A technique for defining agent-oriented engineering processes with tool support

    Eng. Appl. Artif. Intell.

    (2010)
  • WauteletY. et al.

    Business and model-driven development of BDI multi-agent systems

    Neurocomputing

    (2016)
  • WauteletY. et al.

    User-story driven development of multi-agent systems: A process fragment for agile methods

    Comput. Lang. Syst. Struct.

    (2017)
  • KorkmazI. et al.

    An analytic hierarchy process and two-sided matching based decision support system for military personnel assignment

    Inform. Sci.

    (2008)
  • AmiriM.P.

    Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods

    Expert Syst. Appl.

    (2010)
  • ChiamY.K. et al.

    Applying a selection method to choose quality attribute techniques

    Inf. Softw. Technol.

    (2013)
  • AsadiM. et al.

    Toward automated feature model configuration with optimizing non-functional requirements

    Inf. Softw. Technol.

    (2014)
  • LuL. et al.

    A novel TOPSIS evaluation scheme for cloud service trustworthiness combining objective and subjective aspects

    J. Syst. Softw.

    (2018)
  • SaatyT.L.

    How to make a decision: the analytic hierarchy process

    European J. Oper. Res.

    (1990)
  • GreenT.R.G. et al.

    Usability analysis of visual programming environments: A ’cognitive dimensions’ framework

    J. Vis. Lang. Comput.

    (1996)
  • GreenT.R.G. et al.

    Cognitive dimensions: Achievements, new directions, and open questions

    J. Vis. Lang. Comput.

    (2006)
  • WeissG.
  • QinJ. et al.

    Recent advances in consensus of multi-agent systems: A brief survey

    IEEE Trans. Ind. Electron.

    (2017)
  • TaziK. et al.

    Multi-agent system for microgrids: design, optimization and performance

    Artif. Intell. Rev.

    (2020)
  • ZambonelliF. et al.

    Challenges and research directions in agent-oriented software engineering

    Auton. Agents Multi-Agent Syst.

    (2004)
  • KoemanV.J. et al.

    Designing a source-level debugger for cognitive agent programs

    Auton. Agents Multi-Agent Syst.

    (2017)
  • MascardiV. et al.

    Engineering multi-agent systems: State of affairs and the road ahead

    ACM SIGSOFT Softw. Eng. Notes

    (2019)
  • KardasG. et al.

    Domain-specific modelling language for belief–desire–intention software agents

    IET Softw.

    (2018)
  • KardasG.

    Model-driven development of multi-agent systems: a survey and evaluation

    Knowl. Eng. Rev.

    (2013)
  • MernikM. et al.

    When and how to develop domain-specific languages

    ACM Comput. Surv.

    (2005)
  • Cited by (8)

    View all citing articles on Scopus
    View full text