Quantitatively assessing the benefits of model-driven development in agent-based modeling and simulation

Abstract

The agent-based modeling and simulation (ABMS) paradigm has been used to analyze, reproduce, and predict phenomena related to many application areas. Although there are many agent-based platforms that support simulation development, they rely on programming languages that require extensive programming knowledge. Model-driven development (MDD) has been explored to facilitate simulation modeling, by means of high-level modeling languages that provide reusable building blocks that hide computational complexity, and code generation. However, there is still limited knowledge of how MDD approaches to ABMS contribute to increasing development productivity and quality. We thus in this paper present an empirical study that quantitatively compares the use of MDD and ABMS platforms mainly in terms of effort and developer mistakes. Our evaluation was performed using MDD4ABMS—an MDD approach with a core and extensions to two application areas, one of which developed for this study—and NetLogo, a widely used platform. The obtained results show that MDD4ABMS requires less effort to develop simulations with similar (sometimes better) design quality than NetLogo, giving evidence of the benefits that MDD can provide to ABMS.

Introduction

Agent-based simulations have been widely used to understand the emergent behavior of complex systems. These systems are composed of multiple entities, or agents, which can interact with each other and are situated in an environment that they can perceive and modify through their actions. Building agent-based simulations is a challenging task that has been widely investigated in the context of agent-based modeling and simulation (ABMS), a simulation paradigm that uses autonomous agents and multiagent systems to reproduce and explore a phenomenon under investigation. The ABMS paradigm has been used in many application areas, such as traffic, ecology, economics, and epidemiology [36]. According to Macal and North [36], ABMS is selected as the simulation paradigm in such areas because it can explicitly incorporate the complexity arising from individual behavior and interactions that exist in real settings. Agents can furthermore be endowed with learning or evolutionary capabilities to adapt to changes in themselves or the environment. Artificial intelligence techniques that provide such capabilities are well established and can be incorporated into agents leading to more realistic simulations [33].

Many alternatives have been proposed for easing the development of agent-based simulations. Platforms, such as NetLogo [61] and Repast [41], have been widely used because they provide programming environments that offer language constructs to explicitly represent agents and their interactions, the environment as well as the creation and initialization of entities and agents. These platforms, however, demand previous expertise in ABMS and programming. Researchers have already argued about the importance of providing alternatives for modeling and creating agent-based simulations by means of building blocks at a higher abstraction level [25], [33], [59]. Such approaches would potentially decrease the development time and effort to build agent-based simulations.

Approaches towards this direction have been built upon the fundamentals of model-driven development (MDD) [4], [50], a software development approach in which models are considered first-class citizens and the development is driven by modeling artifacts [54]. These models are used to (semi-)automatically generate the source code of software systems and thus play a leading role in the development process. Previous work on the use of MDD in industry showed that MDD approaches can increase productivity by a 5–10x factor [56] and that the productivity increases because the modeling effort is focused on domain concerns instead of programming statements [53].

An MDD approach is composed of a domain-specific language (DSL) and model transformations for code generation [50]. The former allows effectively expressing domain concepts, while the latter introduces automation. A DSL is a modeling or programming language designed for a particular domain that trades generality for expressiveness. DSLs provide high-level, off-the-shelf, abstractions for business-related concepts and processes. By doing so, MDD approaches raise the abstraction level of models, allowing developers to refer to the desired functionality of the system instead of the details related to its development and deployment [8]. As noted by Stahl et al. [54], abstraction in MDD models does not mean vagueness, but compactness and reduction to the essence.

Despite qualitative arguments that motivate the use of MDD in ABMS, there is a need for concrete evidence that it in fact promotes its assumed benefits, i.e. by raising the level of ABMS by means of domain-specific abstractions, there is an increase in productivity. Previous studies provide examples of use [19], [21], [23], [34], [42], evaluation of comprehension [48] and comparison of software size as an estimation of effort [49]. Nevertheless, there is a lack of an assessment of the development effort and quality involving development tasks performed by simulation developers.

In response, we in this paper present an empirical study to quantitatively assess the benefits promoted by MDD in ABMS. More specifically, we take an existing approach in this context, namely MDD4ABMS [47], [48], and compare it with NetLogo, a widely used agent-based simulation platform. We selected MDD4ABMS because it not only provides common ABMS concepts as part of its graphical domain-specific language but also has an extension that includes concepts specific to an application area [49], which is adaptive traffic signal control. This approach thus effectively exploits the trade-off between expressivity and generality that is made in MDD [54]. In order to generalize our results to other domains, we provide a novel extension of MDD4ABMS, focusing on an alternative application area, the spread of diseases. Our empirical study consists of a comparison between MDD4ABMS and NetLogo when they are used by developers to build agent-based simulations in these two application areas. We collected data associated with the design quality of developed simulations and time spent to develop them. We also made a subjective evaluation of the two development approaches by means of questionnaire answered by participants. The obtained results show that MDD4ABMS requires less effort to develop simulations with similar (sometimes better) design quality than NetLogo, giving evidence of the benefits that MDD can provide to ABMS.

Our contributions thus are: (i) an extension of the MDD4ABMS approach, targeting the application area of spread of diseases; (ii) a study protocol to evaluate MDD approaches for ABMS; and (iii) an empirical study that compares MDD4ABMS and NetLogo in two application areas.