Elsevier

Information Systems

Volume 103, January 2022, 101874
Information Systems

Assessing and improving measurability of process performance indicators based on quality of logs

https://doi.org/10.1016/j.is.2021.101874Get rights and content

Highlights

  • Define PPI measurability in respect of the data quality of process event logs.

  • Propose a model for assessment of event log data quality in respect of PPI definitions.

  • Propose guidelines to improve the process monitoring infrastructure to achieve higher PPI measurability.

  • A case study in the service industry demonstrates the feasibility and value of the proposed framework.

Abstract

The efficiency and effectiveness of business processes are usually evaluated by Process Performance Indicators (PPIs), which are computed using process event logs. PPIs can be insightful only when they are measurable, i.e., reliable. This paper proposes to define PPI measurability on the basis of the quality of the data in the process logs. Then, based on this definition, a framework for PPI measurability assessment and improvement is presented. For the assessment, we propose novel definitions of PPI accuracy, completeness, consistency, timeliness and volume that contextualise the traditional definitions in the data quality literature to the case of process logs. For the improvement, we define a set of guidelines for improving the measurability of a PPI. These guidelines may concern improving existing event logs, for instance through data imputation, implementation or enhancement of the process monitoring systems, or updating the PPI definitions. A case study in a large-sized institution is discussed to show the feasibility and the practical value of the proposed framework.

Introduction

Business processes are at the heart of modern organisations, since they allow to design companies considering business goals and objectives, rather than simply focusing on functional separation [1]. As a consequence, evaluating the effectiveness and efficiency of business processes becomes crucial and a particular class of Key Performance Indicators (KPIs), usually referred to as Process Performance Indicators (PPIs) [2], are in charge of supporting the organisation’s management on this purpose. Through the analysis of PPI values, organisations can make decisions that may affect their short and long term profitability, reputation or even survival. Therefore, PPIs (i) must be insightful, i.e., they must capture relevant aspects of business processes, and (ii) their computation must produce a correct result. The former aspect relates with the definition of PPIs, whereas the latter focuses on their measurability.

As far as PPI measurability is concerned, it is normally assumed in the literature that PPIs can be measured as long as so-called process event logs [3] containing relevant data are available [4], [5]. On this basis, the literature about PPIs mainly concentrates on PPI definition, with several examples of conceptual models and notations to define PPIs (e.g., [5], [6]). More recent approaches also consider how to derive formal PPI definitions from natural language descriptions [7], [4].

Although the research on PPIs often assumes that event logs are available and error-free, in practice, like any other data produced by IT systems, even when available, event logs are actually error-prone [8], [9]. In fact, data for event logs can be collected by Business Process Management Systems (BPMS) or they can be obtained by combining, on an ad-hoc basis, local logging information of systems supporting the execution of the tasks composing the process, e.g., ERP modules, machines, ad-hoc systems. As a consequence, a malfunctioning of the BPMS or the other supporting systems could result in missing data in logs, as well as a lack of information to reconcile data between event logs that are involved in the same PPI definition. In addition to these basic problems, especially under the recent influence of Industry 4.0 and Internet of Things (IoT) trends, process-related artefacts have been equipped with high quality sensors [10], to gather more and more data at a finer grained level of detail. If, on the one hand, these improvements have significantly increased the quantity of data stored in the event logs, on the other hand, the quality of these data has not necessarily increased at the same pace. In fact, smart devices, which are fundamental in Industry 4.0, are often resource constrained, and under some circumstances, e.g., low battery level, they could generate unreliable monitoring data. Hence, considering as usually done in the current approaches only the availability of event logs is not sufficient to guarantee the measurability of PPIs. An analysis of the quality of the data included in the event logs must be also considered. This work fills this research gap by introducing a novel notion of PPI measurability as the ability to measure a reliable value of a PPI starting from the available data collected during the process enactment, also considering the granularity of this information especially with respect to the time dimension.

The contributions made by this paper to the business process management literature are the following:

  • To propose a model that links the PPI measurability to the quality of the data logged during the process execution. This model enables the analysis of the effects on PPI measurability of low quality logs, in which attributes may be missing or assume abnormal or inconsistent values, up to the situation in which some logs are not available at all. Here, the accuracy, consistency, completeness, maturity, and volume dimensions are considered, but the proposed approach is open to be extended with other dimensions. This model is novel in the literature because of two reasons: first, a PPI measurability model is currently lacking and PPI measurability is usually taken for granted as long as process event logs are available [4], [5]; second, the literature considers the data quality of event logs a crucial issue only in the event log extraction phase, neglecting the impact that it can have on other phases of the business process lifecycle, such as process monitoring;

  • To propose guidelines for improving PPI measurability. These guidelines range from improving the process monitoring infrastructure, for collecting logs of higher quality, to updating PPI definitions, in order to better exploit existing logs. While existing methods and guidelines for event log quality improvement target specific attributes, mainly timestamps [11], in the context of traditional process mining use cases, the proposed guidelines can be applied to any attribute in an event log and focus on the more general use case of process monitoring.

Besides addressing the research gap identified above, from a practical standpoint the proposed model and guidelines also address the need of data engineers to be able to design and build a reliable pipeline feeding high quality data gathered from operational systems into data analytics tool [12].

For the sake of clarity, a running example loosely derived from a real world process in hospital patient treatments is initially used along the paper to explain the different aspects of the proposed approach. In addition, the applicability of the proposed framework in practice is shown through a more realistic case study in a large European institution in the service industry.

As far as research methods are concerned, this paper adopts a design science approach [13], [14]. The PPI measurability assessment model and improvement guidelines can be seen as the artefact produced by this research. The design of this artefact is informed by the traditional literature on data quality and data integration [15] and by more recently emerging research insights regarding the importance of data quality for event log-based process analytics [9], [11]. From a research standpoint, the artefact proposed closes a gap in the literature, by being the first model that links event log data quality to PPI measurability. From a managerial standpoint, the proposed artefact shows clear practical relevance, as demonstrated by its application in a large real-world case study.

The paper is organised as follows. Related work is discussed in Section 2. Section 3 introduces the motivating example. PPI measurability and the overall framework considered in this paper are introduced in Section 4. The assessment of PPI measurability based on the quality of event logs is presented in Section 5 and guidelines for PPI measurability improvement are discussed in Section 6. The case study is presented in Section 7. Finally, conclusions are drawn in Section 8.

Section snippets

Background and related work

In this section we first review the literature about KPIs and their quality (Section 2.1), then we look in depth at previous work about quality of process event logs (Section 2.2).

Motivating case

To properly clarify the relevance of the PPI measurability and to clearly introduce the proposed approach, we consider a patient treatment process in an emergency situation, as shown by the BPMN process model in Fig. 1. After a first visit in the emergency room, patients receive an X-ray and a blood test. Depending on the outcome of these first tests, they may receive also an electrocardiogram (ECG). Depending on the outcome of a second visit, performed in one of the wards of the hospital

A framework for PPI measurability

In a data-driven process analysis perspective, the measurability of PPIs plays a fundamental role. For instance, to measure the PPI TreatmentCaseDur in the running example, the activity execution timestamps are required. If these timestamps are not properly logged, e.g., logged with unpredictable delays, such a PPI is unlikely to be significant in practice for the managers of the hospital. In other words, even if a PPI has been defined according to S.M.A.R.T. principles, its value for the

PPI measurability assessment

The assessment of the PPI measurability concerns the computation of the selected data quality dimensions for the event logs that are linked to the process model.

Definition 10

The measurability of a ppi is a function of the quality of the system logs collecting data of activity attributes required for evaluating the value of the ppi, i.e., Meas(ppi)={dqm(ppi)}m=1,,Mwhere dqk(ppi) are the considered data quality dimensions.

Note that Meas(ppi) could be expressed also by mean of a single value, such as a

PPI measurability improvement

The PPI measurability assessment allows the identification of a set of improvement actions, which aim to enhance the quality of event logs, so as to improve the measurability of PPIs. Improvement actions may target (i) the existing logged values in ILogppi, (ii) the definition of an indicator ppi, and (iii) the logging infrastructure of the systems S producing data to calculate an indicator ppi: (i) focuses on improving the quality of logging data that have already been acquired, whereas in

Case study and discussion

The practical applicability of the proposed framework has been tested by considering some ITIL compliant processes of a large institution in the service industry in Europe. The analysis was performed from the point of view of decision makers conducting an Internal Audit to assess the risks and performance of the processes. Section 7.1 describes in detail the process considered, while Section 7.2 discusses the PPI measurability assessment and improvement. Finally, Section 7.3 discusses the

Conclusions

Process performance indicators are usually defined to capture critical insights about a process. While the definition of insightful PPIs is fundamental, in practice the ability of a PPI to give critical insights often relies on the extent to which it can be calculated reliably using the data available about process execution, i.e., event logs.

This paper has presented an approach to define, assess, and improve the measurability of a PPI. PPI measurability is defined by considering the data

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (55)

  • AwadA. et al.

    Analyzing and repairing overlapping work items in process logs

    Inf. Softw. Technol.

    (2016)
  • NguyenH.T.C. et al.

    Autoencoders for improving quality of process event logs

    Expert Syst. Appl.

    (2019)
  • DumasM. et al.

    Fundamentals of Business Process Management, Vol. 1

    (2013)
  • van der AalstW. et al.

    Workflow mining: Discovering process models from event logs

    IEEE Trans. Knowl. Data Eng.

    (2004)
  • Van LooyA. et al.

    Business process performance measurement: A structured literature review of indicators, measures and metrics

    SpringerPlus

    (2016)
  • HompesB.F. et al.

    A generic framework for context-aware process performance analysis

  • del Río-OrtegaA. et al.

    Using templates and linguistic patterns to define process performance indicators

    Enterp. Inf. Syst.

    (2016)
  • BoseR.J.C. et al.

    Wanna improve process mining results?

  • FischerD.A. et al.

    Enhancing event log quality: Detecting and quantifying timestamp imperfections

  • HarperK.E. et al.

    Industrial analytics pipelines

  • HevnerA.R. et al.

    Design science in information systems research

    MIS Q.

    (2004)
  • WieringaR.J.

    Design Science Methodology for Information Systems and Software Engineering

    (2014)
  • BatiniC. et al.

    Data and information quality - dimensions, principles and techniques

  • DoranG.T.

    There’s a S.M.A.R.T. way to write management’s goals and objectives

    Manage. Rev.

    (1981)
  • YemmG.

    Essential Guide to Leading Your Team: How to Set Goals, Measure Performance and Reward Talent

    (2013)
  • WatsonH.J. et al.

    The current state of business intelligence

    Computer

    (2007)
  • BallouD.P. et al.

    Enhancing data quality in data warehouse environments

    Commun. ACM

    (1999)
  • Cited by (5)

    View full text