Assessing and improving measurability of process performance indicators based on quality of logs
Introduction
Business processes are at the heart of modern organisations, since they allow to design companies considering business goals and objectives, rather than simply focusing on functional separation [1]. As a consequence, evaluating the effectiveness and efficiency of business processes becomes crucial and a particular class of Key Performance Indicators (KPIs), usually referred to as Process Performance Indicators (PPIs) [2], are in charge of supporting the organisation’s management on this purpose. Through the analysis of PPI values, organisations can make decisions that may affect their short and long term profitability, reputation or even survival. Therefore, PPIs (i) must be insightful, i.e., they must capture relevant aspects of business processes, and (ii) their computation must produce a correct result. The former aspect relates with the definition of PPIs, whereas the latter focuses on their measurability.
As far as PPI measurability is concerned, it is normally assumed in the literature that PPIs can be measured as long as so-called process event logs [3] containing relevant data are available [4], [5]. On this basis, the literature about PPIs mainly concentrates on PPI definition, with several examples of conceptual models and notations to define PPIs (e.g., [5], [6]). More recent approaches also consider how to derive formal PPI definitions from natural language descriptions [7], [4].
Although the research on PPIs often assumes that event logs are available and error-free, in practice, like any other data produced by IT systems, even when available, event logs are actually error-prone [8], [9]. In fact, data for event logs can be collected by Business Process Management Systems (BPMS) or they can be obtained by combining, on an ad-hoc basis, local logging information of systems supporting the execution of the tasks composing the process, e.g., ERP modules, machines, ad-hoc systems. As a consequence, a malfunctioning of the BPMS or the other supporting systems could result in missing data in logs, as well as a lack of information to reconcile data between event logs that are involved in the same PPI definition. In addition to these basic problems, especially under the recent influence of Industry 4.0 and Internet of Things (IoT) trends, process-related artefacts have been equipped with high quality sensors [10], to gather more and more data at a finer grained level of detail. If, on the one hand, these improvements have significantly increased the quantity of data stored in the event logs, on the other hand, the quality of these data has not necessarily increased at the same pace. In fact, smart devices, which are fundamental in Industry 4.0, are often resource constrained, and under some circumstances, e.g., low battery level, they could generate unreliable monitoring data. Hence, considering as usually done in the current approaches only the availability of event logs is not sufficient to guarantee the measurability of PPIs. An analysis of the quality of the data included in the event logs must be also considered. This work fills this research gap by introducing a novel notion of PPI measurability as the ability to measure a reliable value of a PPI starting from the available data collected during the process enactment, also considering the granularity of this information especially with respect to the time dimension.
The contributions made by this paper to the business process management literature are the following:
- •
To propose a model that links the PPI measurability to the quality of the data logged during the process execution. This model enables the analysis of the effects on PPI measurability of low quality logs, in which attributes may be missing or assume abnormal or inconsistent values, up to the situation in which some logs are not available at all. Here, the accuracy, consistency, completeness, maturity, and volume dimensions are considered, but the proposed approach is open to be extended with other dimensions. This model is novel in the literature because of two reasons: first, a PPI measurability model is currently lacking and PPI measurability is usually taken for granted as long as process event logs are available [4], [5]; second, the literature considers the data quality of event logs a crucial issue only in the event log extraction phase, neglecting the impact that it can have on other phases of the business process lifecycle, such as process monitoring;
- •
To propose guidelines for improving PPI measurability. These guidelines range from improving the process monitoring infrastructure, for collecting logs of higher quality, to updating PPI definitions, in order to better exploit existing logs. While existing methods and guidelines for event log quality improvement target specific attributes, mainly timestamps [11], in the context of traditional process mining use cases, the proposed guidelines can be applied to any attribute in an event log and focus on the more general use case of process monitoring.
Besides addressing the research gap identified above, from a practical standpoint the proposed model and guidelines also address the need of data engineers to be able to design and build a reliable pipeline feeding high quality data gathered from operational systems into data analytics tool [12].
For the sake of clarity, a running example loosely derived from a real world process in hospital patient treatments is initially used along the paper to explain the different aspects of the proposed approach. In addition, the applicability of the proposed framework in practice is shown through a more realistic case study in a large European institution in the service industry.
As far as research methods are concerned, this paper adopts a design science approach [13], [14]. The PPI measurability assessment model and improvement guidelines can be seen as the artefact produced by this research. The design of this artefact is informed by the traditional literature on data quality and data integration [15] and by more recently emerging research insights regarding the importance of data quality for event log-based process analytics [9], [11]. From a research standpoint, the artefact proposed closes a gap in the literature, by being the first model that links event log data quality to PPI measurability. From a managerial standpoint, the proposed artefact shows clear practical relevance, as demonstrated by its application in a large real-world case study.
The paper is organised as follows. Related work is discussed in Section 2. Section 3 introduces the motivating example. PPI measurability and the overall framework considered in this paper are introduced in Section 4. The assessment of PPI measurability based on the quality of event logs is presented in Section 5 and guidelines for PPI measurability improvement are discussed in Section 6. The case study is presented in Section 7. Finally, conclusions are drawn in Section 8.
Section snippets
Background and related work
In this section we first review the literature about KPIs and their quality (Section 2.1), then we look in depth at previous work about quality of process event logs (Section 2.2).
Motivating case
To properly clarify the relevance of the PPI measurability and to clearly introduce the proposed approach, we consider a patient treatment process in an emergency situation, as shown by the BPMN process model in Fig. 1. After a first visit in the emergency room, patients receive an X-ray and a blood test. Depending on the outcome of these first tests, they may receive also an electrocardiogram (ECG). Depending on the outcome of a second visit, performed in one of the wards of the hospital
A framework for PPI measurability
In a data-driven process analysis perspective, the measurability of PPIs plays a fundamental role. For instance, to measure the PPI TreatmentCaseDur in the running example, the activity execution timestamps are required. If these timestamps are not properly logged, e.g., logged with unpredictable delays, such a PPI is unlikely to be significant in practice for the managers of the hospital. In other words, even if a PPI has been defined according to S.M.A.R.T. principles, its value for the
PPI measurability assessment
The assessment of the PPI measurability concerns the computation of the selected data quality dimensions for the event logs that are linked to the process model.
Definition 10 The measurability of a ppi is a function of the quality of the system logs collecting data of activity attributes required for evaluating the value of the ppi, i.e., where are the considered data quality dimensions.
Note that could be expressed also by mean of a single value, such as a
PPI measurability improvement
The PPI measurability assessment allows the identification of a set of improvement actions, which aim to enhance the quality of event logs, so as to improve the measurability of PPIs. Improvement actions may target (i) the existing logged values in , (ii) the definition of an indicator , and (iii) the logging infrastructure of the systems producing data to calculate an indicator : (i) focuses on improving the quality of logging data that have already been acquired, whereas in
Case study and discussion
The practical applicability of the proposed framework has been tested by considering some ITIL compliant processes of a large institution in the service industry in Europe. The analysis was performed from the point of view of decision makers conducting an Internal Audit to assess the risks and performance of the processes. Section 7.1 describes in detail the process considered, while Section 7.2 discusses the PPI measurability assessment and improvement. Finally, Section 7.3 discusses the
Conclusions
Process performance indicators are usually defined to capture critical insights about a process. While the definition of insightful PPIs is fundamental, in practice the ability of a PPI to give critical insights often relies on the extent to which it can be calculated reliably using the data available about process execution, i.e., event logs.
This paper has presented an approach to define, assess, and improve the measurability of a PPI. PPI measurability is defined by considering the data
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (55)
- et al.
On the definition and design-time analysis of process performance indicators
Inf. Syst.
(2013) - et al.
Transforming unstructured natural language descriptions into measurable process performance indicators using hidden Markov models
Inf. Syst.
(2017) - et al.
Event log imperfection patterns for process mining: Towards a systematic approach to cleaning event logs
Inf. Syst.
(2017) - et al.
Multi-party business process compliance monitoring through IoT-enabled artifacts
Inf. Syst.
(2018) - et al.
A taxonomy for key performance indicators management
Comput. Stand. Interfaces
(2019) - et al.
Data quality management data usage experience and acquisition intention of big data analytics
Int. J. Inf. Manage.
(2014) - et al.
Big data adoption: State of the art and research challenges
Inf. Process. Manage.
(2019) - et al.
Big data-enabled customer relationship management: A holistic approach
Inf. Process. Manage.
(2018) - et al.
Business process analysis in healthcare environments: A methodology based on process mining
Inf. Syst.
(2012) - et al.
Evaluating the effect of best practices for business process redesign: An evidence-based approach based on process mining techniques
Decis. Support Syst.
(2017)
Analyzing and repairing overlapping work items in process logs
Inf. Softw. Technol.
Autoencoders for improving quality of process event logs
Expert Syst. Appl.
Fundamentals of Business Process Management, Vol. 1
Workflow mining: Discovering process models from event logs
IEEE Trans. Knowl. Data Eng.
Business process performance measurement: A structured literature review of indicators, measures and metrics
SpringerPlus
A generic framework for context-aware process performance analysis
Using templates and linguistic patterns to define process performance indicators
Enterp. Inf. Syst.
Wanna improve process mining results?
Enhancing event log quality: Detecting and quantifying timestamp imperfections
Industrial analytics pipelines
Design science in information systems research
MIS Q.
Design Science Methodology for Information Systems and Software Engineering
Data and information quality - dimensions, principles and techniques
There’s a S.M.A.R.T. way to write management’s goals and objectives
Manage. Rev.
Essential Guide to Leading Your Team: How to Set Goals, Measure Performance and Reward Talent
The current state of business intelligence
Computer
Enhancing data quality in data warehouse environments
Commun. ACM
Cited by (5)
Process-Data Quality: The True Frontier of Process Mining
2023, Journal of Data and Information QualityDesign of a Productivity Monitoring System for an Asset Maintenance Group Using Design Thinking Methodology
2023, ACM International Conference Proceeding SeriesThe Impact of Process Complexity on Process Performance: A Study Using Event Log Data
2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)