1 Introduction

Emerging technologies, such as robotic process automation, machine learning, neural networks and complex algorithms, often referred to as Artificial Intelligence (AI), are expected to impact organisations significantly (Willcocks 2020). Recent studies on AI reveal its potential to help solve current problems in the field of data processing and business process optimisation. On the other hand, concerns are raised about the potential of AI to support wrong decision-making (Dignum 2018; Floridi et al. 2018). Market research shows that the 2025 market size of AI will grow to $232 billion compared to an estimated $12.4 billion today (KPMG 2018). McKinsey’s survey report on AI shows that AI adoption is highest in product or service development and service operations (McKinsey Research 2020), where Gartner indicates the most important trend for 2020 as ‘smarter, faster, more responsible AI’ and that ‘by the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures’ (Gartner 2020). This fits within the service-oriented approach of organisations today to improve internal business services (Plugge et al. 2020). As AI skills are globally in demand, outsourcing of AI is a means to address this shortage in general (Anderson et al. 2020; Dawson 2021) or in specific domains, such as cyber security (Smiths 2018).

Although there is no consensus on a common definition of AI (Legg and Hutter 2007; Wang 2019), Dwivedi et al. (2021, p. 2) argue that ‘the common thread amongst AI definitions is the increasing capability of machines to perform specific roles and tasks currently performed by humans within the workplace and society in general’.

This description includes machines that mimic cognitive functions, such as learning, speech and problem-solving (Russell and Norvig 2016) and the ability to independently interpret and learn from external data (Kaplan and Haenlein 2019). Also responsibilities for AI lack clarity. This includes gaps in four areas: culpability, moral and public accountability and active responsibility (de Sio and Macacci 2021). Especially the latter is relevant in the context of AI outsourcing.

Given the fast pace and complexity of AI technologies, larger organisations adopting AI usually engage in outsourcing relationships with leading suppliers of AI systems, consulting and services. This is part of a broader digital transformation outsourcing trend (Beulen and Ribbers 2020), where artificial intelligence enables digital transformations (Magistretti et al. 2019; Calp 2020; Holmstrom 2021).

Leading suppliers in this field include Accenture, Atos, Boston Consulting Group, EY, HCL, KPMG, McKinsey and Tata Consultancy Services, as well as many other established and niche suppliers. As the impact of AI on business is of strategic importance, AI outsourcing needs to be carefully governed. Kranz (2021) argues that there is statistical evidence that contractual governance plays a key role in Information Systems (IS) arrangements. This is supported by industry, for example by the COBIT framework that consists of 40 Governance and Management Objectives, including the objectives ‘managed service agreements’ (APO09) and ‘managed vendors’ (APO10) (De Haes et al. 2020). However, there is no research on how formal AI contracts are established. Moreover, the way clients and suppliers cope with relational governance, including psychological contracts (Argyris 1960; Levinson et al. 1962), in the context of AI is unknown.

We aim to fill this research gap by presenting a study that examines governance of AI outsourcing in an in-depth, qualitative manner. To broaden our view on governance for AI outsourcing, we examine formal and relational governance elements, including a psychological contract, as part of the client-supplier outsourcing arrangement. These elements have to be taken into account, not only by clients and suppliers, but also by IS sourcing advisors. To the best of our knowledge, formal and informal governance in the context of AI outsourcing has not been studied before. In doing so, we answer the call of Kranz (2021) to study the implementation of contractual models and its effect on governance and relationship outcomes. Moreover, we extend the study by Linden and Rosenkranz (2019) on relational governance in the context of AI outsourcing. Hence, the main research question of this paper is:

How are formal and relational governance used in artificial intelligence outsourcing?

The contribution of this study is to make an inventory of the current AI outsourcing practicesby focusing on formal and relational governance. This paper is organised as follows. First, our theoretical background addresses the concepts of governance for AI outsourcing. Subsequently, we introduce a research framework, based on the outsourcing governance framework of Lioliou et al. (2014). The research method is presented in Sect. 3, and the data analysis and results are described in Sect. 4. Discussions and conclusions are presented in Sect. 5 and 6, respectively.

2 Theoretical background

2.1 Artificial intelligence

Haenlein and Kaplan argue that in the 1950’s the academic discipline of AI has been established, however “AI remained an area of relative scientific obscurity and limited practical interest for over half a century.” (2019, p. 5). Literature shows that AI has been studied from an augmentation perspective, arguing that AI should support humans in decision-making (Miller 2018; Duan et al. 2020). This is consistent with the research of Davenport and Ronanki (2018) in which the authors argue that AI systems are not able to explain the reasoning process of decision-making, and as such are supportive of human judgement. As an example, the authors address the role of deep learning as a means to learn from large volumes of labelled data. However, it is almost impossible to understand the models that are designed to execute its functionality. This AI “black box” may cause problematic issues, for instance in regulated sectors, as regulators insist on knowing why and how decisions are made. Thus, AI can be perceived as an awareness system that is supportive of humans in order to increase the visibility of events or states (Krancher et al. 2018) and may add business value to organisations by means of technology (Sabherwal and Jeyaray 2015). Tarafdar et al. (2019) demonstrates in their study on AI that successful organisations which are able to create value radically improved their business processes. In doing so, AI aids organisations to process more data that support human decision-making.

However, when AI systems, such as intelligent Information Technology (IT) service management systems (Zhang 2021), rely on dispersed, low quality, missing or inconsistent data, conventional organisations are challenged by successfully integrating data, which in turn may result in biased or incorrect decision-making. Meaningful human control is required (de Sio and Mecacci 2021). Due to involving suppliers, ensuring this has become more difficult, as AI services in outsourcing are performed at arms-length.

This may also lead to ethical and legal concerns specifically around the analysis of how decisions are made. Managing compliance regulations, which is perceived to be a business challenge (Haenlein and Kaplan 2019—micro perspective; Dwivedi et al. 2021), is influenced by the degree of AI complexity, for instance deep learning. As a result, this may create compliance issues. Recently, Coombs et al (2020, p. 13) find that ‘there is limited research that examines how organisations design governance arrangements and structures associated with Intelligent Automation decision-making’. By applying an IS governance lens organisations may develop strategies to overcome compliance challenges and contribute to support AI decision-making.

2.2 IS governance

We performed a literature review to identify attributes that correspond to IS governance. Search terms used include “formal governance”, “relational governance”, “contract”, “services”, “pricing”, “trust”, “dependency”, “commitment”, “communication”, and “psychological contract”. The selection was based on two criteria. First, publications have been selected that include at least one search term in the title, abstract or keywords. Second, we focus on IS related journals and books (chapters) to create a fit with the context of our study.

Our review reveals that IS governance, taking formal and relational attributes into account, has been extensively researched (see Table 1). Importantly, there does not appear to be a single accepted definition of IS governance. As such, it is difficult to formulate an exact description. Based on our literature review, we define that ‘IS governance is an integral part of corporate governance [that] addresses the definition and implementation of information system processes, structures and relational mechanisms in the organisation, enabling both business and Information Systems people to execute their responsibilities in support of business/IS alignment and the creation of business value from IS-enabled business investments’ [adapted from Van Grembergen and De Haes (2009)]. As organisations have to manage uncertainties and unanticipated obligations in the provisioning and reliability of external services (Kotlarsky et al. 2020), governance is a prerequisite to encourage desirable behaviour in the use of resources.

Table 1 Overview of relevant IS governance journal publications addressing formal and relational attributes
Full size table

Importantly, the literature reveals two main research streams that include formal and relational governance. Scholars studied formal governance that aims to coordinate activities between outsourcing partners to prevent opportunistic behaviour through the mutually agreed and legally binding behaviours (Beulen and Ribbers 2007, 2021; Hoetker and Mellewigt 2009; Chang et al. 2017). Other scholars considered relationships (Lee and Cavusgil 2006; Rai et al. 2012) in which the work of Macneil (1980) is used as a starting point. Relational governance attempts to address some of the deficiencies in contract governance: the failure to account for social structures within which the inter-organisation exchanges are embedded, as well as the overestimation of hazardous elements in the exchange (Xiao et al. 2012).

More recently, the literature on IS governance shows more complex and dynamic interrelationships between formal and relational governance than the previously assumed dichotomy of complementarity and substitution (Oshri et al. 2015; Lacity et al. 2016; Kranz 2021). These studies emphasise the role of relation-specific boundary conditions and find that formal and relational governance mechanisms can act as substitutes (Rai et al. 2012), simultaneously work as substitutes and complements (Tiwana 2010; Lioliou et al. 2014) or have an impact that oscillates over time (Huber et al. 2013).

2.2.1 Formal IS governance

In formal IS governance, contract mechanisms need to be in place to monitor and enforce contractual commitments (Lacity et al. 2016). Most common formal governance-related attributes address contracts (Oshri et al. 2015), Service Level Agreements (SLAs) and Key Performance Indicators (KPIs) (Lioliou and Willcocks 2019), and flexibility (Goo et al. 2009). Moreover, pricing mechanisms, such as time and materials contracts and fixed price contracts, are important formal attributes as well (Oshri et al. 2015). An increasing number of contracts also include profit-sharing, also known as gainsharing, as a pricing mechanism (Lioliou and Willcocks 2019). A recent study of Kranz (2021), who studied formal and relational IS governance in the field of innovation IT outsourcing shows that contractual mechanisms primarily determine joint performance. Kranz (2021, p. 13) observed that ‘while relationship learning profited from higher levels of alignment between the two governance modes (e.g. formal and relational), joint innovation is highest when contractual governance is high and relational governance is at a medium level’. However, a question that remains unanswered in their research is to understand how control- and coordination-focused clauses affect governance and relationship outcomes. Our aim is to explore this question in the context of Artificial Intelligence outsourcing.

2.2.2 Relational IS governance

Literature on relational IS governance indicate five common attributes, namely; (1) trust, (2) commitment, (3) dependency, (4) communication and (5) psychological contract. Trust between the client and the supplier is essential (Li et al. 2010; Lacity et al. 2016) to achieve agreed outcomes. Closely related to trust is commitment (Goo et al. 2009; Lioliou et al. 2014; Kranz 2021), as a two-sided approach drives achieving results and will contribute to trust. Dependency contributes only if there is a supplier dependency or a mutual dependency (Lioliou et al. 2014). Furthermore, communication enables proactive responses from both the supplier and the client (Lacity et al. 2016; Lioliou and Willcocks 2019). Finally, psychological contracts are important—these are not signed contracts, but are a set of beliefs and perceptions related to contractual commitments (Robinson and Rousseau 1994) to resolve the limitations of formal contracting and create flexibility for the client as well as the supplier (Han et al. 2017). From a relational IS governance perspective Linden and Rosenkranz (2019) studied the activities of third-party advisors in IT outsourcing engagements. This novel study on third-party advisors shows that relational mechanisms are perceived to be essential in building the relationship between client and supplier. However, this relevant study does not include the role of suppliers. Hence, in our research we focus on suppliers explicitly and as such, contribute to literature.

2.3 AI outsourcing governance research framework

As we study AI in the context of outsourcing, the unit of analysis in this research is the supplier side. With our research we want to gain a better understanding on how IS governance for AI is structured. Importantly, IS governance and its embedded attributes (formal governance, relational governance and psychological contracts, as detailed in the Sects. 2.2.1 and 2.2.2) is used as a moderator. The framework (see Fig. 1) is applicable to IS governance in regular supplier relationships. Our framework is based on the general IS governance framework of Lioliou et al. (2014)—Fig. 2, p. 520. However, as AI is an emerging and innovative technology, there is additional complexity in IS governance. In this context trust, collaboration and a network perspective require an increased focus (Hoecht and Trott 2006), as well as there is an increased dependency, due to the client specifics of the AI outsourced services (Gopalakrishnan and Zhang 2019). Also protecting IP is essential in innovative services (Cammarano et al. 2019), such as AI services. This complexity also guides the analysis of the expert interviews.

Fig. 1
figure 1

Governance of Artificial Intelligence outsourcing research framework (interrelations between formal and relational governance and psychological contract are based on Lioliou et al. (2014)—Fig. 2, p. 520)

Full size image

3 Research method

Due to the complex nature of the multi-faceted domain of AI governance for outsourcing relationships, we decided to adopt an exploratory, case study-based approach, based on the assumption that this would give us a deeper understanding of the subject under examination (Yin 2017). Case study research is a common method in the field of IS (Orlikowski and Iacono 2001) and useful when it comes to answering ‘how’ and ‘why’ questions (Benbasat et al. 1987). On the other hand, case study research does not allow for statistical generalisations because the number of entities described in case studies is too small. However, for our purposes, analytical generalisation is more important than statistical generalisation (Yin 2017). Our approach allowed us to focus on formal governance and relational governance practices of suppliers in their AI-outsourcing contracts. Due to commercial confidentiality, the suppliers under study were reluctant to share contractual insights with the research team. Consequently, we did not had access to contractual obligations between the suppliers and their clients. However, by signing Non-Disclosure Agreements (NDAs) we were able to interview various supplier representatives and as such, create an unique insight in how they handle IS governance and their way of working in the context of AI outsourcing. To select relevant suppliers, we used two main criteria. First, we made a distinction with regard to the range within AI provided services. The range of a supplier’s portfolio affects the extent to which governance has to be established. Second, we realised that the role of each supplier within the client-supplier relationship may vary. Some suppliers only focus on client engagements, while others provide solutions to all ecosystem partners. By selecting eight suppliers offering different types of AI, supplemented with two market research advisors to understand the market better (see Table 2), the two criteria outlined above are met.

Table 2 Overview interviewed experts of ten case study organisations
Full size table

3.1 Data collection

We applied a two-phased approach in which we collected primary and secondary data based on an interview protocol. We began by collecting publicly available data related to the type of IS outsourcing suppliers for AI, which helped us understand the core of their business. The data were collected based on two sources: (1) archival data (desk research) and (2) direct observations. The documentation we examined included website information, whitepapers, factsheets and public webinars. We also looked at outsourcing seminars and market conferences and interviewed two market research advisors (interviewees 16–18). In addition, direct observations in the form of field notes were drafted during the interviews, which provided relevant background information on how suppliers dealt with AI governance.

In the second phase, we conducted a number of formal (online) and informal (telephone) interviews. The interviews were semi-structured and based on a protocol that included open questions (see Appendix A) that were designed to contribute to the consistency and reliability of the results (Denzin 2009). In all, 18 in-depth interviews were conducted with various supplier representatives combining two key roles in each interview, namely, a client representative (partners, directors, senior vice presidents) and AI Subject Matter Experts (SMEs), in order to gain insight into day-to-day operations. All the interviewees were selected based on their involvement in establishing AI-related services towards clients. The interviewees 1–4, 6, 8, 10, 12, 14–18 have C-level and C-level -1 leaderships positions in global leading suppliers and + 20 years of experience, and are leading global AI engagements, where the interviews 5, 7, 9 and 11 are senior AI SMEs delivering AI engagements to global top tier clients with + 10 years of experience.

The researchers conducted the interviews predominantly in English, with a few in Dutch. Because the interviews were confidential, we anonymised the names of the interviewees and provided their roles in the organisation, as detailed in Table 2. The interviews were held between November 2020 and May 2021 and lasted between 30 and 90 min. They were all digitally recorded and subsequently transcribed, after which the transcripts were sent to the participants for confirmation. To ensure a reliable data analysis, we built a research database.

3.2 Data analysis

We analysed the data in a number of systematic steps in order to make sure the process was replicable. First, we studied context-related information from a broader organisational client-supplier relationship. The aim is to create a basic understanding of what type of services are provided by suppliers. Second, we conducted a thorough analysis of interview transcripts and archival data, verifying the data as needed via follow-up telephone calls and e-mails. By triangulating the interview transcripts with supplementary key documents (e.g. webinar input, whitepapers, factsheets, field notes), we were also able to triangulate sources (Patton 2002; Denzin 2009). This allowed us to validate the steps included in the research process and improve the internal validity of the expert interviews. All interview data were subject to cross-examination by three researchers, and any errors were corrected, resulting in additional triangulation of the available data.

We used techniques such as coding and clustering (Yin 2017) and followed the advice of Miles and Huberman (1994) to divide coding among two researchers, who each coded the interview notes (Miles and Huberman 1994, p. 64). More specifically, Atlas.ti was used to code and combine interview data of the 18 respondents into concept maps which resulted in a total of 141 codes. We conducted four rounds of concept aggregation. During the first three rounds, a number of concepts that are closely aligned to each other were aggregated into a single code. For example, the code ‘co-develop with clients’ is closely aligned to the code ‘co-development’. Moreover, we reduced the number of codes by ignoring concepts that are perceived as influencing attributes. In round four, we identified 42 codes (concepts) that represent the most important elements of our research framework (see Appendix B). Labelling the relationships between concepts we used the names: ‘Has impact on’, ‘Is associated with’, and ‘Is cause of’. Next, we discussed the findings and clarified any disagreements. Based on the coding process, we were able to create insight in relevant concepts and relationships. Consequently, we draw conclusions on what types of AI were provided by the suppliers under study and how they handle governance of AI outsourcing.

4 Findings

In the following sections, we highlight the governance attributes by answering the following research question:

How are formal and relational governance used in Artificial Intelligence Outsourcing?

We clustered our findings by means of the governance of AI outsourcing and its embedded elements and have added interview quotes to illustrate our findings.

4.1 Client—supplier AI outsourcing arrangement

Interviews show that the initiator of AI at the client side often varies across various departments, such as data analytics, data science, innovation, IS and business intelligence. All interviewees argue that clients’ business processes and IS-enabled systems are interrelated and form a prerequisite to discuss and apply AI. With regard to clients’ AI approach, we distinguish two phases. First, clients establish a multidisciplinary team that comprise of business and IS representatives exploring potential AI opportunities. Next, potential business process candidates are analysed from different perspectives, like risk, compliance, privacy and legality.

Clients’ IT and innovation teams often lack the knowledge of what's really important for their business, how processes work, what the right procedures are, and where failures may occur. But it's not that they already have that within their background. And we don't pretend that all our data scientists know this already, but we do have people specialised in industries and if we work together with them, we can already bring in the right teams (source: Technology supplier—Accenture—Client Partner).

The typical set of profiles of clients involved in AI is 50% business stakeholders and 50% technology experts. However we see more and more clients initiating AI initiatives from a business side (source: Market research advisory firm—Gartner—Client Partner).

Second, the multidisciplinary team is responsible for building, testing and deploying the AI solution by applying an agile (sprint) approach. By using proof-of-concepts, AI functionality is tested in practice.

What’s really important is to train AI, like algorithms. By setting up a minimum viable product (MVP) and train an algorithm during sprints step by step you can increase its functionality. So we create a kind of a jumpstart and by exploring different use cases in parallel we show our competitive edge at the same time (source: Technology supplier—TCS—CTO).

Start small initiatives, and focus on continuous improvement. What is important is to monitor the availability of capabilities closely from the start. The ability to scale up is an important success factor (source: Market research advisory firm—Gartner—AI SME).

Interviews consistently show that the agile approach is a good practice in the current emerging market maturity for AI.

4.2 Formal governance of AI outsourcing

All three attributes of formal governance of AI outsourcing were fully recognised by our interviewees. The next subsection describes the findings in more detail per attribute.

4.2.1 Contract

From a contractual perspective we observed two findings that relate to Intellectual Property (IP) and Terms and Conditions (T&Cs). With regard to IP, we noticed that suppliers argue that AI-related IP should be owned by their clients as they are responsible for applying AI. Technology supplier in contrast distinguish between commodity and customised type of IP, in which the latter should be owned by the client.

Essentially we are bounded by the contract that we agreed with. If we build an AI algorithm, then the client wants us to sign that we do not build the same algorithm for the other clients. So, IP is transferred to the client (source: Technology supplier—Accenture—AI SME).

Similar to software development, if a supplier includes their IP in AI, they grant a perpetual licence to protect their interest. Interestingly, suppliers also offer the opportunity to clients to co-develop AI services. In these circumstances, both the client and supplier draft specific contractual clauses on how to deal with IP-related topics.

By co-creating IP, we are able to monetarise our client’s assets that affect their balance sheet positively. In case we co-develop IP with a client we may come to an agreement that this IP can be used in other client engagements. If that’s the case, the original client receives a royalty fee (source: Technology supplier—HCL—SVP).

Suppliers that are involved in co-IP development argue that contractual clauses are rapidly evolving. Our interviews show that standard T&Cs are used in AI-related contracts. Some suppliers argue that existing framework contracts are used to contract AI and therefore standard T&Cs are applied, but specific AI solution provisions are embedded in the underpinning AI solution Statements of Work.

Typically, clauses are related to a responsible user, as we're building a product and leave it behind. The responsible user clauses are dealing with client obligations on using the product. This is not limited to adjusting the source code but also includes training the algorithm (source: Strategic advisor—BCG—MD & Partner 1).

Interestingly, none of the interviewees argued that decreased liability caps were applied in contracts for AI. Apparently, the suppliers do not recognise the additional risks of provisioning AI or have instead incorporated the additional risks into their fees.

4.2.2 Services—SLAs and KPIs

Addressing the attribute services, we find that, oftentimes, existing framework agreements are used. Suppliers argue that existing SLAs are applied to measure AI performance. However, we notice that the vast majority of AI-type engagements reflect a project approach (e.g. design, build, test and deploy). In these projects, service agreements reflect the skills of data designers, data engineers and project lead times. Importantly, suppliers can agree on applying experience-driven xLAs with their clients. This finding demonstrates their willingness to focus on AI solution outcomes in contrast to technical availability.

We see a shift towards xLAs (experience-driven service levels). Based on the insights that we gain during AI tests we can predict how AI will perform in practice. That means that we are willing to apply a service level driven by experience (source: Technology supplier—TCS—VP & Global Head).

None of the interviewees has yet to experience penalty clauses for missed service levels in providing AI. Therefore, there are no observations related to KPIs, which are typically used in contracts, only to measure the service level but also to include a penalty for missing the agreed-upon service level.

4.2.3 Pricing mechanisms

The third formal governance attribute corresponds to pricing mechanisms. The interviews show pricing mechanisms that can be categorised into transaction-based (e.g. fixed price, time and materials, consumption) and outcome-based (e.g. bonus/malus) mechanisms. Fixed Price agreements are used in case project related activities are performed that represent less risks, such as setting up a proof-of-concept to test an AI solution. In case the degree of uncertainty (e.g. risks) of a project increases, clients and suppliers agree to apply the mechanism of time and materials. This relates for example to the unknown number of sprints that are required to develop, test and deploy AI.

Considering our AI support the financial model is based on a fixed fee type of model. If the degree of uncertainty in a project is difficult to calculate, we use a time and materials model. We do not use or see risk and reward types of financial models in practice (source: Accounting and Consulting firm—KPMG—Senior Manager).

Technology suppliers that offer ‘AI as a Service’ apply a consumption-driven model. Depending on the number of client users and types of AI solutions, the degree of consumption affects the price directly. We find that two technology suppliers offer outcome-based pricing agreements (e.g. HCL and TCS). After co-developing and implementing an AI solution, performance is measured for a certain period; based on the performance and learnings, an agreement is made for the desired outcome. These suppliers applied a bonus/malus mechanism based on the outcome provided.

AI as an outcome-based model is part of our service portfolio. By collaborating with our clients in co-creating AI we are transparent in how AI works in daily life. In specifying a desired outcome, we agree on a bonus/malus principle. An example is the development of a proactive application maintenance solution, which we have based on AI. The focus is on cost reduction and improved quality of services (source: Technology supplier—HCL—SVP).

The interviews reveal that strategy advisors apply the identification of relevant use cases for their AI, which is a similar principle as for their other offerings—potential benefit / investment ratio.

We will not touch anything which is less than 10X, and on average we typically pitch for opportunities between 50 to 100X, both in terms of benefit/investment ratio (source: Strategic advisor—McKinsey—Client Partner).

4.3 Relational governance of AI outsourcing

All five attributes of relational governance of AI outsourcing were fully recognised by our interviewees. The next subsection describes the findings per attribute in detail.

4.3.1 Trust

Interviews reveal that the attribute trust in supplier capabilities is perceived as essential. All suppliers developed practical frameworks that are used to support AI, for instance for design, development and implementation. The generally considered factors that are discussed are risks (e.g. data, design, performance), technology, reliability, ethics, social impact and accountability. By developing and applying frameworks, they establish transparency, which in turn contributes to trust.

Importantly, we developed criteria to assure the quality of our work. In case we develop AI for clients, we also apply our internal AI quality framework. We share and discuss this quality framework with our clients and provide full transparency as an extra guarantee to inform them proactively. Next, the client accepts our product and becomes responsible for the use of the AI solution, such as an algorithm (source: Accounting & Consulting firm—KPMG—AI SME).

Trust is also related to ethics. All suppliers explicitly address this aspect in delivering AI. Continuously monitoring ethical compliance is perceived as important.

Of course, everyone agrees that ethics is important, but how do you really implement it? Actually, when you look at certain elements that are relevant for ethics and fairness, and then testing your output among different groups that isn't rocket science. But you can also make it transparent. That will contribute to trust (source: Accounting & Consulting firm—EY—AI SME).

In addition, we noticed that providing AI assurance, including ethical compliance, is emerging.

Going forward, clients might ask for, or even due to regulatory compliance might be in need of, an AI assurance certification (source: Market research advisory firm—HfS—SVP).

Business leaders are increasingly very closely involved in ensuring ethical compliance (Technology supplier—TCS—VP & Global Head).

I think we are five to ten years away from people thinking about it systematically and proactively. However, in all of our engagements we explicitly integrate ethical compliance in our engagements (source: Strategic advisor—BCG—MD and Partner 2).

We have our set of ethical principles and a set of actionable policies. Any engagement team has to answer 10 yes/no questions to flag risks, potentially followed by a full assessment of 50 questions. We use this to assess the ethical risk profile. During the engagement, the high-risk cases are closely monitored by a committee. These senior subject matter experts are also guiding engagement teams (source: Strategic advisor—BCG—MD and Partner 1).

4.3.2 Commitment

Addressing the attribute of commitment, our research findings show that relationship management is key for suppliers. We notice that most AI solution initiatives start at the executive level due to the strategic impact that AI may have within organisations. The motivation to create executive-level sponsorship is twofold. First, AI may affect an organisation’s corporate image seriously, for example when poor designed solutions result in wrong decision-making. Second, due to AI complexity, projects require multiple internal stakeholders (e.g. business representative, designer, developer, data engineer). Consequently, executive sponsorship is required to manage and align internal departments and create insights in both opportunities and risks. By achieving executive commitment first, suppliers build commitment and trust before starting an AI project.

AI may have a serious effect within a firm or towards their customers that may be impactful. Therefore, it is important to ensure that executive management is aware of the consequences. That’s why we focus on C-level commitment first (source: Strategic advisor—McKinsey—Client Partner).

We found no differences in the initiation of AI solution engagements across the different types of suppliers. All initiated AI solution engagements are with both existing clients and with new clients.

4.3.3 Dependencies

Regarding the attribute dependency, our analysis demonstrates two relevant dependencies that relate to clients and its suppliers. Importantly, when suppliers and their clients co-develop IP, the AI solution may comprise inconsistencies.

By co-creating IP we are able to monetarize our client’s assets that affect their balance sheet positively. In case we co-develop IP with a client we may come to an agreement that this IP can be used in other client engagements. If that’s the case, the original client receives a royalty fee (source: Technology supplier—HCL—AI SME).

Interviewees state that data quality (e.g. correctness, completeness, compliance) is essential and can be seen as the foundation for a profound AI solution. If an AI solution is functioning incorrectly or even not at all, data quality may cause a serious dependency. As data is primarily a clients’ responsibility, the reputation of the supplier may be at risk.

It does start with the data you know because AI will use available data in which the quality is key. Dependent on the client’s data quality, we select a relevant type of AI technology and then start to co-develop. Not just apply the same AI technology because you're used to it, but the client’s specific situation will influence the choice of a technology (source: Accounting & Consulting firm—EY—AI SME).

Compliance is also an important topic if proprietary software is used for providing the AI services.

We are mailing using open source products. Therefore, we didn't have to go down to too many legal issues or other kind of compliance implications, rather than technical standardization (source: Accounting & Consulting firm—Atos—AI SME).

In order to ensure good data quality, data cleansing can also be added to the engagement.

Ensuring the data quality at all levels is often a significant part of our engagements. The data needs to be of high quality to enable a long-term training. For instance, six to eighteen months’ time for algorithm training data sets is not unusual (source: Accounting & Consulting firm—EY—Client Partner).

In turn, clients’ data quality inconsistencies may affect AI governance negatively.

A second dependency is the demand for skilled professionals. The labour market for AI professionals, such as data scientist and data engineers, is stressed. AI skills are in demand by both suppliers and clients. Access to qualified resources is essential for IS governance.

Data scientists and data engineers are in high demand. We see a lot of emphasis on training, also as part of our engagements (source: Technology supplier—Accenture—AI SME).

We find that educating client staff in AI reduces the dependency and improves IS governance.

4.3.4 Communication

Our study shows that as the number of clients’ stakeholders involved in AI solution increases, intensive communication is required to align AI goals and potential outcomes. We find examples of agile teams, in which client and supplier representatives collaborate in sprints, and where the supplier also educates their clients. Moreover, bi-weekly management meetings are used to discuss progress. Due to the complexity of designing and developing AI and the number of stakeholders involved, we find that proactive communication and collaboration enables trust between a client and their supplier.

I think we quite often make AI something that is really far away and is a kind of a black box. But actually, it's a human team that is behind it. And the more you communicate and get involved with the client team that in the end is using it, the more aware you become of AI functionalities (source: Technology supplier—Atos—Client Partner).

4.3.5 Psychological contract

Our study shows that as the number of client stakeholders involved in AI solution increases, the awareness related to reputational risks also increases. Combined with the complexity of AI outsourcing contracts, a psychological contract is established. As in any outsourcing contract, there are mutual responsibilities that are addressed in a formal contract as well as in a psychological contract. However, we find that suppliers’ knowledge and experience, combined with their (long-term) client relationship and commercial interests, leads to proactive responses to handle incidents and decreased performance, regardless of formal agreed responsibilities.

If my customers get worried that our algorithm is not providing the right results for them anymore, then all the attention will go to fixing it now, and then the question of who is responsible for that is no longer relevant anymore because we just need to fix it. We make it happen (source: Technology supplier—Atos—Client Partner).

We find that a psychological contract enables transparency with regard to the obligations between clients and suppliers, which supports governance of AI outsourcing.

4.3.6 Summary

Although adoption of AI services is still emerging, the need to implement IS governance is fully confirmed by all our interviewees. The general understanding was that formal governance and relational governance are interwoven. However, as AI is still an innovative technology and both the client and supplier currently have limited experience, we find that relational governance is dominant over formal governance. We summarized the core statements and findings in Table 3.

Table 3 Overview of core statements and findings
Full size table

5 Discussion

In this section we reflect on formal governance in Sect. 5.1, addressing IP, experience-driven service levels and pricing mechanisms, and on relational governance in Sect. 5.2, addressing fostering innovation and information symmetry. In Sect. 5.3 we reflect on combine both perspectives and discuss the co-development in AI outsourcing relationships.

5.1 Formal governance

Our results show that suppliers who co-develop innovative AI go beyond the traditional type of outsourcing contract agreements, which are more rigid by nature (Lioliou et al. 2014; Aubert et al. 2015). Importantly, our findings on co-development suggest that collaboration between clients and suppliers contribute to the creation of common insights that in turn strengthen trust between both parties. Based on trust, both parties are willing to establish an outcome-based model that reflects a desired outcome. This shows the interwoven nature of formal and relational governance. Bapna et al. (2016) suggested that the outsourcing experience of clients has a negative impact on contract outcome. We expand on Bapna’s analysis by demonstrating that co-development positively contributes to establishing an outcome-based contract.

5.1.1 Intellectual property

Both Strategic Advisors and Accounting & Consulting firms argue that IP of co-developed AI services should remain with the client, while a technology supplier shares this opinion in the case of customised AI. Lacity and Rottman (2008) indicated that an efficient manner to protect IP is to separate projects into a series of segments given to different suppliers; however, this could limit the innovation potential. The fact that clients become the owner of co-developed IP contributes to future innovations, which is in contrast to the fragmented IP approach suggested by Rottman and Lacity (2006). Our findings operationalise a recent study of Kranz (2021) in which contractual mechanisms primarily determine joint innovation performance. Based on our analysis we argue that both formal and relational IS governance contribute to AI outsourcing. More specifically, by establishing a set of beliefs and perceptions (psychological contract), such as AI proof of concept and co-development of IP, clients are better able to deal with contract uncertainties. This correspond to Kranz (2021, p. 12) who argues that ‘positively framed contracts offer an alternative instrument for coping with the substantial exchange hazards and uncertainty risks involved in embedded Information Technology Outsourcing (ITO) relationships aimed at co-creating innovation.’

5.1.2 Experience-driven service levels

Our analysis reveals that experience-driven service levels prevail over metrics-based service levels. We suggest that implementing experience-driven service levels may decrease transaction costs and at the same time limit ex post vendor opportunism and create incentives for mutual cooperative behaviour (Kranz 2021). Thus, by using experience-driven service levels, parties may overcome rigid contract characteristics and adapt to changes as contracts evolve over time. Based on the AI co-development process, clients and suppliers mutually increase the level of trust by applying experience-driven service levels. Therefore, we argue that experience-driven service levels foster relational governance.

5.1.3 Pricing mechanisms

Our results indicate that various types of pricing mechanisms are applied in practice, which are consistent with previous IS governance studies (Oshri et al. 2015; Amiri et al. 2021). Interestingly, we found two technology suppliers who provide outcome-based contracts by using a bonus/malus pricing mechanism. Both technology suppliers co-developed AI with their clients and, in doing so, they achieved transparency about business rules and decision-making principles. This finding provides evidence for the assumption of Oshri et al. (2015) that outcome-based mechanisms ‘increase transparency regarding commitment and profit-sharing involved in such a setting’ (p. 213). The decision to include dissimilar resources in the client-supplier relationship increases collaboration and contributes to shared value creation, a view that is supported by Vitasek and Manrodt (2012). Our findings on outcome-based pricing models go beyond a recent study of Amiri et al. (2021) who find that pricing models in the context of IS sourcing arrangements are predominantly based on times and materials and fixed price. The role of an active client involved in co-developing AI services may explain our findings. In sum, we argue that suppliers who focus on joint collaboration with clients to establish experience-driven services shift from a transactional type of relationship to a relational type of outsourcing arrangement.

5.2 Relational governance

The results of this study confirm that well-governed client-supplier outsourcing relationships positively affect the degree of trust between parties in creating AI solutions. Our findings are consistent with previous studies in which parties that are willing to share knowledge (e.g. co-develop IP) invest in trusted interorganisational relationships (Muthusamy and White 2005; Whitley and Willcocks 2011). Our study demonstrates that the development and maintenance of social ties between suppliers and their clients contributes to building trust and reduces transaction costs (Poppo and Zenger 2002).

5.2.1 Foster innovation

Suppliers who seek commitment with clients’ executive management, create solidarity that encourages a bilateral approach to joint innovation through mutual adjustment. We suggest that mutual commitment forms a prerequisite in developing AI, taking the various stakeholders at client and supplier side into account. This is supported by prior research in which mutual adjustments would fit in the relationship between the client and suppliers to deal with unanticipated events (Plugge and Janssen 2020). Specifically, accounting and consulting firms and technology suppliers become mutually dependent on their clients in their collaboration to develop AI. This corresponds to Pfeffer and Salancik (1978), who argue that resource concentration is related to the extent in which power and authority are widely dispersed. Therefore, we conclude that both clients and suppliers are equally interdependent on each other. In general, our analysis shows that suppliers that share their research insights or collaborate with client subject matter experts experience communication as a key success factor. Proactive communication of suppliers also supports a trustworthy relationship with their clients. We showed how suppliers apply a communication strategy by communicating research outcomes with their clients (e.g. face-to-face, whitepaper, websites). Moreover, our findings show how suppliers build relationships with their clients when managing the relationship and building trust by providing subject matter AI support.

5.2.2 Information asymmetry

Importantly, prior literature examined various devices to reduce information asymmetry between clients and suppliers as part of IS outsourcing arrangements including, supplier reputation, client-supplier prior relationship and technological diversity of the supplier (Bapna et al. 2016). We argue that the suppliers can reduce information asymmetry by jointly collaborating with their clients. In other words, co-developing AI by means of mutual adjustments is a strategy to overcome information asymmetry risks. This includes a focus on communication and putting in place a psychological contract. Attribute communication aligns with COBIT’s suggestion to implement governance boards, as detailed by De Haes et al. (2020). Regarding the psychological contract, this follows the suggestion by Monquin et al. (2020) to balance the commercial interest of suppliers with client interests. Our findings enhanced the model of Linden and Rosenkranz (2019) as we included the view of suppliers in AI outsourcing.

5.3 Implications for governance of AI outsourcing

Recent studies showed an increased complexity between formal and relational governance as relationships are affected by interfirm contingencies (Tiwana 2010; Huber et al. 2013; Lioliou et al. 2014). Our findings concerning AI suppliers demonstrate two relevant client contingencies that influence formal and relational governance, including a psychological contract.

5.3.1 Co-develop artificial intelligence

The first contingency corresponds to the need of clients to co-develop AI with suppliers, specifically. Our findings suggest that collaboration between both parties strengthens the degree of trust. This interaction effect between a contractually agreed activity and the relational mechanism of trust indicates complementarity. The second contingency addresses the willingness of the client and supplier to agree on experience-driven service levels. In doing so, the contractual agreement to measure service metrics that are based on a client’s perception underpin the relational governance attribute of dependency. Within this context, our results indicate that formal and relational governance are complementary, as the combined use of both governance mechanisms positively foster the arrangement. Similar to previous research (Goo et al. 2009; Lioliou et al. 2014), relational governance of technology suppliers to support client AI enables the effectiveness of formal governance. Finally, the importance of ethics in AI assurance has been mentioned by the interviewees. This requires ethical standards and committees on both the client and the supplier side. The significance of ethics in AI governance is growing (Renda 2019; Kerr et al. 2020). This includes addressing both the ‘demand’ and ‘supply’ side (Renda 2019, pp. 113–114), and in addition to ‘lawful’ and ‘responsible’ AI, a third level of ‘sustainable’ AI is also being added (Renda 2019, p. 115). Furthermore, this includes taking into account societal expectations (Kerr et al. 2020).

6 Conclusions, limitations and future research

6.1 Conclusion

To the best of our knowledge, this is the first study that examines the governance of AI outsourcing from the perspective of suppliers. Our study extends previous research on IS governance by studying formal and relational governance, including a psychological contract, by applying the governance model of Lioliou et al. (2014). In doing so, we answer the call of Kranz (2021) to study the implementation of contractual models in an innovation driven context and enhanced the study of Linden and Rosenkranz (2019) by including the role of suppliers. Secondly, our study reveals various forms of contractual models in which some cater for client needs specifically (e.g. outcome-based, experience-driven agreements). Taking this context into account, our analysis acknowledges that AI outsourcing shifts the emphasis from a transactional type of arrangement to a relational type of outsourcing arrangement. Third, this paper posits that formal and relational governance are complementary in cases where clients and suppliers co-develop AI. The combined use of both governance mechanisms positively fosters AI outsourcing arrangements. Finally, the growing importance of ethics in AI will positively contribute to governance. However, this is an emerging topic which will have an increasing significance over time.

6.2 Limitations

While our study provides important implications for clients and suppliers involved in the design, development and delivery of AI solutions, there are also some limitations. Because our research is based on a limited number of expert interviews (#18) of eight supplier firms and two market research advisors, the analytical generalizability of the results is limited due to differences in the types of suppliers. Second, we only studied the use of formal and relational governance. However, the literature reveals that the context in which a study is conducted matters (Lacity et al. 2016). We have not studied other context-related factors that may affect the client-supplier relationship, as we conducted expert interviews, and our case studies are based on various types of AI consulting services, market and technology providers. While these global and leading AI firms can build on a large client base and rich experience, the experiences of client organisations and their perspective on governance of AI outsourcing would further enrich the findings presented. Thirdly in addition to our qualitative research we could have conducted an qualitative study by increasing the number of participants and a simple coherent questionnaire as the basis for a survey.

6.3 Future research

The expert-driven interviews identify multiple avenues that require further research. We recommend additional expert interviews to increase the generalisation of the results, which is related to our first research limitation. These additional expert interviews should include interviews with experts from Tier 2 and Tier 3 suppliers and suppliers operating from developing continents, e.g. Latin America and Africa.

We recommend more detailed research with regard to the attributes in IS governance in the context of emerging technologies such as AI. By including case study specific elements such as the relationship between the client and the supplier and the outsourcing contract and related documents such as service level and audit reports additional insights can be gained. Furthermore, as emerging technologies like AI mature, we assume that the focus will transform from relational to formal governance. This supports to re-do this research in due time. Secondly, we may assume that the impact of maturing technologies will potentially impact the supplier landscape. We expect that maturing will trigger a gradual exit of the strategy advisors in the AI market. Third, we suggest examining the effect of knowledge creation and capturing in case a client and supplier co-develop IP. Interestingly, by using a social capital lens, researchers may explore how co-development of IP may develop mutual trust. These suggestions address the context limitation, which is our second research limitation. Finally, we would recommend, in line with our third research limitation, combining multiple case studies to validate and statistically generalise our findings.