Farming smarter with big data: Insights from the case of Australia's national dairy herd milk recording scheme

https://doi.org/10.1016/j.agsy.2020.102811Get rights and content

Highlights

  • Farmer interest in data relates to business stage and the decision’s importance.

  • Cooperative governance structures facilitate farmer trust and use of big data.

  • Support services reduce the impact of low digital literacy, capacity and capability.

  • Inter-dependence occurs between existing data processes and optimising data use.

Abstract

Digitalization and the use of Smart Farming Technologies are considered a major opportunity for the future of agriculture. However, realisation of full benefits is constrained by: (1) farmers' interest in and use of big data to improve farm decision making; (2) issues of data sovereignty and trust between providers and users of data and technology; (3) institutional arrangements associated with the governance of data platforms. This paper examines the case of Australia's dairy herd milk recording system, arguably one of agriculture's first cases of ‘big data’ use, which collects, analyses and uses farm-level data (milk production, lactation and breeding records) to provide individual cow and herd performance information, used by individual farmers for farm management decisions. The aim of this study was to 1) examine the use of big data to add value to farm decision making; and 2) explore factors and processes, including institutional arrangements, which influence farmer engagement with and use of big data. This paper traces the Australian history of the organisation of dairy herd recording (established in 1912 and digitalized in late 1970s) and then uses findings from a longitudinal study of 7 case study dairy farms, which were incentivised to become involved in herd recording in 2015. Applying a conceptual framework linking path dependency in farm decision making and collaborative governance capacity, we find three new important dimensions of the farm user context influencing farmer demand for big data applications: 1) the transition to a new business stage; 2) the additionality farmers seek from data generated in one component of the farm system to other subsystems, and 3) the use of data in long term or strategic decision making. Further, we identified critical attributes of support services in addressing digital literacy, capacity and capability issues at farm level, including diversity in data presentation formats and facilitation of the on-farm transition process through intermediary herd test organisations. The role of farmers as governance actors, or citizens in the decisions of the trajectory of big data applications, adds to understanding of the nature of collaborative governance arrangements that support farm engagement.

Introduction

The application of big data in smart farming has been a focus of recent scholarly interest. Big data, the collection, aggregation and analysis of large volumes of data and information sets (Bronson and Knezevic, 2016), is suggested to revolutionise timely decision making on farm and realise large gains in productivity and profitability. On-farm application of big data could generate up to up to AUD $19.1bn of additional value across the Australian agricultural sector (Perrett et al., 2017) through capture, discovery, and analysis (Kamilaris et al., 2017) in a way that allows farmers and related organisations to extract economic value from data. However, to maximise the benefits of big data necessitates large investments in data storage and processing infrastructure. Big data analysis includes practices that advance traditional analytics through the amalgamation of real-time, multi-sourced information – derived on and off-farm – providing farmers better opportunities to diagnose risks, identify alternatives, and project future consequences (Lioutas et al., 2019). Smart farming involves the enhancement of existing management tasks and decision making in a specified context, situation and/or location (Kamilaris et al., 2017). In smart farming, digital technologies assist in the continuous monitoring and measurement of the physical environment. This results in large quantities of data – often from heterogeneous sources - requiring large-scale collection, storage, pre-processing, modelling and analysis. Recently the term ‘digital agriculture’ has been used to encompass the concepts of smart farming, precision agriculture, decision agriculture and Agriculture 4.0 (Klerkx et al., 2019).

There are a range of impediments for farmers to realise the benefits of digital agriculture. Constraints surrounding data sovereignty and the socio-ethical dimensions of use of data include; a lack of structure and governance related to agricultural big data (Kamilaris et al., 2017), data collection and use arrangements which favour multinational commercial interests in agriculture over user capability and autonomy (Boyd and Crawford, 2012; Bronson and Knezevic, 2016; Wolfert et al., 2017; Carolan, 2018) and inequality amongst beneficiaries from big data (Carbonell, 2016; Eastwood et al., 2017b; Fleming et al., 2018; Zhang et al., 2017). Big data can also disrupt long-standing relationships between actors such as farmers and agribusinesses or supply chain organisations (Bronson and Knezevic, 2016). A lack of trust between farmers (as data contributors) and those third parties who collect, aggregate and share their data has been noted(Wiseman et al., 2019; Klerkx et al., 2019). Public-private partnerships (PPP) are suggested as a mechanism to balance public and private interests in progressing agricultural innovation more broadly (Hermans et al., 2019). However, the sheer scale of private interests pushing proprietary big data platforms and solutions to farmers is a challenge for the organisation of beneficial digitalization in agriculture (Wolfert et al., 2017; Carolan, 2018).

Secondly, numerous ongoing technical challenges have been reported (Wolfert et al., 2017 provide a comprehensive review) which also impact farm interest and adoption of big data applications. Technical challenges include: data interpretation at scale; data integration; reliable infrastructures to collect and analyse big data; human resources and expertise; and a need for new business models that are attractive to solution providers while enabling a fair share between different stakeholders (Kamilaris et al., 2017 pg 29; Wolfert et al., 2017). The cost of smart farming technologies, ease of use, lack of compatibility between devices and the availability of impartial advisory services have been identified as key farm-level constraints (Tey and Brindal, 2012; Pierpaoli et al., 2013; Eastwood et al., 2017a; DEDJTR, 2018). Better support for learning is important for progressing from data to practical farming decisions (Evans et al., 2017) with digital literacy and the skills and capacity of some groups of farmers to manage digital tools reported as impediments (Pierpaoli et al., 2013; Carolan, 2018; Fleming et al., 2018). The need for incentives to increase farm exposure and transition in digital agriculture, particularly first-time users, has been noted as an important support to adoption (Tey and Brindal, 2012; Yigezu et al., 2018). Evans et al. (2017 p79) suggest information and services will need to be provided flexibly, and at varying frequencies for ‘different user segments’. The lack of proven results in transforming data to better decisions, and the focus of many digital technologies on measurement not decisions is a key concern amongst farmers in the European Union and Australia (Jochinke et al., 2007; Zhang et al., 2017; Fleming et al., 2018; Knierim et al., 2018)

With few proven applications of big data, this range of impediments represents large uncertainties and risks for farmers so it is conceivable to question why farmers would engage with big data. Regan (2019) argues that ‘risk’ in big data applications is ambiguous and perceived differently amongst different actors. They suggest risk management needs to be discourse-based and negotiated collaboratively over time. This requires collaborative governing. Because farmers are citizens (Carolan, 2018) and/or co-creators (Lioutas et al., 2019), we need to consider ways to involve them in governing, Ayre et al., 2019 argue that digital agriculture requires harnessing and mobilizing diverse skills, knowledge/s, materials and representations for translating digital data, digital infrastructure and digital capacities into better decisions for farm management. The role of farm advisors in big data applications has received increasing attention. The importance of back-office advisory roles and how these will need to adapt with the demands of big data has been highlighted by Eastwood et al. (2019). They suggest new roles will emerge beyond information gathering, to include remote data interpretation and sensemaking, as well as roles for technology suppliers as specialist advisors at the technology and farm management interface. These intermediary roles are suggested to continually adapt to enable greater value from data-driven smart farming to be captured by farmers.

Addressing the gap between opportunity and reality of big data application for farm decision making has been suggested as a key area for scholarly concern. Carolan (2018) calls for multiple understandings of agri-digital applications with attention to the interdependencies between individual and collective arrangements and the implications for farmer autonomy and capability. While Wolfert et al. (2017) called for social research related to practical and commercial implications of big data developments, including organisation and governance issues. Bronson and Knezevic (2016) suggest systematically tracing the digital revolution in agriculture as an important goal for big data scholarship. To generate value from big data applications at farm level, Lioutas et al. (2019) emphasise the importance of understanding how farmers interact with data and their motivations in doing so. Primarily it is argued farmers' engagement with big data is about improving farm decision making, hence a need to follow farmers' decisions though case studies.

By extension, current farmer involvement in big data applications and use of data to make informed decisions and the institutional arrangements that have supported or hindered this engagement can feasibly provide insight to these gaps. We use the case of farmer participation in dairy herd recording in Australia to: 1) examine the use of big data to add value to farm decision making; and 2) explore factors and processes, including institutional arrangements, which influence farmer engagement with and use of big data.

With a history spanning more than a century, dairy herd recording (herd testing or milk recording) is one of the oldest examples of digitalization and big data in agriculture, thus it lends itself well as a case study. A herd test captures information about milk volume and quality attributes of individual cows through the collection (and analysis) of individual milk samples (Andrews, 2016). Milk sample analysis and data compilation is undertaken by herd test organisations (HTO); intermediary organisations with multi-faceted roles in the data pipeline (Fig. 1). Characteristic of big data analysis, HTO amalgamate herd test records with data captured from other sources on-farm (Fig. 1). Reports and summary statistics generated can be used to assist on-farm decision making in areas such as cow health, milk quality and feeding (Zottl et al., 2015). With one herd test milk sample capable of generating 899 spectral data points and ~20 milk parameter estimates, dairy cows accumulate hundreds of pieces of information within a year, illustrated in Fig. 2. Nationally, the Australian dairy industry's central data repository has close to 200 million herd test records and is growing by 2.9 million herd test records per year (DataGene Ltd, 2018). HTO also play a role in the aggregation of data from multiple farms for submission into this database (see Section 4.1.1) where it is used collectively for regional reporting, research and national herd genetic improvement. Such data sharing arrangements between HTO and organisations undertaking genetic evaluation services are seen world-wide. These relationships enables additional value from data, without creating extra work on-farm (Zottl et al., 2015).

The paper is organised as follows. First the conceptual framework for the study is presented, followed by a description of the case study methodology and details of the studies conducted. This is followed by results and discussion.

Section snippets

Conceptual framework

To address the research questions, we required an analytical framework that theorized both farmers' engagement with and perception of value in big data, including the role of intermediaries such as farm advisors, and the institutional arrangements that support farmer engagement in big data applications. We drew on theories of farm decision making, intermediary practice and collaborative governance.

Method

To trace the development of herd recording as an example of farmer engagement in big data applications, a case study methodology was chosen. Having the national dairy herd recording system as the focus of the case study allows an in-depth appreciation of farm decision making and collaborative governance relating to big data in a real context. Whilst there are different types and uses of case studies in scientific inquiry, in this study we chose herd testing as an ‘instrumental’ case (Stake, 2005

Big data emergence & a history of collaborative investment and adaptation

Within the evolution of herd recording over 100 years, three key processes are identified as influential in the continuation of herd recording: continual innovation in data collection, analysis and reporting practices; strategic cycles of government investment; and collaborative effort from major stakeholders to develop a sectoral strategy and implement governance arrangements summarised in Table 2. Fig. 5 outlines changes in the number of cows under herd test in Victoria alongside key events

Discussion

In this paper, we drew on the case of herd recording in the Australian dairy sector to investigate factors and processes influencing farmer engagement with big data applications and how farmers use big data in decision making. The importance of the user production context in adoption, and the usability and usefulness of data to decision support has been further reinforced in our study (Evans et al., 2017 p79; Rose et al., 2018). Lioutas et al. (2019) emphasise that an understanding of farmer

Conclusion

In this paper, we have traced the development of herd recording in Australia as a case for understanding: 1) the use of big data to add value to farm decision making; and 2) factors and processes which influence farmer engagement with and use of big data, including institutional arrangements. Following an evolution over 100 years through cycles of digitization and digitalization, the case provides some insights to the broader context of farmer’s acceptance of big data applications.

In this study

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was completed as part of the ImProving Herds Project. ImProving Herds is funded by the Gardiner Dairy Foundation (Melbourne, Australia) and Dairy Australia (Melbourne, Australia) and led by Agriculture Victoria (Victoria, Australia) with collaborative support from DataGene Ltd (Melbourne, Australia), Holstein Australia (Melbourne, Australia) and the National Herd Improvement Association of Australia (Melbourne, Australia). We also acknowledge the important contributions of

References (62)

  • E.D. Lioutas et al.

    Key questions on the use of big data in farming: An activity theory approach

    NJAS - Wageningen Journal of Life Sciences

    (2019)
  • R. Nettle et al.

    How private-sector farm advisors change their practices: An Australian case study

    Journal of Rural Studies

    (2018)
  • E. Pierpaoli et al.

    Drivers of precision agriculture technologies adoption A literature review

    Procedia Technol.

    (2013)
  • J.E. Pryce et al.

    Symposium review: Building a better cow—The Australian experience and future perspectives

    Journal of Dairy Science

    (2018)
  • Á. Regan

    ‘Smart farming’ in Ireland: A risk perception study with key governance actors

    NJAS - Wageningen Journal of Life Sciences

    (2019)
  • H. Soyeurt et al.

    Mid-infrared prediction of bovine milk fatty acids across multiple breeds, production systems, and countries

    Journal of Dairy Science

    (2011)
  • L. Sutherland et al.

    Triggering change: Towards a conceptualisation of major change processes in farm decision-making

    Journal of Environmental Management

    (2012)
  • T. van den Broek et al.

    Governance of big data collaborations: How to balance regulatory compliance and disruptive innovation

    Technological forecasting and social change.

    (2018)
  • S. van der Burg et al.

    Ethics of smart farming: Current questions and directions for responsible innovation towards the future

    NJAS - Wageningen Journal of Life Sciences

    (2019)
  • J.I. Weller et al.

    Invited review: A perspective on the future of genomic selection in dairy cattle

    Journal of Dairy Science

    (2017)
  • S. Wolfert et al.

    Big Data in Smart Farming - A review

    Agricultural Systems

    (2017)
  • YA Yigezu et al.

    Enhancing adoption of agricultural technologies requiring high initial investment among smallholders

    Technological forecasting and social change

    (2018)
  • G.P.S. Anderson et al.

    Comparison of records from in-line milk meters and conventional herd testing for management and genetic evaluation of dairy cows

  • A. Andrews

    The Hico story. Dairy Herd Improvement in Gippsland and Colac

    (2016)
  • N. Beilharz

    Dairy co-op Murray Goulburn cuts milk prices, MD Gary Helou departs

  • J.P. Bennett

    Australian dairy herd improvement scheme

    Proc. Assoc. Advmt. Anim. Breed. Genet.

    (1984)
  • D. Boyd et al.

    Critical questions for big data - Provocations for a cultural, technological, and scholarly phenomenon

    Information, Communication and Society.

    (2012)
  • K. Bronson et al.

    Big Data in food and agriculture

    Big Data Soc.

    (2016)
  • J.M. Bryson et al.

    The design and implementation of cross-sector collaborations: Propositions form the literature

    Public Administration Review

    (2006)
  • I. Carbonell

    The ethics of big data in big agriculture

    Internet Policy Rev.

    (2016)
  • M. Carolan

    Smart farming technologies as political ontology: access, sovereignty and the performance of neo liberal and not so neo liberal worlds

    Sociol. Rural.

    (2018)
  • Cited by (51)

    • The enabling and constraining connections between trust and digitalisation in incumbent value chains

      2023, Technological Forecasting and Social Change
      Citation Excerpt :

      These path dependencies are difficult to overcome when there is a limited basis of trust, or even distrust, that supports a collaborative approach towards digitalisation and its possibilities. In a sense organisations or even entire value chains become ‘locked-out’ or excluded from new (digitalisation) opportunities, due to the level of (dis)trust (Newton et al., 2020). A key implication for those concerned with managing innovation in value chains is that there needs to be sufficient space for exploration to see what the possibilities and challenges are in both the short and longer term.

    • Dimensions of digital transformation in the context of modern agriculture

      2022, Sustainable Production and Consumption
      Citation Excerpt :

      Still, on the subject of data security and data sharing, Van Es and Woodard (2017) points out that coordinating data sharing and making access to this information more transparent are vital to improving data security. To help farmers have more choices in their interactions with their own data, Newton et al. (2020) recommend establishing data-sharing pipelines and policies. As one of the more effective components of DT for agroecological and small farmers, Rotz et al. (2019b) cite data sharing and open-source technologies.

    View all citing articles on Scopus
    View full text