Crowdsourced Behavior-Driven Development☆,☆☆
Graphical abstract
Introduction
Crowdsourcing software engineering offers developers new ways to contribute to software projects, enabling workers outside a traditional software development team to take part in building software (LaToza and van der Hoek, 2016). One form of crowdsourcing is microtask crowdsourcing, in which a workflow is used to decompose tasks into a sequence of microtasks. By reducing the necessary task context for newcomers to learn, microtask programming reduces the cost of onboarding and enables developers to contribute more quickly. Small contributions also open the possibility of increasing parallelism in software development. As many hands make light work, decomposing software development tasks into microtasks might enable some of this work to be completed in less time by parallelizing work across many developers.
A variety of systems have explored the promise of applying microtask crowdsourcing to programming (Chen et al., 2017, Goldman et al., 2011, Goldman, 2012, Lasecki et al., 2015, LaToza et al., 2015a, LaToza et al., 2018, LaToza et al., 2015b, LaToza et al., 2014, LaToza et al., 2013, Nebeling et al., 2012, Schiller and Ernst, 2012, Mao et al., 2017). For example, in Apparition (Lasecki et al., 2015), a client developer narrates a description for a user interface in natural language, and crowd workers translate this description into user interface elements, visual styles, and behavior. In CodeOn (Chen et al., 2017), a client developer narrates a request for help from their IDE, and crowd workers use this request and relevant context to author answers and code.
A long-term vision of microtask programming is to enable software to be built entirely through microtasks (LaToza et al., 2013). This differs from current approaches in scale: rather than facilitating individual tasks to be manually requested and managed by a requester, mechanisms must be found for the crowd to coordinate and work more effectively together. In microtasking, required coordination is described through a workflow, describing the microtasks which developers will complete, the handoffs that occur between microtasks, and the resulting dependencies between microtasks (Kittur et al., 2013). A key challenge in applying microtask crowdsourcing approaches to the domain of software engineering is the decontextualized nature of microtask work. Developers work without awareness of the complete program, reducing the necessary context which must be learned to contribute but increasing the potential for work going off track. Key to the success of microtasking approaches is to ensure effective mechanisms for coordination and aggregation exist, enabling workers to obtain feedback as quickly as possible to ensure their contributions are beneficial and coordinate contributions so that they do not conflict. When this does not occur, problems may ensue. Developers may write code which references fields and functions that do not exist, making poor implementation choices, or making implementation choices which conflict with other choices (LaToza et al., 2014, LaToza et al., 2018). As a result, time and effort are wasted, as further work is required to fix these issues, and the potential for undetected defects increases.
In this paper, we introduce a new approach for feedback based on behavior-driven development. In contrast to existing approaches which rely on either a client or manager (e.g., Lasecki et al., 2015, Goldman, 2012) or later crowd contributions (e.g., LaToza et al., 2014, LaToza et al., 2018), we enable developers to receive initial feedback within the microtask itself. We accomplish this by re-envisioning the scope of the microtask. In our workflow, developers receive feedback from three different sources while they are programming: through syntax errors, through running unit tests, and through the ability to debug their code. Adapting the idea of behavior-driven development to crowdsourcing work (North, 2006, Beck, 2003), each microtask encompasses the work to test and implement a behavior, a specific identifiable use case of a function. Developers work on a behavior end-to-end, identifying it from a high-level description of a function, writing a test to exercise it, implementing it in code, and debugging any issues that emerge. Through the use of stubs, developers can work on an individual function in isolation from the rest of the program being constructed while still receiving information about how their code executes. As developers join a new project, they complete a tutorial and can immediately begin making small contributions to the project.
We apply our behavior-driven approach to microtask programming to implementing a microservice. Large and complex web applications are often built as an interconnected network of smaller single-function microservices. Microservices offer a well-defined interface between a client (e.g., a team consuming a future microservice) and a crowd of developers, defined by the behavior of a set of HTTP endpoints handled by the microservice. Microservices offer a natural decomposition boundary in large web applications, offering a mechanism for a traditional software project to quickly crowdsource a module within a larger project. Rather than require an extended onboarding process for new developers to become familiar with a large software project, our approach enables a team to simply describe the desired functionality and it to be constructed separately by a crowd.
We instantiated our approach in a novel cloud IDE, CrowdMicroservices.1 CrowdMicroservices includes an editor for clients to describe requirements for the system as a set of endpoints, a web-based programming environment in which crowd workers can identify, test, implement, and debug behaviors, and an infrastructure for automatically generating and managing microtasks. Persistence is exposed through an external API, which supports a sandboxed environment for development. After completion, the microservice may be automatically deployed to a hosting site.
To evaluate our approach, we conducted a user study in which 9 crowd workers implemented a simple ToDo microservice. Our results offer initial evidence for the feasibility of the approach. Participants submitted their first microtask 24 min after beginning, successfully submitted 350 microtasks, implemented 13 functions and 36 tests, completed microtasks in a median time under 5 min, and correctly implemented 27 of 34 behaviors.
In this paper, we contribute
- 1.
a novel behavior-driven microtask programming which offers immediate feedback from syntax errors, unit tests, and debugging.
- 2.
CrowdMicroservices, the first programming preconfigured environment for implementing microservices through microtasks
- 3.
initial evidence that behavior-driven microtasks can be quickly completed by developers and used to successfully implement a microservice.
- 4.
evidence that the approach has low onboarding time and a high potential for parallelism.
In the rest of the paper, we review related work, present our approach, illustrate the approach with an example, and report on a user study evaluation. We conclude with a discussion of limitations and threats to validity as well as opportunities and future directions.
Section snippets
Related work
Our work builds on a broad body of work in crowdsourcing, particularly prior approaches to crowdsourcing software development. This work has investigated several key aspects of microtask workflow design, which we focus on below: decomposition and context, parallelism and conflicts, enabling fast onboarding, and achieving quality. A summary of these approaches is presented in Table 1.
CrowdMicroservices workflow
In this section, we describe a behavior-driven approach to microtask programming. The main goal of the workflow is to reduce onboarding time and increase the potential for parallelism. In behavior-driven development, developers first write a unit test for each behavior they will implement, offering a way to verify that their implementation works as intended by running a test. As a workflow for microtasking, behavior-driven development offers a number of potential advantages. As developers work,
Example
We illustrate using CrowdMicroservices to build a microservice through an example. A client first requests a microservice (Section 4.1), crowd workers implement behaviors, review contributions, and create new functions (Section 4.2), and the client may then deploy the microservice (Section 4.3).
Evaluation
To investigate the feasibility of applying behavior-driven microtask programming to implementing microservices, we conducted a user study in which a crowd of workers built a small microservice. Since there are no prior systems for implementing microservices through microtasks, we conducted a study evaluating the feasibility of the system rather than comparing it against existing approaches. A key goal of microtask programming is to enable short contributions by transient contributors.
Limitations and threats to validity
Our study had several limitations and potential threats to the internal and external validity of the results.
In our study, we chose to recruit a wide range of participants, recruiting participants locally from our university as well as globally through social networking sites. This yielded participants with a wide range of backgrounds, with their experience in JavaScript ranging from 2 months to 6 years. We chose this process as it mirrors the process of an open call, where contributors with a
Discussion
Microtask programming offers a software development process in which large crowds of transient workers build software through short and self-contained microtasks, reducing barriers to onboarding and increasing participation in open source projects. In this paper, we explored a novel workflow for organizing microtask work and offered the first approach capable of microtasking the creation of web microservices. In our behavior-driven microtask workflow, each microtask involves a worker
Conclusion
In this paper, we offer a novel behavior-driven approach for microtasking programming work and apply this approach to implementing microservices. Our results offer initial evidence for its feasibility. Workers were able to successfully complete a microtask which asked them to identify, test, implement, and debug a behavior in less than 5 min as well as onboard onto a project in less than 30 min. Together, these contributions were able to be aggregated into a functioning microservice that was
CRediT authorship contribution statement
Emad Aghayi: Conceptualization, Methodology, Software, Conducting study, Data curation, Visualizing, Investigation. Thomas D. LaToza: Supervision, Conceptualization, Methodology, Software, Conducting study, Data curation, Investigation. Paurav Surendra: Conceptualization, Methodology, Software, Investigation. Seyedmeysam Abolghasemi: Software.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Emad Aghayi is currently a Ph.D. student in the Department of Computer Science at George Mason University. He received degrees in information technology from University of Tehran and Shiraz University of Technology in 2014 and 2010 respectively . He is a member of the Developer Experience Design Lab, which studies how humans interact with code and designs new ways of building software. He works at the intersection of software engineering and human–computer interaction. His research goal is to
References (48)
- et al.
Intergroup and intragroup competition and cooperation
J. Exp. Soc. Psychol.
(1977) - et al.
A survey of the use of crowdsourcing in software engineering
J. Syst. Softw.
(2017) - et al.
A systematic literature review on the barriers faced by newcomers to open source software projects
Inf. Softw. Technol.
(2015) - et al.
Community, joining, and specialization in open source software innovation: a case study
Res. Policy
(2003) - Aghayi, E., 2020. Large-scale microtask programming. In: Symposium on Visual Languages and Human-Centric Computing. pp....
- et al.
A contingency approach to software project coordination
J. Manage. Inf. Syst.
(2002) Test-Driven Development: by Example
(2003)- Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K., 2010....
- Bounov, D., DeRossi, A., Menarini, M., Griswold, W.G., Lerner, S., 2018. Inferring loop invariants through...
- et al.
Puzzle-based automatic testing: Bringing humans into the loop by solving puzzles
Crowdsourcing systems on the world-wide web
Commun. ACM
Crowd microservices hackathon: Utilizing crowdsourcing for microtask programming on a large-scale
J. Stud.-Sci. Res.
Onboarding in open source projects
IEEE Softw.
Understanding free software developers: Findings from the FLOSS study
Perspect. Free Open Sour. Softw.
Software Development with Real-time Collaborative Editing
Analysis of the Impact of Test Based Development Techniques (TDD, BDD, and ATDD) to the Software Life Cycle
A multidimensional model of knowledge sharing behavior in mobile social networks
Kybernetes
Behavior driven development
Cited by (6)
Behaviour driven development: A systematic mapping study
2023, Journal of Systems and SoftwareAdaptive decoupling planning method for the product crowdsourcing design tasks based on knowledge reuse
2022, Expert Systems with ApplicationsAI-enchanced Crowdsourcing as an Element of Information Systems Development
2023, Lecture Notes in Networks and SystemsA controlled experiment on the impact of microtasking on programming
2023, Empirical Software EngineeringCan microtask programming work in industry?
2020, ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software EngineeringCan microtask programming work in industry?
2020, arXiv
Emad Aghayi is currently a Ph.D. student in the Department of Computer Science at George Mason University. He received degrees in information technology from University of Tehran and Shiraz University of Technology in 2014 and 2010 respectively . He is a member of the Developer Experience Design Lab, which studies how humans interact with code and designs new ways of building software. He works at the intersection of software engineering and human–computer interaction. His research goal is to make developers more productive by providing better development tools and approaches. To achieve this goal, his research involves understanding developer behaviors, identifying problems, and designing tools and approaches to help with those problems using various HCI methods at each stage.
Thomas D. LaToza received a Ph.D. in software engineering from Carnegie Mellon University in 2012 and degrees in psychology and computer science from the University of Illinois, Urbana–Champaign in 2004. He is currently an Assistant Professor of Computer Science at George Mason University. He serves as director of the Developer Experience Design Lab, which studies how humans interact with code and designs new ways of building software. He has conducted over 20 studies of software developers and designed numerous programming tools, including tools for designing, understanding, reusing, editing, and debugging code. His recent work has focused on programming environments which crowdsource insights. He served as co-chair of the Workshop on Crowdsourcing in Software Engineering and the Workshop on the Evaluation of Programming Languages and Tools. He is a recipient of the NSF CAREER award for his research and teaching on debugging mental models.
Paurav Surendra received a Masters degree in Software Engineering at George Mason University, Virginia and his Bachelors degree in Information Science and Engineering at Visvesvaraya Technological University in India. He is currently working as a Software Engineer at the Student Opportunity Center. His interests are in Software Architecture, Software Design, and Software Development for Web Platforms.
Seyedmeysam Abolghasemi received a Masters degree in Computer Science at Old Dominion University in 2017 and a Bachelors degree in Information Technology and Systems from Monash University in 2012. He is currently working as a Senior Software Developer at Old Dominion University. His area of interests are Software Architecture and High-Performance Computing.
- ☆
This research project was supported in part by the National Science Foundation under grants CCF-1414197 and CCF-1845508.
- ☆☆
CrowdMicroservices: Crowdsourced Behavior-Driven Development.