Ethical issues in empirical studies using student subjects: Re-visiting practices and perceptions

Abstract

Context

Using student subjects in empirical studies has been discussed extensively from a methodological perspective in Software Engineering (SE), but there is a lack of similar discussion surrounding ethical aspects of doing so. As students are in a subordinate relationship to their instructors, such a discussion is needed.

Objective

We aim to increase the understanding of practices and perceptions SE researchers have of ethical issues with student participation in empirical studies.

Method

We conducted a systematic mapping study of 372 empirical SE studies involving students, following up with a survey answered by 100 SE researchers regarding their current practices and opinions regarding student participation.

Results

The mapping study shows that the majority of studies does not report conditions regarding recruitment, voluntariness, compensation, and ethics approval. In contrast, the majority of survey participants supports reporting these conditions. The survey further reveals that less than half of the participants require ethics approval. Additionally, the majority of participants recruit their own students on a voluntary basis, and use informed consent with withdrawal options. There is disagreement among the participants whether course instructors should be involved in research studies and if they should know who participates in a study.

Conclusions

It is a positive sign that mandatory participation is rare, and that informed consent and withdrawal options are standard. However, we see immediate need for action, as study conditions are under-reported, and as opinions on ethical practices differ widely. In particular, there is little regard in SE on the power relationship between instructors and students.

Appendices

Appendix A: List of Included Venues

The list of publication venues is as follows (Table 4):

Table 4 Publication venues with acronyms

Appendix B: Mapping Study: Data Extraction

To extract the study conditions and key metrics from the primary studies, we followed the following process.

• 1. Check whether paper should actually be included. By default, papers are included in this step. Only exclude if one of the following points applies. Whenever a paper is excluded, note down the reason.

  • Is it at least 8 pages long? If not, exclude.

  • Is it published at one of the included venues (see Appendix A)? If not, exclude. Note that only the technical tracks at the conferences are included, no workshops or `special’ tracks (like SEET at ICSE). Only exclude based on the Publication Title field, or on the venue name printed in the PDF – no extra online search.

  • If there are no student subjects or no empirical study, exclude (this is also caught by the search later on).

• 2. Extract study information. First, do keyword search. If necessary (if any of the five categories below remain unanswered), read introduction, method (usually needed for the study type), and threats to validity.

  • Which study type is it? Use the same term as used in the paper (e.g., “Controlled experiment”, “Family of quasi-experiments”, “qualitative study”). If there is no empirical study conducted, exclude the paper.

  • How many students are participating in the study? If students and professional subjects participate, count only the students. If it cannot clearly be determined how many students participated, use `NA’ (as long as it’s clear that students indeed participated, otherwise exclude).

    Keyword search (one after the other, until you find the information): student, subject, graduate, master, bachelor, recruit, invit, participa

  • Was it voluntary to participate? Possible values: voluntary (also if it is mentioned that subjects were recruited openly, e.g., “via flyers”, “university-wide”, “via mailing lists”), part of a course (if it is only mentioned that participation was in the scope of a course, not whether it was voluntary or mandatory), mandatory, and NA.

    Keyword search: Same as for voluntariness. Additionally: withdraw, option, volunt, mandatory, compulsory

  • How were (student) subjects compensated? Use the same term as used in the paper. If it cannot clearly be determined, use `NA’.

    Keyword search: financ, money, monetary, reward, extra, bonus, compensat, voucher, receiv, gift, points

  • Was ethical approval obtained? Yes, No, Other, or NA. `Other’ if it is described that mandatory procedures (e.g., from university, state, or country) were followed, but it is clear that this is not an ethical approval.

    Keyword search: ethic, irb, board, approv, consent

Appendix C: Questionnaire

The online questionnaire consisted of the following questions. The entire questionnaire with introduction and final page, and the precise layout is found in the online dataset (Liebel 2021).

• 1. In which country are you employed?

  If several, state the country in which you spend most of your work time.

  (Free-text with suggestions based on standard country list)

• 2. What is your highest degree?

  (Higher education entrance qualification, Bachelor degree (or equivalent), Master degree (or equivalent), PhD degree (or equivalent), Habilitation degree (or equivalent), Other (free text))

• 3. What is your current, or last held, academic rank?

  (For example, doctoral student, associate professor)

  (Free text)

• 4. Have you previously conducted an empirical study which involved student subjects?

  Note: We understand “subject” broadly in this survey. That is, studies such as a case study involving students would also be included.

  (Yes/No)

New Page

• 5. Which of the following statements regarding ethics approval processes applies to your work?

  (Single selection with free-text option)

  • For studies with student subjects, I am required to obtain ethical approval

  • I am not required to obtain ethical approval, but there are mandatory steps for human-subject studies from my employer/the state (clarify details, if any)

  • Neither of the above (clarify details, if any)

• 6. How many studies with student subjects have you performed in the last 5 years?

  (> 10 studies, 6-10 studies, 1-5 studies, None)

• 7. In your most recent research study with students, did you recruit your own students?

  We refer to “own” students as students that are either supervised (thesis project, PhD supervision) by the respondent, or take a course given by the respondent.

  (Yes/No)

• 8. In your most recent research study with students, was their participation mandatory or voluntary?

  (Voluntary, Voluntary, but part of a graded course component (e.g., assignment), Mandatory)

• 9. In your most recent research study with students, what forms of compensation did you offer to student subjects?

  Please mark all options that apply.

  (None (besides potential learning experience), Bonus points in course, Monetary reward (e.g., fixed rate, a raffle/lottery), Snacks/Food, Other (please specify))

• 10 In your most recent research study with students, did the subjects in your study/studies give consent to participate?

  (Yes/No)

• 11 In your most recent research study with students, did the subjects in your study/studies have the option to withdraw from the study at any time?

  (Yes/No)

• 12 Based on your answers above, would you like to clarify any details?

  (Free text)

New Page

• 13 Please state your agreement to the following statements regarding educational value of research studies.

  (Per item: 5-point Likert agreement scale and “don’t know” option)

  • If course content is prescribed by a curriculum, it should not be changed just to make a research study fit in the course.

  • Research studies should be connected to course projects.

  • Reviewers (of a study design/protocol) should pay close attention to students receiving adequate educational value from the research study.

• 14 Please state your agreement to the following statements regarding consent.

  (Per item: 5-point Likert agreement scale and “don’t know” option)

  • Every research study involving student subjects should be based on informed consent.

  • Student subjects should be permitted to withdraw from a research study at any time.

  • Participation in a research study may be mandatory if it fits into the course context.

  • Participation in a research study conducted in a course should always be voluntary for students.

• 15 Please state your agreement to the following statements regarding the course-study relationship.

  The following statements all apply to a situation in which a course instructor conducts a research study within his/her course.

  (Per item: 5-point Likert agreement scale and “don’t know” option)

  From an ethical standpoint, it is acceptable...

  • to use the enrolled students as subjects in a research study.

  • to base a part of the assessment on the participation in a research study.

  • to base a part of the assessment on the performance in a research study.

  • to encourage students to participate in a research study.

  • to withhold information from the students with respect to the study goals.

  • to withhold information from the students as to how the data they provide will be used.

  • for the instructor to know who participated in the research study.

  • that the researcher conducting the research study is the same person as the course instructor.

• 16 According to your opinion, how relevant is it to include the following information in a publication that uses student subjects?

  (Per item: 5-point Likert scale from “very irrelevant” to “very relevant”, and “don’t know” option)

  • Status of ethical approval or similar measures.

  • Voluntariness of student participation.

  • Compensation of student subjects.

  • Use of informed consent.

• 17 Do you have additional comments?

  (Free text)