Designing a Clinical Education Tracking System: An Innovative Approach

Introduction to the Problem

One of the main challenges that clinical educators (i.e., clinical instructors, directors of clinical education, program evaluators) face in their job is tracking a students’ progress throughout their clinical experience. When a student rotates through their clinical assignments, there are objectives and competency that require verification, daily evaluation and assessment of their performance from their preceptors, and time management (attendance verification). Typically, a clinical educator reviews all students' clinical competency-collected data during each rotation for confirmation that the student can progress into the next clinical rotation and that all required clinical competency objectives for the course have been met. Usually, each program has their own method of documenting and tracking clinical data (i.e., attendance, competencies and evaluations). For example, some clinical educators use a paper-based tracking system in which they provide students with a file that has a list of all required objectives and competencies, and attendance tracking forms. Each of these forms must be completed daily, during each clinical rotation and/or by the end of the term or course. These paper-based forms have their own collection of additional paper-based forms that require further review. In addition, these forms also require a signature and documentation of progress and comments by their preceptors or clinical instructors. There are cost-based proprietary clinical electronic systems that provide clinical educators with a clinical tracking system specifically developed for select professions to manage the many different processes involved in clinical education. However, this comes with not only the additional costs tostudents and the program, but is lacking in configurable program design for specific requests adding to the overall cost of the program. Both cost-based proprietary clinical electronic systems and paper-based tracking system solutions have their own challenges and limitations.

Challenges with Proprietary Cost-Based Systems

Cost-based proprietary clinical electronic systems offer products tailored to specific health professions (i.e., physician assistant, rspiratory therapy, nursing, etc.). Examples of current system applications include, but are not limited to, daily evaluations, time management, procedures completed, tracking clinical attendance, clocking systems, feedback to clinical sites, student feedback, clinical faculty and program documentation, and clinical scheduling, etc. The challenge programs and clinical educators face with these available solutions is cost, both direct and indirect, and the ability to configure specific program needs. Indirect costs are in time, troubleshooting, downtime of the software (scheduled or unscheduled), learning the software, educating students and preceptors on the software. There is also the potential issue of clinical sites allowing students and clinical instructors to access the online program through their information technology (IT) security and firewall system. Most programs do not have the financial resources to afford the upfront and yearly upgrade and management costs in purchasing these systems or software. To cover such costs, programs may charge their students additional access fees in addition to tuition in order to cover the ongoing costs of the system. Another potential challenge is with the software vendors' customer service availability on weekends or holidays, and after normal business hours. The cost and availability of customer services or a help desk 24 hours per day, seven days per week to create new accounts; assist with password resets or login issues; and educate students, preceptors, and clinical faculty, due to day and night shifts, and the weekend nature of clinical rotations, as well as educating other end users on how to use the software and program. In short, costs and customer service availability can be considered the largest challenges with cost-based proprietary electronic systems.

Challenges with a Paper-Based Tracking System

Some programs, on the other hand, would prefer not to incur the addition costs to the program or students and utilize a paper-based tracking system. In this traditional system, students turn in their evaluations daily, or based on rotation days, and/or overall clinical evaluations at the end of the course or term. In addition, the signed evaluations, competencies, and time stamps, completed by the clinical instructor are then sent by hand or email to their director of clinical education or course instructor. The paper workflow now goes to the clinical educator to sort, document, maintain and store the paper forms for evaluation and program assessment. In addition, the documents from the students and preceptors need to be evaluated in writing for further documentation, resulting in large amounts of paper documents to evaluate and assess the students’ performance, provide a grade, as well as aid in accreditation assessment data and analysis. The student is then notified of their performance by email, in person, or via a learning management software system (such as, but not limited to, Canvas and Blackboard, etc.). This process also creates an increased faculty workload due to the added amount of faculty time and resources needed to accomplish these tasks. In short, additional time, increased workload and decreased efficiency can be considered some of the largest challenges in a traditional paper-based system.

The literature reports that celectronic tracking systems are superior to paper-based tracking systems.^1–3 Thus, the purpose of this perspective article is not to compare paper-based vs electronic tracking systems, but rather to demonstrate an innovative approach utilizing available resources, as well as a streamlined concept employing an electronic tracking system at a low to no costs alternative.

This innovative approach is currently in used in a respiratory care program to overcome the described challenges and limitations described above of a paper-based system, using a business intelligence (BI) framework and tools that are available in our institution and available at most institutions at little or no additional costs. This approach is primarily useful for programs and professions that do not have a standardized clinical tracking system required by their professional accrediting body, or the extra resources to purchase a clinical tracking system.

Frustration Drives Innovation

As described above, these challenges and limitations can cause a huge burden for all stakeholders involved in the clinical education process. In this section, an online model will be proposed to overcome these challenges and limitations in a simplified matter. This online model has worked well in our Respiratory Care program and can be implemented in other programs as well.

First, a simplified description will be presented to understand the data that are collected in clinics from students. The online clinical evaluation and assessment model in Figure 1 is from one of our respiratory care programs clinical education course sequence. Students are required to electronically document their attendance in clinics, complete a daily evaluation form, and a specific number of objectives every quarter, as well as having each clinical competency signed as completed electronically by the clinical instructor and preceptor providing the bedside education. Over the course of the clinical rotation(s), this evaluation and assessment interaction between the student, preceptor and clinical director is documented and completed, and made available online to the student, preceptor and clinical director in real time. Should the student or preceptor identify a clinical deficiency at any point in the process, the clinical director will further evaluate and assess the specific needs of the students’ progress and institute-appropriate remediation on a timely matter. The overall assessment of the students and clinical experiences are also collected and analyzed on an ongoing basis for program and institutional accreditation purposes.

Figure 1 (A) A sample time stamp question using an electronic online survey platform. Students have access to the link and select one of the options listed in the figure. Once submitted, data is immediately stored in an electronic database. (B) An example of the electronic survey form that represent a sample clinical competency check-off for the student. Once submitted, data is immediately stored electronically in a database.

To begin the process of creating an online clinical education tracking system, a web-based survey application is needed to collect and produce a digital “form” based on currently utilized paper forms that can be transferred and designed in any web-based survey application service. Most institutions have an educational institutional licenses to such web-based survey services such as, but not limited to, Google forms, Microsoft forms, Survey Monkey, Qualtrics, etc. The following sections will provide steps and examples how these forms, are created, designed, and utilized in our program.

Attendance – The Time Stamp

To create and design the time stamp, for tracking clinical attendance and hours is as follow: when a student clocks in using a digital system (online web-based survey/digital form), they are basically recording their arrival time in a database by entering the actual time they enter the information in clinic. This can be achieved by using any available web-based survey application website. The student would be entering the information via their mobile electronic device or using one of the clinical sites computer system if appropriate. To place this information into perspective, as a form converted to an online survey application, a student is basically answering a multiple-choice question, that is easily accomplished in an online survey application format. Figure 1A shows a sample TimeStamp question used in our clinical course using one of the online survey platforms:

The result or outcome in the above example would be an accurate time stamp or attendance of the students’ clinical time clock in, in and out for lunch, and clocking out at the end of shift. Some software applications allow the record of the Internet Protocol (IP) address that can further ensure the actual place the student entered the timestamp for attendance.

Clinical Competency Check-Off

Clinical competency check-offs were designed in a similar method as the time stamp. The paper competency form, along with the same clinical performance criteria, are used to design and create the online form. When a preceptor or a clinical educator is validating the students clinical performance based on a specific competency or procedure, they are basically answering a series of question(s) with a satisfactory/unsatisfactory option (yes or no) to create the online form. Figure 1B shows an example of the competency check-off. The results or outcome of this example would be a record as either satisfactory or unsatisfactory in a database under a student name. This data, as well as, all data collected can be easily exported into an excel sheet for further analysis, evaluation, and assessment. To put it all in perspective, Figure 2 shows a sample daily evaluation in both paper and electronic form that a student fills out at the end of their shift along with the preceptors verification and evaluation.

Figure 2 The paper form that students used to fill out on a daily basis and the same form converted into an online form with a link provided to the students ahead of time to access it via their clinical site computers. As noted, the evaluation categories are the same but converted online.

When comparing clinical and traditional classroom education assessment and evaluation, there are distinct and different processes to be considered. For example, there are many variables, information, and data that is essential and requires tracking to provide the clinical educator with the progress and evaluation of students, as well as outcome assessment of the clinical experience for program and accreditation purposes. The more data points that are available at the disposal of clinical educator, the possibilities of additional analysis and outcomes of students' performance and clinical competence are enhanced. Every time a student performs a task or a skill in clinics, data is entered, and is stored in a database that can be available for analysis and outcome reporting in a multitude of creative methods in a timely manner. Most, if not all, of the survey websites applications mentioned, and not mentioned offer reporting systems or features that provides data visualization tools of database outcomes.

A Simple Feedback System

A clinical tracking system, large or small, simple or complex, requires inputs, outputs, processes, and feedback that should be easy to access and navigate for all stakeholders. For example, inputs of any type come in as data, for example, satisfactory or unsatisfactory clinical competency or objective performance, that can then be processed to create desired data outputs in the form of reports and visualizations. There is the possibility that the ability to access and navigate the amount and data variables create a loss of information available to all stakeholders. Without a clear feedback loop of these variables, there can be a gap or loss of continuity of information to the students and reporting of collection data. The student may have deficiencies, learning challenges, behavioral issues that need immediate attention, or patterns of concern over time requiring educational interventions. For example, if a student is not able to complete a competency or objective due to a lack of skill or knowledge, feedback may come in the form of remediation, course correction, re-training, or providing additional information and resources of education to reach the desired outcome. To close this potential gap in input and output systems, a feedback loop, based on the outputs produced in the system are needed. The software and technology components currently available at most institutions, as mentioned above can be interchanged and integrated for improved efficiencies. Any survey platform for data collection (input), and data storage can be based in cloud storage, or stored locally in an electronic process. An electronic online tracking system reduces the time required to process inputs and develop outputs (reports). For example, computation of frequency count can be done much quicker in an electronic format rather than in a manual paper-based system for assessment and evaluation criteria. Therefore, a clinical educator with the aid of technology can be more efficient and accurate with their communication to students and providing program outcomes.

Business Intelligence in Clinical Education: An Innovative Approach

The term BI has revolutionized the business industry “by taking the industry to the next level,” by understanding customer needs, preferences, and patterns for improved projection analysis. For example, BI helps businesses identify potential customers and target the right products to the right consumer. BI has been found to improve efficiency and allowed companies and entrepreneurs to better understand their customers to better prepare for future trends. Data collection and analysis is at the heart of BI initiatives. BI involves data mining for trends, data visualization, machine learning, and artificial intelligence to create prediction models.⁴ An educator requires a repository of data and mechanisms to continuously collect and analyze data. The ease and availability of an educator to build their own database and use BI as a powerful visualizing tool is possible. In health care management, BI has become a vital resource to mine data, provide machine learning and artificial intelligence applications to improve efficiencies. Research has shown BI to be effective in other non-business-related applications, such as sleep medicine, internal medicine, and radiology, etc.^4–10 As part of BI, the concept of Key Performance Indicators (KPI) are one of the significant tools that can be used to understand customers (stakeholders, such as students, preceptors, clinical sites, accreditation agencies, etc.).^11,12 We believe that clinical education is an obvious application of this technology where data mining and predicting student’s success can be developed and achieved as well as using KPIs to understand its data. BI tools can bring together all the databases data on one platform for ease of use, and enhanced visualization for all of stakeholders (students, clinical preceptors, accreditation, hospital management, etc.) for the improvement of students' clinical progress and data collection and outcome reporting. BI benefits can be fully realized once a significant amount of historical data has been collected over time for program outcome assessment. Further, the development of algorithms, models, and predictive analytics can be an added benefit for any clinical program to identify and project whether a student is succeeding, failing, or needing assistance based on multiple variables.

Our Own Experience with BI

In our institution, faculty have access to select web-based survey application platforms at the enterprise level. In addition, the department and program did not have the resources to purchase, nor was the idea of adding additional student fees or increase in tuition an option to the program for a clinical tracking software system. Thus, it was decided to upgrade from a paper-based tracking system, to an electronic digital format as indicated above. Moreover, the program decided to add a BI piece where a live dashboard containing all clinical education data was viewable by the clinical faculty and the students in real time. BI dashboards include multiple visualizations on one screen, providing configurable key data or graphs side-by-side. The analogy of a vehicle’s dashboard will suffice as an example; you are able to view your gas level, speed, engine temperature, and many other key data for the driver in real time, and is similar to the dashboard provided to the program and clinical faculty. This dashboard was developed by gathering the data collected and displaying what is most important to the end user, our students, faculty, program. In a clinical tracking model, this could be the number, and name of procedures completed, the average performance and other analytics as well. This method and process can be implemented in any BI application software provided by educational institutions. Figure 3 shows the model that was used as a simple feedback system that bring all concepts together (input, process, output). In addition, Figure 4 illustrates a sample fictitious storyboard when putting all the data pieces together from a paper form system to a BI dashboard that is live and accessible to all clinical educators and students. Lastly, Figure 5 shows a step-by-step guide to assist clinical educators in converting an existing paper tracking system to an online electronic tracking system. One of the limitations that is encountered is when the system fail to sync the data and requires a manual refresh or intervention. Other possible limitations are some firewalls that hospitals and clinical sites, might place on external links to the surveys for students to complete their evaluation or check-off. Some clinical sites do not allow extranet access to all websites.

Figure 3 The model depicts the concept of a simple system with feedback and its application to the real-world case of a Clinical Education Tracking System.

Figure 4 A sample fictitious database showing how the data is being stored after being submitted electronically via an online form. As shown, data is stored in a database (Excel) that can also be downloaded and stored for the department's records (A). (B) represents a sample dashboard based on the entries represented in (A) using BI software designed for clinical faculty to track the whole class performance. It can be filtered by date, name, and clinical site.

Figure 5 A Guide to clinical educators on how to start a clinical tracking system for their program/department.

Conclusion

In conclusion, this innovative approach can save a department direct and indirect costs in, reduced faculty time, and provide a real-time feedback loop system between students and the faculty. Students will be able to view and track their own performance using their own dashboards, and the faculty are able to track and manage student progress more efficiently and accurately. Clinical educators should consider investigating and implementing KPIs that can be used in clinical education assessment and evaluation. It is our hope that this innovative experience and perspective will spark and ignite other clinical educators to find other innovative ways, if not better, using BI tools to take clinical education to the next level. Lastly, studies to validate this proposed perspective are needed as they will test and validate its function from the user perspective.