Background Smartphone to-do list app was hypothesized to be more efficient than a paper-based list in the management of workflow and to provide additional benefits. Purpose To analyze the impact of a mobile task-management application on the workflow of an ICU medical staff. Methods Superiority by a margin test, quasi-experimental study comparing the use of a smartphone application versus standard practice regarding tasks management in an academic ICU. Superiority margin was set at 8% based on a pilot study. During two periods of 20 working days each (October 2018 and January 2019), medical staff managed tasks with both methods on a weekly rotation basis. Primary outcome was the proportion of daily tasks completed. Secondary outcomes assessed users’ satisfaction and the impact of the app in terms of changes in clinical practice. Results 25 ICU physicians were enrolled. A total of 1983 tasks were recorded. The proportion of completed tasks per day was higher when using the smartphone app (99% [96-100] versus 95% [93-98] for the standard group, p = 0.006), but did not reach the superiority margin. Smartphone application was perceived as positive experience, as participants felt that they forgot fewer tasks (p = 0.02), were more aware of their progress on ongoing or remaining tasks (p = 0.03) and observed an improvement in communication among the medical staff (p = 0.03). Conclusion This study failed to demonstrate the superiority of a smartphone app over paper-based lists regarding the proportion of daily tasks completed. However, positive feedback regarding the application was received from the medical staff.

Background and objective This study proposes an approach to evaluation and measuring of presence for man-machine interaction in the virtual reality based on electroencephalographic data. Materials and methods It analyzes stable electroencephalographic patterns that allow us to trace a connection between a brain activity and purposeful actions of an individual in various environments. The subjects of the study were experienced downhill skiers equipped with electroencephalographs, who performed real-life skiing on a downhill course, after which they were offered a virtual simulation of downhill skiing using an HTCVive headset and a programmed 2D or desktop simulator. Results The results of measurement showed neuropatterns similar in the cases of virtual reality simulation and physical downhill skiing (in part of changes in space and power parameters of electroencephalograms in the different frequency ranges) and different from a 2D simulator. This observation enables us to make an assumption of realism of a virtual reality simulator in the context of reproduction of the subjects' similar cognitive and semantic connections and motor programs. Discussion Further research work will focus on evaluation of efficiency in performing psychophysiological tests (time response to a mobile object) in the virtual reality and 2D desktop application.

Objective Despite showing comparable levels of efficacy, internet-based psychological interventions (IPI) exhibited lower acceptance and intention of use as compared to psychological treatment delivered by face-to-face methods. Surprisingly, no research has inspected whether IPI acceptance is associated with variables linked with intentions of technology use and with barriers to seeking professional psychological help, such as personal depression stigma. Informed by the Unified Theory of Acceptance and Use of Technology, the current study tested the role of technology and mental health-related determinants as predictors of acceptance of IPI for depression. Methods Participants were 417 community Portuguese adults, who completed a pencil-and-paper survey. Results Our results indicated that performance expectancy, social influence, and personal stigma against depression were significantly associated with the acceptance of IPI for depression. Conclusions These results suggest that barriers to seeking professional psychological help should be considered in the understanding of IPI acceptance.

Purpose The purpose of our study was to predict blood pressure variability from time-series data of blood pressure measured at home and data obtained through medical examination at a hospital. Previous studies have reported the blood pressure variability is a significant independent risk factor for cardiovascular disease. Methods We adopted a standard deviation for a certain period and predicted the variability and mean values of blood pressure for 4 weeks using multi-input multi-output deep neural networks. In designing the prediction model, we prepared a dataset from a clinical study. The dataset included past time-series data for blood pressure and medical examination data such as gender, age, and others. As evaluation metrics, we used the standard deviation ratio (SR) and the root mean square error (RMSE). Moreover, we used cross-validation as the evaluation method. Results The prediction performance of blood pressure variability and mean value after 1 to 4weeks showed a standard deviation ratio as variability metrics of “0.69” to “0.70”, an RMSE of “5.04” to “6.65” mmHg, respectively. Additionally, our models were able to work for a participant with high variability in blood pressure values due to its multi-output nature. Conclusion The results of this study show that our models can predict blood pressure over 4 weeks. Our models work for an individual with high variability of blood pressure. Therefore, we consider that our prediction models are valuable for blood pressure management.

Background Non-adherence to tuberculosis (TB) treatment jeopardizes the individual’s health and contributes to disease transmission and drug resistance. New patient-centered strategies are needed to improve TB treatment outcomes. Purpose To convert and expand a texting-based intervention into a mobile optimized application (app), evaluate the feasibility of an added self-administered paper-based drug metabolite test, and identify needs and preferences to inform their iterative design. Methods Qualitative methods using focus groups and field testing with patients in active TB treatment were used to gather initial input on the converted intervention design, content and issues using at home test strips to report medication adherence. Seven participants were recruited from an outpatient clinic within a regional public reference hospital specialized in respiratory diseases in Argentina. Thematic analyses were conducted on the transcripts and session notes. Results Participants considered interactive communication, access to answers to frequently asked questions, and tracking of progress in treatment as important. Participants reported having many questions and uncertainties at initiation of treatment and emphasized a need for reliable information, assurance and support from both providers and peers. Other suggestions included streamlining the graphical user interface for easier and shorter data entry times and usability. Conclusions Overall feedback from the participants regarding the intervention was positive, reporting that it was useful and relevant, and they were eager to contribute their ideas for improvement and additional functionality. Valuable feedback to improve functionality and meet the needs of end-users were obtained to inform the generation of new design ideas for refinement and testing in a pilot study.

Background Clinical pathways (CPs) increase in popularity and are known to lead to several benefits in the hospital environment. Clinical pathways can be either paper-based or software-based. It is known that paper-based CPs can result in more paperwork instead of simplifying daily routines of healthcare workers. Insufficient research has been done on the acceptance of software-based CPs by different user groups. Our aim in this study was to assess the effectiveness of the software-based CPs (CPS) from the perspective of healthcare professionals in the hospital environment as well as to investigate the differences in perceived effectiveness between user groups. Methods Using surveys and interviews, data were collected in four departments of an academic medical center. A distinction was made between decision makers (DM) and executive staff (ES). The surveys contained questions based on the Technology Acceptance Model and four objectives of the software defined by the hospital. Statistical tests were used to investigate the effectiveness of CPS and study the differences between DM and ES. Interviews were recorded and transcribed based on grounded theory principals. Results After implementation, monitoring protocol-based working was significantly improved (p = .026) and significantly higher efficiency on the work floor was reported (p = .046). ES perceived the software as less useful than expected (Md = 3.25 vs. Md = 2.75, p = .028) compared to DM and were less convinced of its ability to improve monitoring protocol-based working. The most important benefits of CPS as perceived by its users are the better overview of tasks it provides and facilitating documentation. Negative aspects mentioned were the lack of usability and the inflexibility of the software, and particularly ES claimed that the software did not increase their effectiveness. Conclusion Our study showed that CPS is effective from healthcare professionals’ perspective due to its ability to increase monitoring of protocol-based working and by enhancing the efficiency on the work floor. However, the users also acknowledge that the software lacks usability and is not flexible enough, which results in an additional workload. Policy makers should be more focused on informing and training executive staff more thoroughly when implementing a CPS. Our results strongly suggest that executive staff members need to be convinced of its usefulness and the added value a CPS provides. Preferably, they should be involved in the design phase of the software.

Background Systems-centered root cause analysis (RCA) of patient safety events presents unique advantages as it aims to disclose vulnerabilities of healthcare systems. However, the increasing number of collected events poses the problems of low efficiency and information overload for traditional RCA. Objectives This study aims to improve systems-centered RCA by developing optimized information extraction and presentation. Methods We experimented supervised machine-learning methods to extract safety-related information from 3333 de-identified patient safety event reports from two independent sources. Based on the extracted information, we further evaluated how optimized information presentation could help facilitate the disclosure of system vulnerabilities in traditional RCA. Results Multilabel text classification is effective in identifying safety-related information from the narrative description of patient safety events. The Pruned Sets in conjunction with Naïve Bayes are the outperformed algorithm in one dataset, with an overall F score of 60.0% and the highest F score of 96.0% for identifying “Adverse Drug Reaction”. The Classifier Chains in conjunction with Naïve Bayes are the outperformed algorithm in another dataset, with an overall F score of 43.2% and the highest F score of 64.0% for identifying “Medication”. During the RCA, human experts applied the optimized presentation of information which showed advantages of identifying system vulnerabilities. Conclusion Our study demonstrated the feasibility of using multilabel text classification for identifying safety-related information from the narrative description of patient safety events. The extracted information when grouped by safety-related information can better aid human experts to conduct systems-centered RCA and disclose system vulnerabilities.

Background Understanding functional and non-functional requirements is essential to successfully implement electronic medical record (EMR) systems. Actual requirements will be different for different contexts. Objective To elicit and prioritize requirements for implementing EMRs in oncology in low and middle income countries (LMICs), and to relate these to requirements from high-income countries. Participants and setting Cancer care stakeholders including oncologists, general doctors, nurses, biostatisticians, information technologists, from different LMICs, were involved. Methods Concept mapping was used. Statements of requirements were obtained during focus group discussions (FGDs) and interviews. Using surveys, the requirements were clustered and ranked on importance and feasibility. Data were analyzed in SPSS using agglomerative hierarchical clustering and multidimensional scaling, to create cluster maps and go-zone maps reflecting the relationships between the requirements and their prioritization. Results Four FGD sessions, with twenty participants, were conducted. In addition, six participants were interviewed. Twenty-two participants clustered the requirements and sixty-three participants ranked them on importance and feasibility. One hundred and sixty requirement statements were generated which were reduced to sixty-four after de-duplication and merging. Nine clusters were obtained encompassing the following domains, in order of importance: Security, Conducive organization, Management/Governance, General EMR functionalities, Computer infrastructure, Data management, Usability, Oncology decision support, and Ancillary requirements. On ranking, the requirements scored between 3.74 and 4.80 on importance, and between 3.55 and 4.46 on feasibility, on a 5-point Likert scale. We generated concept maps for use when communicating with stakeholders. Conclusion For oncology EMRs in LMICs, requirements overlap those from high-income countries, but generic EMR functionalities, Infrastructural and organizational requirements are still considered priority in LMICs compared to oncology-specific requirements or advanced EMR features e.g. computerized decision support or interoperability. Concept mapping is a fast and cost-effective method for eliciting and prioritizing EMR requirements in a user-centered manner.

Background Risk visualizations are often employed to support risk communication. However, their effectiveness in communication of single absolute risks remains unclear. We investigated the effectiveness of risk visualizations in conveying verbatim knowledge of single absolute risks among the general population. Methods Randomly sampled members of the general Dutch population completed four basic risk conversions from percentages to natural frequencies and vice versa. By random investigator-blinded allocation, these conversions were supported by either icon arrays, pie charts, bar graphs or no visualization. Verbatim risk knowledge was scored as the number of conversions completed correctly. Results 393 subjects were included. Overall, 60% of respondents answered all four questions correctly. Risk format (percentages vs. natural frequencies, p = 0.677) and risk magnitude (p = 0.532) were not associated with verbatim risk knowledge score. Younger age (p = 0.001) and higher education level (p < 0.001) were independently associated with higher scores. The use of risk visualizations was not associated with higher scores (OR = 1.08; 95% confidence interval: 0.69-1.69; p = 0.745). All three forms of risk visualization were equally ineffective. These findings held when stratifying by risk format, risk magnitude and user preference for a certain form of risk visualization. There were no significant interactions with age or education level. Conclusion Risk visualizations did not improve conveyance of verbatim knowledge of single absolute risks, irrespective of age, education level, risk magnitude, risk format and form of risk visualization. Risk visualizations may therefore be less suitable for settings in which detailed conveyance of single absolute risks is the main objective, although their effect on user experience and perception of risk communication and subsequent patient activation and participation remains to be elucidated.

Background and objective To exchange patient health information using Health Information Exchange (HIE) projects, such information first should be collected thoroughly using an appropriate data entry interface that reinforces information quality (IQ). Assessment of the given data interface based on its structure level may give us a better understanding of patients’ attitudes toward information-sharing efforts. The main objective of this study is to examine the effects of data structure on perceptions and attitudes of patients toward the quality of health information that may be shared through HIE networks. Materials and Methods Eight experiments were conducted to examine the impact of different design of information collection interfaces (structured vs. unstructured) to record two types of health information (sensitive vs. non-sensitive) that can be used for two types of sharing purposes (health care vs. marketing). Results Results show that the degree of data entry structure can significantly influence patients’ perceptions of completeness, accuracy, psychological risk, accessibility of data, concise representation, and understandability of health information. Discussion There is a connection between data entry interface design and patients’ perceptions of the quality of health information used in HIE networks, which in turn, could lead to the development of best practices in interface design and data collection techniques. This may also improve interactions between patients and healthcare entities, enhance patients’ attitudes toward data collection procedures and HIE, and help healthcare providers use complete and accurate databases. Conclusions We propose that healthcare professionals can tailor data entry interfaces based on the sensitivity of medical data and the purpose of information exchange.

Objective To simulate the clinical reasoning of doctors, retrieve analogous patients of an index patient automatically and predict diagnoses by the similar/dissimilar patients. Methods We proposed a novel patient-similarity-based framework for diagnostic prediction, which is inspired by the structure-mapping theory about analogy reasoning in psychology. Patient similarity is defined as the similarity between two patients’ diagnoses sets rather than a dichotomous (absence/presence of just one disease). The multilabel classification problem is converted to a single-value regression problem by integrating the pairwise patients’ clinical features into a vector and taking the vector as the input and the patient similarity as the output. In contrast to the common k-NN method which only considering the nearest neighbors, we not only utilize similar patients (positive analogy) to generate diagnostic hypotheses, but also utilize dissimilar patients (negative analogy) are used to reject diagnostic hypotheses. Results The patient-similarity-based models perform better than the one-vs-all baseline and traditional k-NN methods. The f-1 score of positive-analogy-based prediction is 0.698, significantly higher than the scores of baselines ranging from 0.368 to 0.661. It increases to 0.703 when the negative analogy method is applied to modify the prediction results of positive analogy. The performance of this method is highly promising for larger datasets. Conclusion The patient-similarity-based model provides diagnostic decision support that is more accurate, generalizable, and interpretable than those of previous methods and is based on heterogeneous and incomplete data. The model also serves as a new application for the use of clinical big data through artificial intelligence technology.

Background To explore attitudes about artificial intelligence (AI) among staff who utilized AI-based clinical decision support (CDS). Methods A survey was designed to assess staff attitudes about AI-based CDS tools. The survey was anonymously and voluntarily completed by clinical staff in three primary care outpatient clinics before and after implementation of an AI-based CDS system aimed to improve glycemic control in patients with diabetes as part of a quality improvement project. The CDS identified patients at risk for poor glycemic control and generated intervention recommendations intended to reduce patients’ risk. Results Staff completed 45 surveys pre-intervention and 38 post-intervention. Following implementation, staff felt that care was better coordinated (11 favorable responses, 14 unfavorable responses pre-intervention; 21 favorable responses, 3 unfavorable responses post-intervention; p < 0.01). However, only 14% of users would recommend the AI-based CDS. Staff feedback revealed that the most favorable aspect of the CDS was that it promoted team dialog about patient needs (N = 14, 52%), and the least favorable aspect was inadequacy of the interventions recommended by the CDS. Conclusions AI-based CDS tools that are perceived negatively by staff may reduce staff excitement about AI technology, and hands-on experience with AI may lead to more realistic expectations about the technology’s capabilities. In our setting, although AI-based CDS prompted an interdisciplinary discussion about the needs of patients at high risk for poor glycemic control, the interventions recommended by the CDS were often perceived to be poorly tailored, inappropriate, or not useful. Developers should carefully consider tasks that are best performed by AI and those best performed by the patient’s care team.

Background Several benefits and problems of electronic prescribing (ePrescribing) are described in scientific literature, though problems remain in the implementation. In this study, we evaluated the pharmacist’s perception of the ePrescription implementation within the community pharmacy software in Belgium, and the frequency and hindrance of encountered problems. Material and methods A cross-sectional study was conducted among community pharmacists in Belgium to measure satisfaction with the ePrescribing implementation and factors influencing this satisfaction. Results In total 246 pharmacists (3.3% response rate) rated the implementation in their software with an average score of 6.46 ± 2.16 (SD) on a scale of 10. In Belgium, French-speaking pharmacists gave a significantly higher satisfaction score compared to Dutch-speaking pharmacists (P = 0.032), whereas Dutch-speaking pharmacists perceived to process significantly more ePrescriptions compared to French-speaking pharmacists (P < 0.001). Satisfaction with the implementation of the ePrescription was significantly associated with the software package (P < 0.001), the knowledge of the ePrescribing workflow (P = 0.036), the frequency of slow responses of the software (P < 0.001) and the perception of unavailability of the system (P = 0.003). Conclusions The Belgian pharmacist was moderately satisfied with the implementation of the ePrescription. Problems with the availability of Belgian eHealth systems and interoperability issues with national codes used between prescriber and dispenser have to be resolved in the future in order to meet the Belgian community pharmacist’s needs.

Background The proper estimate of the risk of recurrences in early-stage oral tongue squamous cell carcinoma (OTSCC) is mandatory for individual treatment-decision making. However, this remains a challenge even for experienced multidisciplinary centers. Objectives We compared the performance of four machine learning (ML) algorithms for predicting the risk of locoregional recurrences in patients with OTSCC. These algorithms were Support Vector Machine (SVM), Naive Bayes (NB), Boosted Decision Tree (BDT), and Decision Forest (DF). Materials and methods The study cohort comprised 311 cases from the five University Hospitals in Finland and A.C. Camargo Cancer Center, São Paulo, Brazil. For comparison of the algorithms, we used the harmonic mean of precision and recall called F1 score, specificity, and accuracy values. These algorithms and their corresponding permutation feature importance (PFI) with the input parameters were externally tested on 59 new cases. Furthermore, we compared the performance of the algorithm that showed the highest prediction accuracy with the prognostic significance of depth of invasion (DOI). Results The results showed that the average specificity of all the algorithms was 71% . The SVM showed an accuracy of 68% and F1 score of 0.63, NB an accuracy of 70% and F1 score of 0.64, BDT an accuracy of 81% and F1 score of 0.78, and DF an accuracy of 78% and F1 score of 0.70. Additionally, these algorithms outperformed the DOI-based approach, which gave an accuracy of 63%. With PFI-analysis, there was no significant difference in the overall accuracies of three of the algorithms; PFI-BDT accuracy increased to 83.1%, PFI-DF increased to 80%, PFI-SVM decreased to 64.4%, while PFI-NB accuracy increased significantly to 81.4%. Conclusions Our findings show that the best classification accuracy was achieved with the boosted decision tree algorithm. Additionally, these algorithms outperformed the DOI-based approach. Furthermore, with few parameters identified in the PFI analysis, ML technique still showed the ability to predict locoregional recurrence. The application of boosted decision tree machine learning algorithm can stratify OTSCC patients and thus aid in their individual treatment planning.

Importance Anticoagulants are high-risk medications with the potential to cause significant patient harm or death. Digital transformation is occurring in hospital practice and it is essential to implement effective, evidence-based strategies for these medications in an electronic medical record (EMR). Objective To systematically appraise the literature to determine which EMR interventions have improved the safety and quality of therapeutic anticoagulation in an inpatient hospital setting. Methods PubMed, Embase, CINAHL, and the International Pharmaceutical Database were searched for suitable publications. Articles that met eligibility criteria up to September 2018 were included. The review was registered with PROSPERO (CRD42018104899). The web-based software platform Covidence® was used for screening and data extraction. Studies were grouped according to the type of intervention and the outcomes measured. Where relevant, a bias assessment was performed. Results We found 2,624 candidate articles and 27 met inclusion criteria. They included 3 randomised controlled trials, 4 cohort studies and 20 pre/post observational studies. There were four major interventions; computerised physician order entry (CPOE) (n = 4 studies), clinical decision support system (CDSS) methods (n = 21), dashboard utilisation (n = 1) and EMR implementation in general (n = 1). Seven outcomes were used to summarise the study results. Most research focused on prescribing or documentation compliance (n = 18). The remaining study outcome measures were: medication errors (n = 9), adverse drug events (n = 5), patient outcomes (morbidity/mortality/length of hospital stay/re-hospitalisation) (n = 5), quality use of anticoagulant (n = 4), end-user acceptance (n = 4), cost effectiveness (n = 1). Conclusion Despite the research cited, limited benefits have been demonstrated to date. It appears healthcare organisations are yet to determine optimal, evidence-based-methods to improve EMR utilisation. Further evaluation, collaboration and work are necessary to measure and leverage the potential benefits of digital health systems. Most research evaluating therapeutic anticoagulation management within an EMR focused on prescribing or documentation compliance, with less focus on clinical impact to the patient or cost effectiveness. Evidence suggests that CPOE in conjunction with CDSS is needed to effectively manage therapeutic anticoagulation. Targets for robust research include the integration of ‘stealth’ alerts, nomograms into digital systems and the use of dashboards within clinical practice.

Introduction A dataset with patient information allows a comparison between different clinical treatments in many fields of medicine as well as the effective use of medical resources. Patient-reported outcome measures (PROMs) collect data directly from patients for use in clinical practice by helping decision making and tailoring treatments according to the patients’ needs. The authors present a novel database to overcome data collection related barriers, calculating automatically the questionnaire results, displayed in graphics on the patients’ dashboard in real time, and focused on the active follow of the patients. Objective To present electronic health record database for monitoring clinical or surgical interventions and assess the usability. Methodology Process modeling and specification of system requirements were performed using the Iconix methodology along with the Post-Study System Usability Questionnaire (PSSUQ) to validate the usability and usefulness of the proposed system. The system and the questionnaires were performed in three languages: Brazilian Portuguese, Spanish, and English. Results The database enables researchers to use the questionnaires defining the time of data collection according to the needs of each clinical study. The system facilitates the patient answers without any personal interference from smartphones, tablets or computers. The questionnaire scores were calculated automatically in real time and displayed in graphics on the patients’ dashboard. The evaluation of the usability and usefulness of the developed database used 8 people divided into two equal groups (4 physicians and 4 medical students). Conclusion The proposed electronic health record database allows a friendly and flexible use of PROMs according to the population, needs in practice and clinical settings. The platform promotes active and direct data collection from patients and physicians in English, Portuguese and Spanish.

Background Mental disorders are a major public health problem leading to premature mortality, homelessness, addiction problems, poor physical health, and suicide. The prevalence of mental disorders in Arab countries, is high. The proliferation and ubiquity of smartphones and their apps in the Arab world may be the long-awaited for digital therapeutic for mental health disorders. However, the evidence about the availability and characteristics of mental health apps available to Arabic speakers remains poor. Objective To conduct a systematic assessment of the features of depression and anxiety mobile apps available for Arabic speakers. Methods A critical review of all the currently available depression and anxiety apps, available free of charge to Arabic speaker. The apps are evaluated using the Mobile App Rating Scale (MARS). Further, a categorization of apps’ main functions, inspired by the mhGAP guidelines, is developed to classify the apps based on their main functions. Results A total of 23 apps are identified with far more apps available on the Google Play Store (n = 21) versus only two apps on the iOS App Store. The majority of the apps (n = 16) provide general information about either anxiety, depression, or both. Six apps are of spiritual nature mainly referring to the Islamic faith and the Holy Quran, with one app referring to the Christian faith. Another five apps provide advice on alternative treatments, mainly concerning herbal medicine recipes. Only two apps provided utilities for users, specifically about medication reminders. Conclusions Mental health digi-threaputics have huge potentials to transform mental health care delivery. However, more empirical studies are needed to assure their quality and efficacy. The results of this study clearly highlight the current gaps to address the needs of Arabic speakers; only 23 apps were identified in this study, mostly with low app quality scores. There is a need to involve expert healthcare professionals in the development of mental health apps and for healthcare providers to empower patients through discussing apps that are useful and discern them from those that can potentially cause harm.

Background The Clinical Practice Research Datalink (CPRD) GOLD is an extremely influential U.K. primary care dataset for epidemiological research having a number of published papers based on its data much bigger than any other U.K. primary care dataset. The Office for National Statistics (ONS) death data for England can be linked to GOLD at the patient level and are considered the gold standard on mortality. GOLD, which also holds death data, has been recently assessed against ONS linked dataset and the accuracy of its dates of death has been deemed sufficient for the majority of observational studies. However, there is a lack of guidance on how to manage the challenges existing when ONS mortality and GOLD datasets are linked, including linkage coverage period, linkage correctness likelihood, linkage regional limitations and data discrepancy. Objectives Provide reconciling guidelines on how to make maximum and at the same time trustworthy use of mortality information coming from both GOLD and ONS linked datasets with the aim of improving the quality, reproducibility, transparency and comparison of clinical research. Method and results We have developed recommendations on how to manage mortality data coming from both GOLD and linked ONS, taking into account linkage coverage period, linkage correctness likelihood, linkage regional limitations and data discrepancies between these two datasets. We have also implemented these guidelines in an SQL algorithm for researchers to use. Conclusion We have provided detailed guidelines on the reconciliation of mortality data between GOLD and ONS linked death datasets, taking into account both their strengths and limitations. The consistent application of these guidelines made practical by an SQL algorithm, has the potential to improve clinical research quality, reproducibility, transparency and comparison.

Objective The objective of this study was to quantify both the competitiveness of the EHR vendor market in the United States of America (US) and the degree of fragmentation of individual Medicare beneficiaries’ medical records across the differing EHR Vendors found in the US healthcare system. Methods and Materials We determined the Part A and Part B Medicare-expenditure weighted market shares of EHR vendors and estimated the rate of attestation of meaningful use (MU) for EHRs among Medicare Part A & B providers from 2011 to 2016. Based on these data we calculated the annual Herfindahl-Hirschman Index to quantify the competitiveness of the EHR market as well as the number of vendors individual Medicare beneficiaries’ medical records were stored in for the period 2014 to 2016. Results We find that as of 2016 the EHR vendor environment was competitive but trending towards becoming highly concentrated soon. We also found that patient medical records were highly fragmented as only 4.5% of expenditure-weighted individual Medicare beneficiaries had their MU medical records associated with a single vendor, while 19.8% of expenditure-weighted beneficiaries had their MU medical records stored in 8 or more vendors. Discussion These results indicate that there are tradeoffs between EHR market competition, and the challenges associated with achieving interoperability across numerous competing vendors. Conclusion Uncertainty of interoperability among different EHR vendors may make transmission of medical records among different providers challenging, mitigating the benefit of vendor competition. This highlights the critical importance of current interoperability efforts moving forward.

PURPOSE Information Technology (IT) plays a critical role in supporting emergency physicians' (EPs) routines. Pagers, personal computers, and smartphones offer fast access to patient data, such as laboratory results. However, due to the inherent features of specimen processing and laboratory instruments, the turnaround time from test ordering to availability of results can be long. Lack of follow-up of abnormal results can lead to missed information that could impact patient care and safety. Despite the increasing use of ubiquitous technologies, a third of physicians remains devoid of reliable methods for ensuring that results have been received. In this feasibility study, we report the potential of using a smartwatch to deliver laboratory results to EPs at the point-of-care and to support efficiency in emergency care. Unlike mobile devices that are increasingly used by EPs, smartwatches are always accessible, even during hands-on procedures. METHOD Two EPs and four experts in human-computer interaction designed the smartwatch application following the Design Science Research Methodology (DSRM). The application was then evaluated in a pediatric emergency department through semi-simulated scenarios by eleven EPs. The primary outcome was to measure both the app perceived usability and satisfaction scores by the aim of the System Usability Scale (SUS), and the perceived usefulness and intention of its use by the aim of the Unified Theory of Acceptance and Use of Technology (UTAUT) scale. Secondary outcomes were to assess the application's efficiency by measuring the delay between the reception of the notification and 1) the access to its details and 2) the visit to the patient. Finally, open questions about the positive and negative aspects of the prototype as well as potential improvements were asked and evaluated qualitatively. RESULTS The prototype obtained a score of 81.4 out of 100 (good) on the SUS and a score of 5.96 out of 7 on the UTAUT scale. EPs using the smartwatch visited patients within 30 seconds receiving the laboratory results. CONCLUSIONS This study demonstrates the capacity of smartwatches to speed up the point-of-care delivery of laboratory results in the ED.

BACKGROUND Internet-based interventions can help empower caregivers of people with chronic diseases and can develop solutions to decrease the physical and psychological consequences resulting from caregiving. OBJECTIVE Analysing the effectiveness of health web-based and/or mobile app-based interventions with regard to the level of well-being and quality of life of informal caregivers in charge of people with chronic diseases. MATERIALS AND METHODS Systematic review of the following databases: Pubmed, Apa PsycINFO, ProQuest Health & Medical Complete and Scopus. Quality standards established by PRISMA and Joanna Briggs Institute Systematic Review Approach have been followed. The two phases of the selection process were carried out independently and a cross-case comparative analysis by three reviewers. RESULTS A total of 17 studies met inclusion criteria. The analysis shows that almost all studies involved web-based interventions with the exception of one which concerned a mobile app-based intervention. Most of them prove their effectiveness in the overall well-being of the caregiver and more specifically in the mental dimension, highlighting a decrease in caregivers' anxiety and/or distress, depression symptoms and sense of competence. CONCLUSIONS The findings support that web-based interventions have an impact mainly on caregivers' well-being. Nevertheless, other dimensions that are necessary for caregiving, such as physical, mental and social dimension, have been scarcely explored. More studies on mobile app-based interventions are needed to know their effectiveness.

INTRODUCTION Clinicians commonly use the modified Rankin Scale (mRS) and the Barthel Index (BI) to measure clinical outcome after stroke. These are potential targets in machine learning models for stroke outcome prediction. Therefore, the quality of the measurements is crucial for training and validation of these models. The objective of this study was to apply and evaluate density-based outlier detection methods for identifying potentially incorrect measurements in multiple large stroke datasets to assess the measurement quality. METHOD We applied three density-based outlier detection methods including density-based spatial clustering of applications (DBSCAN), hierarchical DBSCAN (HDBSCAN) and local outlier factor (LOF) based on a large dataset obtained from a nationwide prospective stroke registry in Taiwan. The testing of each method was done by using four different NINDS funded stroke datasets. RESULT The DBSCAN achieved a high performance across all mRS values where the highest average accuracy was 99.2 ± 0.7 at mRS of 4 and the lowest average accuracy was 92.0 ± 4.6 at mRS of 3. The LOF also achieved similar performance, however, the HDBSCAN with default parameters setting required further tuning improvement. CONCLUSION The density-based outlier detection methods were proven to be promising for validation of stroke outcome measures. The outlier detection algorithm developed from a large prospective registry dataset was effectively applied in four different NINDS stroke datasets with high performance results. The tool developed from this detection algorithm can be further applied to real world datasets to increase the data quality in stroke outcome measures.

BACKGROUND Early postoperative discharge after joint arthroplasty may lead to decreased wound monitoring. A mobile woundcare app with an integrated algorithm to detect complications may lead to improved monitoring and earlier treatment of complications. In this study, the ease of use and perceived usefulness of such a mobile app was investigated. OBJECTIVE Primary objective was to investigate the ease of use and perceived usefulness of using a woundcare app. Secondary objectives were the number of alerts created, the amount of days the app was actually used and patient-reported wound infection. METHODS Patients that received a joint arthroplasty were enrolled in a prospective cohort study. During 30 postoperative days, patients scored their surgical wound by daily answering of questions in the app. An inbuilt algorithm advised patients to contact their treating physician if needed. On day 15 and day 30, additional questionnaires in the app investigated ease of use and perceived usefulness. RESULTS Sixty-nine patients were included. Median age was 68 years. Forty-one patients (59.4%) used the app until day 30. Mean grade for ease of use (on a Likert-scale of 1-5) were 4.2 on day 15 and 4.2 on day 30; grades for perceived usefulness were 4.1 on day 15 and 4.0 on day 30. Out of 1317 days of app use, an alert was sent to patients on 29 days (2.2%). Concordance between patient-reported outcome and physician-reported outcome was 80%. CONCLUSIONS Introduction of a woundcare app with an alert communication on possible wound problems resulted in a high perceived usefulness and ease of use. Future studies will focus on validation of the algorithm and the association between postoperative wound leakage and the incidence of prosthetic joint infection.

BACKGROUND Pain gained recognition as a vital sign in the early 2000s, underscoring the importance of accurate documentation, characterization, and treatment of pain. No prior studies have demonstrated the utility of the 0-10 pain scale with respect to discharge opioid prescriptions, nor characterized the most influential factors in discharge prescriptions. METHODS Inpatient and emergency department(ED) encounters from July 1, 2012 to April 1, 2018 resulting in a discharge prescription for tablet opioid medications were identified. The primary outcome was to determine if pain levels in 24 h prior to discharge correlated with opioids (in milligrams of morphine equivalents (MME)) prescribed. Secondary outcomes included the impact of patient and prescriber demographics, demographics. A generalized linear model was created to investigate factors affecting the quantity of prescribed opioids. RESULTS n = 78,691 patient encounters. Overall mean adjusted MME for non-ED visits was 378 versus 197 for ED visits. Whites received the highest quantities; those identifying as non-white and non-black received the lowest. Women received significantly fewer discharge MMEs in both the ED and inpatient cohorts. Provider prescribing patterns exhibited the most profound effect on discharge MMEs. The most prolific (≥300 prescriptions over the study period) writing the largest amount. In the ED, there was a significant negative correlation between documented pain levels and discharge MMEs(ρ = 0.074,p < 0.001). CONCLUSIONS Pain scale was significantly negatively correlated with discharge MMEs in the ED and positively correlated in the inpatient population. Individual prescriber characteristics were the more influential variable, with prolific high prescribers writing for the largest MME amounts. The inverse association of pain and MMEs at discharge in the ED, and the large effect pre-existing prescriber patterns exhibited, both improved methodology for assessing and appropriately treating pain, and effective prescriber-targeted interventions, must be a priority.

It is known that excessive levels of occupational stress affect professionals' technical and non-technical skills and surgeons are no exception. However, very few studies address this problem in neurosurgeons. A system for monitoring cardiovascular strain and autonomic imbalance during intracranial aneurysm procedures is proposed in order to obtain overall cardiac measures from those procedures. Additionally, this study also allows to detect stressful events and compare their impact with the surgeon's own appraisal. Linear and nonlinear heart rate variability (HRV) features were extracted from surgeon's electrocardiogram (ECG) signal using wearable ECG monitors and mobile technology during 10 intracranial aneurysm surgeries with two surgeons. Stress appraisal and cognitive workload were assessed using self-report measures. Findings suggest that the surgeon associated to the main role during the clipping can be exposed to high levels of stress, especially if a rupture occurs (pNN20 = 0%), while the assistant surgeon tends to experience mental fatigue. Cognitive workload scores of one of the surgeons were negatively correlated with AVNN, SDNN, pNN20, pNN50, 1 V, 2 L V, SD2 and CVI measures. Cognitive workload was positively related with stress appraisal, suggesting that more mentally demanding procedures are also assessed as more stressful. Finally, pNN20 seems to better mirror behavior during stress moments than pNN50. Additionally, a sympathovagal excitation occurs in one of the professionals after changing to main role. The present methodology shows potential for the identification of harmful events. This work may be of importance for the design of effective interventions in order to reduce surgeons stress levels. Furthermore, this approach can be applied to other professions.

BACKGROUND Automatic extraction of morbid disease or conditions contained in Death Certificates is a critical process, useful for billing, epidemiological studies and comparison across countries. The fact that these clinical documents are written in regular natural language makes the automatic coding process difficult because, often, spontaneous terms diverge strongly from standard reference terminology such as the International Classification of Diseases (ICD). OBJECTIVE Our aim is to propose a general and multilingual approach to render Diagnostic Terms into the standard framework provided by the ICD. We have evaluated our proposal on a set of clinical texts written in French, Hungarian and Italian. METHODS ICD-10 encoding is a multi-class classification problem with an extensive (thousands) number of classes. After considering several approaches, we tackle our objective as a sequence-to-sequence task. According to current trends, we opted to use neural networks. We tested different types of neural architectures on three datasets in which Diagnostic Terms (DTs) have their ICD-10 codes associated. RESULTS AND CONCLUSIONS Our results give a new state-of-the art on multilingual ICD-10 coding, outperforming several alternative approaches, and showing the feasibility of automatic ICD-10 prediction obtaining an F-measure of 0.838, 0.963 and 0.952 for French, Hungarian and Italian, respectively. Additionally, the results are interpretable, providing experts with supporting evidence when confronted with coding decisions, as the model is able to show the alignments between the original text and each output code.

OBJECTIVE Electronic Health Records (EHRs) interventions hold the promise for enabling better healthcare. However, the implementation of EHR systems has been scarce in developing countries. The objective of this study is to investigate the state of EHRs implementation in Morocco; and draw insights for potential improvements. MATERIALS AND METHODS University Medical Centers, known by locals in French as Centres Hospitalier Universitaires (CHU), are the largest and most advanced public healthcare centers in Morocco. A two-phase qualitative study was conducted in four out of the five CHUs. Phase One involved data collection through semi-structured interviews with 27 clinician champions, administrators, and medical directors. Phase Two included a brainstorming session during a health informatics conference held in Fes, Morocco. The data were analyzed using inductive analysis. RESULTS We identified five main categories of challenges due to silo strategies: (1) EHRs selection and weak bargaining power, (2) identical errors repeated across silos, (3) a lack of interoperability standards, (4) insufficient human and financial, and (5) missed cooperation and collaboration opportunities. DISCUSSION While identifying these silo challenges is an important milestone, proposing guidelines to address these challenges can bring Morocco and similar developing countries a step closer to improving healthcare through the use of health informatics and EHRs. Our recommendations for public healthcare organizations are threefold: (1) recognize the power of partnerships among all CHUs, (2) establish an e-health framework, and (3) seek national and international collaborations to drive and shape the eHealth agenda. Furthermore, we align our recommendations with the World Health Organization toolkit for an eHealth strategy to further benefit developing countries. CONCLUSION This study identifies the challenges faced by the Moroccan EHRs implementation silo-ed strategy, and it proposes practical and fundamental guidelines to address these challenges and develop an interoperable and sustainable national eHealth system in Morocco and similar developing countries.

OBJECTIVES To identify characteristics of office-based provider used as a usual source of care (USC) associated with secure electronic messaging (SM) use. DATA SOURCE 2015 Medical Expenditure Panel Survey Household Component and the supplemental Medical Organizations Survey. STUDY DESIGN Cross-sectional analysis. EXTRACTION METHODS Patients are linked to characteristics of their usual source of care provider. MAIN FINDINGS We found that 89 percent of patients whose USC had electronic health records were able to exchange secure messages with their provider. Patients whose USC reported being patient-centered medical homes (PCMHs) or that used other health information technology (HIT) were also more likely to have been able to exchange SM with their provider. Patients of independent group or solo practices were less likely to have been able to exchange SM relative to patients whose USC practice was hospital owned. CONCLUSIONS Patients were more likely to have visited a USC that exchanged SMs if that practice also used other electronic health records functionalities. Study findings suggest that while patients' USC practices were likely to exchange secure messages, there is a disparity in SM use between physician-owned practices, and hospital-owned practices.

OBJECTIVE Structured rounding tools have shown to improve the overall efficiency and perceived satisfaction with the rounding process. However, little is known about how EHR-integrated rounding tools impact the content, structure and interactivity of communication during rounds. METHOD We conducted a prospective pre-post evaluation with two rounding tools: a Microsoft Word-based fillable rounding tool (usual tool), and an EHR-integrated rounding report tool (RRT). 27 clinicians across two teams participated in rounds for 169 patients (nusual=84, nRRT=85). We audio-recorded and coded communication during rounds using conversational analysis methods. Using the coded communication interactions, we investigated differences between the two tools on: clinical content discussed, questions raised, and breakdowns in interactive communication. Additionally, we gathered clinician perspectives on the rounding tools through follow-up interviews. RESULTS We found that the use of RRT was associated with significantly more discussion of patient identifiers (e.g., name), and action items (e.g., to-do list) and significantly less discussion of imaging (e.g., X-rays) than the usual tool. RRT was also associated with fewer questions (t = 3.1, p = 0.03), and correspondingly, fewer responses (t = 3.2, p = 0.02). Communication breakdowns related to incorrect responses was fewer during the use of RRT (t = 0.5, p = 0.01). There were no statistically significant differences in the time spent for rounding between the two tools. CONCLUSIONS Our findings showed that RRT impacted rounding workflow: during pre-rounding, by saving time and effort in gathering information from multiple sources; during rounding, by streamlining content of the conversations using the structured RRT template; and during post-rounding, by supporting explicit discussion of patient tasks and action items for patient care planning and management.

BACKGROUND Computer-aided diagnosis (CAD) can assist physicians in effective and efficient diagnostic decision-making. CAD systems are currently essential tools in some areas of clinical practice. In addition, it is one of the established fields of study in the interface of medicine and computer science. There are, however, still some critical challenges that CAD systems face. METHODS This paper first describes a new literature review protocol, the Dynamic PRISMA approach based on the well-known PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) approach. This new approach enhances the traditional approach by integrating a feedback mechanism module. As a result of the literature review, this paper identifies seven major challenges that occur today in CAD and inhibit the next major developments. RESULTS The seven challenges described in this paper involve some technical weaknesses in the interface of medicine and computer science. These challenges are related to various algorithmic limitations, the difficulty of medical professionals to adopt new systems, problems when dealing with patient data, and the lack of guidelines and standardization regarding many aspects of CAD. This paper also describes some of the recent research developments towards these challenges. CONCLUSION If these seven key challenges are addressed properly, then the ways for dealing with them will become the R&D pillars needed to bring CAD to the next level. This would require additional well-coordinated collaboration between researchers and practitioners in the fields of medicine and computer science.

BACKGROUND Multidisciplinary treatment for head and neck carcinoma offers the best curative results but generates acute toxicities, which negatively affect both patients' quality of life and treatment compliance. Usually, the patient's clinical condition is recorded during scheduled, time-limited office visits and patients might forget to discuss symptoms occurred weeks before. They could also have difficulties contacting their clinicians outside of these limited encounters. Technology-based interventions for oncological patients have already been proved to encourage accurate symptoms report through regular inquiries of their clinical conditions. OBJECTIVES The aim of this work is to present the results of a pilot study about the assessment of a novel mobile application for reporting clinical parameters, quality of life, and symptoms of home patients affected by head and neck carcinoma, during chemo-radiotherapy and the subsequent follow-up period. Results will inform app designers about the necessary modifications to face a full-scale trial. METHODS Ten patients used the app for the foreseen period (up to 65 days, median 50.5), at the end of which they answered a paper questionnaire addressing user satisfaction with the app. The questionnaire included 8 questions and a free text comment field. Patients were followed by three clinicians, who also answered a similar paper questionnaire at the end of the pilot study. Questionnaires total score ranged 0-25 and a threshold of 16 was set in the study protocol to represent an overall positive outcome. However, to consider the individual constructs, questions about usability, perceived usefulness and user acceptance were also analyzed separately, and association among them was investigated. Finally, the feasibility of the intervention was analyzed in terms of the actual use of the app, i.e. dropout rates and compliance with the required data input. Statistics were only performed on patients' data, due to the small number of doctors involved in the study. RESULTS The median of the total score per patient was 18.5 (interquartile range 11.2-20.5), and per doctor was 16 (range 11-20), thus showing a positive overall satisfaction with the app. Concerning patients, only 4 out of a total of 80 answers (10 patients × 8 questions) expressed a definite negative feeling. Perceived usefulness was a critical issue for some patients. It was positively correlated with usability, and both aspects were independent predictors of acceptance. Feasibility was demonstrated by the low percentage of dropouts (9%) and noncompliance with assignments (10%). A significant (p = 0.007) negative correlation between the severity of reported symptoms and the EuroQoL questionnaire scores was found, supporting the consistency of the entered data. Free comments were reported by 6 Patients. CONCLUSIONS This study was meant to explore the context of outpatients' remote monitoring through the collection of patient-reported outcomes. The intervention for a proactive approach to symptoms monitoring in curatively treated head and neck cancer patients resulted feasible and acceptable by both patients and oncologists. The study revealed a criticality on the perceived usefulness, but, at the same time, the patients' comments suggested how to improve this aspect. Further actions will need to focus on measuring the impact of HeNeA on the process of care and on the health outcomes.

OBJECTIVE To characterise the use of an electronic medication safety dashboard by exploring and contrasting interactions from primary users (i.e. pharmacists) who were leading the intervention and secondary users (i.e. non-pharmacist staff) who used the dashboard to engage in safe prescribing practices. MATERIALS AND METHODS We conducted a 10-month observational study in which 35 health professionals used an instrumented medication safety dashboard for audit and feedback purposes in clinical practice as part of a wider intervention study. We modelled user interaction by computing features representing exploration and dwell time through user interface events that were logged on a remote database. We applied supervised learning algorithms to classify primary against secondary users. RESULTS We observed values for accuracy above 0.8, indicating that 80% of the time we were able to distinguish a primary user from a secondary user. In particular, the Multilayer Perceptron (MLP) yielded the highest values of precision (0.88), recall (0.86) and F-measure (0.86). The behaviour of primary users was distinctive in that they spent less time between mouse clicks (lower dwell time) on the screens showing the overview of the practice and trends. Secondary users exhibited a higher dwell time and more visual search activity (higher exploration) on the screens displaying patients at risk and visualisations. DISCUSSION AND CONCLUSION We were able to distinguish the interactive behaviour of primary and secondary users of a medication safety dashboard in primary care using timestamped mouse events. Primary users were more competent on population health monitoring activities, while secondary users struggled on activities involving a detailed breakdown of the safety of patients. Informed by these findings, we propose workflows that group these activities and adaptive nudges to increase user engagement.

AIM A feasibility study was conducted to evaluate implementation of a voice-enabled automated platform for collection of medical data from patients with cardiovascular disease: CardioCube. METHODS The study enrolled 22 individuals (10 males, 45.5%) including 9 patients with cardiovascular disease and 13 healthy participants. Utilizing (1) voice-enabled patient registration software implemented on the Amazon Echo and (2) web-based electronic health record (EHR) system, study participants verbally answered a set of clinical questions. Primary endpoint: accuracy of the CardioCube system. Secondary endpoints: acceptability, usability and technical performance. The study was performed at the Outpatient Cardiology Clinic, Cedars-Sinai Medical Center, Los Angeles, CA, USA. RESULTS The CardioCube system collected 432 data points with a high agreement level between verbally provided data and corresponding EHR information (accuracy 97.51%). The CardioCube was able to automatically generate a summarized medical report, which was instantly available for a doctor in the web-based EHR system. Patients reported CardioCube was "easy to use". Applicability of the system was graded excellent by the medical staff. A single session utilized less than 0.002% of available computational resources. CONCLUSION CardioCube can collect, index and document medical data using a voice interface. In this pilot study, CardioCube supported healthcare professionals by performing time-consuming paperwork during patient registration.

OBJECTIVE Telemonitoring of cardiac implantable electronic devices (CIEDs) has many advantages. However, telemonitoring involves clinical work that is often overlooked or considered a burden, such as the work performed during telephone contact with patients. The objective of this study was to scrutinize telephone calls to and from patients to understand the clinical workload in CIED remote monitoring. The focus was on time spent, type of work, and the content of telephone contact with patients. METHODS A combined quantitative and qualitative observational study was conducted at a large CIED remote monitoring center. The unit 'encounter' was used to describe either a telephone call between patient and clinician and/or a complete review of a CIED data transmission. The time spent on different encounters was measured, the telephone call content was identified and described, and the different types of clinical work were described. RESULTS A total of 260 encounters were analyzed. Encounters that involved patient telephone contact were more time consuming. Telephone calls were mostly about the home monitoring box, CIED transmission data, and symptoms. In most telephone calls, two or more topics appeared. Five types of clinical work were performed: inclusion work, coordination work, diagnostic work, education work, and comfort work. Inclusion work and diagnostic work were the dominant types. DISCUSSION Patient telephone contact in CIED telemonitoring is typically described as a "burden". This study unpacks the contents and functions of telephone calls between patients and clinicians and suggests that the function of telephone contact should be recognized as integral, rather than burdensome, to the clinical work in CIED telemonitoring.

BACKGROUND AND OBJECTIVE Hypoglycemia is a common safety event when attempting to optimize glycemic control in diabetes (DM). While electronic medical records provide a natural ground for detecting and analyzing hypoglycemia, ICD codes used in the databases may be invalid, insensitive or non-specific in detecting new hypoglycemic events. We developed text preprocessing methods to improve automatic detection of hypoglycemia from analysis of clinical encounter text notes. METHODS We set out to improve hypoglycemia detection from clinical notes by introducing three preprocessing methods: stop word filtering, medication signaling, and ICD narrative enrichment. To test the proposed methods, we selected clinical notes from VA Maryland Healthcare System, based on various combinations of three criteria that are suggestive of hypoglycemia, including ICD-9 code of diabetes and hypoglycemia, laboratory glucose values < 70 md/dL, and text reference to a proximate hypoglycemia event. In addition, we constructed one dataset of 395 clinical notes from year 2009 and another of 460 notes from year 2014 to test the generality of the proposed methods. For each of the datasets, two physician judges manually reviewed individual clinical notes to determine whether hypoglycemia was present or absent. A third physician judge served as a final adjudicator for disagreements. RESULTS Each of the proposed preprocessing methods contributed to the performance of hypoglycemia detection by significantly increasing the F1 score in the range of 5.3∼7.4% on one dataset (p < .01). Among the methods, stop word filtering contributed most to the performance improvement (7.4%). Combining all the preprocessing methods led to greater performance gain (p < .001) compared with using each method individually. Similar patterns were observed for the other dataset with the F1 score being increased in the range of 7.7%∼9.4% by individual methods (p < .001). Nevertheless, combining the three methods did not yield additional performance gain. CONCLUSION The proposed text preprocessing methods improved the performance of hypoglycemia detection from clinical text notes. Stop word filtering achieved the most performance improvement. ICD narrative enrichment boosted the recall of detection. Combining the three preprocessing methods led to additional performance gains.

OBJECTIVE This study aims to estimate the association between health information technology (HIT) use and influenza vaccine uptake among US adults. MATERIALS AND METHODS Data analysis was conducted using 2011-2015 National Health Interview Survey (NHIS) adult data (n = 169,912). HIT use was defined as having used computers (past 12 months) to seek health information, fill prescriptions, schedule appointments, communicate with health providers via email, and/or use online health chat groups. Crude and multivariable logistic regression models were used to estimate the odds of influenza vaccine uptake among HIT users versus non-users. Interactions were tested and stratified results were reported. RESULTS Among US adults, 39.8% received an influenza vaccine in the past 12 months, while 48.6% reported any HIT use. After adjusting for covariates, any HIT users had 1.23 times greater odds (95% CI = 1.19, 1.27) of influenza vaccine uptake relative to non-HIT users. HIT use for looking up health information on the internet (OR = 1.19, 95% CI = 1.15, 1.23), filling prescriptions (OR = 1.56; 95% CI = 1.50, 1.66), scheduling appointments (OR = 1.56; 95% CI = 1.50, 1.66), and communicating with providers via email (OR = 1.51; 95% CI = 1.44, 1.59) were significantly associated with influenza vaccine uptake. DISCUSSION HIT use is positively associated with influenza vaccine uptake. Each category of HIT use was independently associated with influenza vaccine uptake. To our knowledge, no other studies have evaluated the relationship between HIT use and influenza vaccine uptake. Our results are exploratory and represent an association, not a causal relationship. Longitudinal, confirmatory studies are also needed to verify our cross-sectional findings.

OBJECTIVE To report on the use of an eye-tracking retrospective think-aloud for usability evaluation and to describe its application in assessing the usability of a mobile health app. MATERIALS AND METHODS We used an eye-tracking retrospective think-aloud to evaluate the usability of an HIV prevention mobile app among 20 young men (15-18 years) in New York City, NY; Birmingham, AL; and Chicago, IL. Task performance metrics, critical errors, a task completion rate per participant, and a task completion rate per task, were measured. Eye-tracking metrics including fixation, saccades, time to first fixation, time spent, and revisits were measured and compared among participants with/without a critical error. RESULTS Using task performance analysis, we identified 19 critical errors on four activities, and of those, two activities had a task completion rate of less than 78%. To better understand these usability issues, we thoroughly analyzed participants' corresponding eye movements and verbal comments using an in-depth problem analysis. In areas of interest created for the activity with critical usability problems, there were significant differences in time spent (p = 0.008), revisits (p = 0.004), and total numbers of fixations (p = 0.007) by participants with/without a critical error. The overall mean score of perceived usability rated by the Health IT Usability Evaluation Scale was 4.64 (SD = 0.33), reflecting strong usability of the app. DISCUSSION AND CONCLUSION An eye-tracking retrospective think-aloud enabled us to identify critical usability problems as well as gain an in-depth understanding of the usability issues related to interactions between end-users and the app. Findings from this study highlight the utility of an eye-tracking retrospective think-aloud in consumer health usability evaluation research.

BACKGROUND A key falls prevention intervention delivered within occupational therapy is the home environment falls-risk assessment process. This involves the clinician visiting the patient's home and using a 2D paper-based measurement guidance booklet to ensure that all measurements are taken and recorded accurately. However, 30% of all assistive devices installed within the home are abandoned by patients, in part as a result of the inaccurate measurements being recorded as part of the home environment falls-risk assessment process. In the absence of more appropriate and effective guidance, high levels of device abandonment are likely to persist. AIM This study presents guidetomeasure-OT, a mobile 3D measurement guidance application designed to support occupational therapists in carrying out home environment falls-risk assessments. Furthermore, this study aims to empirically evaluate the performance of guidetomeasure-OT compared with an equivalent paper-based measurement guidance booklet. METHODS Thirty-five occupational therapists took part in this within-subjects repeated measures study, delivered within a living lab setting. Participants carried out the home environment falls-risk assessment process under two counterbalanced treatment conditions; using 3D guidetomeasure-OT; and using a 2D paper-based guide. Systems Usability Scale questionnaires and semi-structured interviews were completed at the end of both task. A comparative statistical analysis explored performance relating to measurement accuracy, measurement accuracy consistency, task completion time, and overall system usability, learnability, and effectiveness of guidance. Interview transcripts were analysed using inductive and deductive thematic analysis, the latter was informed by the Unified Theory of Acceptance and Use of Technology model. RESULTS The guidetomeasure-OT application significantly outperformed the 2D paper-based guidance in terms task efficiency (p <  0.001), learnability (p <  0.001), system usability (p <  0.001), effectiveness of guidance (p =  0.001). Regarding accuracy, in absolute terms, guidetomeasure-OT produced lower mean error differences for 11 out of 12 items and performed significantly better for six out of 12 items (p = < 0.05). In terms of SUS, guidetomeasure-OT scored 83.7 compared with 70.4 achieved by the booklet. Five high-level themes emerged from interviews: Performance Expectancy, Effort Expectancy, Social Influence, Clinical Benefits, and Augmentation of Clinical Practice. Participants reported that guidetomeasure-OT delivered clearer measurement guidance that was more realistic, intuitive, precise and usable than the paper-based equivalent. Audio instructions and animated prompts were seen as being helpful in reducing the learning overhead required to comprehend measurement guidance and maintain awareness of task progression. CONCLUSIONS This study reveals that guidetomeasure-OT enables occupational therapists to carry out significantly more accurate and efficient home environment falls-risk assessments, whilst also providing a measurement guide tool that is considered more usable compared with the paper-based measurement guide that is currently used by clinicians in practice. These results are significant as they indicate that mobile 3D visualisation technologies can be effectively deployed to improve clinical practice, particularly within the home environment falls-risk assessment context. Furthermore, the empirical findings constitute overcoming the challenges associated with the digitisation of health care and delivery of new innovative and enabling technological solutions that health providers and policy makers so urgently need to ease the ever-increasing burden on existing public resources. Future work will focus on the development and empirical evaluation of a mobile 3D application for patient self-assessment and automated assistive equipment prescription. Furthermore, broader User Experience aspects of the application design and the interaction mechanisms that are made available to the user could be considered so as to minimize the effect of cognitive overloading and optimise user performance.

PURPOSE To study the number of disruptions in patient processes in a radiotherapy centre after the replacement of an Electronic Health Record (EHR), integrating information tools for patient care and billing. METHODS Our self-made Electronic Medical Record was replaced by a new EHR, including clinical path and workflow-management. A social-technological approach was used to reduce complexity. We measured disruptions in patient processes by the number and type of EHR related root causes and EHR-related incidents that reached patients, in our patient safety system 12 months before implementing the new EHR, 6 months after implementation (transition period) and 24 months after the transition period. We used Mann-Whitney U and X² tests to compare data before and after implementation. RESULTS An increase of disruptions occurred only temporarily during 6 months. After this period, the number stabilized to the level before implementation while having more functionalities and benefits. Neither the number nor the severity of incidents reaching patients increased. CONCLUSIONS Disruptions in patient processes are considered as a main barrier for implementing an EHR. Using a social/technical approach, the increase in disruptions did only temporarily occur and did not reach patients. We think it is important to share this insight with physicians because literature shows that their long-term opinion regarding the usefulness of the EHR is often based on the experience in the first months after implementation. Management of expectations is recommended. ADVANCES IN KNOWLEDGE This study is the first of its kind measuring long-term effects of EHR on patient processes in radiotherapy.

OBJECTIVE Electronic health record (EHR) systems contain structured data (such as diagnostic codes) and unstructured data (clinical documentation). Clinical insights can be derived from analyzing both. The use of natural language processing (NLP) algorithms to effectively analyze unstructured data has been well demonstrated. Here we examine the utility of NLP for the identification of patients with non-alcoholic fatty liver disease, assess patterns of disease progression, and identify gaps in care related to breakdown in communication among providers. MATERIALS AND METHODS All clinical notes available on the 38,575 patients enrolled in the Mount Sinai BioMe cohort were loaded into the NLP system. We compared analysis of structured and unstructured EHR data using NLP, free-text search, and diagnostic codes with validation against expert adjudication. We then used the NLP findings to measure physician impression of progression from early-stage NAFLD to NASH or cirrhosis. Similarly, we used the same NLP findings to identify mentions of NAFLD in radiology reports that did not persist into clinical notes. RESULTS Out of 38,575 patients, we identified 2,281 patients with NAFLD. From the remainder, 10,653 patients with similar data density were selected as a control group. NLP outperformed ICD and text search in both sensitivity (NLP: 0.93, ICD: 0.28, text search: 0.81) and F2 score (NLP: 0.92, ICD: 0.34, text search: 0.81). Of 2281 NAFLD patients, 673 (29.5%) were believed to have progressed to NASH or cirrhosis. Among 176 where NAFLD was noted prior to NASH, the average progression time was 410 days. 619 (27.1%) NAFLD patients had it documented only in radiology notes and not acknowledged in other forms of clinical documentation. Of these, 170 (28.4%) were later identified as having likely developed NASH or cirrhosis after a median 1057.3 days. DISCUSSION NLP-based approaches were more accurate at identifying NAFLD within the EHR than ICD/text search-based approaches. Suspected NAFLD on imaging is often not acknowledged in subsequent clinical documentation. Many such patients are later found to have more advanced liver disease. Analysis of information flows demonstrated loss of key information that could have been used to help prevent the progression of early NAFLD (NAFL) to NASH or cirrhosis. CONCLUSION For identification of NAFLD, NLP performed better than alternative selection modalities. It then facilitated analysis of knowledge flow between physician and enabled the identification of breakdowns where key information was lost that could have slowed or prevented later disease progression.

PURPOSE To identify factors that influence the acceptance of technology for ageing in place by seniors in the post-implementation stage. This review is among very few that focus on acceptance in post-implementation phase. METHODS A literature review. We searched six databases (Cinahl, Medline, PsycINFO, PubMed, Science Direct, Scopus). Inclusion criteria were: 1) original and peer-reviewed research written in English, 2) Articles published in 2010-2018, 3) Empirical research papers, 4) Research in which participants are seniors aged above 60 years, 5) Research aimed at investigating factors that influence the acceptance of digital technology for ageing in place, 6) Research conducted in home environment, 7) Focus on post-implementation stage. RESULTS Twenty-three out of 2181 papers were included. The results show that acceptance of technology in the post-implementation stage is influenced by 36 factors, divided into six themes: concerns/problems regarding technology (technical errors, etc.), experienced positive characteristics of technology (e.g., ease of use factors, privacy implications), expected benefits of technology (e.g., increased safety, companionship, increased security, etc.), need for technology (e.g., perceived need to use), social influence (e.g., influence by peers, family or surroundings) and characteristics of older adults (e.g., past experiences/attitudes, physical environment). The articles considered different types of technology: health monitoring, ADL, safety and communication. The level of technology readiness for digital technologies supporting ageing in place in post-implementation stage is still low within the scientific literature, since only seven out of 23 articles studied mature technologies (TRL 8-9 of the technology readiness level scale). The majority of the studies were conducted in Western Europe or the US, and only two were conducted in other regions (Australia, and Taiwan). Qualitative and quantitative methods were equally used in the analysed articles. CONCLUSIONS Acceptance of technology in the post-implementation stage is influenced by multiple factors. An interesting finding was that the seniors' views of technology change between the pre- and post-implementation stages. Some negative concerns that appeared in the pre-implementation stage appear as positive characteristics in the post-implementation stage. In the post-implementation stage, seniors realize the wide variety of benefits that technology can have on their lives. We expect that findings of this review can be utilised by academics and policy-makers for gaining insights for further research and successful implementation of technology for ageing in place.

BACKGROUND Assessment of the performance of Intensive Care Units (ICU) is of vital importance for an effective healthcare system. Such assessment ensures that the limited resources of the healthcare system are allocated where they are most needed. Severity scoring systems are employed for this purpose and improving these systems is a continuing area of research which has focused on the use of more complex techniques and new variables. OBJECTIVES This paper investigates whether scoring systems could be improved through use of metrics which better summarise the high frequency data collected by automated systems for patients in the ICU. METHODS AND DATA 3128 admissions to the Gold Coast University Hospital ICU are used to construct three logistic regressions based on the most widely used scoring system (APACHE III) to compare performance with and without predictors leveraging available high frequency information. Performance is assessed based on model accuracy, calibration, and discrimination. High frequency information was considered for existing pulse and mean arterial pressure physiology fields and resulting models compared against a baseline logistic regression using only APACHE III physiology variables. RESULTS Model discrimination and accuracy were better for models which included high frequency predictors, with calibration remaining good in all cases. The most influential high frequency summaries were the number of turning points in a patient's mean arterial pressure or pulse in the first 24 h of ICU admission. CONCLUSIONS The findings indicate that scoring systems can be improved by better accounting for high frequency data.

BACKGROUND AND OBJECTIVE Autism spectrum disorder (ASD) is a heterogeneous disorder. Research has explored potential ASD subgroups with preliminary evidence supporting the existence of behaviorally and genetically distinct subgroups; however, research has yet to leverage machine learning to identify phenotypes on a scale large enough to robustly examine treatment response across such subgroups. The purpose of the present study was to apply Gaussian Mixture Models and Hierarchical Clustering to identify behavioral phenotypes of ASD and examine treatment response across the learned phenotypes. MATERIALS AND METHODS The present study included a sample of children with ASD (N = 2400), the largest of its kind to date. Unsupervised machine learning was applied to model ASD subgroups as well as their taxonomic relationships. Retrospective treatment data were available for a portion of the sample (n  = 1034). Treatment response was examined within each subgroup via regression. RESULTS The application of a Gaussian Mixture Model revealed 16 subgroups. Further examination of the subgroups through Hierarchical Agglomerative Clustering suggested 2 overlying behavioral phenotypes with unique deficit profiles each composed of subgroups that differed in severity of those deficits. Furthermore, differentiated response to treatment was found across subtypes, with a substantially higher amount of variance accounted for due to the homogenization effect of the clustering. DISCUSSION The high amount of variance explained by the regression models indicates that clustering provides a basis for homogenization, and thus an opportunity to tailor treatment based on cluster memberships. These findings have significant implications on prognosis and targeted treatment of ASD, and pave the way for personalized intervention based on unsupervised machine learning.

BACKGROUND AND PURPOSE Manual annotation and categorization of non-standardized text ("free-text") of drug orders entered into electronic health records is a labor-intensive task. However, standardization is required for drug order analyses and has implications for clinical decision support. Machine learning could help to speed up manual labelling efforts. The objective of this study was to analyze the performance of deep machine learning methods to annotate non-standardized text of drug order entries with their therapeutically active ingredients. MATERIALS AND METHODS The data consisted of drug orders entered 8/2009-4/2014 into the electronic health records of inpatients at a large tertiary care academic medical center. We manually annotated the most frequent order entry patterns with the active ingredient they contain (e.g. "Prograf"⟵"Tacrolimus"). We heuristically included additional orders by means of character sequence comparisons to augment the training dataset. Finally, we trained and employed character-level recurrent deep neural networks to classify non-standardized text of drug order entries according to their active ingredients. RESULTS A total of 26,611 distinct order patterns were considered in our study, of which the top 7.6% (2028) had been annotated with one of 558 distinct ingredients, leaving 24,583 unlabeled observations. Character-level recurrent deep neural networks achieved a Mean Reciprocal Rank (MRR) of 98% and outperformed the best representative baseline, a trigram-based Support Vector Machine, by 2 percentage points. CONCLUSION Character-level recurrent deep neural networks can be used to map the active ingredient to non-standardized text of drug order entries, outperforming other representative techniques. While machine learning might help to facilitate categorization tasks, still a considerable amount of manual labelling and reviewing work is required to train such systems.

INTRODUCTION Insufficient patient enrollment in clinical trials remains a serious and costly problem and is often considered the most critical issue to solve for the clinical trials community. In this project, we assessed the feasibility of automatically detecting a patient's eligibility for a sample of breast cancer clinical trials by mapping coded clinical trial eligibility criteria to the corresponding clinical information automatically extracted from text in the EHR. METHODS Three open breast cancer clinical trials were selected by oncologists. Their eligibility criteria were manually abstracted from trial descriptions using the OHDSI ATLAS web application. Patients enrolled or screened for these trials were selected as 'positive' or 'possible' cases. Other patients diagnosed with breast cancer were selected as 'negative' cases. A selection of the clinical data and all clinical notes of these 229 selected patients was extracted from the MUSC clinical data warehouse and stored in a database implementing the OMOP common data model. Eligibility criteria were extracted from clinical notes using either manually crafted pattern matching (regular expressions) or a new natural language processing (NLP) application. These extracted criteria were then compared with reference criteria from trial descriptions. This comparison was realized with three different versions of a new application: rule-based, cosine similarity-based, and machine learning-based. RESULTS For eligibility criteria extraction from clinical notes, the machine learning-based NLP application allowed for the highest accuracy with a micro-averaged recall of 90.9% and precision of 89.7%. For trial eligibility determination, the highest accuracy was reached by the machine learning-based approach with a per-trial AUC between 75.5% and 89.8%. CONCLUSION NLP can be used to extract eligibility criteria from EHR clinical notes and automatically discover patients possibly eligible for a clinical trial with good accuracy, which could be leveraged to reduce the workload of humans screening patients for trials.

BACKGROUND Cleft palate patients have inability to produce adequate velopharyngeal closure, which results in hypernasal speech. In clinic, hypernasal speech is assessed through subject assessment by speech language pathologists. Automatic hypernasal speech detection can provide aided diagnoses for speech language pathologists and clinicians. OBJECTIVES This study aims to develop Long Short-Term Memory (LSTM) based Deep Recurrent Neural Network (DRNN) system to detect hypernasal speech from cleft palate patients, thus to provide aided diagnoses for clinical operation and speech therapy. Meanwhile, the feature mining and classification abilities of LSTM-DRNN system are explored. METHODS The utilized speech recordings are 14,544 vowels in Mandarin. Speech data is collected from 144 children (72 children with hypernasality and 72 controls) with the age of 5-12 years old. This work proposes a LSTM based DRNN system to achieve automatic hypernasal speech detection, since LSTM-DRNN can learn short-time dependences of hypernasal speech. The vocal tract based features are fed into LSTM-DRNN to achieve deep mining of features. To verify the feature mining ability of LSTM-DRNN, features projected by LSTM-DRNN are fed into shallow classifiers instead of the following two fully connected layers and a softmax layer. And the features without the projecting process of LSTM-DRNN are directly fed into shallow classifiers as a comparison. Hypernasality-sensitive vowels (/a/, /i/, and /u/) are analyzed for the first time. RESULTS This LSTM-DRNN based hypernasal speech detection method reaches higher detection accuracy than that using shallow classifiers, since LSTM-DRNN mines features through time axis and network depth simultaneously. The proposed LSTM-DRNN based hypernasality detection system reaches the highest accuracy of 93.35%. According to the analysis of hypernasality-sensitive vowels, the experimental result concludes that vowels /i/ and /u/ are the most sensitive vowels to hypernasal speech. CONCLUSIONS The results show that LSTM-DRNN has robust feature mining ability and classification ability. This is the first work that applies the LSTM-DRNN technique to automatically detect hypernasality in cleft palate speech. The experimental results demonstrate the potential of deep learning on pathologist speech detection.

INTRODUCTION Predictive survival modeling offers systematic tools for clinical decision-making and individualized tailoring of treatment strategies to improve patient outcomes while reducing overall healthcare costs. In 2015, a number of machine learning and statistical models were benchmarked in the DREAM 9.5 Prostate Cancer Challenge, based on open clinical trial data for metastatic castration resistant prostate cancer (mCRPC). However, applying these models into clinical practice poses a practical challenge due to the inclusion of a large number of model variables, some of which are not routinely monitored or are expensive to measure. OBJECTIVES To develop cost-specified variable selection algorithms for constructing cost-effective prognostic models of overall survival that still preserve sufficient model performance for clinical decision making. METHODS Penalized Cox regression models were used for the survival prediction. For the variable selection, we implemented two algorithms: (i) LASSO regularization approach; and (ii) a greedy cost-specified variable selection algorithm. The models were compared in three cohorts of mCRPC patients from randomized clinical trials (RCT), as well as in a real-world cohort (RWC) of advanced prostate cancer patients treated at the Turku University Hospital. Hospital laboratory expenses were utilized as a reference for computing the costs of introducing new variables into the models. RESULTS Compared to measuring the full set of clinical variables, economic costs could be reduced by half without a significant loss of model performance. The greedy algorithm outperformed the LASSO-based variable selection with the lowest tested budgets. The overall top performance was higher with the LASSO algorithm. CONCLUSION The cost-specified variable selection offers significant budget optimization capability for the real-world survival prediction without compromising the predictive power of the model.

BACKGROUND EHealth solutions are envisaged to contribute significantly to a sustainable healthcare system. Between 2016 and 2018 the eight Dutch University Medical Centers (UMCs) received Dutch Government's funding to undertake research into the clinical impact, cost-effectiveness and ethical consideration of eHealth. The UMCs collaborated within the consortium 'Citrien fund (CF) program eHealth' and found that, in order to increase the value of eHealth in routine care, a national vision on eHealth developed by the UMCs was warranted. OBJECTIVE The objective of this paper was to elucidate the process of the 'Netherlands Federation of UMCs (NFU) eHealth vision' development by describing the results of the performed concept mapping study. METHODS A concept mapping approach was followed. Sixteen members of the steering committee of the CF program eHealth were selected as participants. First, each member selected relevant objectives from the eight individual UMC eHealth vision documents, which was to be incorporated into the overall 'NFU eHealth vision'. Second, objectives were rated for necessary to be included in the vision document and the need to achieve the objective within five years. Thereafter, the objectives were sorted into self-created thematic clusters. And finally, the concept map with the thematic clusters and corresponding objectives was discussed with the steering committee to determine the major themes of the 'NFU eHealth vision'. RESULTS 38 objectives were determined by the steering committee and grouped into the following 6 thematic clusters on the concept map: 'patient participation and empowerment'; 'infrastructure'; 'education and research'; 'multi-disciplinary care'; 'organisational restructuring'; and 'essential conditions for development of eHealth solutions'. After discussing the concept mapping results with the steering committee, the following five major themes were determined to be addressed in the vision document: 'patient and caregiver'; 'research and innovation'; 'education'; 'organisation of care'; and 'essential conditions for development of eHealth solutions'. CONCLUSION Concept mapping was successfully applied to conceptualise the different values and opinions of the eight Dutch UMCs in order to develop a national vision on eHealth. This vision document will give direction to the development, evaluation and implementation of eHealth in the eight Dutch UMCs and their adherent healthcare providers.

BACKGROUND Wearable devices with an ability to collect various type of physiological data are increasingly becoming seamlessly integrated into everyday life of people. In the area of electronic health (eHealth), many of these devices provide remote transfer of health data, as a result of the increasing need for ambulatory monitoring of patients. This has a potential to reduce the cost of care due to prevention and early detection. OBJECTIVE The objective of this study was to provide an overview of available wearable sensor systems with data exchange possibilities. Due to the heterogeneous capabilities these systems possess today, we aimed to systematize this in terms of usage, where there is a need of, or users benefit from, transferring self-collected data to health care actors. METHODS We searched for and reviewed relevant sensor systems (i.e., devices) and mapped these into 13 selected attributes related to data-exchange capabilities. We collected data from the Vandrico database of wearable devices, and complemented the information with an additional internet search. We classified the following attributes of devices: type, communication interfaces, data protocols, smartphone/PC integration, connection to smartphone health platforms, 3rd party integration with health platforms, connection to health care system/middleware, type of gathered health data, integrated sensors, medical device certification, access to user data, developer-access to device, and market status. Devices from the same manufacturer with similar functionalities/characteristics were identified under the same device family. Furthermore, we classified the systems in three subgroups of relevance for different actors in mobile health monitoring systems: EHR providers, software developers, and patient users. RESULTS We identified 362 different mobile health monitoring devices belonging to 193 device families. Based on an analysis of these systems, we identified the following general challenges: CONCLUSIONS: Few of the identified mobile health monitoring systems use standardized, open communication protocols, which would allow the user to directly acquire sensor data. Use of open protocols can provide mobile health (mHealth) application developers an alternative to proprietary cloud services and communication tools, which are often closely integrated with the devices. Emerging new types of sensors, often intended for everyday use, have a potential to supplement health records systems with data that can enrich patient care.

BACKGROUND Guidelines from the Royal College of Radiologists and National Patient Safety Agency highlight the crucial importance of "fail-safe" alert systems for the communication of critical and significant clinically unexpected results between imaging departments and referring clinicians. Electronic alert systems are preferred, to minimise errors, increase workflow efficiency and improve auditability. To date there is a paucity of evidence on the utility of such systems. We investigated i) how often emailed radiology alerts were acknowledged by referring clinicians, ii) how frequently follow-up imaging was requested when indicated and iii) whether practise improved after an educational intervention. METHODS 100 cases were randomly selected before and after an educational intervention at a tertiary referral centre in London, where the email-based 'RadAlert' system (Rivendale Systems, UK) has been in operation since May 2017. RESULTS Following educational intervention, 'accepted' alerts increased from 39% to 56%, 'abandoned' alerts reduced from 55% to 37% and 'declined' alerts decreased from 5% to 3%. There was evidence to confirm that, when indicated, further imaging had been requested for 78% of all alerts, 78% of 'accepted' alerts and 76% of 'abandoned' alerts both before and after educational intervention. CONCLUSIONS Acknowledgment of report alerts by referring clinicians increased after departmental education / governance meetings. However, a proportion of email alerts remained unacknowledged. It is incumbent on reporting radiologists to be aware that electronic alert systems cannot be solely relied upon and to take the necessary steps to ensure significant and clinically unsuspected findings are relayed to referring clinical teams in a timely manner.

BACKGROUND The Bedside Paediatric Early Warning System (BedsidePEWS) is a clinical decision support tool designed to augment clinician expertise, objectively identify children at risk for clinical deterioration, and standardize and prioritize care to improve outcomes in community settings. Although the paper-based BedsidePEWS documentation record has been shown to improve clinicians' perception of their ability to detect deterioration and follow care recommendations, research is needed to asses this impact empirically. Furthermore, as hospitals progressively move toward electronic clinical systems, knowledge regarding the impact of BedsidePEWS' novel electronic interface on clinicians' performance and user experience is required. OBJECTIVES The primary objectives of this study were (1) to compare adherence to evidence-based care recommendations using a) electronic health record software, b) paper BedsidePEWS, and c) a novel electronic BedsidePEWS interface, and (2) to describe end-users' experiences of usability and opportunities for improvement of both paper and electronic BedsidePEWS. METHODS Paediatric nurses participated in a repeated measures simulation study. Participants assessed simulated patients, documented patient data, and responded to a series of questions regarding follow-up care for each patient. Three patient types (i.e., stable, mild deterioration, severe deterioration) were assessed in each of three intervention conditions (i.e., electronic health record, paper BedsidePEWS, electronic BedsidePEWS). Following simulation scenarios, participants provided comments regarding the usability of the paper and electronic tools. RESULTS Participants made 12.7% and 18.0% more appropriate care decisions with paper and electronic BedsidePEWS, respectively, than with the electronic health record intervention (p < 0.001). Accurate BedsidePEWS severity of illness score calculation was related to better adherence to evidence-based care recommendations (65%), compared to inaccurate calculation (55%), and electronic BedsidePEWS was associated with 15.7% fewer calculation errors than paper (p < 0.005). Electronic BedsidePEWS demonstrated usability benefits over its paper predecessor, including automatic score calculation and data plotting, and the potential to eliminate double charting, and participants expressed a preference for electronic BedsidePEWS in all aspects of the debrief questionnaire (p < 0.001). CONCLUSIONS BedsidePEWS in both paper and electronic formats significantly improved participants' ability to detect deterioration and follow care recommendations compared to electronic health record software. Furthermore, results suggest that electronic BedsidePEWS would afford improved patient care in excess of the paper-based original and further contribute to the standardization, prioritization, and improvement of care in community settings.

BACKGROUND Physical activity (PA) smartphone applications (apps) featuring at least one behavioral change technique (BCT) are ubiquitous. Although BCTs in PA apps and their quality have been evaluated in Western countries, such research in China is new. This study (1) characterizes the extent that BCTs are implemented in Chinese PA apps and (2) evaluates their features and quality. METHODS Of 5,253 PA apps identified in five Android app stores in China, 51 top-ranked PA apps with more than 2 million total downloads and user rating above 4 (out of 5) were selected and assessed; their BCTs were scored using the BCT taxonomy(V1), and app quality was evaluated with the Mobile App Rating Scale (MARS). Correlations among the number of BCTs, app quality, app features, app downloads, and user ratings were examined. RESULTS Of the top-ranked apps, 62.7% allow behavioral tracking (i.e., pedometer), 9.8% promote Chinese dancing, and 17.6% offer monetary incentives. The average number of BCT categories included in the apps analyzed was 11 (SD = 3.51; range = 3-16), with the most common ones comprising feedback and monitoring (88.2%), goals and planning (82.4%), social support (81.4%), and reward and threat (80.4%). The average scores for objective and subjective quality of these apps were 3.90 (SD = 0.44) and 2.27 (SD = 0.61), respectively, as assessed by MARS. Moderate positive correlations were observed among the number of BCT incorporated, apps quality, app features, and their total downloads. CONCLUSIONS Chinese PA apps vary in the number of BCTs incorporated, with goals setting, feedback, social support, and reward being typical. App objective quality is moderate, with relatively low subjective quality. Generally, higher quality PA apps in China include more BCTs with more app features and downloads. These findings can guide the development of PA apps to mitigate physical inactivity in China-and elsewhere.

INTRODUCTION Population health programs are increasingly reliant on Health Information Technology (HIT). Program HIT architecture description is a necessary step prior to evaluation. Several sociotechnical frameworks have been used previously with HIT programs. The Informatics Stack is a novel framework that provides a thorough description of HIT program architecture. The Emergency Department Return Visit Quality Program (EDRVQP) is a population-level continuous quality improvement (QI) program connecting EDs across Ontario. The objectives of the study were to utilize the Informatics Stack to provide a description of the EDRVQP HIT architecture and to delineate population health program factors that are enablers or barriers. MATERIALS AND METHODS The Informatics Stack was used to describe the HIT architecture. A qualitative study was completed with semi-structured interviews of key informants across stakeholder organizations. Emergency departments were selected randomly. Purposive sampling identified key informants. Interviews were conducted until saturation. An inductive qualitative analysis using grounded theory was completed. A literature review of peer-reviewed background literature, and stakeholder organization reports was also conducted. RESULTS 23 business actors from 15 organizations were interviewed. The EDRVQP architecture description is presented across the Informatics Stack levels. The levels from most comprehensive to most basic are world, organization, perspectives/roles, goals/functions, workflow/behaviour/adoption, information systems, modules, data/information/knowledge/wisdom/algorithms, and technology. Enabling factors were the high rate of electronic health record adoption, legislative mandate for data collection, use of functional data standards, implementation flexibility, leveraging validated algorithms, and leveraging existing local health networks. Barriers were privacy legislation and a high turn-around time. DISCUSSION The Informatics Stack provides a robust approach to thoroughly describe the HIT architecture of population health programs prior to program replication. The EDRVQP is a population health program that illustrates the pragmatic use of continuous QI methodology across a population (provincial) level.

OBJECTIVE One of the most important methods for evaluating information systems is usability evaluation. Usability is a context-dependent qualitative feature that is measured by multiple quality components that can be related to users' characteristics. This study was conducted to evaluate the usability of a comprehensive national health information system (SIB; an abbreviation for the Persian equivalent of 'integrated health system') from the perspective of different users and to determine the relationship between quality components and users' characteristics. METHOD The study population were users of the national health information system (n = 309) at health centers and health homes affiliated to Kashan University of Medical Sciences, Kashan, Iran. Data were collected using Software Usability Measurement Inventory (SUMI) questionnaire which measures users' experiences of software interface in five quality components (i.e. affect, efficiency, helpfulness, control, and learnability) and provides a global usability score. SUMI scores were analyzed according to an extensive reference database (SUMISCO). The relationships between quality components and users' characteristics were investigated by one-way analysis of variance (ANOVA), independent t-test, and Pearson's correlation coefficient. RESULTS A total of 250 users completed the questionnaire (response rate = 81%). The mean scores of all quality components were significantly lower than the mean of SUMISCO. Learnability score had significant relationships with the user's position, education level, and field of education (P < 0.001). Physicians scored significantly lower than other users in efficiency, helpfulness and global usability (P < 0.05). Users' practice experience and age had significant linear inverse relationships with efficiency, helpfulness, and learnability (P < 0.05). CONCLUSIONS The national health information system which is used by a large number of users across a developing country have low usability. Given the significant relationships between the users' characteristics of such systems and quality components, it is essential to consider the characteristics and needs of various user groups during the processes of system analysis and design.

PURPOSE To evaluate quality and readability of online information on dental treatment for snoring and obstructive sleep apnea. METHODS An Internet search was done using three engines (Bing, Google, and Yahoo) with the combination of terms, "snoring sleep apnea dental treatment". The first 100 sites from the search of each engine were screened. Subject sites were evaluated with Health on the Net(HON) criteria, Journal of American Medical Association(JAMA) benchmarks, DISCERN, Ensuring Quality Information for Patients(EQIP), Flesch-Kincaid Grade level and Flesch Reading Ease(FRE) score. RESULTS One hundred and thirty websites were evaluated. The HON, DISCERN, EQIP, and FRE score were each 39.4%, 47.3%, 49.7%, and 51.6% of the maximum possible score, respectively. According to JAMA benchmarks fewer than 50% of the sites displayed attribution and currency. There was only one site displaying the HON seal. HON score, DISCERN score and EQIP score showed significant inter-correlation. CONCLUSION Based on this study, the current quality and readability of searchable websites on dental treatment for snoring and sleep apnea are low and poorly maintained on average. Clinicians should be able to evaluate and give accurate online information on this issue to patients.

BACKGROUND Patient transfers involve the physical movement of patients, along with the transfer of their care-related information, responsibility, and control between sending and receiving clinicians. Patient transfers between critical care units are complex and vulnerable to bottlenecks. OBJECTIVE To examine the patient transfer process from emergency department (ED) to medical intensive care unit (MICU). MATERIALS AND METHOD A qualitative study on transfers from ED to MICU was conducted at two academic hospitals. Using a process-based methodological approach supported by shadowing of patient transfers and clinician interviews, we examined the process-based similarities and differences in barriers and strategies used across hospitals. RESULTS Phases underlying ED-MICU transfer process included: pre-transfer phase involving ED care coordination and MICU transfer decision-making; transfer phase involving ED-MICU resident handoff, and post-transfer phase involving MICU care planning and management. DISCUSSION AND CONCLUSION Transfer of information, responsibility and control between sending and receiving clinicians is key to effective management of interdependencies between the pre-transfer, transfer and post-transfer phases underlying the patient transfer process. Evidence-based strategies to address challenges related to transfer of information, responsibility and control include the use of videophones and communication checklists, the allocation of a crash bed, engagement of sending, receiving and consulting teams in the physical movement of patients, and in-hospital transfer protocols.

INTRODUCTION Research in family medicine is necessary to improve the quality of care. The number of publications in general medicine remains low. Databases from Electronic Medical Records can increase the number of these publications. These data must be coded to be used pertinently. The objective of this study was to assess the quality of semantic annotation by a multi-terminological concept extractor within a corpus of family medicine consultations. METHOD Consultation data in French from 25 general practitioners were automatically annotated using 28 different terminologies. The data extracted were classified into three groups: reasons for consulting, observations and consultation results. The first evaluation led to a correction phase of the tool which led to a second evaluation. For each evaluation, the precision, recall and F-measure were quantified. Then, the inter- and intra-terminological coverage of each terminology was assessed. RESULTS Nearly 15,000 automatic annotations were manually evaluated. The mean values for the second evaluation of precision, recall and F-measure were 0.85, 0.83 and 0.84 respectively. The most common terminologies used were SNOMED CT, SNOMED 3.5 and NClt. The terminologies with the best intra-terminological coverage were ICPC-2, DRC and CISMeF Meta-Terms. CONCLUSION A multi-terminological concepts extractor can be used for the automatic annotation of consultation data in family medicine. Integrating such a tool into general practitioners' business software would be a solution to the lack of routine coding. Developing the use of a single terminology specific to family medicine could improve coding, facilitate semantic interoperability and the communication of relevant information.