Skip to main content

Advertisement

SpringerLink
Log in

Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists

  • Original Paper
  • Published:
The European Journal of Health Economics Aims and scope Submit manuscript

Abstract

Background

Published health utility studies are increasingly cited in cost–utility analyses to inform reimbursement decision-making. However, there is limited guidance for investigators looking to systematically evaluate the methodological quality of health utility studies or their applicability to decision contexts.

Objective

To describe how health utility concepts are reflected in tools intended for use with the health economic literature, particularly with respect to the evaluation of methodological quality and context applicability.

Methods

We reviewed the critical appraisal and reporting tools described in a 2012 report published by the Agency for Healthcare Research and Quality (AHRQ), supplemented with a keyword search of MEDLINE and EMBASE, to identify existing tools which include health utility constructs. From these tools, a list of relevant items was compiled and grouped into domain categories based on the methodological or applicability aspect they were directed toward.

Results

Of the 24 tools we identified, 12 contained items relevant to the evaluation of health utilities. Sixty-five items were considered relevant to the evaluation of quality, while 44 were relevant to the evaluation of applicability. Items were arranged into four domains: health state descriptions; selection and description of respondents; elicitation and measurement methods; and other considerations.

Conclusion

As key inputs to cost–utility analyses, health utilities have the potential to significantly impact estimates of cost-effectiveness. Existing tools contain only general items related to the conduct or use of health utility studies. There is a need to develop tools that systematically evaluate the methodological quality and applicability of health utility studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Klarman, H.E., Francis, J.O.S., Rosenthal, G.D.: Cost Effectiveness Analysis Applied to the Treatment of Chronic Renal Disease. Med. Care 6(1), 48–54 (1968)

    Article  Google Scholar 

  2. Bremner, K.E., Chong, C.A., Tomlinson, G., Alibhai, S.M., Krahn, M.D.: A review and meta-analysis of prostate cancer utilities. Med. Decis. Making 27(3), 288–298 (2007). https://doi.org/10.1177/0272989x07300604

    Article  PubMed  Google Scholar 

  3. Sturza, J.: A review and meta-analysis of utility values for lung cancer. Med. Decis. Making 30(6), 685–693 (2010). https://doi.org/10.1177/0272989x10369004

    Article  PubMed  Google Scholar 

  4. Paracha, N., Thuresson, P.O., Moreno, S.G., MacGilchrist, K.S.: Health state utility values in locally advanced and metastatic breast cancer by treatment line: a systematic review. Expert Rev. Pharmacoecon. Outcomes Res. 16(5), 549–559 (2016). https://doi.org/10.1080/14737167.2016.1222907

    Article  PubMed  Google Scholar 

  5. Peasgood, T., Ward, S.E., Brazier, J.: Health-state utility values in breast cancer. Expert Rev. Pharmacoecon. Outcomes Res. 10(5), 553–566 (2010). https://doi.org/10.1586/erp.10.65

    Article  PubMed  Google Scholar 

  6. Hao, Y., Wolfram, V., Cook, J.: A structured review of health utility measures and elicitation in advanced/metastatic breast cancer. Clin. Econ. Outcomes Res. CEOR 8, 293–303 (2016). https://doi.org/10.2147/ceor.s100448

    Article  Google Scholar 

  7. Schiller-Fruhwirth, I.C., Jahn, B., Arvandi, M., Siebert, U.: Cost-effectiveness models in breast cancer screening in the general population: a systematic review. Appl. Health Econ. Health Policy 15(3), 333–351 (2017). https://doi.org/10.1007/s40258-017-0312-3

    Article  PubMed  Google Scholar 

  8. Carter, G.C., King, D.T., Hess, L.M., Mitchell, S.A., Taipale, K.L., Kiiskinen, U., Rajan, N., Novick, D., Liepa, A.M.: Health state utility values associated with advanced gastric, oesophageal, or gastro-oesophageal junction adenocarcinoma: a systematic review. J. Med. Econ. 18(11), 954–966 (2015). https://doi.org/10.3111/13696998.2015.1066380

    Article  PubMed  Google Scholar 

  9. Djalalov, S., Rabeneck, L., Tomlinson, G., Bremner, K.E., Hilsden, R., Hoch, J.S.: A review and meta-analysis of colorectal cancer utilities. Med. Decis. Making 34(6), 809–818 (2014). https://doi.org/10.1177/0272989x14536779

    Article  PubMed  Google Scholar 

  10. Jeong, K., Cairns, J.: Systematic review of health state utility values for economic evaluation of colorectal cancer. Heal. Econ. Rev. 6(1), 36 (2016). https://doi.org/10.1186/s13561-016-0115-5

    Article  Google Scholar 

  11. Richardson, J., Khan, M.A., Iezzi, A., Maxwell, A.: Comparing and explaining differences in the magnitude, content, and sensitivity of utilities predicted by the EQ-5D, SF-6D, HUI 3, 15D, QWB, and AQoL-8D multiattribute utility instruments. Med. Decis. Making 35(3), 276–291 (2015). https://doi.org/10.1177/0272989x14543107

    Article  PubMed  Google Scholar 

  12. Dolan, P.: Modeling valuations for EuroQol health states. Med. Care 35(11), 1095–1108 (1997). https://doi.org/10.1097/00005650-199711000-00002

    Article  CAS  PubMed  Google Scholar 

  13. Mulhern, B.J., Bansback, N., Norman, R., Brazier, J.: Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration. Med. Care 58(6), 566–573 (2020). https://doi.org/10.1097/mlr.0000000000001324

    Article  PubMed  Google Scholar 

  14. Galante, J., Augustovski, F., Colantonio, L., Bardach, A., Caporale, J., Marti, S.G., Kind, P.: Estimation and comparison of EQ-5D health states' utility weights for pneumoccocal and human papillomavirus diseases in Argentina, Chile, and the United Kingdom. Value Health 14(5, Supplement), S60–S64 (2011). https://doi.org/10.1016/j.jval.2011.05.007

  15. Takemoto, M.L., Lopes da Silva, N., Ribeiro-Pereira, A.C., Schilithz, A.O., Suzuki, C.: Differences in utility scores obtained through Brazilian and UK value sets: a cross-sectional study. Health Qual. Life Outcomes 13, 119 (2015). https://doi.org/10.1186/s12955-015-0318-1

    Article  PubMed  PubMed Central  Google Scholar 

  16. Pollard, C., Hartz, S., Leage, S.L., Paget, M.A., Cook, J., Enstone, A.: Elicitation of health state utilities associated with varying severities of flares in systemic lupus erythematosus. Health Qual. Life Outcomes 13, 66 (2015). https://doi.org/10.1186/s12955-015-0262-0

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Brazier, J., Ara, R., Azzabi, I., Busschbach, J., Chevrou-Severac, H., Crawford, B., Cruz, L., Karnon, J., Lloyd, A., Paisley, S., Pickard, A.S.: Identification, review, and use of health state utilities in cost-effectiveness models: an ISPOR Good practices for outcomes research task force report. Value Health 22(3), 267–275 (2019). https://doi.org/10.1016/j.jval.2019.01.004

    Article  PubMed  Google Scholar 

  18. Xie, F., Zoratti, M., Chan, K., Husereau, D., Krahn, M., Levine, O., Clifford, T., Schunemann, H., Guyatt, G.: Toward a centralized, systematic approach to the identification, appraisal, and use of health state utility values for reimbursement decision making: introducing the health utility book (HUB). Med. Decis. Making 39(4), 370–378 (2019). https://doi.org/10.1177/0272989x19837969

    Article  PubMed  Google Scholar 

  19. Walker, D.G., Wilson, R.F., Sharma, R., Bridges, J., Niessen, L., Bass, E.B., Frick, K.: Best practices for conducting economic evaluations in health care: a systematic review of quality assessment tools. AHRQ Publication No. 12(13)-EHC132-EF. Agency for Healthcare Research and Quality (2012)

  20. Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A.H., Mauskopf, J., Loder, E.: Consolidated health economic evaluation reporting standards (CHEERS)–explanation and elaboration: a report of the ISPOR health economic evaluation publication guidelines good reporting practices task force. Value Health 16(2), 231–250 (2013). https://doi.org/10.1016/j.jval.2013.02.002

    Article  PubMed  Google Scholar 

  21. Evers, S., Goossens, M., de Vet, H., van Tulder, M., Ament, A.: Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. Int. J. Technol. Assess. Health Care 21(2), 240–245 (2005)

    Article  Google Scholar 

  22. Nerich, V., Saing, S., Gamper, E.M., Holzner, B., Pivot, X., Viney, R., Kemmler, G.: Critical appraisal of health-state utility values used in breast cancer-related cost-utility analyses. Breast Cancer Res. Treat. (2017). https://doi.org/10.1007/s10549-017-4283-8

    Article  PubMed  Google Scholar 

  23. Chiou, C.F., Hay, J.W., Wallace, J.F., Bloom, B.S., Neumann, P.J., Sullivan, S.D., Yu, H.T., Keeler, E.B., Henning, J.M., Ofman, J.J.: Development and validation of a grading system for the quality of cost-effectiveness studies. Med. Care 41(1), 32–44 (2003). https://doi.org/10.1097/01.mlr.0000039824.73620.e5

    Article  PubMed  Google Scholar 

  24. Simoens, S.: Assessment of methodological quality of economic evaluations in belgian drug reimbursement applications. PLoS One 8(12), e85411 (2013). https://doi.org/10.1371/journal.pone.0085411

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Stalmeier, P.F., Goldstein, M.K., Holmes, A.M., Lenert, L., Miyamoto, J., Stiggelbout, A.M., Torrance, G.W., Tsevat, J.: What should be reported in a methods section on utility assessment? Med. Decis. Making 21(3), 200–207 (2001)

    Article  CAS  Google Scholar 

  26. Petrou, S., Rivero-Arias, O., Dakin, H., Longworth, L., Oppe, M., Froud, R., Gray, A.: Preferred reporting items for studies mapping onto preference-based outcome measures: the MAPS statement. Pharmacoeconomics 33(10), 985–991 (2015). https://doi.org/10.1007/s40273-015-0319-2

    Article  PubMed  PubMed Central  Google Scholar 

  27. Xie, F., Pickard, A.S., Krabbe, P.F., Revicki, D., Viney, R., Devlin, N., Feeny, D.: A checklist for reporting valuation studies of multi-attribute utility-based instruments (CREATE). Pharmacoeconomics 33(8), 867–877 (2015). https://doi.org/10.1007/s40273-015-0292-9

    Article  PubMed  PubMed Central  Google Scholar 

  28. Brazier, J., Deverill, M., Green, C.: A review of the use of health status measures in economic evaluation. J. Health Serv. Res. Policy 4(3), 174–184 (1999). https://doi.org/10.1177/135581969900400310

    Article  CAS  PubMed  Google Scholar 

  29. Drummond, M.F., Jefferson, T.O.: Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ 313(7052), 275–283 (1996)

    Article  CAS  Google Scholar 

  30. Ungar, W.J., Santos, M.T.: The pediatric quality appraisal questionnaire: an instrument for evaluation of the pediatric health economics literature. Value Health 6(5), 584–594 (2003). https://doi.org/10.1046/j.1524-4733.2003.65253.x

    Article  PubMed  Google Scholar 

  31. Clemens, K., Townsend, R., Luscombe, F., Mauskopf, J., Osterhaus, J., Bobula, J.: Methodological and conduct principles for pharmacoeconomic research. Pharmaceutical Research and Manufacturers of America. PharmacoEconomics 8(2), 169–174 (1995)

    Article  CAS  Google Scholar 

  32. Adams, M.E., McCall, N.T., Gray, D.T., Orza, M.J., Chalmers, T.C.: Economic analysis in randomized control trials. Med. Care 30(3), 231–243 (1992)

    Article  CAS  Google Scholar 

  33. Gerard, K.: Cost-utility in practice: a policy maker’s guide to the state of the art. Health Policy 21(3), 249–279 (1992)

    Article  CAS  Google Scholar 

  34. Sacristan, J.A., Soto, J., Galende, I.: Evaluation of pharmacoeconomic studies: utilization of a checklist. Ann. Pharmacother. 27(9), 1126–1133 (1993). https://doi.org/10.1177/106002809302700919

    Article  CAS  PubMed  Google Scholar 

  35. Drummond, M., Manca, A., Sculpher, M.: Increasing the generalizability of economic evaluations: recommendations for the design, analysis, and reporting of studies. Int. J. Technol. Assess. Health Care 21(2), 165–171 (2005)

    Article  Google Scholar 

  36. Ramsey, S., Willke, R., Briggs, A., Brown, R., Buxton, M., Chawla, A., Cook, J., Glick, H., Liljas, B., Petitti, D., Reed, S.: Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA Task Force report. Value Health 8(5), 521–533 (2005). https://doi.org/10.1111/j.1524-4733.2005.00045.x

    Article  PubMed  Google Scholar 

  37. Goetghebeur, M.M., Wagner, M., Khoury, H., Levitt, R.J., Erickson, L.J., Rindress, D.: Evidence and value: Impact on decisionMaking–the EVIDEM framework and potential applications. BMC Health Serv. Res. 8, 270 (2008). https://doi.org/10.1186/1472-6963-8-270

    Article  PubMed  PubMed Central  Google Scholar 

  38. Davis, J.C., Robertson, M.C., Comans, T., Scuffham, P.A.: Guidelines for conducting and reporting economic evaluation of fall prevention strategies. Osteoporos Int. 22(9), 2449–2459 (2011). https://doi.org/10.1007/s00198-010-1482-0

    Article  CAS  PubMed  Google Scholar 

  39. Vintzileos, A.M., Beazoglou, T.: Design, execution, interpretation, and reporting of economic evaluation studies in obstetrics. Am. J. Obstet. Gynecol. 191(4), 1070–1076 (2004). https://doi.org/10.1016/j.ajog.2004.05.021

    Article  PubMed  Google Scholar 

  40. Grutters, J.P., Seferina, S.C., Tjan-Heijnen, V.C., van Kampen, R.J., Goettsch, W.G., Joore, M.A.: Bridging trial and decision: a checklist to frame health technology assessments for resource allocation decisions. Value Health 14(5), 777–784 (2011). https://doi.org/10.1016/j.jval.2011.01.005

    Article  PubMed  Google Scholar 

  41. Russell, L.B., Gold, M.R., Siegel, J.E., Daniels, N., Weinstein, M.C.: The role of cost-effectiveness analysis in health and medicine. Panel on cost-effectiveness in health and medicine. Jama 276(14), 1172–1177 (1996)

    Article  CAS  Google Scholar 

  42. Siegel, J.E., Weinstein, M.C., Russell, L.B., Gold, M.R.: Recommendations for reporting cost-effectiveness analyses. Panel on cost-effectiveness in health and medicine. Jama 276(16), 1339–1341 (1996)

    Article  CAS  Google Scholar 

  43. Weinstein, M.C., Siegel, J.E., Gold, M.R., Kamlet, M.S., Russell, L.B.: Recommendations of the panel on cost-effectiveness in health and medicine. JAMA 276(15), 1253–1258 (1996)

    Article  CAS  Google Scholar 

  44. Sanders, G.D., Neumann, P.J., Basu, A., Brock, D.W., Feeny, D., Krahn, M., Kuntz, K.M., Meltzer, D.O., Owens, D.K., Prosser, L.A., Salomon, J.A., Sculpher, M.J., Trikalinos, T.A., Russell, L.B., Siegel, J.E., Ganiats, T.G.: Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA 316(10), 1093–1103 (2016). https://doi.org/10.1001/jama.2016.12195

    Article  PubMed  Google Scholar 

  45. Higgins, J.P.T., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savović, J., Schulz, K.F., Weeks, L., Sterne, J.A.C.: The Cochrane Collaboration’s tool for assessing risk of bias in randomized trials. BMJ 343 (2011). https://doi.org/10.1136/bmj.d5928

  46. Mokkink, L.B., Terwee, C.B., Patrick, D.L., Alonso, J., Stratford, P.W., Knol, D.L., Bouter, L.M., de Vet, H.C.: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality Life Res. 19(4), 539–549 (2010). https://doi.org/10.1007/s11136-010-9606-8

    Article  Google Scholar 

Download references

Funding

No funding was received for this research.

Author information

Authors and Affiliations

  1. Department of Health Research Methodology, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

    Michael J. Zoratti, Mitchell Levine, Lehana Thabane & Feng Xie

  2. Department of Pharmacy Systems, Outcomes and Policy, College of Pharmacy, The University of Illinois at Chicago, Chicago, IL, USA

    A. Simon Pickard

  3. Health Evidence, Radboud University Medical Centre, Nijmegen, The Netherlands

    Peep F. M. Stalmeier

  4. Center for the Evaluation of Value and Risk in Health, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, USA

    Daniel Ollendorf

  5. Acaster Lloyd Consulting Ltd, London, UK

    Andrew Lloyd

  6. Sunnybrook Odette Cancer Centre, Toronto, ON, Canada

    Kelvin K W Chan

  7. Canadian Centre for Applied Research in Cancer Control, Toronto, Canada

    Kelvin K W Chan

  8. School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada

    Don Husereau

  9. School of Health and Related Research, University of Sheffield, Sheffield, UK

    John E. Brazier

  10. Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada

    Murray Krahn

  11. Department of Medicine, Division of Clinical Pharmacology and Toxicology, McMaster University, Hamilton, ON, Canada

    Mitchell Levine

  12. St Joseph’s Healthcare, Hamilton, ON, Canada

    Lehana Thabane

Authors
  1. Michael J. Zoratti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Simon Pickard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peep F. M. Stalmeier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Ollendorf
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrew Lloyd
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kelvin K W Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Don Husereau
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. John E. Brazier
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Murray Krahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Mitchell Levine
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Lehana Thabane
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Feng Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

Health Utility Book (HUB) Working Group

Corresponding author

Correspondence to Feng Xie.

Ethics declarations

Conflicts of interest

None to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 59 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zoratti, M.J., Pickard, A.S., Stalmeier, P.F.M. et al. Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists. Eur J Health Econ 22, 723–733 (2021). https://doi.org/10.1007/s10198-021-01286-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10198-021-01286-0

Keywords

Search

Navigation