Studying the Effects of ACGME Duty Hours Limits on Resident Satisfaction
Background As the Accreditation Council on Graduate Medical Education (ACGME) deliberates over further limiting duty hours of graduate medical education (GME) trainees, few large-scale studies have shown residents to be satisfied with the effect the 2003 standards have had on clinical care, education outcomes, or working environments. This study measures the effect of the 2003 duty hours limits on resident-reported satisfaction with GME training during their rotations through the Department of Veterans Affairs (VA) medical centers from 2001 through 2007.
Method Self-reported satisfaction with clinical care and education environments were assessed by comparing responses to VA's annual Learners' Perceptions Survey administered before 2003 with responses administered after 2003. To measure duty hours effects on satisfaction, before–after differences were adjusted for covariate biases modeled after an exhaustive covariate search with 10-fold cross-validation. Because nonteaching controls are not available in satisfaction studies, we used a robust differencing variable technique to control before–after differences for trend biases in the simultaneous presence of missing data and possible model misspecification.
Results There were 19,605 responders. Adjusting for covariate and trend biases, after the 2003 ACGME standards, 25% more residents in medicine specialties reported satisfaction with VA clinical environment and 11% more with VA preceptors and faculty. For surgery, 33% more residents reported satisfaction with VA clinical environment and 12% more with VA preceptors and faculty. Satisfaction with working environment was mixed.
Conclusions The 2003 ACGME duty hours standards were associated with improved satisfaction for resident clinical training and learning environments.
Standards governing duty hours limits have generally been considered necessary in graduate medical education (GME) to protect the safety of both patients and residents. Resident sleep deprivation as a result of long duty hours has been linked to higher rates of medical errors, poorer clinical performance, adverse events, and attentional failures in observational, pre–post, and experimental studies. Longer duty hours have also been linked to resident motor-vehicle-related injuries, obstetric complications, depression, burnout, poorer quality of life and neuropsychological performance, including memory loss and reduced response times.
In response to these safety concerns, the Accreditation Council for Graduate Medical Education (ACGME) implemented mandatory standards on July 1, 2003, that limited duty hours for medical residents in accredited U.S. GME programs. Although benefits from ACGME duty hours limits continue to be debated, few studies have described how duty hours limits may be affecting clinical training environments, trainee learning, resident access to preceptors and faculty, and resident education. For instance, residents have complained that mandatory duty hours rules interfere with continuity of care, increase cross-coverage errors, shift the education focus away from professionalism, create fear that new regulations will add additional training years, and cause frustration when residents are faced with heavy workloads and must reconcile actual hours against ACGME duty hours rules. Underscoring these concerns are the contrasting missions of the teaching hospitals, who need staff to provide professional care; faculty, who balance attending, practice, service, and research responsibilities; and residents, who need access to faculty and supervised clinical experiences to properly prepare them to enter independent practice.
To assess the effects of ACGME duty hours rules on training environments, researchers have surveyed residents using post and pre–post survey designs. Post surveys were administered after ACGME duty hours rules became mandatory. These surveys asked residents and fellows about their views of the success or failure of the mandatory standards. Although informative about how residents perceived duty hours limits, postsurvey results are often colored by memory loss, cohort confounds when all of the responders who have prelimits experiences have become upper-level residents by the time the survey is administered, and reporting biases when residents mimic faculty attitudes and beliefs in their survey responses.
Pre–post designs compare responses to surveys that were administered in 2003 and earlier with responses to the same surveys readministered in 2004 and later. Pre–post designs are subject to covariate biases whenever responders who took the survey prelimits (2003 and earlier) differ significantly from responders who took the survey postlimits (2004 and later). Pre–post designs are also subject to trend biases whenever naturally occurring time trends in the data confound pre–post differences. Covariate biases have been addressed by computing outcomes that are adjusted to reflect the influences of variations in responder characteristics. Trend biases are addressed by difference-in-differences methods where effect sizes are computed by subtracting the pre–post difference in mean responses among physician residents rotating through "effect" settings minus the pre–post difference in mean responses computed for comparable residents who rotated through "control" settings. Control settings have been identified as (1) nonteaching hospitals where duty hours limits are irrelevant, (2) training programs in teaching hospitals where duty hours limits were openly not enforced, or (3) responders whose duty schedules were not changed, for whatever reason, by duty hours limits. Facility-level controls are limited to outcomes that can be observed in both teaching and nonteaching settings, such as patient outcomes and medical errors, and are thus not practical for resident satisfaction surveys. Program-level controls are often difficult to implement because few program directors openly defy ACGME standards. Responder-level controls can be identified by asking respondents if the 2003 duty hours limits had any impact on their actual duty schedules. However, such questions were not answerable before 2003, when ACGME duty hours limits were first implemented. We call this the "missing-data problem."
For this report, we introduce and apply a methodology that uses responder-level controls to assess the influence of the 2003 mandatory ACGME duty hours limits on how physician residents perceived their clinical training environments in the Department of Veterans Affairs (VA) medical centers between July 1, 2000 and June 30, 2007. The study addresses covariate confounds, trend biases, and missing-data problems in three important aspects. First, we used the Learners' Perceptions Survey (LPS), a structured interview administered annually by the VA Office of Academic Affiliations (VA-OAA) to residents rotating through VA medical centers. Second, respondents were classified into effects or control groups based on LPS survey questions that asked respondents whether duty hours limits actually changed their hours worked during scheduled VA rotations. Third, we adjusted for covariate and trend biases using a robust differencing variable technique, an advanced statistical method designed to handle the missing-data problem caused by failing to identify controls among pre-2003 responders.
Abstract and Introduction
Abstract
Background As the Accreditation Council on Graduate Medical Education (ACGME) deliberates over further limiting duty hours of graduate medical education (GME) trainees, few large-scale studies have shown residents to be satisfied with the effect the 2003 standards have had on clinical care, education outcomes, or working environments. This study measures the effect of the 2003 duty hours limits on resident-reported satisfaction with GME training during their rotations through the Department of Veterans Affairs (VA) medical centers from 2001 through 2007.
Method Self-reported satisfaction with clinical care and education environments were assessed by comparing responses to VA's annual Learners' Perceptions Survey administered before 2003 with responses administered after 2003. To measure duty hours effects on satisfaction, before–after differences were adjusted for covariate biases modeled after an exhaustive covariate search with 10-fold cross-validation. Because nonteaching controls are not available in satisfaction studies, we used a robust differencing variable technique to control before–after differences for trend biases in the simultaneous presence of missing data and possible model misspecification.
Results There were 19,605 responders. Adjusting for covariate and trend biases, after the 2003 ACGME standards, 25% more residents in medicine specialties reported satisfaction with VA clinical environment and 11% more with VA preceptors and faculty. For surgery, 33% more residents reported satisfaction with VA clinical environment and 12% more with VA preceptors and faculty. Satisfaction with working environment was mixed.
Conclusions The 2003 ACGME duty hours standards were associated with improved satisfaction for resident clinical training and learning environments.
Introduction
Standards governing duty hours limits have generally been considered necessary in graduate medical education (GME) to protect the safety of both patients and residents. Resident sleep deprivation as a result of long duty hours has been linked to higher rates of medical errors, poorer clinical performance, adverse events, and attentional failures in observational, pre–post, and experimental studies. Longer duty hours have also been linked to resident motor-vehicle-related injuries, obstetric complications, depression, burnout, poorer quality of life and neuropsychological performance, including memory loss and reduced response times.
In response to these safety concerns, the Accreditation Council for Graduate Medical Education (ACGME) implemented mandatory standards on July 1, 2003, that limited duty hours for medical residents in accredited U.S. GME programs. Although benefits from ACGME duty hours limits continue to be debated, few studies have described how duty hours limits may be affecting clinical training environments, trainee learning, resident access to preceptors and faculty, and resident education. For instance, residents have complained that mandatory duty hours rules interfere with continuity of care, increase cross-coverage errors, shift the education focus away from professionalism, create fear that new regulations will add additional training years, and cause frustration when residents are faced with heavy workloads and must reconcile actual hours against ACGME duty hours rules. Underscoring these concerns are the contrasting missions of the teaching hospitals, who need staff to provide professional care; faculty, who balance attending, practice, service, and research responsibilities; and residents, who need access to faculty and supervised clinical experiences to properly prepare them to enter independent practice.
To assess the effects of ACGME duty hours rules on training environments, researchers have surveyed residents using post and pre–post survey designs. Post surveys were administered after ACGME duty hours rules became mandatory. These surveys asked residents and fellows about their views of the success or failure of the mandatory standards. Although informative about how residents perceived duty hours limits, postsurvey results are often colored by memory loss, cohort confounds when all of the responders who have prelimits experiences have become upper-level residents by the time the survey is administered, and reporting biases when residents mimic faculty attitudes and beliefs in their survey responses.
Pre–post designs compare responses to surveys that were administered in 2003 and earlier with responses to the same surveys readministered in 2004 and later. Pre–post designs are subject to covariate biases whenever responders who took the survey prelimits (2003 and earlier) differ significantly from responders who took the survey postlimits (2004 and later). Pre–post designs are also subject to trend biases whenever naturally occurring time trends in the data confound pre–post differences. Covariate biases have been addressed by computing outcomes that are adjusted to reflect the influences of variations in responder characteristics. Trend biases are addressed by difference-in-differences methods where effect sizes are computed by subtracting the pre–post difference in mean responses among physician residents rotating through "effect" settings minus the pre–post difference in mean responses computed for comparable residents who rotated through "control" settings. Control settings have been identified as (1) nonteaching hospitals where duty hours limits are irrelevant, (2) training programs in teaching hospitals where duty hours limits were openly not enforced, or (3) responders whose duty schedules were not changed, for whatever reason, by duty hours limits. Facility-level controls are limited to outcomes that can be observed in both teaching and nonteaching settings, such as patient outcomes and medical errors, and are thus not practical for resident satisfaction surveys. Program-level controls are often difficult to implement because few program directors openly defy ACGME standards. Responder-level controls can be identified by asking respondents if the 2003 duty hours limits had any impact on their actual duty schedules. However, such questions were not answerable before 2003, when ACGME duty hours limits were first implemented. We call this the "missing-data problem."
For this report, we introduce and apply a methodology that uses responder-level controls to assess the influence of the 2003 mandatory ACGME duty hours limits on how physician residents perceived their clinical training environments in the Department of Veterans Affairs (VA) medical centers between July 1, 2000 and June 30, 2007. The study addresses covariate confounds, trend biases, and missing-data problems in three important aspects. First, we used the Learners' Perceptions Survey (LPS), a structured interview administered annually by the VA Office of Academic Affiliations (VA-OAA) to residents rotating through VA medical centers. Second, respondents were classified into effects or control groups based on LPS survey questions that asked respondents whether duty hours limits actually changed their hours worked during scheduled VA rotations. Third, we adjusted for covariate and trend biases using a robust differencing variable technique, an advanced statistical method designed to handle the missing-data problem caused by failing to identify controls among pre-2003 responders.
SHARE
