Quality of care associated with number of cases seen and self-reports of clinical competence for Japanese physicians-in-training in internal medicine

Sample

An anonymous survey was administered in departments of internal medicine at 11 teaching hospitals in Japan, a convenience sample from 6 university-affiliated and 5 non-affiliated teaching hospitals (with oversampling of university settings). All physicians-in-training within 10 years of graduation from medical school were eligible for the study (n = 375). To avoid contamination, the survey was administered on the same day (03/14/2003) in all institutions.

Measurement method

Quality of care was defined as the delivery of patient care in a manner that leads to better outcomes for individuals and populations [8]. Clinical vignettes have been used to measure variations in quality of care [9]. The scores derived from the vignettes reliably reflected actual levels of physician practice, resulting in higher criterion validity compared to scores derived from chart abstractions. Based on disease prevalence in Japan, we began by selecting six clinical vignettes to measure quality of care: four common outpatient chronic conditions (diabetes mellitus, chronic obstructive pulmonary disease, vascular disease, and depression) and two acute emergency room conditions (subarachnoid hemorrhage and gastrointestinal bleeding).

Two detailed clinical vignettes were developed for each chronic condition, for a total of 8 vignettes. These vignettes were originally developed to measure quality of care in the United States. The vignettes were translated in Japanese and partly revised to match clinical practice in Japan, for example, using equivalent drugs and screening procedures. In addition, we developed two original Japanese vignettes for the two acute conditions. From the 10 vignettes available, each participant received five randomly selected vignettes, one from each condition (4 chronic and 1 acute). The vignettes required open-ended responses to questions that were presented in sections characteristic of a typical patient encounter: presenting complaint, history, physical examination, radiological or laboratory tests, diagnosis, and treatment and management plans. Each section began with the presentation of new information. After answering a given section, participants could not return to previous sections to revise (possibly improve) their answers. Participants were given 85 minutes to complete all 5 vignettes.

Clinical exposure was measured using participant self-estimates of the number of patients seen in in-patient wards, outpatient clinics, and emergency rooms. Data was also collected on the number of years after graduation, type of institution (university-affiliated teaching hospitals or non-affiliated), self-reports of clinical competence (i.e., problem-solving ability, basic procedural skills [e.g., venipuncture, bone marrow aspiration], and basic medical knowledge), and communication ability (i.e., attitude toward patients and their family and cooperativeness with other medical staff). Self-reports were rated using a five-point ordinal rating scale (i.e., unsatisfactory, satisfactory, good, excellent, or outstanding). The overall model consisted of one quality-of-care outcome variable, portrayed by the vignettes, and four predictor variables, that is, self-estimates of total number of patients seen, type of institution, and self-reports of clinical competence and communication ability. The latter two variables were also summed to create a global self-reported competence variable.

Scoring

The responses to the vignettes were scored by the authors. To ensure consistency in scoring, given conditions were scored by the same author. With regard to chronic conditions, we used the scoring criteria developed by the original American authors who based their criteria on national guidelines [9]. These criteria were then reviewed and ratified by expert panels of academic and community physicians in Japan, in fields relevant to each condition; in the end the original criteria were adopted. Scoring criteria for the acute conditions were developed de novo, using expert panels of Japanese physicians. To verify the equivalence of the Japanese version with the original English version, the 10 vignettes were back-translated into English and verified by the original American authors. Based on their recommendations and consensus among the authors, the vignettes and scoring criteria were finalized. Each vignette contained an average of 37 criteria (range = 26–50). Each criterion was rated according to a three-level quality-of-care scale: adequate, unnecessary, and inappropriate care. A one-point credit was assigned for each criterion when adequate care was proposed. An overall vignette score was assigned by summing the scores from the individual criteria.

First, we used the general linear model to test for a vignette (disease condition) effect; there was no such effect (p = 0.239). Thus the scores from all five vignettes were added for each participant and then converted to a standardized t-score with a mean of 50 and a standard deviation of 10; t-scores were used as the outcome (criterion) variable for quality of care in the analyses. This transformation facilitated the interpretation of the relative importance of the predictor variables by comparing the corresponding t-scores to means of 50.

Predictor variables

The amount of clinical exposure was computed by adding the number of cases that each subject had seen in each setting (inpatient, outpatient and ER) and then scored according to five ordinal categories: 0–100, 201–300, 301–400, 401–600, 601–800, >800 cases. The data distribution was skewed to the left and consequently ordinal categories were used because they fit the model better than log transformations (based on Akaike's information criteria – AIC [10]). We also used a broader range (200 vs. 100) for numbers above 401 because the data were skewed and sparse in those categories. The proportion of cases in the in-patient setting was also calculated and incorporated into the analyses because training occurs mostly in in-patient settings in Japan. The number of years after graduation could influence the amount of clinical exposure and was thus incorporated into the analysis using three ordinal categories (because again they fit the model better than log transformations): one year after graduation (PGY1); two years after graduation (PGY2); and more than 3 years after graduation (≥PGY3).

In addition to examining the relationship between overall clinical exposure and t-scores, we also looked specifically at the exposure to disorders similar to the ones in the vignettes. This was measured using a common disease index (CDI), defined as (numerator:) the number of cases seen that were similar to the diseases in the vignettes (i.e., stroke, gastrointestinal bleeding, COPD, heart failure, ischemic heart disease, depression, and diabetes mellitus), divided by (denominator:) the total number of cases seen. The t-scores were plotted against CDI to interpret graphically the relationship between the CDI and t-scores. (a measure of quality of care). We also examined the relationship between CDI and quality of care, adjusting for overall clinical exposure in order to verify whether simply increasing the proportion of clinical exposure to similar disorders would lead to higher quality for a fixed amount of exposure.

Clinical competence was defined as the sum of the self-assessed ratings for the three elements of competence (i.e., problem-solving ability, basic procedural skills, and basic medical knowledge). CDI and clinical competence scores were each further divided into three-level variables: low (up to 33^rd percentile), middle (up to 67^th percentile), and high (greater than 67^th percentile). The maximum score within each level of competence was 15, that is, the sum of 5 points maximum for each element of competence (e.g., problem solving, procedural skills, and knowledge).

Quality of care can vary depending on the type of institution [9], and thus it was also incorporated into the analyses. Two types of teaching hospitals were included: university-affiliated and non-university-affiliated (community) teaching hospitals. While this variable was included in the analyses, specific nominal results about this factor are not reported because some hospitals did not consent to revealing type of institution.

Analyses

Descriptive statistics included rates and proportions for categorical data and means and standard deviations (SD) for continuous data. We first performed univariate analyses to evaluate the relationship between predictor variables and quality of care. Analysis of covariance or pooled t-test was used for categorical data. Pearson or Spearman correlation coefficients were used for continuous data.

Multivariate linear regression models were then constructed to examine the association between clinical exposure and quality of care. We incorporated all predictor variables into the model because all of the variables were thought to be important factors that could potentially be associated with levels of quality of care. We tested the interaction between the amount of clinical exposure and type of institution and self-reports of clinical competence as well as for a case (vignette type) main effect. For all analyses, alpha was set at 0.05. Analyses were done using commercially available software (Intercooled STATA 8.0; STATA Corporation, TX, USA). Ethics approval was granted for this study by the Kyoto University Faculty of Medicine Institutional Review Board.