Evaluating the effectiveness of a radiation safety training intervention for oncology nurses: a pretest – intervention – posttest study

Design

Pretest-intervention-posttest designs are uniquely appropriate for investigating the effects of educational innovations[7] and are commonly used in educational research [8–10]. Strict experimental designs suggest the use of a two-group pretest-intervention-posttest design with a control group that receives no training intervention and a group that receives the training. In the occupational training environment required to meet regulatory compliance requirements, it is often not possible to allow differential services (i.e. different levels of training) for the staff. This was the case for radiation safety training at our cancer center where it was decided not to have a control group because the withholding of training from the control group would represent a differential service inconsistent with a worker's 'right to know' about the potential hazards. In this case, a one-group (no control) pretest-intervention-posttest experimental design was utilized. The absence of a control group was not considered a significant threat to the internal validity of the experiment because the likelihood that extraneous factors account for the change was small[5]. It was assumed that in the absence of the intervention (i.e. the specific radiation safety training programs) there were minimal or nonexistent outside variables that would have significantly changed a nurse's cognitive knowledge or attitude with regard to radiation safety regulations and policies over the pretest to posttest interval timeframes. It was concluded that the use of an experimental design without a control group was justified.

Setting

The study was carried out in 2004 for inpatient staff registered nurses working with oncology patients at MSKCC, a National Cancer Institute designated comprehensive cancer center in New York City.

Sample

All inpatient staff registered nurses (750 nurses) were considered as the study population. Participants were recruited by nurse educators and nurse leaders. A total of 15% of the registered nurses (i.e. 113 nurses) completed the study pretests and posttests. All nurses received the same training information.

Intervention

The intervention consisted of a multifaceted set of improvements (Figure 1) including: nursing procedure revisions, a core concepts video, two types of inservice training, and enhanced "radiation precaution" signs and labels.

Radiation related nursing procedures were revised to maintain a consistent format and were edited to include only information that was considered to be essential information for nursing care situations. The procedures were validated by nursing and radiation safety experts for both scope and clarity.

A twelve-minute digital video was developed for incorporation into nursing orientation training and annual mandatory training for all nurses. Video topics were selected specifically to address the perceived knowledge gaps as well as to directly address fears of radiation. The video cast included a narrator, a professional male actor chosen by the nurse members of the multidisciplinary team, and representatives of the nursing and radiation safety staff community depicting various radiation safety precaution actions and discussions. An effort was made to include as many staff members as possible to heighten interest in the movie, help validate the material, and instill a sense of ownership. The video was digitized to facilitate presentation during training workshops as well as making the video available on the MSKCC intranet. Hyperlinks from the nursing procedures website allowed nurses to review the video at any time from virtually anywhere in the hospital.

Two types of inservice training sessions were held. The first was an interactive, hands-on workshop developed specifically for nursing leadership (managers, clinical nurse specialists, nurse practitioners, and nurse educators) with the goal of increasing their ability to problem solve in the area of radiation and radioactive precautions. The second was a didactic session for nursing staff on units that regularly housed patients requiring radioactive precautions. This inservice focused on the various radiation treatments and the associated precautions most likely to be experienced by nurses.

Highly visible "radiation precaution" door signs and chart labels were developed to specifically detail the required actions by nursing staff for maintaining the safety of patients and hospital staff while providing care for patients. The improved signs and labels better differentiated among the precautions required for permanent implants, temporary implants, and radiopharmaceutical therapy than the signs previously used.

Instruments

Two instruments were designed for this evaluation. The first was designed to measure cognitive knowledge of radiation and radiation protection practices. The second was designed to measure attitudes of nurses with regard to radiation.

The present research relied upon a criteria-based multiple-choice/true-false test of cognitive knowledge [11–16]. The same instrument was utilized for both pretest and posttest evaluations. The cognitive test included 15 questions with 4 choices for each question. Face validity for the knowledge instrument was assessed by a local group radiation safety specialists and nursing leaders. Each question was scrutinized to ensure that it represented an accurate measure of desired parameters. Content validity was assessed through the use of radiation safety subject matter experts to ensure that proper topical coverage had been afforded by the overall test. [See additional file 1].

The cognitive questions addressed the following areas of knowledge: background radiation dose, annual limit, when to wear a badge, how to find exposure records, declared pregnant limit, radiosensitivity of the fetus, external beam treatment, seed implant treatment, visitor precautions, temporary implant treatment, permanent implant treatment, systemic radioiodine precautions, systemic radioiodine contamination, monitoring patient dose rate, and contamination cleanup protocols.

A Likert-scaled attitude evaluation included 9 questions. Face validity was assessed by a local group of radiation safety specialists and nursing leaders. [See additional file 1]. The instrument addressed the following attitudinal areas: I feel that radiation safety policies are clear, I know whom to contact for information, I know what steps to take, I can explain precautions well, I feel safe, policies are based on regulations, I am monitored, there is oversight, and I feel safe to have a child.

Data analysis

Data were captured from the response sheets using the ReMark Office OMR software[17]. Two levels of data analysis were performed on the aggregate of pretest and posttest responses. The first used descriptive statistics and the second employed hypothesis evaluation statistics. The significance alpha level was chosen for all statistical tests to be 0.05, the most typical value for social research[6, 18].

Descriptive findings

The frequency distribution and the cumulative description of the cognitive test scores are shown in Figures 2 and 3. Figure 2 displays the results of the pretest which show a typical Gaussian result with scores centered around 60%. Figure 3 displays the results of the posttest and demonstrates scores skewed to the right (i.e. shifted toward higher scores).

To assist in describing the attitude evaluation responses, a weighted average Likert score was calculated for the pretest and posttest attitude questions (Figure 4). For 8 out of 9 of the questions, the attitudes were generally positive, i.e. the weighted Likert score was greater than 3, for both the pretest and posttest responses. Responses to the question on oversight were generally negative, (i.e. the weighted Likert score was less than 3, for both the pretest and posttest. Figure 4 also shows that the weighted Likert score was higher in all cases for the posttest evaluation when compared to the lower scores in the pretest evaluation.

Hypothesis evaluation findings

There was a significant relationship between the nurses' attitudes toward the radiation safety program and participation in the intervention program for the attitude survey questions concerning the following: I feel that radiation safety policies are clear, I know what steps to take, I can explain precautions well, I feel safe, and policies are based on regulations. Although the weighted average Likert score was higher in the posttests, the differences in pretest and posttest responses were not statistically significant for the following attitude survey questions: I know whom to contact for information, I am monitored, there is oversight, and I feel safe to have a child.

Recommendations for general practice

The training intervention has been incorporated into ongoing training programs. In addition, several strengths of the program can be adopted for use in other training program improvements. The involvement of stakeholders was essential for the development of the training intervention as well as the development and implementation of the research design. The use of a multidisciplinary team for the development of the training intervention elements resulted in high quality training tools that have been shown to be successful with the target audience.

The attitude evaluation results demonstrated that four areas (I know whom to contact for information, I am monitored, there is oversight, and I feel safe to have a child) should be emphasized in future radiation training intervention implementations. The training elements focused strongly on cognitive knowledge, with the assumption that an increase in knowledge would result in a concomitant improvement of attitudes. It may be possible to develop an additional training element that specifically addresses underlying assumptions and fears. Such an intervention might utilize open discussions or hands-on approaches. The addition of a behavioral psychologist to the multidisciplinary team may improve the outcome.

Recommendations for future research

Further research is needed to focus on specifics. While this study evaluated the impact of the entire multi-element training intervention, no conclusions could be drawn for the individual elements of the intervention such as the video, the inservice training sessions, the policies and procedure, or the door signs. Which of these intervention elements had the most impact on increasing cognitive knowledge and/or attitudes is conjectural. The present study was not stratified by age, experience level, gender, hospital unit, or floor, to see if we could identify a nursing sub-population which is more difficult to educate.

Further research could broaden the study. While this study evaluated the performance and attitude of registered nurses on inpatient floors, a comparison of intervention results for other groups such as registered nurses versus nursing assistants may help to answer the question as to whether education level would affect outcomes. The study should also be extended to other practice settings.

While the present study utilized a specific set of attitude evaluation questions that concentrated on what the multidisciplinary team believed represented appropriate concerns of oncology nurses, all of the specific fears of the nurses were not requested or learned during this study. An expanded attitude evaluation survey and questionnaire could be beneficial in identifying real fears with respect to radiation. The intervention elements themselves could then be modified to include other methods designed to specifically address identified fears and/or misconceptions.