Mohammed saqr, College of Medicine , Qassim University, E-learning unit
22 January 2009
The adaptive mastery testing (AMT) procedure developed by Kingsbury and Weiss (1979) is designed to make high-precision classifications concerning students’ mastery of specific content areas within a course of instruction. The procedure is also intended to minimize the number of test questions needed to make these classifications in order to increase the amount of class time available for actual instruction. The AMT procedure makes use of item response theory (IRT; Lord, 1980; Lord & Novick, 1968) to adapt the test items administered to suit each student. <br>That has not become practical until computers became more prevalent with time. And generated what is known as computer adaptive testing.<br>IRTs probability-based estimation strategy is what makes CAT possible. A measure is adaptive only when the selection of items is adapted to prior estimates of the respondent’s level of the construct being measured. <br>After thorough evaluation of their clarity, content, sensitivity, and other properties, have been calibrated to an IRT model. The developmental items must be administered to a large sample of persons who are representative of the population of interest. These responses are used to tests the assumptions of IRT and to calibrate the items to an IRT model. <br>CAT relies on large banks of previously collected and calibrated responses as described, which means that banks will need time to develop first and would be exposed to large number of students, and hence will be consumed earlier contrary to what is proposed in the paper.<br>The preparation of the bank is a very complicated process and requires lots of time, technical experience, time and technology. Much to our dismay only few models do actually exist in real life. A fact that complicates matters further.<br>Unfortunately, the technology and infrastructure needed to implement CAT are currently beyond the reach, Existing CAT software is highly technical. And needs much expertise to run, Most CATs are developed “in house.” Which means every new implementation will start from the ground up, an unwarranted and arduous process to begin a bank with. <br>The relatively high start-up costs which includes research, programming, bank development, training and? Outcome assessment of the methodology makes CAT impractical as a startup project.<br>Another expense in implementing CAT is the “delivery device.” Because CATs are administered by computer, either respondents or an interviewer must interface with a computer. In addition to being expensive, these interfaces have other drawbacks. <br>Although the methodology is there more than three decades ago, it has not gained much acceptance in academic settings,. Nonetheless few examples do exist; the NCLEX is an example. However the NCLEX is mainly a pass/fail exam and level of proficiency is hard to get using their methodology. <br>Using computer adaptive testing Increases measurement efficiency. And in many occasions saves time, we argue here that using it in a Saudi Bank for medical examination is not only difficult, but could be problematic and may not be the best option. Further research is needed and time before effective banks and exams could be created using CAT.<br><br>References<br><br>1-Karon F Cook, Kimberly J O'Malley, and Toni S Roddey. Dynamic Assessment of Health Outcomes: Time to Let the CAT Out of the Bag? Health Serv Res. 2005 October; 40(5 Pt 2): 1694–1711. <br>2-Jette AM, Haley SM, Ni P, Olarsch S, Creating a computer adaptive test version of the late-life function and disability instrument. Moed R.J Gerontol A Biol Sci Med Sci. 2008 Nov;63(11):1246-56.Click here to read Links<br>3-McGlohen M, Chang HH. Combining computer adaptive testing technology with cognitively diagnostic assessment. Behav Res Methods. 2008 Aug;40(3):808-21.<br><br><br><br><br><br>
Competing interests
None declared
Congratulation and a comment
Abdulwahab Telmesani, College of medicine, Um Al-Qura University, Saudi Arabia
22 January 2009
Good job for Dr Sohail Bajammal and the group. A well written article in the sense that the idea was discussed from different angles and a semi SWOT analysis was conducted.<br>I totally agree with the idea of having a licensing examination in Saudi Arabia. I can give one more reason that is very difficult to argue against. Few days back, one of the local news papers reported the discovery of many health care workers ,from doctors to nurses, who were holding fake certificates and practicing medicine for sometime i.e. fake doctors and nurses?!!. A licensing exam would have identified and checked such a crime.<br>The proposed format of the exam is excellent, but we need to take in consideration the contest of the governing bodies of the heath care and practice i.e., choosing the appropriate format and implantation of the exam that fits the unique circumstances of country. I believe that the SCHS is taking a steady steps in the right direction toward the licensing exam. It is a matter of time and triggers like your proposal. I expect the SCHS giving the proposal the right attention and probably form a task force to put together a format(the proposed or modified one) for a licensing exam for any physician wish to practice in the Saudi Arabia.<br><br>Abdulwahab Telmesani<br>School of Medicine<br>Umm Al-Qura University<br>Makkah, Saudi Arabia<br>
Competing interests
None
Comments and Questions
Anees Sindi, King Abdulaziz University/ McMaster University
23 January 2009
Dear Dr. Bajammal and Colleagues,
Thank you for publishing this well written article. I have some comments and questions:
1. Page 14:
“Given the comprehensiveness and standardization in the planning, writing and conduct of these rigorous examinations, these licensing examinations can eventually replace the multiple exams required currently from medical students which include the final year examinations, the SCHS’ “Acceptance Test”, and the specialty-specific screening exams.”
Do you think replacing the final year examination is something doable given that the grading system is different in each university? If yes, how?
Should it just replace the SLE and the specialty-specific exams instead?
2. Page 15
“The established medical schools are at a disadvantage due to relocation of some of their experienced medical educators to the new schools, while the new medical schools are often disadvantaged due to being “new” in the business.”
One more point, this test will help us evaluate the newly established private medical schools and compare their graduates to the government-based medical schools.
3. Page 17:
“Scholarships for undergraduate medical students:
The expansion of international scholarship of undergraduate medical studies will bring many Saudi physicians with variable qualities of undergraduate medical education and from different cultural backgrounds into the country. The licensing exam will serve as a method of unifying the minimum standard of acceptable competency in the health service among these graduates.”
I think the test should be done before they go abroad instead of doing it after years of practice outside the country. It may not be easy to convince somebody who is already board certified to write such a basic test after 7-10 years of training in what is supposed to be a highly qualified center.
4. Page 19:
“The standardization of the assessment of medical students prior to their entry into medical practice, whether they join residency or not, is needed to complete the first and last step of the standardization process already in practice.”
Strongly agree since we started to see more Saudi graduates who are not interested in joining a local program or waiting to be accepted somewhere abroad.
5. Page 21:
“ The SCHS’ “Acceptance Test” is not suitable to provide this information for medical schools because only students who want to enroll in a Saudi residency program sit for this test.”
Is there any data/study looked at the score of this exam among different medical schools graduates who were enrolled in a local program?
6. Page 21:
“Exposing potential weakness in medical schools education system:
Medical schools might resist this call for change fearing that they will be exposing the weaknesses of their medical schools publicly.”
I personally don’t think that this should be an issue. I think it is going to be a very good assessment tool for the Ministry of Higher Education (MOHE) to evaluate our medical schools, new vs. old, government vs. private.
7. Page 21:
“The fear of impeding flexibility within medical school’s curriculum:
Medical schools educators might resist the change because they believe that national learning outcomes and a national standardized examination will restrict the flexibility of customizing their curriculum and choice of assessment methods.”
If this test got approved by the MOHE as an assessment tool then no body can complain or resist.
8. Page 22:
“Language barrier for non-Arabic speaking physicians
English is the language of instruction and examination in all medical schools in Saudi Arabia. Implementing a mandatory OSCE part which entails communicating with patients, who are mostly Arabic-speaking, might be a major issue for non-Arabic speaking physicians.”
Excellent point. Strongly agree. This may avoid a lot of the miscommunication we have been seeing in our hospitals. I would add the following point:
6. Religious Aspects for non-Muslims physicians:
This should include how to communicate with a female Muslim patient from the religious point of view, end of life decisions from the religious point of view.
Computer adaptive testing
22 January 2009
The adaptive mastery testing (AMT) procedure developed by Kingsbury and Weiss (1979) is designed to make high-precision classifications concerning students’ mastery of specific content areas within a course of instruction. The procedure is also intended to minimize the number of test questions needed to make these classifications in order to increase the amount of class time available for actual instruction. The AMT procedure makes use of item response theory (IRT; Lord, 1980; Lord & Novick, 1968) to adapt the test items administered to suit each student. <br>That has not become practical until computers became more prevalent with time. And generated what is known as computer adaptive testing.<br>IRTs probability-based estimation strategy is what makes CAT possible. A measure is adaptive only when the selection of items is adapted to prior estimates of the respondent’s level of the construct being measured. <br>After thorough evaluation of their clarity, content, sensitivity, and other properties, have been calibrated to an IRT model. The developmental items must be administered to a large sample of persons who are representative of the population of interest. These responses are used to tests the assumptions of IRT and to calibrate the items to an IRT model. <br>CAT relies on large banks of previously collected and calibrated responses as described, which means that banks will need time to develop first and would be exposed to large number of students, and hence will be consumed earlier contrary to what is proposed in the paper.<br>The preparation of the bank is a very complicated process and requires lots of time, technical experience, time and technology. Much to our dismay only few models do actually exist in real life. A fact that complicates matters further.<br>Unfortunately, the technology and infrastructure needed to implement CAT are currently beyond the reach, Existing CAT software is highly technical. And needs much expertise to run, Most CATs are developed “in house.” Which means every new implementation will start from the ground up, an unwarranted and arduous process to begin a bank with. <br>The relatively high start-up costs which includes research, programming, bank development, training and? Outcome assessment of the methodology makes CAT impractical as a startup project.<br>Another expense in implementing CAT is the “delivery device.” Because CATs are administered by computer, either respondents or an interviewer must interface with a computer. In addition to being expensive, these interfaces have other drawbacks. <br>Although the methodology is there more than three decades ago, it has not gained much acceptance in academic settings,. Nonetheless few examples do exist; the NCLEX is an example. However the NCLEX is mainly a pass/fail exam and level of proficiency is hard to get using their methodology. <br>Using computer adaptive testing Increases measurement efficiency. And in many occasions saves time, we argue here that using it in a Saudi Bank for medical examination is not only difficult, but could be problematic and may not be the best option. Further research is needed and time before effective banks and exams could be created using CAT.<br><br>References<br><br>1-Karon F Cook, Kimberly J O'Malley, and Toni S Roddey. Dynamic Assessment of Health Outcomes: Time to Let the CAT Out of the Bag? Health Serv Res. 2005 October; 40(5 Pt 2): 1694–1711. <br>2-Jette AM, Haley SM, Ni P, Olarsch S, Creating a computer adaptive test version of the late-life function and disability instrument. Moed R.J Gerontol A Biol Sci Med Sci. 2008 Nov;63(11):1246-56.Click here to read Links<br>3-McGlohen M, Chang HH. Combining computer adaptive testing technology with cognitively diagnostic assessment. Behav Res Methods. 2008 Aug;40(3):808-21.<br><br><br><br><br><br>
Competing interests
None declared
Congratulation and a comment
22 January 2009
Good job for Dr Sohail Bajammal and the group. A well written article in the sense that the idea was discussed from different angles and a semi SWOT analysis was conducted.<br>I totally agree with the idea of having a licensing examination in Saudi Arabia. I can give one more reason that is very difficult to argue against. Few days back, one of the local news papers reported the discovery of many health care workers ,from doctors to nurses, who were holding fake certificates and practicing medicine for sometime i.e. fake doctors and nurses?!!. A licensing exam would have identified and checked such a crime.<br>The proposed format of the exam is excellent, but we need to take in consideration the contest of the governing bodies of the heath care and practice i.e., choosing the appropriate format and implantation of the exam that fits the unique circumstances of country. I believe that the SCHS is taking a steady steps in the right direction toward the licensing exam. It is a matter of time and triggers like your proposal. I expect the SCHS giving the proposal the right attention and probably form a task force to put together a format(the proposed or modified one) for a licensing exam for any physician wish to practice in the Saudi Arabia.<br><br>Abdulwahab Telmesani<br>School of Medicine<br>Umm Al-Qura University<br>Makkah, Saudi Arabia<br>
Competing interests
None
Comments and Questions
23 January 2009
Dear Dr. Bajammal and Colleagues,
Thank you for publishing this well written article. I have some comments and questions:
1. Page 14:
“Given the comprehensiveness and standardization in the planning, writing and conduct of these rigorous examinations, these licensing examinations can eventually replace the multiple exams required currently from medical students which include the final year examinations, the SCHS’ “Acceptance Test”, and the specialty-specific screening exams.”
Do you think replacing the final year examination is something doable given that the grading system is different in each university? If yes, how?
Should it just replace the SLE and the specialty-specific exams instead?
2. Page 15
“The established medical schools are at a disadvantage due to relocation of some of their experienced medical educators to the new schools, while the new medical schools are often disadvantaged due to being “new” in the business.”
One more point, this test will help us evaluate the newly established private medical schools and compare their graduates to the government-based medical schools.
3. Page 17:
“Scholarships for undergraduate medical students:
The expansion of international scholarship of undergraduate medical studies will bring many Saudi physicians with variable qualities of undergraduate medical education and from different cultural backgrounds into the country. The licensing exam will serve as a method of unifying the minimum standard of acceptable competency in the health service among these graduates.”
I think the test should be done before they go abroad instead of doing it after years of practice outside the country. It may not be easy to convince somebody who is already board certified to write such a basic test after 7-10 years of training in what is supposed to be a highly qualified center.
4. Page 19:
“The standardization of the assessment of medical students prior to their entry into medical practice, whether they join residency or not, is needed to complete the first and last step of the standardization process already in practice.”
Strongly agree since we started to see more Saudi graduates who are not interested in joining a local program or waiting to be accepted somewhere abroad.
5. Page 21:
“ The SCHS’ “Acceptance Test” is not suitable to provide this information for medical schools because only students who want to enroll in a Saudi residency program sit for this test.”
Is there any data/study looked at the score of this exam among different medical schools graduates who were enrolled in a local program?
6. Page 21:
“Exposing potential weakness in medical schools education system:
Medical schools might resist this call for change fearing that they will be exposing the weaknesses of their medical schools publicly.”
I personally don’t think that this should be an issue. I think it is going to be a very good assessment tool for the Ministry of Higher Education (MOHE) to evaluate our medical schools, new vs. old, government vs. private.
7. Page 21:
“The fear of impeding flexibility within medical school’s curriculum:
Medical schools educators might resist the change because they believe that national learning outcomes and a national standardized examination will restrict the flexibility of customizing their curriculum and choice of assessment methods.”
If this test got approved by the MOHE as an assessment tool then no body can complain or resist.
8. Page 22:
“Language barrier for non-Arabic speaking physicians
English is the language of instruction and examination in all medical schools in Saudi Arabia. Implementing a mandatory OSCE part which entails communicating with patients, who are mostly Arabic-speaking, might be a major issue for non-Arabic speaking physicians.”
Excellent point. Strongly agree. This may avoid a lot of the miscommunication we have been seeing in our hospitals. I would add the following point:
6. Religious Aspects for non-Muslims physicians:
This should include how to communicate with a female Muslim patient from the religious point of view, end of life decisions from the religious point of view.
Sincerely,
Anees A.Ramadani Sindi, MBChB, ABIM
Clinical Fellow, Division of Respirology
McMaster University
Hamilton,ON
Canada
Demonstrator, Division of Critical Care Medicine
Faculty of Medicine, King Abdulaziz Univeristy
Jeddah, Saudi Arabia
Competing interests
None declared