Table 5 Examples of interviewees’ quotes under the sub-themes

Validity

Assessment blueprinting

“I think open book is the way forward for us because we are teaching professional courses which involves a lot of decision making … decision making ability is one of the key outcomes. So that should really be tested right from the beginning.” (P8)

“For certain modules, practice related modules, I think this is a suitable exam format. But of course not all modules can do that. I would say majority of the modules, especially those lower semesters which are mainly knowledge based, I think conventional face-to-face is still more suitable.” (P3)

Faculty competencies

“Setting the question is a major challenge for the faculty. If every faculty can be trained to set higher order thinking questions, then I think the faculty are ready.” (P10)

“During the vetting session, we sat down together and looked at the appropriateness, the level and the rubric. So we always checked that and we improved (the questions) during the vetting session.” (P21)

Quality assurance

“I think having the vetting is really essential, because we were able to help each other a lot during the vetting and we were really able to, at least at the department level we're definitely able to help each other make better questions” (P2)

Reliability

Student performance

“As a teacher, of course initially I doubt the reliability. But after conducting the exercise, I think it is quite okay. I mean the result is quite reliable” (P5)

Standard setting

“When we did the standard setting, we specifically remind the faculty to keep in mind that these are open book exams, meaning the students can access the resources to answer those questions.” (P1)

Educational impact

Promotion of higher order cognitive skills

“If we just test on the recall questions then we expect the students to just do a search, find and then if they find the answers they will just click on the correct answer. But if we are testing on the higher order questions, then the students they will realise that "now I need to study and I need to understand the concept". So a total different game that they will have to play now.” (P10)

“In the future I think we are going that direction. So our values to society then comes into the equation … the strategizing, analysis, individualisation to different situations, and that is what we call higher order … And if you want to survive as a university we must be competitive and our students must all be prepared for this future.” (P7)

Acceptability

Recognition by stakeholders

“I'm just wondering from an external point of view, let's say if you have a student who graduated by taking this type of open book exam, I don't know will there be any biasness from the employers, can they rely on the scroll received by that student?” (P12)

Change management (faculty and student readiness)

“Move forward, I think if we think positively, I think the change… I think we can actually embrace the change. I think it's a good way, I think if it doesn't happen we also would not know how things will turn out. So as I said it's a learning phase for me as well as the students, so I think it's a good way.” (P13)

“There are enough literatures to support that these open book exams do work, and they can actually work better for questions where critical analysis is required.” (P4)

“I think the contingency plan, so a more complete contingency plan… so that like when this happens, what is (plan) B and C for different scenarios that we actually encounter, so it will be nice if this happens, immediately we know what to do.” (P6)

“Initially I think the students were very resistant about open book exams. They were concerned, I think their idea is that when you are having an open book exam, meaning that your questions will be tougher and they don't know how to answer, meaning that they cannot find the answers in the textbook at all.” (P10)

“We try to prepare them before they sit for the paper. You know, on what's expected, so during the tutorial, we do give some cases so that they can practise on to help them. Some are able to catch up, so when they respond to the OBA questions, they are okay. But some, not able to….. I think it's no harm to proceed with open book as long as we prepare them well ahead, you know. And I think they should be okay….. I think we can but we need to do it slowly. Not too drastic, you know.” (P13)

Impact of COVID-19 pandemic

“We never really took up that opportunity to convert them to all open book assessment…I think this COVID-19 pandemic has open up those doors.” (P1)

Academic integrity

“To be honest, every time when I conduct this online exam, I was wondering what are they doing at the other side. Whether they are actually discussing among themselves or get the help from other people, we wouldn't know.” (P17)

“Of course at the same time we have to inculcate academic honesty…. I think is to start by letting them know their rights and what academic honesty is all about, what intellectual property is all about, you know. Then to have a certain sense of pride in their own learning and their own attainment.” (P20)

Cost

IT softwares

“Ideally we should have a really good system so that we can do the questions online and the proctoring.” (P2)

Internet connection

“I think some of them face challenges with the internet connection, I think that is the main issue because of stability of internet. And some of the students … when they are downloading their questions, take a bit of time to download the questions. (P10)

Technical support

“I think also having the IT people and the eLearning people present when all these things (issues) happen … I think they were really important.” (P2)

“If they closely monitor, all these (technical issues will be actually resolved, can be resolved immediately.” (P6)

Faculty time

“I think on the faculty opinion right, normally they will prefer to have the on campus exam…It's just mainly on the time that the faculty has to spend to set higher order thinking questions.” (P10)