The evaluation of the app-based surgical e-logbook system, which found 67.3% matches with HRs and 32.7% overreported procedures, indicates a need to critically reflect on both the accuracy and the quality of this newly implemented digital system. Compliance with surgical logbook-keeping, be it analog or digital, in an environment with limited resources and technological know-how should also be critically examined. There might be several explanations for the discrepancies between the participants’ expressed satisfaction with the app and the low rate of accuracy found by the quantitative analysis. First, the e-logbook system was newly introduced and still at an early stage of adoption when data for this study were collected. The users were still in the process of getting to know the app and how to use it. However, this also opens up the possibility of learning from the most crucial time of implementation. Second, participants’ limited experience with app technology may have further decreased the percentage of matches. Some participants expressed that the information and training provided before they started to use the app were insufficient, and more training might increase the accuracy of registered procedures. Third, challenges related to the app design, particularly finding the right procedure in the drop-down menus, may have affected the process of cross-checking, as “identical procedure” was one of the study criteria for matching. Drop-down menus also pose a risk that users may mistakenly select the wrong option without noticing, causing errors in registration.
The heterogeneity among the participants’ distribution of match percentages indicates considerable individual variability in registration practices. In the case of two participants, 100% of the procedures were either overreported or underreported. For one of them, this is explained by a very limited number (n = 2) of procedures registered in the e-logbook database. For the other, it was explained during the interviews that procedures might not have been recorded in the HRs during a free hernia surgery camp, as someone else was assigned this task. All procedures of the participant in question were performed within the time span of this camp, leading to a potentially large number of overreported procedures. The distribution of underreported procedures also shows heterogeneity, and the percentage of underreported procedures (50.7%) is evidence that the app does not capture all procedures registered in the HRs. Reporting habits for cesarean sections is a possible reason for this, as these procedures might be registered in the obstetric section instead of the surgical section of the app, with only the latter being included in this study. A closer investigation revealed that some participants had, indeed, registered cesarean sections in the obstetric section. Cesarean sections constitute a large number of the entries in the HRs, as they are the second most frequently performed operation in the STP [17], which further emphasizes the importance of this issue.
A previous study that analyzed the analog logbook system within the same training scheme [14] found an estimated proportion of 73.1% (95% CI, 56–85) matching and close-matching entries; these categories combined are equivalent to the category “match” in the present study. A direct comparison between the studies suggests that the app-based system, with a 67.3% match rate, might be less accurate. A possible reason for the perceived lower accuracy of the e-logbook system, as mentioned above, is that the study was conducted shortly after its implementation. Moreover, the previous study from 2016 [14] is not necessarily an accurate representation of how the analog logbook system works today. At the time of the prior study, there were fewer trainees and graduates in the STP, and following up on the multistep analog system was easier. Furthermore, the methodology for the quantitative analysis was designed for the evaluation of the analog logbook system. To be able to compare the two systems, the methodology for cross-checking e-logbook entries with hospital records was maintained for the evaluation of the e-logbook system. However, it is not necessarily the most suitable study design for the quantitative analysis of the e-logbook system, as it may be more descriptive of the registration habits of the users than of the system itself. It might be argued, therefore, that the current results are not sufficient evidence to discard the e-logbook system.
During the participant interviews, it was revealed that some hospitals did not register minor procedures, as they are often performed on the wards and not in the operation theaters. Participants indicated that the app would capture more minor procedures, which would cause these procedures to be overreported in the cross-checking and, consequently, not be considered a match. However, this is not necessarily a weakness of the e-logbook system, as it will capture information that is missing from the HRs. The tendency of overreporting of minor procedures is further suggested by the subanalysis of the quantitative data.
The increased use of e-logbooks in LICs requires both technical competence and availability of technology, which have been identified as important barriers to the implementation of eHealth innovations [8]. Study participants described both of these barriers during the semi-structured interviews, and addressing these challenges might be crucial for a successful implementation. Although there are challenges in implementing an eHealth solution in a low-resource setting, this does not justify inaction. Digitalization is considered to be a driving force to improve access to surgical care and reduce health care inequality globally [8, 18]. It has the potential to increase workforce capacity, as technology automates work that was previously required to be performed by people [19]. A transition from a paper-based system to a digital system may also facilitate and increase the efficiency of patient care, especially when the amount of patient information is extensive. The substantial constraints on accessible health care in LICs, including the limited access to human resources, further justify the need to try innovative approaches.
Strengths and limitations
Evaluating the e-logbook system at an early stage can be seen as a strength of the study. The insight gained from the assessment of eHealth solutions still under development can contribute to promoting user-friendliness and secure data management, leading to a successful implementation of the innovation [8]. In addition, the combination of qualitative and quantitative data enabled a fuller picture to be formed based on the available data. This approach is also supported by a Cochrane review that stated that more qualitative research is required to determine if and why eHealth solutions are successful [20].
A limitation of this study is the scarce amount of available quantitative data due to the short time between the implementation of the e-logbook system and the study period, resulting in a limited number of study participants, study hospitals, and procedures in the e-logbook database. Given the limited amount of data, detailed statistical analysis will add little value to the interpretation of the results. There is much to learn from descriptive statistics alone, which has therefore been presented instead. The use of different statistical methods complicates the comparison of the outcomes of this study to the evaluation of the analog logbook system. In addition, the reliability of the quantitative results as a representation of how the e-logbook system will work in the long term is questionable. Furthermore, the HRs collected for this study consisted exclusively of handwritten paper books. It has been shown that data collection from paper-based medical records in LICs varies in quality from inadequate [21, 22] to very accurate [23]. The assessment of the HRs was intricate, as the handwriting could be difficult to interpret, the information they contained varied between the study hospitals, and paper documents may have tears and creases that complicate the process. This may further have disrupted the cross-checking of the procedures, as the HRs are not a perfect standard of reference. This issue as well as the aforementioned constraints related to the sample size makes it difficult to draw conclusions based on the quantitative analysis.
However, an equally representative group of trainees and graduates of the STP were included during the evaluation of both the analog and the e-logbook systems, which contributes to the value of this study. This was the case for both the quantitative analysis and the semi-structured interviews, which is an advantage in regard to the comparison of the systems. Nevertheless, the interviews for the evaluation of the e-logbook system were conducted via telephone. The lack of nonverbal communication, such as gestures and facial expressions, may have caused important cues to go unnoticed. For both evaluative studies, cultural differences and linguistic barriers may have had a similar effect on communication.
Key recommendations
Based on the results and discussion of this article, the following key recommendations can be made:
-
A proper introduction, sufficient training, and mentoring on the use of the e-logbook system should be offered, including guidance on the use of the necessary technology and the app itself.
-
Access to the necessary technology, including devices compatible with the app, reliable internet access, and power to charge devices should be ensured.
-
Further development of the app design and modifications to achieve greater user-friendliness. A potential alteration is the inclusion of a free-text field where the user can add additional information if necessary, for example as a supplement to the drop-down menus.
-
Enabling users to easily retrieve their own procedures registered in the e-logbook system. This information should be anonymized to adhere to data protection regulations and confidentiality.
-
The distinction between the surgical and obstetric sections of the e-logbook system should be clarified for users, and information on which type of procedures should be entered into each section should be provided.