raRoB-VetThe objectives of work package 4 of SA13 were to investigate the applicability of the tool designed for evaluating the risk of bias in human epidemiological studies (raRoB) to epidemiological studies with animals (raRoB-vet), as well as to explore its potential for further refinement (Deliverable 1). Consequently, a new instrument specifically designed for the assessment of epidemiological animal studies was to be established (Deliverable 2).To test the existing structure of domains and items for the assessment of human studies and their further development into a separate instrument for animal studies, a modified three-stage Delphi study was conducted with 16 internationally recruited experts in veterinary medicine and measurement theory. In the course of the development, three versions of the instrument were created and two tests were carried out with experts and independently recruited test reviewers. During the course of development, the existing structure was not only extended to epidemiological animal studies, but was also developed and implemented as a Shiny-webapp. In addition to that, an overall risk of bias score was developed and integrated into the webapp.On the basis of the three-stage Delphi study, a web-based tool for the qualitative and structured assessment of the risk of bias in epidemiological animal studies was ultimately developed in an iterative and consensus-oriented process. By further developing the tool into a webapp, the project's objectives were not only achieved, but exceeded. raRoB-vet is available as a Shiny-webapp at http://shiny.bfr.bund.de/apps/raRoB-vet/ and is hosted by the German Federal Institute of Risk Assessment. The instrument includes seven domains with 14 items, potentially covering all primary sources of bias, including Selection-, Information-, Confounding-, Attrition- and Reporting bias. It is designed to be applied on a variety of epidemiological study designs, including Cohort-, Case-control, Case-cohort, Cross-sectional, Ecological- and Nested case-control studies. For all items and domains, signal questions and help functions have been developed to assist non-expert users during the guided analysis. In addition to the item-based assessment, the tool provides an overall risk of bias score at the end of each analysis. Furthermore, reports are supported in MS Word and MS Excel formats, making the webapp suitable for use in meta-analyses and weight-of-evidence approaches. The landing page offers an introduction to the tool, along with links to the manual and contact information for support inquiries.So far, the raRoB-vet web app has been presented at two conferences as part of the project, once at DAGSTAT 2024 in Salzburg and once at ISVEE2024 in Sydney. At these conferences, the potential for integrating the tool with automated procedures using LL models was discussed and proposed as a future direction for its development. Another potential goal is to conduct further testing of the instrument, particularly to estimate a kappa in order to quantify its interrater reliability.
shiny rriskEFSAshort forEuropean Food Safety Authority recognizes the importance of quantitative risk estimates in their risk assessments to enhance the usability of their conclusions and improve risk communication. Quantitative approaches offer several advantages over purely narrative descriptions:· They provide a measurable level of risk· They incorporate inherent uncertainties in the assessment· They allow for more precise comparisons and decision-makingRisk assessments often involve multiple influencing factors and sub-questions. To propagate evidence and uncertainties through various stages to the final quantitative outcome, mathematical calculations are frequently employed. These calculations require specialized software such as:· Commercial (e.g. @RISk) or open-source (e.g. Poptools) spreadsheet based risk modeling (SRBM) tools· Open-source applications such as the shiny rrisk, which was implemented in R and utilises necessary functionalities of the R universe.
EFSAshort forEuropean Food Safety Authority is committed to transparency in its work, allowing stakeholders and the general public to review and scrutinize their conclusions. To achieve this,
EFSAshort forEuropean Food Safety Authority:· publishes all evidence, methodologies, and intermediate results· releases software code (e.g., @RISK files) as additional material in their opinions· shares more general content in the Zenodo repositoryFor @RISK files, specific functions are deactivated to enable review using Microsoft Excel. However, recalculating results requires an @RISK license to activate the necessary functions and perform Monte Carlo simulations.By adopting tools like shiny rrisk and maintaining transparency in their processes,
EFSAshort forEuropean Food Safety Authority could enhance the quality, reliability, and accessibility of their risk assessments across various domains.The users of SBRMs are not only in the regulatory field, but also in many other fields, including the financial industry. This means that these tools cannot by themselves map the specific needs of the regulatory authorities alone; these must be introduced and maintained by the respective users themselves in the SBRMS through self-discipline. This can work, as in case study 1. However, it can also fail, as in the other two case studies, where either the Excel file was difficult to reproduce or the documentation of the model itself was difficult to follow.Unlike the SBRM tools, shiny rrisk provides guidance for modelling, documentation of and risk models as used in regulatory authorities. Also, it saves the whole model in an open data format. This should make reliable and transparent risk and exposure models that are fully available and executable the norm rather than the exception.shiny rrisk offers the possibility of 2D Monte Carlo simulation, a functionality that is not available in SBRM tools. The mapping of uncertainty is an absolute must for risk models, because of the wide range of available parameter estimates, in order to ensure that the risk assessment is realistic.
EFSAshort forEuropean Food Safety Authority and
BfRshort forGerman Federal Institute for Risk Assessment have worked together in this area in the past.The ability to run shiny rrisk does not depend on the availability of MS Excel and a commercial add-on, rather it can be used as a free web service on all platforms.Shiny rrisk is also well suited for training in risk and exposure modelling (e.g., EU-FORA platform) because it is free to run, has a structured graphical user interface and can be run on all platforms.We were able to show that shiny rrisk can be applied to main areas of
EFSAshort forEuropean Food Safety Authority's quantitative risk modelling needs. Furthermore, we were able to demonstrate the customisability and extendibility of shiny rrisk within this project.The software and the manual will be made available via the Github BfRstats. The web tool is currently available at the
BfRshort forGerman Federal Institute for Risk Assessment via shiny rrisk as a running web tool, as well as shiny rrisk distribution.Once the gap in functionality in shiny rrisk for distribution fitting has been filled, the application should be tested and validated by the relevant
EFSAshort forEuropean Food Safety Authority modelers. If this is positive, shiny rrisk can be used at
EFSAshort forEuropean Food Safety Authority as a modelling and documentation tool.In the future, the application
shiny rrisk distributions will complement the app
shiny rrisk. shiny rrisk distribution will enable transparent, reproducible and well-documented distribution fitting (both as density and cumulative distribution function fits). Results from shiny rrisk distribution can be directly incorporated into shiny rrisk by users. This will replace the currently existing ad hoc implementation for the
EFSAshort forEuropean Food Safety Authority-EKE cdf fit.However, the application can already be used as a learning and training environment, e.g., within the EU-FORA programme. To do this, there would have to be a running shiny rrisk application on
EFSAshort forEuropean Food Safety Authority's own R4EU server.