SETAC Spotlight: The CATs are out of the bag: Experiences to date with Critical Appraisal Tools

CEA presented 1 platform and 9 posters at SETAC Seville in May 2024. We will be showcasing each of these presentations in a series of ‘SETAC Spotlight’ articles. This week it is:

SETAC Spotlight: The CATs are out of the bag: Experiences to date with Critical Appraisal Tools

Authors: Helena Crosland, Sarah Priestly, Amy Brooks

In the ecotoxicology risk assessment for pesticides, studies are often submitted for which internationally agreed guidelines are not available. This can result in inconsistent evaluations across regulatory bodies and unexpected outcomes for industry. In October 2022, EFSA (European Food Safety Authority) published a proposal document for the use of Critical Appraisal Tools (CATs) for non-standard ecotoxicology studies. The document outlines the theory behind the development of the CATs, why they are considered relevant, and the seven types of studies they have been created for.

The CATs consist of spreadsheets (as annexes to the proposal document) listing assessment criteria for relevance and reliability of studies. These can be assessed as ‘fulfilled’, ‘partly fulfilled’, ‘not fulfilled’, or ‘not reported’. Some ‘key criteria’ are assessed as being more significant than the others. Word document handbooks are also available to aid with the use of each CAT. CATs for two additional study types are also provided as part of the revised EFSA 2023 bird and mammal guidance document. CATs are currently available for the following study types:

• Modified aquatic exposure studies
• Aquatic mesocosm studies
• Honeybee brood tests
• Non-target arthropod (NTA) extended laboratory studies
• NTA aged residue studies
• NTA field studies
• Residue decline studies and related kinetics field studies
• Bird and mammal (B&M) ecological field studies
• B&M field effect studies

The premise of these tools is that they provide a framework for the evaluation of non-standard studies (i.e. where guidelines are not available), in theory making the evaluation of such studies easier, and making outcomes more consistent across studies and regulators.

CEA has performed scoping exercises to investigate the potential impact of the use of these CATs in the evaluation of several existing studies of differing types. Specific and general  issues identified are summarised here.

Modified exposure studies with aquatic organisms

• Study plans and statistical reports should be included in the study reports.
• If non-standard/wild organisms used:
  • Keying out should be reported
  • Need to justify source
  • Need to justify life-stage chosen.
• Check that specific reporting criteria are not missed – e.g. water solubility.
• Careful monitoring of environmental parameters is required. Where variations occur, justification/explanation of impact should be reported in study report.

Field residue studies

• Uncontrollable variables can impact result of the CAT appraisal. Not always possible to plan for/mitigate these.
• Expert evaluation of historical weather data/implications required. But how appropriate is historical weather data?
• Surrounding landscape must be reported. Trials being performed in an agricultural area may be enough for this criterion to be ‘partially fulfilled’.
• Soil concentrations should be measured for earthworm residue studies.
• CROs may disagree with the requirements outlined in the CAT – e.g. analytical measurements in control both at the start and end of the study.
• For further information on the designing of residues studies, see our previous poster via this link.

General issues

• Though not their intended use, consideration of CATs is useful when planning/ designing/reporting studies. I could save time and money, and potentially avoids the need for repeat studies.
• Note that although assessment criteria in CATs are formalised, they can still be vague and are open to interpretation!
• Reporting of studies is important. Older studies may be assessed poorly due inappropriate reporting. But if the information is available via the performing laboratory, study reports could be updated to avoid issues. Some CROs have already updated their reporting templates to take into account the requirements of the CATs.
• It can be difficult for studies to meet the category of ‘high’ relevance/reliability as defined by CATs due to the ‘key’ criteria requirements, and because small failures/shortfalls have a big impact on the result of the appraisal.
• The use of CATs by regulators (as is intended) is not currently mandatory, but their inclusion in the new bird and mammal guidance document suggests that they are likely to be used.
• As the tools are new, feedback from regulators is lacking.

Follow this link to see our poster on this topic presented at SETAC 2024.

You can read our previous news article on this topic via this link.