Inter- and intra-rater reliability of two grading systems for equine gastric glandular disease.

Overview

• This study compares the reliability of two different scoring systems for diagnosing equine gastric glandular disease based on gastroscopy images.
• The research evaluates how consistent the scoring is both within the same rater over time (intra-rater) and between different raters (inter-rater), considering varying levels of clinical experience.

Background and Purpose

• Equine gastric glandular disease involves lesions in the stomach lining of horses, requiring accurate diagnosis for effective management.
• Two primary methods exist for grading these lesions:
  • A recommended combined descriptive terminology system, which uses clinical descriptors to categorize lesions.
  • A published numerical scoring system that assigns numerical values based on lesion characteristics.
• Previous studies suggested that the descriptive system has poor reliability, especially among raters with varying clinical experience, whereas the numerical system shows better consistency.
• This study aims to:
  • Compare the intra- and inter-rater reliability of these two systems.
  • Determine whether clinical experience impacts reliability differently for each system.
  • Examine correlations between numerical scores and specific descriptive terms such as ‘severity’.

Methods

• A prospective, blinded observer study was conducted.
• Thirty gastroscopy images featuring glandular lesions were duplicated to allow repeated assessments.
• Twelve raters assessed these images using both scoring systems. The raters were divided into four groups based on clinical experience:
  • Internal medicine specialists
  • Residents
  • General practitioners
  • New graduates
• Krippendorff’s alpha statistic was used to evaluate:
  • Intra-rater reliability (consistency of ratings by the same rater on duplicate images).
  • Inter-rater reliability (agreement among different raters).
• The study also analyzed the association between numerical scores and individual descriptive criteria, especially the ‘severity’ term.

Results

• Overall reliability:
  • The numerical scoring system had higher intra-rater reliability (α = 0.89) and inter-rater reliability (α = 0.67) across all raters.
  • The combined descriptive terminology system showed lower intra-rater (α = 0.61) and very poor inter-rater reliability (α = 0.09).
• Effect of clinical experience on intra-rater reliability:
  • Numerical scoring showed high consistency across all groups regardless of clinical experience.
  • Descriptive system had high intra-rater reliability only for specialists; less experienced groups showed poorer consistency.
• Effect of clinical experience on inter-rater reliability:
  • With the numerical system, moderate reliability was observed among specialists, residents, and recent graduates.
  • Inter-rater reliability was poor in all groups using the descriptive system.
• Association between scoring systems:
  • There was a strong, statistically significant correlation between the numerical score and the descriptive criterion ‘severity’ (η_H = 0.66; p < 0.001).

Conclusions

• The numerical scoring system is more reliable than the combined descriptive terminology system for grading equine gastric glandular disease across all levels of clinical experience.
• Clinical experience influences the reliability of the descriptive scoring system more strongly than the numerical system.
• The numerical score correlates strongly with the perceived severity described in the terminology system, supporting its clinical relevance.
• Using still images reviewed individually on screens, the study suggests that numerical scoring is preferable for consistent, reproducible assessment of glandular lesions in horses.