FREE SHIPPING over $40
OVER 9000 FIVE-STAR REVIEWS
5% OFF ALL SUBSCRIPTIONS
100% HAPPINESS GUARANTEE
Analyze Diet
0
Home / Equine Research Bank / J Vet Intern Med / Interobserver and intraobserver reliability for 2 grading sy...
Journal of veterinary internal medicine2020; 35(1); 571-579; doi: 10.1111/jvim.15987

Interobserver and intraobserver reliability for 2 grading systems for gastric ulcer syndrome in horses.

Abstract: Grading of equine gastric ulcer syndrome (EGUS) is undertaken in clinical and research settings, but the reliability of EGUS grading systems is poorly understood. Objective: Investigate interobserver and intraobserver reliability of an established ordinal grading system and a novel visual analog scale (VAS), and assess the influence of observer experience. Methods: Sixty deidentified gastroscopy videos. Methods: Six observers (3 specialists and 3 residents) graded videos using the EGUS Council (EGUC) system and VAS. Observers graded the videos three 3 for each system, using a cross-over design with at least 1 week between each phase. The order of videos was randomized for each phase. Methods: Interobserver and intraobserver reliability were estimated using Gwet's agreement coefficient with ordinal weights applied (AC2) for the EGUC system and the intraclass correlation coefficient (ICC) for the VAS. Results: Using the EGUC system, interobserver reliability was substantial for squamous (AC2 = 0.69; 95% confidence interval [CI], 0.57-0.80) and glandular mucosa (AC2 = 0.72; 95% CI, 0.70-0.75), and intraobserver reliability was substantial for squamous (AC2 = 0.80; 95% CI, 0.71-0.90) and glandular mucosa (AC2 = 0.80; 95% CI, 0.74-0.86). Interobserver reliability using the VAS was moderate for squamous (ICC = 0.64; 95% CI, 0.31-0.96) and poor for glandular mucosa (ICC = 0.35; 95% CI, 0.06-0.64), and intraobserver reliability was moderate for squamous (ICC = 0.74; 95% CI, 0.62-0.86) and glandular mucosa (ICC = 0.56; 95% CI, 0.39-0.72). Conclusions: The EGUC system had acceptable intraobserver and interobserver reliability and performed well regardless of observer experience. Familiarity and observer experience improved reliability of the VAS.
© 2020 The Authors. Journal of Veterinary Internal Medicine published by Wiley Periodicals LLC. on behalf of the American College of Veterinary Internal Medicine.
Get Full Text
Publication Date: 2020-12-07 PubMed ID: 33284465PubMed Central: PMC7848314DOI: 10.1111/jvim.15987Google Scholar: Lookup
The Equine Research Bank provides access to a large database of publicly available scientific literature. Inclusion in the Research Bank does not imply endorsement of study methods or findings by Mad Barn.
LinkedInCopy
  • Journal Article

Summary

This research summary has been generated with artificial intelligence and may contain errors and omissions. Refer to the original study to confirm details provided. Submit correction.

This research examines the reliability of two grading systems for diagnosing Equine Gastric Ulcer Syndrome (EGUS) and how observer experience affects this reliability. The study found that the EGUS Council (EGUC) system showed substantial reliability, regardless of the observer’s experience, while the reliability of the Visual Analog Scale (VAS) improved with the observer’s experience.

Objective and Methods

  • The research aimed at determining the interobserver and intraobserver reliability of two EGUS grading systems: an established ordinal grading system and a newly introduced visual analog scale (VAS). Interobserver reliability refers to the degree of agreement among different observers, while intraobserver reliability refers to the consistency of an individual observer’s assessments.
  • A total of 60 anonymous gastroscopy videos were used in the study. Three specialists and three residents graded these videos using both the EGUC and VAS systems. Each video was graded three times using a crossover design with at least one week gap between each phase. The video order was randomized for every phase.
  • The researchers used Gwet’s agreement coefficient (AC2) with ordinal weights to estimate the reliability for the EGUC system, and the intraclass correlation coefficient (ICC) for the VAS.

Results

  • The EGUC system had substantial interobserver reliability for both the squamous (AC2=0.69) and glandular mucosa (AC2=0.72). Additionally, it also displayed substantial intraobserver reliability for squamous (AC2=0.80) and glandular mucosa (AC2=0.80). The results suggest that different observers and the same observer at different times arrived at similar results, marking high reliability of this system.
  • For the VAS, the interobserver reliability was moderate for squamous mucosa (ICC=0.64) and poor for glandular mucosa (ICC=0.35), and intraobserver reliability was moderate for squamous (ICC=0.74) and glandular mucosa (ICC=0.56). This indicates that the VAS might not produce consistent results among different observers or even for the same observer over time.

Conclusions

  • The study concluded that the EGUC system performed reliably, irrespective of the observer’s experience level. Observer familiarity and experience were found to improve the reliability of the VAS system.
  • This implies clinicians and researchers can use the EGUC system confidently for grading EGUS, while the VAS might require a more experienced observer for more consistent results.

Cite This Article

APA
Wise JC, Wilkes EJA, Raidal SL, Xie G, Crosby DE, Hale JN, Hughes KJ. (2020). Interobserver and intraobserver reliability for 2 grading systems for gastric ulcer syndrome in horses. J Vet Intern Med, 35(1), 571-579. https://doi.org/10.1111/jvim.15987

Publication

Journal of veterinary internal medicine
ISSN: 1939-1676
NlmUniqueID: 8708660
Country: United States
Language: English
Volume: 35
Issue: 1
Pages: 571-579

Researcher Affiliations

Wise, Jessica C
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Wilkes, Edwina J A
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Raidal, Sharanne L
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Xie, Gang
  • Quantitative Consultant Unit, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Crosby, Danielle E
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Hale, Josephine N
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.
Hughes, Kristopher J
  • School of Animal and Veterinary Sciences, Charles Sturt University, Wagga Wagga, New South Wales, Australia.

MeSH Terms

  • Animals
  • Gastroscopy / veterinary
  • Horse Diseases / diagnosis
  • Horses
  • Humans
  • Observer Variation
  • Reproducibility of Results
  • Stomach Ulcer / diagnosis
  • Stomach Ulcer / veterinary

Conflict of Interest Statement

Kristopher J. Hughes serves as Associate Editor for the Journal of Veterinary Internal Medicine. He was not involved in review of this manuscript.

References

This article includes 37 references
  1. Sykes BW, Hewetson M, Hepburn RJ, Luthersson N, Tamzali Y. European College of Equine Internal Medicine Consensus Statement—equine gastric ulcer syndrome in adult horses.. J Vet Intern Med 2015;29:1288‐1299.
    pmc: PMC4858038pubmed: 26340142
  2. Macallister CG, Andrews FM, Deegan E. A scoring system for gastric ulcers in the horse.. Equine Vet J 1997;29:430‐433.
    pubmed: 9413714
  3. Equine Gastric Ulcer Council. Recommendations for the diagnosis and treatment of equine gastric ulcer syndrome (EGUS): The Equine Gastric Ulcer Council.. Equine Vet Educat 1999;11:262‐272.
  4. Sykes BW, Sykes KM, Hallowell GD. A comparison of two doses of omeprazole in the treatment of equine gastric ulcer syndrome: a blinded, randomised, clinical trial.. Equine Vet J 2014;46:416‐421.
    pubmed: 24102898
  5. Sykes BW, Sykes KM, Hallowell GD. A comparison of three doses of omeprazole in the treatment of equine gastric ulcer syndrome: a blinded, randomised, dose‐response clinical trial.. Equine Vet J 2015;47:285‐290.
    pubmed: 24761780
  6. MacAllister CG, Sifferman RL, McClure SR. Effects of omeprazole paste on healing of spontaneous gastric ulcers in horses and foals: a field trial.. Equine Vet J 1999;31:77‐80.
    pubmed: 10696300
  7. Fuller CJ, Bladon BM, Driver AJ. The intra‐ and interassessor reliability of measurement of functional outcome by lameness scoring in horses.. Vet J 2006;171:281‐286.
    pubmed: 16490710
  8. Andrews FM, Nadeau JA. Clinical syndromes of gastric ulceration in foals and mature horses.. Equine Vet J 1999;29:30.
    pubmed: 10696290
  9. Andrews FM, Sifferman RL, Bernard W. Efficacy of omeprazole paste in the treatment and prevention of gastric ulcers in horses.. Equine Vet J 1999;31:81‐86.
    pubmed: 10696301
  10. Begg LM, O'Sullivan CB. The prevalence and distribution of gastric ulceration in 345 racehorses.. Aust Vet J 2003;81:199‐201.
    pubmed: 15080440
  11. Dionne RM, Vrins A, Doucet MY, Pare J. Gastric ulcers in standardbred racehorses: prevalence, lesion description, and risk factors.. J Vet Intern Med 2003;17:218‐222.
    pubmed: 12683624
  12. Bell RJ, Kingston JK, Mogg TD. A comparison of two scoring systems for endoscopic grading of gastric ulceration in horses.. N Z Vet J 2007;55:19‐22.
    pubmed: 17339912
  13. Aabakken L, Larsen S, Osnes M. Visual analogue scales for endoscopic evaluation of nonsteroidal anti‐inflammatory drug‐induced mucosal damage in the stomach and duodenum.. Scand J Gastroenterol 1990;25:443‐448.
    pubmed: 2193376
  14. de Lange T, Larsen S, Aabakken L. Inter‐observer agreement in the assessment of endoscopic findings in ulcerative colitis.. BMC Gastroenterol 2004;4:9‐9.
    pmc: PMC434504pubmed: 15149550
  15. McCormack HM, Horne DJ, Sheather S. Clinical applications of visual analogue scales: a critical review.. Psychol Med 1988;18:1007‐1019.
    pubmed: 3078045
  16. Landis JR, Koch GG. The measurement of observer agreement for categorical data.. Biometrics 1977;33:159‐174.
    pubmed: 843571
  17. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research.. J Chiropr Med 2016;15:155‐163.
    pmc: PMC4913118pubmed: 27330520
  18. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.. Psychol Assess 1994;6:284‐290.
  19. Fuller CJ, Bladon BM, Driver AJ, Barr ARS. The intra‐ and inter‐assessor reliability of measurement of functional outcome by lameness scoring in horses.. Vet J 2006;171:281‐286.
    pubmed: 16490710
  20. Olsen E, Dunkel B, Barker WHJ. Rater agreement on gait assessment during neurologic examination of horses.. J Vet Intern Med 2014;28:630‐638.
    pmc: PMC4857973pubmed: 24612411
  21. Menzies‐Gow NJ, Stevens KB, Sepulveda MF, Jarvis N, Marr CM. Repeatability and reproducibility of the obel grading system for equine laminitis.. Vet Rec 2010;167:52‐55.
    pubmed: 20622203
  22. Menzies‐Gow NJ, Knowles EJ, Rogers I, Rendle DI. Validity and application of immunoturbidimetric and enzyme‐linked immunosorbent assays for the measurement of adiponectin concentration in ponies.. Equine Vet J 2019;51:33‐37.
    pubmed: 29679396
  23. Kottner J, Audige L, Brorson S. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed.. J Clin Epidem 2011;64:96‐106.
    pubmed: 21130355
  24. Zannoni L, Savelli L, Jokubkiene L. Intra‐ and interobserver reproducibility of assessment of Doppler ultrasound findings in adnexal masses.. Ultrasound Obstet Gynecol 2013;42:93‐101.
    pubmed: 23065868
  25. McGivney CL, Sweeney J, David F. Intra‐ and interobserver reliability estimates for identification and grading of upper respiratory tract abnormalities recorded in horses at rest and during overground endoscopy.. Equine Vet J 2017;49:433‐437.
    pubmed: 27859573
  26. McGivney CL, Sweeney J, Gough KF. Serial evaluation of resting and exercising overground endoscopic examination results in young Thoroughbreds with no treatment intervention.. Equine Vet J 2019;51:192‐197.
    pubmed: 30003586
  27. Wongpakaran N, Wongpakaran T, Wedding D. A comparison of Cohen's kappa and Gwet's AC1 when calculating inter‐rater reliability coefficients: a study conducted with personality disorder samples.. BMC Med Res Methodol 2013;13:61.
    pmc: PMC3643869pubmed: 23627889
  28. Tammaa A, Fritzer N, Lozano P. Interobserver agreement and accuracy of non‐invasive diagnosis of endometriosis by transvaginal sonography.. Ultrasound Obstet Gynecol 2015;46:737‐740.
    pubmed: 25766661
  29. Bignotti B, Calabrese M, Signori A. Background parenchymal enhancement assessment: inter‐ and intra‐rater reliability across breast MRI sequences.. Europ J Radiol 2019;114:57‐61.
    pubmed: 31005177
  30. Gwet KL. Handbook of Inter‐Rater Reliability, 4th Edition: The Definitive Guide to Measuring The Extent of Agreement Among Raters.. US: Advanced Analytics, LLC; 2014.
  31. Birkmann K, Junge HK, Maischberger E, Wehrli Eser M, Schwarzwald CC. Efficacy of omeprazole powder paste or enteric‐coated formulation in healing of gastric ulcers in horses.. J Vet Intern Med 2014;28:925‐933.
    pmc: PMC4895457pubmed: 24628650
  32. Sykes BW, Sykes K, Hallowell GD. Comparison of the effect of two doses of omeprazole on the squamous gastric mucosa in thoroughbred racehorses.. Vet Rec 2014;175:249.
    pubmed: 25096591
  33. Sykes BW, Sykes KM, Hallowell GD. A comparison between pre‐ and post exercise administration of omeprazole in the treatment of equine gastric ulcer syndrome: a blinded, randomised, clinical trial.. Equine Vet J 2014;46:422‐426.
    pubmed: 24102898
  34. Fowler P, Bellardie H, Shaw B. Reliability of a Categorical Scale (GOSLON) and a continuous scale (10‐cm visual analog scale) for assessing dental arch relationships using conventional plaster and 3D digital orthodontic study models of children with complete unilateral cleft lip and palate.. Cleft Palate Craniofacial J 2019;56:84‐89.
    pubmed: 29698112
  35. Sim J, Reid N. Statistical inference by confidence intervals: issues of interpretation and utilization.. Phys Ther 1999;79:186‐195.
    pubmed: 10029058
  36. Akobeng AK. Confidence Intervals and p‐Values in Clinical Decision Making.. Oxford, UK: Blackwell Publishing; 2008:1004‐1007.
    pubmed: 18462462
  37. Scaffidi MA, Grover SC, Carnahan H. A prospective comparison of live and video‐based assessments of colonoscopy performance.. Gastrointest Endosc 2018;87:766‐775.
    pubmed: 28859953

Citations

This article has been cited 12 times.
  1. Irving J, Pineau V, Shultz S, Ter Woort F, Julien F, Lambey S, van Erck-Westergren E. Impact of Low-Starch Dietary Modifications on Faecal Microbiota Composition and Gastric Disease Scores in Performance Horses. Animals (Basel) 2025 Jun 28;15(13).
    doi: 10.3390/ani15131908pubmed: 40646806google scholar: lookup
  2. Birkmann K, Waldern N, Jucker S, Balaschitsch K, Zablotski Y, Fettelschoss-Gabriel A. Validation of the Equine Urticaria Activity Score for the assessment of chronic recurrent urticaria in horses. Vet Dermatol 2025 Oct;36(5):630-637.
    doi: 10.1111/vde.13358pubmed: 40384485google scholar: lookup
  3. Muñoz-Prieto A, Rubić I, Rešetar Maslov D, González-Sánchez JC, Mrljak V, Cerón JJ, Hansen S. Towards the Identification of New Biomarkers in Saliva and Serum for Treatment Monitoring of Equine Gastric Ulcer Syndrome: A Liquid Proteomic Approach. Animals (Basel) 2024 Oct 28;14(21).
    doi: 10.3390/ani14213105pubmed: 39518828google scholar: lookup
  4. Muñoz-Prieto A, Llamas-Amor E, Contreras-Aguilar MD, Ayala I, Martín Cuervo M, Cerón JJ, Hansen S. Automated Spectrophotometric Assays for the Measurement of Ammonia and Bicarbonate in Saliva of Horses: Analytical Validation and Changes in Equine Gastric Ulcer Syndrome (EGUS). Metabolites 2024 Feb 28;14(3).
    doi: 10.3390/metabo14030147pubmed: 38535307google scholar: lookup
  5. Williams Louie E, Nieto J, Wensley F, Morgan JM, Finno CJ, Berryhill EH. Efficacy of the oral supplement, Equine Omega Complete, for the prevention of gastric ulcers and alpha-tocopherol supplementation in horses. J Vet Intern Med 2023 Nov-Dec;37(6):2529-2534.
    doi: 10.1111/jvim.16877pubmed: 37775973google scholar: lookup
  6. Van Driessche L, Fecteau G, Arsenault J, Miana L, Chorfi Y, Villettaz-Robichaud M, Hélie P, Buczinski S. Inter-Rater Reliability of Scoring Systems for Abomasal Lesions in Quebec Veal Calves. Animals (Basel) 2023 May 17;13(10).
    doi: 10.3390/ani13101664pubmed: 37238094google scholar: lookup
  7. Ferlini Agne G, May BE, Lovett A, Simon O, Steel C, Santos L, Guedes do Carmo L, Barbosa B, Werner LC, Daros RR, Somogyi AA, Sykes B, Franklin S. Horse Grimace Scale Does Not Detect Pain in Horses with Equine Gastric Ulcer Syndrome. Animals (Basel) 2023 May 12;13(10).
    doi: 10.3390/ani13101623pubmed: 37238054google scholar: lookup
  8. Vokes J, Lovett A, Sykes B. Equine Gastric Ulcer Syndrome: An Update on Current Knowledge. Animals (Basel) 2023 Apr 5;13(7).
    doi: 10.3390/ani13071261pubmed: 37048517google scholar: lookup
  9. Muñoz-Prieto A, Contreras-Aguilar MD, Cerón JJ, Ayala I, Martin-Cuervo M, Gonzalez-Sanchez JC, Jacobsen S, Kuleš J, Beletić A, Rubić I, Mrljak V, Tecles F, Hansen S. Changes in Proteins in Saliva and Serum in Equine Gastric Ulcer Syndrome Using a Proteomic Approach. Animals (Basel) 2022 May 2;12(9).
    doi: 10.3390/ani12091169pubmed: 35565595google scholar: lookup
  10. Pratt S, Bowen I, Hallowell G, Shipman E, Redpath A. Assessment of agreement using the equine glandular gastric disease grading system in 84 cases. Vet Med Sci 2022 Jul;8(4):1472-1477.
    doi: 10.1002/vms3.807pubmed: 35412651google scholar: lookup
  11. Contreras-Aguilar MD, Rubio CP, González-Arostegui LG, Martín-Cuervo M, Cerón JJ, Ayala I, Henriksen IH, Jacobsen S, Hansen S. Changes in Oxidative Status Biomarkers in Saliva and Serum in the Equine Gastric Ulcer Syndrome and Colic of Intestinal Aetiology: A Pilot Study. Animals (Basel) 2022 Mar 7;12(5).
    doi: 10.3390/ani12050667pubmed: 35268236google scholar: lookup
  12. Hewetson M, Tallon R. Equine Squamous Gastric Disease: Prevalence, Impact and Management. Vet Med (Auckl) 2021;12:381-399.
    doi: 10.2147/VMRR.S235258pubmed: 35004264google scholar: lookup
Subscribe to our newsletter
Join 99,357 horse owners like you who receive our email updates on horse health and nutrition.