FREE SHIPPING over $40
OVER 9000 FIVE-STAR REVIEWS
5% OFF ALL SUBSCRIPTIONS
100% HAPPINESS GUARANTEE
Analyze Diet
0
Home / Equine Research Bank / Equine Vet J / Interobserver agreement during clinical magnetic resonance i...
Equine veterinary journal2024; 57(2); 406-418; doi: 10.1111/evj.14126

Interobserver agreement during clinical magnetic resonance imaging of the equine foot.

Abstract: Agreement between experienced observers for assessment of pathology and assessment confidence are poorly documented for magnetic resonance imaging (MRI) of the equine foot. Objective: To report interobserver agreement for pathology assessment and observer confidence for key anatomical structures of the equine foot during MRI. Methods: Exploratory clinical study. Methods: Ten experienced observers (diploma or associate level) assessed 15 equine foot MRI studies acquired from clinical databases of 3 MRI systems. Observers graded pathology in seven key anatomical structures (Grade 1: no pathology, Grade 2: mild pathology, Grade 3: moderate pathology, Grade 4: severe pathology) and provided a grade for their confidence for each pathology assessment (Grade 1: high confidence, Grade 2: moderate confidence, Grade 3: limited confidence, Grade 4: no confidence). Interobserver agreement for the presence/absence of pathology and agreement for individual grades of pathology were assessed with Fleiss' kappa (k). Overall interobserver agreement for pathology was determined using Fleiss' kappa and Kendall's coefficient of concordance (KCC). The distribution of grading was also visualised with bubble charts. Results: Interobserver agreement for the presence/absence of pathology of individual anatomical structures was poor-to-fair, except for the navicular bone which had moderate agreement (k = 0.52). Relative agreement for pathology grading (accounting for the ranking of grades) ranged from KCC = 0.19 for the distal interphalangeal joint to KCC = 0.70 for the navicular bone. Agreement was generally greatest at the extremes of pathology. Observer confidence in pathology assessment was generally moderate to high. Conclusions: Distribution of pathology varied between anatomical structures due to random selection of clinical MRI studies. Observers had most experience with low-field MRI. Conclusions: Even with experienced observers, there can be notable variation in the perceived severity of foot pathology on MRI for individual cases, which could be important in a clinical context.
© 2024 EVJ Ltd.
Get Full Text
Publication Date: 2024-06-30 PubMed ID: 38946165DOI: 10.1111/evj.14126Google Scholar: Lookup
The Equine Research Bank provides access to a large database of publicly available scientific literature. Inclusion in the Research Bank does not imply endorsement of study methods or findings by Mad Barn.
LinkedInCopy
  • Journal Article

Summary

This research summary has been generated with artificial intelligence and may contain errors and omissions. Refer to the original study to confirm details provided. Submit correction.

This research examined the degree of agreement between different experienced observers while assessing pathology in horse foot using Magnetic Resonance Imaging(MRI). The findings indicate that variations in pathology assessments can occur even among experienced observers, which can potentially affect clinical decision-making.

Research Methodology

  • The researchers carried out an exploratory clinical study which included ten experienced observers. These observers were either diploma holders or associates, bringing professional perspectives into the assessment process.
  • The assessment involved 15 horse foot MRI studies viz. the data were drawn from clinical databases of three different MRI systems, thus providing a broad range of sources for the study.
  • For the analysis, the observers were made to grade the pathology in seven central anatomical structures. Four grades were established: Grade 1 for no pathology, Grade 2 for mild pathology, Grade 3 for moderate pathology, and Grade 4 for severe pathology.
  • The observers also had to provide a confidence grade for each of their pathology assessments. It ranged from Grade 1 for high confidence to Grade 4 for no confidence.
  • The interobserver agreement was measured using Fleiss’ kappa coefficient (k), which is a statistical measure for calculating the degree of agreement among observers. The overall interobserver agreement for pathology was determined using Fleiss’ Kappa and Kendall’s coefficient of concordance (KCC).
  • The researchers also used bubble charts to visualize the distribution of grading.

Key Findings

  • Finding showed that the interobserver agreement for the presence or absence of pathology for individual anatomical structures was generally poor-to-fair. Only the evaluation of the navicular bone showed moderate agreement with a kappa score of 0.52.
  • Relative agreement for pathology grading, which accounts for the ranking of grades, varied from a low of 0.19 (for the distal interphalangeal joint) up to 0.70 (for the navicular bone).
  • The agreement was typically highest at the extremes of pathology, that is, when the disease was either very severe or non-existent.
  • Interestingly, observer confidence in pathology assessment was generally moderate to high, despite the evident differences in their pathology evaluations.

Conclusion

  • Pathology distribution varied across different anatomical structures due to the random selection procedure of clinical MRI studies.
  • Most observers had more experience with low-field MRI, which might have influenced their evaluations.
  • The study concluded that even experienced observers can have marked differences in their evaluations of the severity of foot pathology in MRI studies. This discrepancy in perception can have significant implications for clinical decisions.

Cite This Article

APA
Byrne CA, Voute LC, Marshall JF. (2024). Interobserver agreement during clinical magnetic resonance imaging of the equine foot. Equine Vet J, 57(2), 406-418. https://doi.org/10.1111/evj.14126

Publication

Equine veterinary journal
ISSN: 2042-3306
NlmUniqueID: 0173320
Country: United States
Language: English
Volume: 57
Issue: 2
Pages: 406-418

Researcher Affiliations

Byrne, Christian A
  • School of Veterinary Medicine, College of Medical, Veterinary and Life Sciences, University of Glasgow, Glasgow, UK.
Voute, Lance C
  • School of Veterinary Medicine, College of Medical, Veterinary and Life Sciences, University of Glasgow, Glasgow, UK.
Marshall, John F
  • School of Veterinary Medicine, College of Medical, Veterinary and Life Sciences, University of Glasgow, Glasgow, UK.

MeSH Terms

  • Animals
  • Horses / anatomy & histology
  • Magnetic Resonance Imaging / veterinary
  • Observer Variation
  • Horse Diseases / diagnostic imaging
  • Horse Diseases / pathology
  • Foot Diseases / veterinary
  • Foot Diseases / diagnostic imaging
  • Foot Diseases / pathology
  • Foot / pathology
  • Foot / diagnostic imaging

Grant Funding

  • SPrj033 / Horserace Betting Levy Board

References

This article includes 51 references
  1. Dyson SJ, Murray RC, Schramme M, Branch M. Magnetic resonance imaging of the equine foot: 15 horses. Equine Vet J 2003;35(1):18–26.
  2. Dyson SJ, Murray RC. Chapter 12: The foot and pastern. In: Murray RC, editor. Equine MRI. Chichester: John Wiley & Sons; 2010. p. 269–314.
  3. Barrett MF, Frisbie DD, King MR, Werpy NM, Kawcak CE. A review of how magnetic resonance imaging can aid in case management of common pathological conditions of the equine foot. Equine Vet Educ 2017;29(12):683–693.
  4. Bolas N. Basic MRI principles. In: Murray RC, editor. Equine MRI. Volume 19. Chichester: John Wiley & Sons; 2010. p. 3–37.
  5. Ganesan A, Alakhras M, Brennan PC, Mello‐Thoms C. A review of factors influencing radiologists' visual search behaviour. J Med Imaging Radiat Oncol 2018;62(6):747–757.
  6. Williams LH, Drew T. What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures. Cogn Res Princ Implic 2019;4(1):21.
  7. Bladon B. Magnetic resonance imaging and foot lameness. Problem solved? Or do we know we know less now that we know more?. Equine Vet J 2014;46(3):264–266.
  8. Krupinski EA. Current perspectives in medical image perception. Atten Percept Psychophys 2010;72(5):1205–1217.
  9. Tourassi G, Voisin S, Paquit V, Krupinski E. Investigating the link between radiologists' gaze, diagnostic decision, and image content. J Am Med Inform Assoc 2013;20(6):1067–1075.
  10. Brunyé TT, Drew T, Weaver DL, Elmore JG. A review of eye tracking for understanding and improving diagnostic interpretation. Cogn Res Princ Implic 2019;4(1):7.
  11. Alexander RG, Waite S, Macknik SL, Martinez‐Conde S. What do radiologists look for? Advances and limitations of perceptual learning in radiologic search. J Vis 2020;20(10):17.
  12. Murray RC, Dyson S, Branch M, Schramme M. Validation of magnetic resonance imaging use in equine limbs. Clin Techn Equine Pract 2007;6(1):26–36.
  13. Evrard L, Audigié F, Bertoni L, Jacquet S, Denoix JM, Busoni V. Low field magnetic resonance imaging of the equine distal interphalangeal joint: comparison between weight‐bearing and non‐weight‐bearing conditions. PLoS One 2019;14(1):e0211101.
  14. Berner D, Mader D, Groß C, Gerlach K. Effect of scan plane and arthrography on visibility and interobserver agreement of the equine distal sesamoidean impar ligament on magnetic resonance images. J Equine Vet Sci 2020;94:103227.
  15. van Zadelhoff C, Schwarz T, Smith S, Engerand A, Taylor S. Identification of naturally occurring cartilage damage in the equine distal interphalangeal joint using low‐field magnetic resonance imaging and magnetic resonance arthrography. Front Vet Sci 2020;6:508.
  16. Baker ME, Kershaw LE, Carstens A, Daniel CR, Brown H, Roberts S. T2 mapping of cartilage in the equine distal interphalangeal joint with corresponding histology using 0.27 T and 3.0 T magnetic resonance imaging. Equine Vet J 2023;55(5):843–852.
  17. Bolen G, Audigié F, Spriet M, Vandenberghe F, Busoni V. Qualitative comparison of 0.27T, 1.5T, and 3T magnetic resonance images of the normal equine foot. J Equine Vet Sci 2010;30(1):9–20.
  18. Whitmore JK, Dakin SG, Weller R, Mair TS, Sherlock CE. Interobserver agreement of standing magnetic resonance imaging for assessment of deep digital flexor tendon pathology. Equine Vet J 2014;46(S47):25.
  19. Black B, Cribb NC, Nykamp SG, Thomason JJ, Trout DR. The effects of perineural and intrasynovial anaesthesia of the equine foot on subsequent magnetic resonance images. Equine Vet J 2013;45(3):320–325.
  20. Ramsay L, Lorenz I, Brix L, Murray R, Gerke O, Hess S. Agreement and reliability of lesion detection in the equine metacarpo(tarso)phalangeal joint using standing low‐field MRI and arthroscopy. 2022.
    doi: 10.2139/ssrn.4258748google scholar: lookup
  21. Byrne CA, Marshall JF, Voute LC. Clinical magnetic resonance image quality of the equine foot is significantly influenced by acquisition system. Equine Vet J 2021;53(3):469–480.
  22. Murray RC, Schramme MC, Dyson SJ, Branch MV, Blunden TS. Magnetic resonance imaging characteristics of the foot in horses with palmar foot pain and control horses. Vet Radiol Ultrasound 2006;47(1):1–16.
  23. Mair TS, Kinns J, Jones RD, Bolas NM. Magnetic resonance imaging of the distal limb of the standing horse: technique and review of 40 cases of foot lameness. Proc Am Assoc Equine Pract 2003;49:29–41.
  24. Dyson SJ, Murray RC, Schramme M. Lameness associated with foot pain: results of magnetic resonance imaging in 199 horses (January 2001‐December 2003) and response to treatment. Equine Vet J 2005;37(2):113–121.
  25. Sampson SN, Schneider RK, Gavin PR, Ho CP, Tucker RL, Charles EM. Magnetic resonance imaging findings in horses with recent onset navicular syndrome but without radiographic abnormalities. Vet Radiol Ultrasound 2009;50(4):339–346.
  26. Gutierrez‐Nibeyro SD, Werpy NM, White NA. Standing low‐field magnetic resonance imaging in horses with chronic foot pain. Aust Vet J 2012;90(3):75–83.
  27. Altman DG. Practical statistics for medical research. New York: Chapman and Hall; 1990.
  28. Gwet KL. Handbook of inter‐rater reliability, 4th edition: the definitive guide to measuring the extent of agreement among raters. Gaithersburg, MD: Advanced Analytics; 2014.
  29. Minitab. Kappa statistics and Kendall's coefficients. 2020 Available from: https://support.minitab.com/en-us/minitab/19/help-and-how-to/quality-and-process-improvement/measurement-system-analysis/supporting-topics/attribute-agreement-analysis/kappa-statistics-and-kendall-s-coefficients/.
  30. Nelson JC, Pepe MS. Statistical description of interrater variability in ordinal ratings. Stat Methods Med Res 2000;9(5):475–496.
  31. Esselman AM, Johnson SA, Frisbie DD, Barrett MF, Zhou T, Contino EK. Substantial variability exists in the interpretation of survey radiographs among equine veterinarians. Equine Vet J 2025;57(1):169–182.
    doi: 10.1111/evj.14045google scholar: lookup
  32. Cohen J, Fischetti AJ, Daverio H. Veterinary radiologic error rate as determined by necropsy. Vet Radiol Ultrasound 2023;64(4):573–584.
  33. Donald JJ, Barnard SA. Common patterns in 558 diagnostic radiology errors. J Med Imaging Radiat Oncol 2012;56(2):173–178.
  34. Schramme M, Segar‐Weisse E. Magnetic resonance imaging. In: Baxter GM, editor. Adams and Stashak's lameness in horses. Hoboken, NJ: Wiley; 2020. p. 387–430.
  35. Drew T, Vo MLH, Olwal A, Jacobson F, Seltzer SE, Wolfe JM. Scanners and drillers: characterizing expert visual search through volumetric images. J Vis 2013;13(10):3.
    doi: 10.1167/13.10.3google scholar: lookup
  36. Wu CC, Wolfe JM. Eye movements in medical image perception: a selective review of past, present and future. Vision 2019;3(2):32.
    doi: 10.3390/vision3020032google scholar: lookup
  37. Kliewer MA, Hartung M, Green CS. The search patterns of abdominal imaging subspecialists for abdominal computed tomography: toward a foundational pattern for new radiology residents. J Clin Imaging Sci 2021;11:1.
  38. Murray RC, Werpy N. Image interpretation and artefacts. In: Murray RC, editor. Equine MRI. Chichester: John Wiley & Sons; 2010. p. 101–145.
  39. Spriet M, Mai W, McKnight A. Asymmetric signal intensity in normal collateral ligaments of the distal interphalangeal joint in horses with a low‐field MRI system due to the magic angle effect. Vet Radiol Ultrasound 2007;48(2):95–100.
  40. Smith MA, Dyson SJ, Murray RC. Is a magic angle effect observed in the collateral ligaments of the distal interphalangeal joint or the oblique sesamoidean ligaments during standing magnetic resonance imaging?. Vet Radiol Ultrasound 2008;49(6):509–515.
  41. Murray RC, Blunden TS, Schramme MC, Dyson SJ. How does magnetic resonance imaging represent histologic findings in the equine digit?. Vet Radiol Ultrasound 2006;47(1):17–31.
  42. Murray RC, Mair TS, Sherlock CE, Blunden AS. Comparison of high‐field and low‐field magnetic resonance images of cadaver limbs of horses. Vet Rec 2009;165(10):281–288.
  43. Dyson SJ, Pool R, Blunden T, Murray RC. The distal sesamoidean impar ligament: comparison between its appearance on magnetic resonance imaging and histology of the axial third of the ligament. Equine Vet J 2010;42(4):332–339.
  44. Kottmeier LK, Seehusen F, Helweg M, Rohn K, Stadler P, Hellige M. High‐field (3 Tesla) MRI of the navicular apparatus of sound horses shows good agreement to histopathology. Vet Radiol Ultrasound 2020;61(1):48–57.
  45. Watson PF, Petrie A. Method agreement analysis: a review of correct methodology. Theriogenology 2010;73(9):1167–1179.
  46. Bowers D. Medical statistics from scratch: an introduction for health professionals. Hoboken, NJ: Wiley; 2019.
  47. Shankar V, Bangdiwala SI. Observer agreement paradoxes in 2x2 tables: comparison of agreement measures. BMC Med Res Methodol 2014;14:100.
  48. Smith MA. Optimising the use of MRI in the management of foot lameness in the horse. Livestock 2015;20(5):290–293.
  49. Hallmarq Veterinary Imaging. Above and beyond: over 100,000 horses scanned in over 100 Standing Equine MRI systems. 2019 Available from: http://hallmarq.net/wp-content/uploads/2019/07/100th-System-and-beyond-PR-310719.pdf.
  50. Werpy NM. Low‐field MRI in horses: practicalities and image acquisition. In: Murray RC, editor. Equine MRI. Chichester: John Wiley & Sons; 2010. p. 75–99.
  51. Minitab. Data considerations for Attribute Agreement Analysis. 2023 Available from: https://support.minitab.com/en‐us/minitab/21/help‐and‐how‐to/quality‐and‐process‐improvement/measurement‐system‐analysis/how‐to/attribute‐agreement‐analysis/attribute‐agreement‐analysis/before‐you‐start/data‐considerations/1.
Subscribe to our newsletter
Join 99,357 horse owners like you who receive our email updates on horse health and nutrition.