Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses.
Abstract: Orthopedic disorders are common among horses, often leading to euthanasia, which often could have been avoided with earlier detection. These conditions often create varying degrees of subtle long-term pain. It is challenging to train a visual pain recognition method with video data depicting such pain, since the resulting pain behavior also is subtle, sparsely appearing, and varying, making it challenging for even an expert human labeller to provide accurate ground-truth for the data. We show that a model trained solely on a dataset of horses with acute experimental pain (where labeling is less ambiguous) can aid recognition of the more subtle displays of orthopedic pain. Moreover, we present a human expert baseline for the problem, as well as an extensive empirical study of various domain transfer methods and of what is detected by the pain recognition method trained on clean experimental pain in the orthopedic dataset. Finally, this is accompanied with a discussion around the challenges posed by real-world animal behavior datasets and how best practices can be established for similar fine-grained action recognition tasks. Our code is available at https://github.com/sofiabroome/painface-recognition.
Publication Date: 2022-03-04 PubMed ID: 35245288PubMed Central: PMC8896717DOI: 10.1371/journal.pone.0263854Google Scholar: Lookup
The Equine Research Bank provides access to a large database of publicly available scientific literature. Inclusion in the Research Bank does not imply endorsement of study methods or findings by Mad Barn.
- Journal Article
- Research Support
- Non-U.S. Gov't
Summary
This research summary has been generated with artificial intelligence and may contain errors and omissions. Refer to the original study to confirm details provided. Submit correction.
This study explores the use of a computer model trained to recognize acute pain in horses from video data as a tool for detecting subtler, chronic pain associated with orthopedic conditions. Via domain transfer methods and baseline expertise, the researchers demonstrate the potential of this technology as an early detection tool for equine orthopedic disorders.
Clarification of Research Purpose
- This research aims to create a machine learning model that can identify subtle signs of long-term orthopedic pain in horses.
- The need for this arises from the difficulty of identifying such pain early on, often leading to late-stage diagnosis and substantial consequences such as euthanasia.
- Currently, consistently accurate labels for training such a model are difficult to establish since it’s challenging even for human experts to distinguish these subtle pain indicators.
Data and Model Training
- A dataset of video footage exhibiting horses with acute, experimental pain was used to train the model. This was done since acute pain manifestations are less ambiguous and are easier to label than subtle, chronic pain.
- The hope was that a model trained in this manner could then identify more nuanced and varied signs of long-term orthopedic pain in horses by transferring knowledge from the acute pain domain.
Application & Study Findings
- The study presents an empirical investigation into various domain transfer methods, that is, how to best train the model on acute pain data to recognize chronic orthopedic pain.
- The effectiveness of the recognition model, trained on clear experimental pain, is then tested on the orthopedic dataset.
- The results revealed the potential of the model in identifying orthopedic pain, implicitly demonstrating the practical use of this technology for early diagnosis.
Significance and Future Remarks
- The findings open up a discussion on the challenges inherent to using machine learning on real-world animal behavior datasets and how best to address these.
- With this study, best practices can be established for similar fine-grained action recognition tasks in the future.
- Overall, this research has important implications for veterinary medicine and for furthering humane treatment of animals, by aiding in the early detection and treatment of painful medical conditions.
- The code used in this research is publicly available, fostering further innovation and application by other researchers in this field.
Cite This Article
APA
Broomé S, Ask K, Rashid-Engström M, Haubro Andersen P, Kjellström H.
(2022).
Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses.
PLoS One, 17(3), e0263854.
https://doi.org/10.1371/journal.pone.0263854 Publication
Researcher Affiliations
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden.
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, Uppsala, Sweden.
- Department of Computer Science, University of California, Davis, California, United States of America.
- Univrses, Stockholm, Sweden.
- Department of Clinical Sciences, Swedish University of Agricultural Sciences, Uppsala, Sweden.
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden.
- Silo AI, Stockholm, Sweden.
MeSH Terms
- Animals
- Communications Media
- Horses
- Pain / veterinary
Conflict of Interest Statement
The authors have declared that no competing interests exist.
References
This article includes 76 references
- Taylor PM, Pascoe PJ, Mama KR. Diagnosing and treating pain in the horse. Where are we today?. Veterinary Clinics of North America—Equine Practice 2002;18(1):1–19.
- Torcivia C, McDonnell S. In-Person Caretaker Visits Disrupt Ongoing Discomfort Behavior in Hospitalized Equine Orthopedic Surgical Patients. Animals 2020;10(2).
- Penell JC, Egenvall A, Bonnett BN, Olson P, Pringle J. Specific causes of morbidity among Swedish horses insured for veterinary care between 1997 and 2000. Veterinary Record 2005;157:470—477.
- Pollard D, Wylie CE, Newton JR, Verheyen KLP. Factors Associated with Euthanasia in Horses and Ponies Enrolled in a Laminitis Cohort Study in Great Britain. Preventive Veterinary Medicine 2020;174(November 2019):104833.
- Slayter J, Taylor G. National Equine Health Survey (NEHS) 2018. 2018.
- Sneddon LU, Elwood RW, Adamo SA, Leach MC. Defining and assessing animal pain. Animal Behaviour 97:201–212.
- Ashley FH, Waterman-Pearson AE, Whay HR. Behavioural assessment of pain in horses and donkeys: application to clinical practice and future studies. Equine Veterinary Journal 2005;37(6):565–575.
- Loeser JD, Treede RD. The Kyoto protocol of IASP Basic Pain Terminology. Pain 2008;137(3):473–477.
- Romero R, Souzdalnitski D, Banack T. Ischemic and Visceral Pain. Springer Verlag; New York; 2011.
- Gu C, Sun C, Ross DA, Vondrick C, Pantofaru C, Li Y. AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2018.
- Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S. The Kinetics Human Action Video Dataset. CoRR 2017;abs/1705.06950.
- Soomro K, Zamir AR, Shah M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR 2012;abs/1212.0402.
- Mahdisoltani F, Berger G, Gharbieh W, Fleet DJ, Memisevic R. Fine-grained Video Classification and Captioning. CoRR 2018;abs/1804.09235.
- Li Y, Li Y, Vasconcelos N. RESOUND: Towards Action Recognition without Representation Bias. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018.
- Shao D, Zhao Y, Dai B, Lin D. FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2020.
- Ask K, Rhodin M, Tamminen LM, Hernlund E, Haubro Andersen P. Identification of Body Behaviors and Facial Expressions Associated with Induced Orthopedic Pain in Four Equine Pain Scales. Animals 2020;10(11).
- Costa ED, Minero M, Lebelt D, Stucke D, Canali E, Leach MC. Development of the Horse Grimace Scale (HGS) as a Pain Assessment Tool in Horses Undergoing Routine Castration. PLoS ONE 2014;9(3).
- Dalla Costa E, Stucke D, Dai F, Minero M, Leach MC, Lebelt D. Using the Horse Grimace Scale (HGS) to Assess Pain Associated with Acute Laminitis in Horses (Equus Caballus). Animals 2016;6(47):1–9.
- Gleerup KB, Lindegaard C. Recognition and quantification of pain in horses: A tutorial review. Equine Veterinary Education 2016;28(1):47–57.
- van Loon JPAM, Van Dierendonck MC. Objective pain assessment in horses (2014–2018). Veterinary Journal 2018;242:1–7.
- Gleerup KB, Forkman B, Lindegaard C, Andersen PH. An equine pain face. Veterinary Anaesthesia and Analgesia 2015;42.
- Broomé S, Gleerup KB, Andersen PH, Kjellström H. Dynamics Are Important for the Recognition of Equine Pain in Video. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2019.
- Islam A, Radke R. Weakly Supervised Temporal Action Localization Using Deep Metric Learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV); 2020.
- Rashid M, Kjellström H, Lee YJ. Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks. In: The IEEE Winter Conference on Applications of Computer Vision; 2020. p. 615–624.
- Paul S, Roy S, Roy-Chowdhury AK. W-TALC: Weakly-supervised Temporal Activity Localization and Classification. In: Proceedings of the European Conference on Computer Vision (ECCV); 2018. p. 563–579.
- Nguyen P, Liu T, Prasad G, Han B. Weakly Supervised Action Localization by Sparse Temporal Pooling Network. In: CVPR; 2018.
- Wang L, Xiong Y, Lin D, Van Gool L. Untrimmednets for weakly supervised action recognition and detection. In: CVPR; 2017.
- Ilse M, Tomczak J, Welling M. Attention-based Deep Multiple Instance Learning. In: Dy J, Krause A, editors. 35th International Conference on Machine Learning, ICML 2018. 35th International Conference on Machine Learning, ICML 2018. International Machine Learning Society (IMLS); 2018. p. 3376–3391.
- Li X, Lang Y, Chen Y, Mao X, He Y, Wang S. Sharp Multiple Instance Learning for DeepFake Video Detection. Proceedings of the 28th ACM International Conference on Multimedia 2020.
- Wang X, Yan Y, Tang P, Bai X, Liu W. Revisiting Multiple Instance Neural Networks. arXiv preprint arXiv:161002501 2016.
- Cinbis RG, Verbeek J, Schmid C. Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 2017;39(1):189–203.
- Carreira J, Zisserman A. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In: CVPR; 2017.
- Zhai Y, Wang L, Liu Z, Zhang Q, Hua G, Zheng N. Action Coherence Network for Weakly Supervised Temporal Action Localization. In: ICIP; 2019.
- Singh KK, Lee YJ. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In: ICCV; 2017.
- Zeng R, Gan C, Chen P, Huang W, Wu Q, Tan M. Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization. IEEE Transactions on Image Processing 2019.
- Arnab A, Sun C, Nagrani A, Schmid C. Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos. In: Vedaldi A, Bischof H, Brox T, Frahm JM, editors. Computer Vision—ECCV 2020. Cham: Springer International Publishing; 2020. p. 751–768.
- Yang P, Hu VT, Mettes P, Snoek CGM. Localizing the Common Action Among a Few Videos. In: Vedaldi A, Bischof H, Brox T, Frahm JM, editors. Computer Vision—ECCV 2020. Cham: Springer International Publishing; 2020. p. 505–521.
- Lu Y, Mahmoud M, Robinson P. Estimating Sheep Pain Level Using Facial Action Unit Detection. In: 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017); 2017.
- Hummel HI, Pessanha F, Salah A, van Loon TM, Veltkamp RC. Automatic Pain Detection on Horse and Donkey Faces. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (FG). Los Alamitos, CA, USA: IEEE Computer Society; 2020. p. 717–724.
- Pessanha P, McLennan K, Mahmoud M. Towards automatic monitoring of disease progression in sheep: A hierarchical model for sheep facial expressions analysis from video. In: 15th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2020); 2020.
- Tuttle AH, Molinaro MJ, Jethwa JF, Sotocinal SG, Prieto JC, Styner MA. A deep neural network to assess spontaneous pain from mouse facial expressions. Molecular Pain 2018;14:1744806918763658.
- Andresen N, Wöllhaf M, Hohlbaum K, Lewejohann L, Hellwich O, Thöne-Reineke C. Towards a fully automated surveillance of well-being status in laboratory mice using deep learning: Starting with facial expression analysis. PLOS ONE 2020;15(4):1–23.
- Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 2015;115(3):211–252.
- He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. p. 770–778.
- Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the Inception Architecture for Computer Vision. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition; 2016.
- Lencioni GC, de Sousa RV, de Souza Sardinha EJ, Corrêa RR, Zanella AJ. Pain assessment in horses using automatic facial expression recognition through deep learning-based modeling. PLOS ONE 2021;16(10):1–12.
- Li Z, Broomé S, Andersen PH, Kjellström H. Automated Detection of Equine Facial Action Units. ArXiv 2021;abs/2102.08983.
- Wathan J, Burrows AM, Waller BM, McComb K. EquiFACS: The Equine Facial Action Coding System. PLOS ONE 2015;10(8):1–35.
- Rashid M, Broomé S, Ask K, Hernlund E, Andersen PH, Kjellström H. Equine Pain Behavior Classification via Self-Supervised Disentangled Pose Representation. In: Winter Conference on Applications of Computer Vision (WACV), to appear.; 2022.
- Cohen S, Beths T. Grimace Scores: Tools to Support the Identification of Pain in Mammals Used in Research. Animals 2020;10(10).
- Mogil JS, Pang DSJ, Silva Dutra GG, Chambers CT. The Development and Use of Facial Grimace Scales for Pain Measurement in Animals. Neuroscience & Biobehavioral Reviews 2020;116:480–493.
- van Weeren PR, de Grauw JC. Pain in Osteoarthritis. Veterinary Clinics of North America: Equine Practice 2010;26(3):619–642.
- Schaible H. Mechanisms of Chronic Pain in Osteoarthritis. Current Rheumatology Reports 2012;14:549–556.
- Van de Water E, Oosterlinck M, Korthagen NM, Duchateau L, Dumoulin M, van Weeren PR. The lipopolysaccharide model for the experimental induction of transient lameness and synovitis in Standardbred horses. The Veterinary Journal 2021;270:105626.
- Bussiéres G, Jacques C, Lainay O, Beauchamp G, Leblond A, Cadoré JL. Development of a composite orthopaedic pain scale in horses. Res Vet Sci 2008;85(2).
- Shi X, Chen Z, Wang H, Yeung D, Wong W, Woo W. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Q, Canada; 2015. p. 802–810.
- Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Comput 1997;9(8):1735–1780.
- Wiese AJ, Gaynor JS, Muir W. Assessing Pain: Pain Behaviors, in Handbook of veterinary pain management (Third Edition). 2015.
- de Camp Nora V, LW Mechthild, G CIE, Thöne-Reineke Christa, B J. EEG based assessment of stress in horses: A pilot study. PeerJ 2020;8:e8629:1–15.
- Andreassen SM, Vinther AML, Nielsen SS, Andersen PH, Tnibar A, Kristensen AT. Changes in concentrations of haemostatic and inflammatory biomarkers in synovial fluid after intra-articular injection of lipopolysaccharide in horses. BMC Veterinary Research 2017;13(1):182.
- Rashid M, Silventoinen A, Gleerup KB, Andersen PH. Equine Facial Action Coding System for determination of pain-related facial responses in videos of horses. PLOS ONE 2020;15(11):1–18.
- Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision 2017;2017-Octob:618–626.
- Andersen PH, Broomé S, Rashid M, Lundblad J, Ask K, Li Z. Towards Machine Recognition of Facial Expressions of Pain in Horses. Animals 2021;11(6), 1643.
- Waran N, Williams V, Clarke N, Bridge I. Recognition of pain and use of analgesia in horses by veterinarians in New Zealand. New Zealand Veterinary Journal 2010;58:274—280.
- Farina S, Valeriani M, Rosso T, Aglioti SM, Tinazzi M. Transient inhibition of the human motor cortex by capsaicin-induced pain. A study with transcranial magnetic stimulation. Neuroscience Letters 2001;314:97–101.
- Tuveson B, Leffler AS, Hansson PT. Time dependant differences in pain sensitivity during unilateral ischemic pain provocation in healthy volunteers. European Journal of Pain 2006;10.
- Lutgendorf SK, Logan H, Kirchner HL, Rothrock NE, Svengalis S, Iverson K. Effects of Relaxation and Stress on the Capsaicin-Induced Local Inflammatory Response. Psychosomatic Medicine 2000;62:524–534.
- Rhodin M, Persson-Sjodin E, Egenvall A, Bragança FMS, Pfau T, Roepstorff L. Vertical movement symmetry of the withers in horses with induced forelimb and hindlimb lameness at trot. Equine Veterinary Journal 2018;50:818—824.
- McLennan KM, Rebelo CJB, Corke MJ, Holmes MA, Leach MC, Constantino-Casas F. Development of a facial expression scale using footrot and mastitis as models of pain in sheep. Applied Animal Behaviour Science 2016;176:19–26.
- Hayashi K, Ikemoto T, Ueno T, Arai YCP, Shimo K, Nishihara M. Discordant Relationship Between Evaluation of Facial Expression and Subjective Pain Rating Due to the Low Pain Magnitude. Basic and Clinical Neuroscience Journal 2018;9(1).
- Dalla Costa E, Pascuzzo R, Leach MC, Dai F, Lebelt D, Vantini S. Can grimace scales estimate the pain status in horses and mice? A statistical approach to identify a classifier. PLOS ONE 2018;13(8):1–17.
- Pilitsis JG, Fahey M, Custozzo A, Chakravarthy K, Capobianco R. Composite Score Is a Better Reflection of Patient Response to Chronic Pain Therapy Compared With Pain Intensity Alone. Neuromodulation: Technology at the Neural Interface 2021;24(1):68–75.
- Trindade PHE, Taffarel MO, Luna SPL. Spontaneous Behaviors of Post-Orchiectomy Pain in Horses Regardless of the Effects of Time of Day, Anesthesia, and Analgesia. Animals 2021;11(6).
- Cao J, Tang H, Fang HS, Shen X, Lu C, Tai YW. Cross-Domain Adaptation for Animal Pose Estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); 2019.
- Mathis A, Biasi T, Schneider S, Yuksekgonul M, Rogers B, Bethge M. Pretraining Boosts Out-of-Domain Robustness for Pose Estimation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV); 2021. p. 1859–1868.
- Lundblad J, Rashid M, Rhodin M, Andersen PH. Effect of transportation and social isolation on facial expressions of healthy horses. PLoS One 2021;16(6).
Use Nutrition Calculator
Check if your horse's diet meets their nutrition requirements with our easy-to-use tool Check your horse's diet with our easy-to-use tool
Talk to a Nutritionist
Discuss your horse's feeding plan with our experts over a free phone consultation Discuss your horse's diet over a phone consultation
Submit Diet Evaluation
Get a customized feeding plan for your horse formulated by our equine nutritionists Get a custom feeding plan formulated by our nutritionists