Identifying Novel Emotions and Wellbeing of Horses from Videos Through Unsupervised Learning.
- Journal Article
Summary
This study utilizes machine learning to learn and identify new emotional states in horses from video footage. It employs a modified version of an established machine learning framework called Momentum Contrast (MoCo) to create a novel dataset and classifier for horse emotions.
Data Collection
The research team compiled a dataset of 3929 images depicting five breeds of wild horses from various global locations. The dataset’s size and variety are key to allowing the unsupervised learning process to capture a wide range of horse behaviours and emotions.
- Unsupervised learning is a type of machine learning where a model learns to identify patterns without being explicitly told what to look for.
- The dataset used in this study, created through fieldwork and video analysis, is large and diverse. It includes different horse breeds located in various geographical areas.
Emotional Framework
The analysis adopted the seven mammalian emotions suggested by neuroscientist Jaak Panksepp: Exploration, Sadness, Play, Rage, Fear, Affection, and Lust. An additional emotion was included – Pain – which is particularly relevant in analysing horse behaviour.
- These emotions were not used as labels for the machine learning process but rather as a lens through which the outcomes could be understood.
Methodology
The Momentum Contrast (MoCo) machine learning framework was used to predict the identified emotions in an unsupervised manner. The MoCo framework was significantly modified to include a custom downstream classifier that connects with a pretrained CNN encoder.
- The MoCo framework was trained using the collected dataset to understand and map out the visual representation of horse emotions.
- The research introduces a new downstream classifier, which is a separate network designed to classify the output of the pretrained CNN encoder. The downstream classifier is not trained on any labelled data, enabling it to discover patterns independently.
Findings
The research methodology allowed the encoder network to autonomously learn the similarities and differences within image groups. The image clusters formed because of this learning process may indicate deeper complexity in a horse’s mood, potentially suggesting the existence of previously unidentified complex equine emotions.
- The image clusters are not pre-defined categories, but groups discovered by the machine learning algorithm. The formation of these clusters indicates that horses could potentially express more nuanced and complex emotions than previously understood.
Cite This Article
Publication
Researcher Affiliations
- Massachusetts Institute of Technology, System Design & Management, Cambridge, MA 02142, USA.
- Equine International, Cambridge CB22 5LD, UK.
- TUM School of Computation, Information and Technology, Technical University of Munich, Arcisstraße 21, 80333 Munich, Germany.
- Massachusetts Institute of Technology, System Design & Management, Cambridge, MA 02142, USA.
MeSH Terms
- Animals
- Horses / physiology
- Emotions / physiology
- Unsupervised Machine Learning
- Video Recording
Conflict of Interest Statement
References
- Li S, Deng W. Deep Facial Expression Recognition: A Survey. IEEE Trans. Affect. Comput. 2022;13:1195–1215.
- Chen T, Kornblith S, Norouzi M, Hinton G.E. A Simple Framework for Contrastive Learning of Visual Representations. arXiv 20202002.05709.
- He K, Fan H, Wu Y, Xie S, Girshick R.B. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv 20191911.05722.
- Grill J, Strub F, Altché F, Tallec C, Richemond P.H, Buchatskaya E, Doersch C, Pires B.Á, Guo Z.D, Azar M.G. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. arXiv 20202006.07733.
- Chen X, He K. Exploring Simple Siamese Representation Learning. arXiv 20202011.10566.
- Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Liu C, Krishnan D. Supervised Contrastive Learning. arXiv 20202004.11362.
- Li M, Yang B, Levy J, Stolcke A, Rozgic V, Matsoukas S, Papayiannis C, Bone D, Wang C. Contrastive Unsupervised Learning for Speech Emotion Recognition. arXiv 20212102.06357.
- Sun L, Lian Z, Liu B, Tao J. HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition. Inf. Fusion 2024;108:102382.
- Panksepp J. Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford University Press; Oxford, UK: 2004.
- Bhave A, Hafner A, Bhave A, Gloor P.A. Unsupervised Canine Emotion Recognition Using Momentum Contrast. Sensors 2024;24:7324.
- Chamove A.S, Crawley-Hartrick O.J, Stafford K.J. Horse reactions to human attitudes and behavior. Anthrozoös 2002;15:323–331.
- Wathan J, McComb K. The eyes and ears are visual indicators of attention in domestic horses. Curr. Biol. 2014;24:R677–R679.
- Proctor H.S, Carder G. Can ear postures reliably measure the positive emotional state of cows?. Appl. Anim. Behav. Sci. 2014;161:20–27.
- Hall C, Randle H, Pearson G, Preshaw L, Waran N. Assessing equine emotional state. Appl. Anim. Behav. Sci. 2018;205:183–193.
- Stewart M, Stratton R, Beausoleil N, Stafford K, Worth G, Waran N. Assessment of positive emotions in horses: Implications for welfare and performance. J. Vet. Behav. Clin. Appl. Res. 2011;5:296.
- Stratton R.B. Assessment of Positive Emotion in Horses. Ph.D. Thesis. Massey University; Manawatū, New Zealand: 2022.
- Waran N, Randle H. What we can measure, we can manage: The importance of using robust welfare indicators in Equitation Science. Appl. Anim. Behav. Sci. 2017;190:74–81.
- Feh C. Social behaviour and relationships of Prezewalski horses in Dutch semi-reserves. Appl. Anim. Behav. Sci. 1988;21:71–87.
- Feh C. Relationships and communication in socially natural horse herds. The Domestic Horse. Cambridge University Press; Cambridge, UK: 2005; pp. 83–93.
- Maeda T, Sueur C, Hirata S, Yamamoto S. Behavioural synchronization in a multilevel society of feral horses. PLoS ONE 2021;16:e0258944.
- Maeda T, Ochi S, Ringhofer M, Sosa S, Sueur C, Hirata S, Yamamoto S. Aerial drone observations identified a multilevel society in feral horses. Sci. Rep. 2021;11:71.
- Heitor F, do Mar Oom M, Vicente L. Social relationships in a herd of Sorraia horses: Part II. Factors affecting affiliative relationships and sexual behaviours. Behav. Process. 2006;73:231–239.
- Heitor F, Vicente L. Maternal care and foal social relationships in a herd of Sorraia horses: Influence of maternal rank and experience. Appl. Anim. Behav. Sci. 2008;113:189–205.
- Heitor F, do Mar Oom M, Vicente L. Social relationships in a herd of Sorraia horses: Part I. Correlates of social dominance and contexts of aggression. Behav. Process. 2006;73:170–177.
- Shimada M, Suzuki N. The contribution of mutual grooming to affiliative relationships in a feral misaki horse herd. Animals 2020;10:1564.
- Farmer K, Krüger K, Byrne R.W, Marr I. Sensory laterality in affiliative interactions in domestic horses and ponies (Equus caballus). Anim. Cogn. 2018;21:631–637.
- Kieson E, Sams J. A preliminary investigation of preferred affiliative interactions within and between select bonded pairs of horses: A first look at equine “Love Languages”. Int. J. Zool. Anim. Biol. 2021;4:000318.
- Zeitler-Feicht M.H, Hartmann E, Erhard M.H, Baumgartner M. Which affiliative behaviour can be used as a valid, reliable and feasible indicator of positive welfare in horse husbandry?. Appl. Anim. Behav. Sci. 2024;273:106236.
- Bartlett E, Cameron L.J, Freeman M.S. A preliminary comparison between proximity and interaction-based methods to construct equine (Equus caballus) social networks. J. Vet. Behav. 2022;50:36–45.
- Benhajali H, Richard-Yris M.A, Leroux M, Ezzaouia M, Charfi F, Hausberger M. A note on the time budget and social behaviour of densely housed horses: A case study in Arab breeding mares. Appl. Anim. Behav. Sci. 2008;112:196–200.
- Kieson E, Goma A.A, Radi M. Tend and Befriend in Horses: Partner Preferences, Lateralization, and Contextualization of Allogrooming in Two Socially Stable Herds of Quarter Horse Mares. Animals 2023;13:225.
- Lansade L, Lemarchand J, Reigner F, Arnould C, Bertin A. Automatic brushes induce positive emotions and foster positive social interactions in group-housed horses. Appl. Anim. Behav. Sci. 2022;246:105538.
- Van Dierendonck M, Sigurjónsdóttir H, Colenbrander B, Thorhallsdóttir A.G. Differences in social behaviour between late pregnant, post-partum and barren mares in a herd of Icelandic horses. Appl. Anim. Behav. Sci. 2004;89:283–297.
- Pugh H.K, Heatwole Shank K.S. Multispecies Occupations Involving Equines: An Action-Oriented Inquiry to Inform Occupational Therapy Practitioners. OTJR Occup. Ther. J. Res. 2024;44:196–204.
- Souris A.C, Kaczensky P, Julliard R, Walzer C. Time budget-, behavioral synchrony-and body score development of a newly released Przewalski’s horse group Equus ferus przewalskii, in the Great Gobi B strictly protected area in SW Mongolia. Appl. Anim. Behav. Sci. 2007;107:307–321.
- Mendonça R.S, Pinto P, Inoue S, Ringhofer M, Godinho R, Hirata S. Social determinants of affiliation and cohesion in a population of feral horses. Appl. Anim. Behav. Sci. 2021;245:105496.
- Wolter R, Stefanski V, Krueger K. Parameters for the analysis of social bonds in horses. Animals 2018;8:191.
- Burke C.J, Whishaw I.Q. Sniff, look and loop excursions as the unit of “exploration” in the horse (Equus ferus caballis) when free or under saddle in an equestrian arena. Behav. Process. 2020;173:104065.
- Pawluski J, Jego P, Henry S, Bruchet A, Palme R, Coste C, Hausberger M. Low plasma cortisol and fecal cortisol metabolite measures as indicators of compromised welfare in domestic horses (Equus caballus). PLoS ONE 2017;12:e0182257.
- Nuñez C.M, Adelman J.S, Smith J, Gesquiere L.R, Rubenstein D.I. Linking social environment and stress physiology in feral mares (Equus caballus): Group transfers elevate fecal cortisol levels. Gen. Comp. Endocrinol. 2014;196:26–33.
- Sauveroche M, Henriksson J, Theodorsson E, Holm A.C.S, Roth L.S. Hair cortisol in horses (Equus caballus) in relation to management regimes, personality, and breed. J. Vet. Behav. 2020;37:1–7.
- Heimbürge S, Kanitz E, Otten W. The use of hair cortisol for the assessment of stress in animals. Gen. Comp. Endocrinol. 2019;270:10–17.
- Johnson R.A, Johnson P.J, Megarani D.V, Patel S.D, Yaglom H.D, Osterlind S, Grindler K, Vogelweid C.M, Parker T.M, Pascua C.K. Horses working in therapeutic riding programs: Cortisol, adrenocorticotropic hormone, glucose, and behavior stress indicators. J. Equine Vet. Sci. 2017;57:77–85.
- Sgorbini M, Bonelli F, Rota A, Aurich C, Ille N, Camillo F, Panzani D. Determination of salivary cortisol in donkey stallions. bioRxiv 2018.
- Sikorska U, Maśko M, Ciesielska A, Zdrojkowski Ł, Domino M. Role of Cortisol in Horse’s Welfare and Health. Agriculture 2023;13:2219.
- Dai F, Cogi N.H, Heinzl E.U.L, Dalla Costa E, Canali E, Minero M. Validation of a fear test in sport horses using infrared thermography. J. Vet. Behav. 2015;10:128–136.
- Forkman B, Boissy A, Meunier-Salaün M.C, Canali E, Jones R.B. A critical review of fear tests used on cattle, pigs, sheep, poultry and horses. Physiol. Behav. 2007;92:340–374.
- McDonnell S.M. Reproductive behavior of stallions and mares: Comparison of free-running and domestic in-hand breeding. Anim. Reprod. Sci. 2000;60:211–219.
- Crowell-Davis S.L. Sexual behavior of mares. Horm. Behav. 2007;52:12–17.
- Stout T. Modulating reproductive activity in stallions: A review. Anim. Reprod. Sci. 2005;89:93–103.
- Boyd L.E. Time budgets of adult Przewalski horses: Effects of sex, reproductive status and enclosure. Appl. Anim. Behav. Sci. 1988;21:19–39.
- McDonnell S.M, Poulin A. Equid play ethogram. Appl. Anim. Behav. Sci. 2002;78:263–290.
- Torcivia C, McDonnell S. Equine discomfort ethogram. Animals 2021;11:580.
- Ludewig A, Gauly M, von Borstel U.K. Effect of shortened reins on rein tension, stress and discomfort behavior in dressage horses. J. Vet. Behav. Clin. Appl. Res. 2013;2:e15–e16.
- Arnold G, Grassia A. Ethogram of agonistic behaviour for thoroughbred horses. Appl. Anim. Ethol. 1982;8:5–25.
- McDonnell S, Haviland J. Agonistic ethogram of the equid bachelor band. Appl. Anim. Behav. Sci. 1995;43:147–188.
- Fureix C, Jego P, Henry S, Lansade L, Hausberger M. Towards an Ethological Animal Model of Depression? A Study on Horses. PLoS ONE 2012;7:e39280.
- Blois-Heulin C, Rochais C, Camus S, Fureix C, Lemasson A, Lunel C, Bezard E, Hausberger M. Animal welfare: Could adult play be a false friend?. Anim. Behav. Cogn. 2015;2:156–185.
- Hausberger M, Fureix C, Bourjade M, Wessel-Robert S, Richard-Yris M.A. On the significance of adult play: What does social play tell us about adult horse welfare?. Naturwissenschaften 2012;99:291–302.
- Sigurjónsdóttir H, Haraldsson H. Significance of group composition for the welfare of pastured horses. Animals 2019;9:14.
- VanDierendonck M.C, Spruijt B.M. Coping in groups of domestic horses–Review from a social and neurobiological perspective. Appl. Anim. Behav. Sci. 2012;138:194–202.
- Cameron E.Z, Linklater W.L, Stafford K.J, Minot E.O. Social grouping and maternal behaviour in feral horses (Equus caballus): The influence of males on maternal protectiveness. Behav. Ecol. Sociobiol. 2003;53:92–101.
- Broomé S, Feighelstein M, Zamansky A, Carreira Lencioni G, Haubro Andersen P, Pessanha F, Mahmoud M, Kjellström H, Salah A.A. Going deeper than tracking: A survey of computer-vision based recognition of animal pain and emotions. Int. J. Comput. Vis. 2023;131:572–590.
- Gleerup K.B, Forkman B, Lindegaard C, Andersen P.H. An equine pain face. Vet. Anaesth. Analg. 2015;42:103–114.
- Dalla Costa E, Minero M, Lebelt D, Stucke D, Canali E, Leach M.C. Development of the Horse Grimace Scale (HGS) as a pain assessment tool in horses undergoing routine castration. PLoS ONE 2014;9:e92281.
- Coneglian M.M, Borges T.D, Weber S.H, Bertagnon H.G, Michelotto P.V. Use of the horse grimace scale to identify and quantify pain due to dental disorders in horses. Appl. Anim. Behav. Sci. 2020;225:104970.
- Andersen P.H, Broomé S, Rashid M, Lundblad J, Ask K, Li Z, Hernlund E, Rhodin M, Kjellström H. Towards machine recognition of facial expressions of pain in horses. Animals 2021;11:1643.
- Andersen P.H, Gleerup K, Wathan J, Coles B, Kjellström H, Broomé S, Lee Y, Rashid M, Sonder C, Resenberg E. Can a machine learn to see horse pain? An interdisciplinary approach towards automated decoding of facial expressions of pain in the horse. Proc. Meas. Behav. 2018:6–8.
- Feighelstein M, Riccie-Bonot C, Hasan H, Weinberg H, Rettig T, Segal M, Distelfeld T, Shimshoni I, Mills D.S, Zamansky A. Automated recognition of emotional states of horses from facial expressions. PLoS ONE 2024;19:e0302893.
- Lencioni G.C, de Sousa R.V, de Souza Sardinha E.J, Corrêa R.R, Zanella A.J. Pain assessment in horses using automatic facial expression recognition through deep learning-based modeling. PLoS ONE 2021;16:e0258672.
- Chiavaccini L, Gupta A, Chiavaccini G. From facial expressions to algorithms: A narrative review of animal pain recognition technologies. Front. Vet. Sci. 2024;11:1436795.
- Lund S.M, Nielsen J, Gammelgård F, Nielsen M.G, Jensen T.H, Pertoldi C. Behavioral Coding of Captive African Elephants (Loxodonta africana): Utilizing DeepLabCut and Create ML for Nocturnal Activity Tracking. Animals 2024;14:2820.
- Blumrosen G, Hawellek D, Pesaran B. Towards automated recognition of facial expressions in animal models. Proceedings of the IEEE International Conference on Computer Vision Workshops Venice, Italy. 22–29 October 2017; pp. 2810–2819.
- Corujo L.A, Kieson E, Schloesser T, Gloor P.A. Emotion recognition in horses with convolutional neural networks. Future Internet 2021;13:250.
- Ashley F, Waterman-Pearson A, Whay H. Behavioural assessment of pain in horses and donkeys: Application to clinical practice and future studies. Equine Vet. J. 2005;37:565–575.
- Leiner L, Fendt M. Behavioural fear and heart rate responses of horses after exposure to novel objects: Effects of habituation. Appl. Anim. Behav. Sci. 2011;131:104–109.
- Yaseen M. What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector. arXiv 20242408.15857.
- Hadsell R, Chopra S, LeCun Y. Dimensionality Reduction by Learning an Invariant Mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) New York, NY, USA. 17–22 June 2006; pp. 1735–1742.
- van den Oord A, Li Y, Vinyals O. Representation Learning with Contrastive Predictive Coding. arXiv 20181807.03748.
- Deng J, Dong W, Socher R, Li L.J, Li K, Fei-Fei L. Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition Miami, FL, USA. 20–25 June 2009; pp. 248–255.
- Assiri Y. Stochastic Optimization of Plain Convolutional Neural Networks with Simple methods. arXiv 20202001.08856.
- Ali J. MNIST-SOPCNN. 2024. [(accessed on 1 December 2024)].
- . Image Classification on MNIST. 2024. [(accessed on 1 December 2024)].
- Hoemann K, Lee Y, Kuppens P, Gendron M, Boyd R.L. Emotional granularity is associated with daily experiential diversity. Affect. Sci. 2023;4:291–306.
- Hoxhallari K, Purcell W, Neubauer T. The potential of Explainable Artificial Intelligence in Precision Livestock Farming. In: Berckmans D., Oczak M., Iwersen M., Wagener K., editors. Precision Livestock Farming 2022: Papers Presented at the 10th European Conference on Precision Livestock Farming. University of Veterinary Medicine Vienna; Vienna, Austria: 2022. pp. 710–717.
- Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A. Unsupervised learning of visual features by contrasting cluster assignments. Adv. Neural Inf. Process. Syst. 2020;33:9912–9924.