Multi-Modal Feature Fusion and Hierarchical Classification for Automated Equine-Human Interaction Behavior Recognition.

Overview

• This research focuses on developing an automated system that recognizes interaction behaviors between horses and humans using video data.
• The system integrates multiple types of computer vision features and a hierarchical classification framework to accurately identify complex behaviors despite challenges like class imbalance and cross-species interaction.

Research Background and Motivation

• Automated recognition of animal behavior from videos is important for applications such as animal welfare, service animal evaluation, and safety monitoring in equestrian activities.
• Previous works mainly emphasize:
  • Recognition of single-species behaviors (either human or animal alone).
  • Analysis based primarily on facial expressions, neglecting full body posture.
  • Flat classification models which struggle with natural datasets containing very uneven distributions of behavior classes.
• No prior framework focuses on simultaneous, cross-species analysis of human and horse body language during direct interactions.

Methodology

• The paper proposes a novel hierarchical classification framework leveraging multimodal computer vision features to recognize horse-human interaction behaviors.
• Three distinct feature extraction pipelines are combined:
  • YOLOv8: To model spatial relationships between human and horse, capturing proximity and positioning in the scene.
  • MediaPipe: To analyze human body posture, capturing their movements and gestures.
  • AP-10K: To interpret equine body language by estimating horse poses and movements.
• Data specifics:
  • 28 annotated videos of horse-human interactions were used.
  • The dataset includes 50,270 temporal samples from five different horse breeds.
  • From these samples, 35 discriminative features were extracted, describing proximity dynamics, body orientations, and species-specific behavioral signs.
• Challenges addressed:
  • The dataset has severe class imbalance: affiliative behaviors are 18.3 times more frequent than avoidant behaviors.
  • A cost-sensitive gradient boosting algorithm with automatic class weight adjustments was implemented to mitigate this imbalance.
• Classification Framework:
  • It is a two-stage hierarchical model:
    • Stage 1: Classifies behaviors into three broad parent categories—affiliative, neutral, and avoidant.
    • Stage 2: Further distinguishes six specific sub-behaviors within the parent categories.

Results and Performance

• Stage 1 (parent category classification) achieved 73.2% balanced accuracy.
• Stage 2 (fine-grained sub-behavior classification) achieved 88.5% balanced accuracy with oracle-level routing (meaning when given perfect parent category input).
• Cascading both stages end-to-end resulted in 62.9% balanced accuracy, with the main error source identified as the initial parent classification bottleneck.
• The system achieved an 85.0% recall rate for safety-critical avoidant behaviors, despite these being only 3.8% of the dataset—highlighting successful recognition of rare but crucial events.
• Ablation studies revealed that equine pose features from AP-10K contributed most significantly to the classification accuracy.
• Cross-validation tests confirmed robust performance across diverse contexts and horse breeds, indicating good generalizability.

Contributions and Significance

• This study presents the first comprehensive multimodal and cross-species framework for automatic behavioral analysis of horse-human interactions.
• The hierarchical two-stage classification effectively handles severe class imbalance and complex behavior taxonomies.
• Integrating human and equine body language cues provides a richer, more nuanced understanding of the interaction dynamics than single-species or single-modality approaches.
• Practical implications include:
  • Enhanced animal welfare monitoring by automatically detecting subtle affiliative or avoidant signals.
  • Improved safety protocols for riders and handlers through early identification of potentially unsafe behaviors.
  • Potential to aid in equine-assisted therapy and service animal program evaluations by providing objective interaction assessments.

Summary

• The paper introduces a novel, multimodal, hierarchical classification system that effectively recognizes horse-human interaction behaviors from video data.
• It overcomes previous limitations by combining spatial, human posture, and equine pose features and mitigating class imbalance through cost-sensitive learning.
• Results demonstrate the method’s robustness and applicability for real-world ethology and equestrian safety use cases.