Accuracy of large language model-based artificial intelligence tools for equine topics.

Accuracy of large language model-based artificial intelligence tools for equine topics evaluated their performance in answering questions on horse care and management.

Objective and Hypothesis

• The study aimed to assess the accuracy, relevance, thoroughness, and source quality of AI-generated responses on equine-related questions.
• Researchers hypothesized that AI platforms would perform well on basic questions but poorly on complex topics.

Methods

• Forty questions were designed covering diverse equine topics: general horse care, facilities management, nutrition, genetics, and reproduction.
• Questions were classified by difficulty into four categories: beginner, intermediate, advanced, and trending.
• Three AI platforms were tested:
  • ChatGPT (CGPT)
  • Microsoft Copilot (MicCP)
  • ExtensionBot (ExtBot)
• Each AI’s response was scored on four criteria: accuracy, relevance, thoroughness, and source quality, with a maximum of 5 points each (total score 20).
• Statistical analysis was performed using PROC GLM in SAS version 9.4 to determine differences by platform, question difficulty, and topic.

Key Results

• Total Score and Difficulty:
  • Question difficulty affected the total score significantly (P = 0.002).
  • Intermediate-level questions received the highest total scores (mean 15.95 out of 20, SD 1.99).
• Accuracy:
  • Platform, difficulty level, and topic significantly influenced accuracy (P < 0.001 for platform and level; P = 0.015 for topic).
  • ChatGPT scored highest on accuracy (4.18 ± 0.93), followed closely by Microsoft Copilot (4.08 ± 0.83).
  • ExtensionBot lagged behind with a lower accuracy score (3.26 ± 1.21).
• Relevance:
  • Platform (P = 0.042) and question difficulty (P < 0.001) significantly affected relevance scores.
• Thoroughness:
  • Platform had a significant effect on thoroughness (P < 0.001), indicating varying depth of responses.
• Source Quality:
  • Significantly differed by platform (P = 0.037), highlighting variability in the reliability of referenced sources.

Conclusions

• AI platforms demonstrate potential as informational resources for equine topics but currently do not match the expertise offered by Equine Extension Specialists.
• Performance was inconsistent, particularly on complex or advanced topics where the AI struggled to provide accurate, thorough, and high-quality sources.
• Among the tested platforms, ChatGPT and Microsoft Copilot generally outperformed ExtensionBot.
• The findings suggest caution in relying solely on AI-generated information for complex equine care decisions and highlight the continued importance of expert human guidance.