FREE SHIPPING over $40
OVER 9000 FIVE-STAR REVIEWS
5% OFF ALL SUBSCRIPTIONS
100% HAPPINESS GUARANTEE
Analyze Diet
0
Home / Equine Research Bank / Sci Rep / Fusion of visible and thermal images improves automated dete...
Scientific reports2023; 13(1); 10385; doi: 10.1038/s41598-023-37295-7

Fusion of visible and thermal images improves automated detection and classification of animals for drone surveys.

Abstract: Visible and thermal images acquired from drones (unoccupied aircraft systems) have substantially improved animal monitoring. Combining complementary information from both image types provides a powerful approach for automating detection and classification of multiple animal species to augment drone surveys. We compared eight image fusion methods using thermal and visible drone images combined with two supervised deep learning models, to evaluate the detection and classification of white-tailed deer (Odocoileus virginianus), domestic cow (Bos taurus), and domestic horse (Equus caballus). We classified visible and thermal images separately and compared them with the results of image fusion. Fused images provided minimal improvement for cows and horses compared to visible images alone, likely because the size, shape, and color of these species made them conspicuous against the background. For white-tailed deer, which were typically cryptic against their backgrounds and often in shadows in visible images, the added information from thermal images improved detection and classification in fusion methods from 15 to 85%. Our results suggest that image fusion is ideal for surveying animals inconspicuous from their backgrounds, and our approach uses few image pairs to train compared to typical machine-learning methods. We discuss computational and field considerations to improve drone surveys using our fusion approach.
© 2023. This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply.
Get Full Text
Publication Date: 2023-06-27 PubMed ID: 37369669PubMed Central: PMC10300091DOI: 10.1038/s41598-023-37295-7Google Scholar: Lookup
The Equine Research Bank provides access to a large database of publicly available scientific literature. Inclusion in the Research Bank does not imply endorsement of study methods or findings by Mad Barn.
LinkedInCopy
  • Journal Article
  • Research Support
  • U.S. Gov't
  • Non-P.H.S.
  • Research Support
  • Non-U.S. Gov't

Summary

This research summary has been generated with artificial intelligence and may contain errors and omissions. Refer to the original study to confirm details provided. Submit correction.

The research investigates how combining visible and thermal images from drones can enhance automated detection and classification of various animal species, with the primary study species being white-tailed deer, domestic cows, and horses.

Combining Visible and Thermal Images

  • The research focuses on the fusion of visible and thermal images attained from drones to improve animal detection and classification in the wild.
  • Eight imaged fusion methods were compared using thermal and visible images from drones, combined with two supervised deep learning models, to detect and categorize three specific species: white-tailed deer, domestic cow, and domestic horse.
  • The investigation aimed to determine whether this fusion technique improved the detection rates of these animals compared to using only visible images.

Results of the Image Fusion Methods

  • For cows and horses, which are generally easily spotted due to their size, color, and shape contrasting with their surroundings, the fusion process offered minimal enhancements in detection compared to using the visible images alone.
  • However, for the detection and classification of white-tailed deer, which are usually hard to detect due to their color blending with the environment and frequently being in shadowy areas, image fusion led to significant improvements.
  • The added thermal information in the fusion process increased the detection rates of white-tailed deer by 15 to 85%.

Noval Approach and Future Considerations

  • The research suggests that image fusion can be extremely beneficial for monitoring animals that blend in with their surroundings.
  • The approach advocated in the study requires fewer image pairs to train, making it a potentially more efficient and manageable technique than conventional machine-learning methods.
  • However, this process includes computational and field considerations that need to be explored further to optimize drone surveys using this fusion approach.

Cite This Article

APA
Krishnan BS, Jones LR, Elmore JA, Samiappan S, Evans KO, Pfeiffer MB, Blackwell BF, Iglay RB. (2023). Fusion of visible and thermal images improves automated detection and classification of animals for drone surveys. Sci Rep, 13(1), 10385. https://doi.org/10.1038/s41598-023-37295-7

Publication

Scientific reports
ISSN: 2045-2322
NlmUniqueID: 101563288
Country: England
Language: English
Volume: 13
Issue: 1
Pages: 10385
PII: 10385

Researcher Affiliations

Krishnan, B Santhana
  • Geosystems Research Institute, Mississippi State University, Mississippi State, Mississippi State, MS, 39762, USA.
Jones, Landon R
  • Department of Wildlife, Fisheries, and Aquaculture, Mississippi State University, Box 9690, Mississippi State, MS, 39762, USA.
Elmore, Jared A
  • Department of Wildlife, Fisheries, and Aquaculture, Mississippi State University, Box 9690, Mississippi State, MS, 39762, USA.
  • Department of Forestry and Environmental Conservation, Clemson University, Clemson, SC, 29634, USA.
Samiappan, Sathishkumar
  • Geosystems Research Institute, Mississippi State University, Mississippi State, Mississippi State, MS, 39762, USA.
Evans, Kristine O
  • Department of Wildlife, Fisheries, and Aquaculture, Mississippi State University, Box 9690, Mississippi State, MS, 39762, USA.
Pfeiffer, Morgan B
  • U.S. Department of Agriculture, Animal and Plant Health Inspection Service, Wildlife Services, National Wildlife Research Center, Ohio Field Station, Sandusky, OH, 44870, USA.
Blackwell, Bradley F
  • U.S. Department of Agriculture, Animal and Plant Health Inspection Service, Wildlife Services, National Wildlife Research Center, Ohio Field Station, Sandusky, OH, 44870, USA.
Iglay, Raymond B
  • Department of Wildlife, Fisheries, and Aquaculture, Mississippi State University, Box 9690, Mississippi State, MS, 39762, USA. ray.iglay@msstate.edu.

MeSH Terms

  • Female
  • Animals
  • Cattle
  • Horses
  • Deer
  • Unmanned Aerial Devices
  • Aircraft

Conflict of Interest Statement

The authors declare no competing interests.

References

This article includes 46 references
  1. Christie KS, Gilbert SL, Brown CL, Hatfield M, Hanson L. Unmanned aircraft systems in wildlife research: Current and future applications of a transformative technology.. Front. Ecol. Environ. 2016;14:241–251.
    doi: 10.1002/fee.1281google scholar: lookup
  2. Linchant J, Lisein J, Semeki J, Lejeune P, Vermeulen C. Are unmanned aircraft systems (UASs) the future of wildlife monitoring? A review of accomplishments and challenges.. Mamm. Rev. 2015;45:239–252.
    doi: 10.1111/mam.12046google scholar: lookup
  3. Elmore JA. Evidence on the efficacy of small unoccupied aircraft systems (UAS) as a survey tool for North American terrestrial, vertebrate animals: a systematic map.. Environ. Evid. 2023;10:1.
  4. Hodgson JC, Baylis SM, Mott R, Herrod A, Clarke RH. Precision wildlife monitoring using unmanned aerial vehicles.. Sci Rep 2016 Mar 17;6:22574.
    doi: 10.1038/srep22574pmc: PMC4795075pubmed: 26986721google scholar: lookup
  5. Hodgson JC. Drones count wildlife more accurately and precisely than humans.. Methods Ecol. Evol. 2018;9:1160–1167.
    doi: 10.1111/2041-210X.12974google scholar: lookup
  6. Corcoran E, Denman S, Hanger J, Wilson B, Hamilton G. Automated detection of koalas using low-level aerial surveillance and machine learning.. Sci Rep 2019 Mar 1;9(1):3208.
    doi: 10.1038/s41598-019-39917-5pmc: PMC6397288pubmed: 30824795google scholar: lookup
  7. Seymour AC, Dale J, Hammill M, Halpin PN, Johnston DW. Automated detection and enumeration of marine wildlife using unmanned aircraft systems (UAS) and thermal imagery.. Sci Rep 2017 Mar 24;7:45127.
    doi: 10.1038/srep45127pmc: PMC5364474pubmed: 28338047google scholar: lookup
  8. Lyons MB. Monitoring large and complex wildlife aggregations with drones.. Methods Ecol. Evol. 2019;10:1024–1035.
    doi: 10.1111/2041-210X.13194google scholar: lookup
  9. Eikelboom JAJ. Improving the precision and accuracy of animal population estimates with aerial image object detection.. Methods Ecol. Evol. 2019;10:1875–1887.
    doi: 10.1111/2041-210X.13277google scholar: lookup
  10. Rey N, Volpi M, Joost S, Tuia D. Detecting animals in African Savanna with UAVs and the crowds.. Remote Sens. Environ. 2017;200:341–351.
    doi: 10.1016/j.rse.2017.08.026google scholar: lookup
  11. Bowley C, Mattingly M, Barnas A, Ellis-Felege S, Desell T. An analysis of altitude, citizen science and a convolutional neural network feedback loop on object detection in unmanned aerial systems.. J. Comput. Sci. 2019;34:102–116.
    doi: 10.1016/j.jocs.2019.04.010google scholar: lookup
  12. Wood SA, Robinson PW, Costa DP, Beltran RS. Accuracy and precision of citizen scientist animal counts from drone imagery.. PLoS One 2021;16(2):e0244040.
    doi: 10.1371/journal.pone.0244040pmc: PMC7899343pubmed: 33617554google scholar: lookup
  13. Zhou M, Elmore JA, Samiappan S, Evans KO, Pfeiffer MB, Blackwell BF, Iglay RB. Improving Animal Monitoring Using Small Unmanned Aircraft Systems (sUAS) and Deep Learning Networks.. Sensors (Basel) 2021 Aug 24;21(17).
    pmc: PMC8433839pubmed: 34502588doi: 10.3390/s21175697google scholar: lookup
  14. Lenzi J, Barnas AF, ElSaid AA, Desell T, Rockwell RF, Ellis-Felege SN. Artificial intelligence for automated detection of large mammals creates path to upscale drone surveys.. Sci Rep 2023 Jan 18;13(1):947.
    doi: 10.1038/s41598-023-28240-9pmc: PMC9849265pubmed: 36653478google scholar: lookup
  15. Chen A, Jacob M, Shoshani G, Charter M. Using computer vision, image analysis and UAVs for the automatic recognition and counting of common cranes (Grus grus).. J Environ Manage 2023 Feb 15;328:116948.
    doi: 10.1016/j.jenvman.2022.116948pubmed: 36516707google scholar: lookup
  16. Torney CJ. A comparison of deep learning and citizen science techniques for counting wildlife in aerial survey images.. Methods Ecol. Evol. 779–787 (2019).
    doi: 10.1111/2041-210x.13165google scholar: lookup
  17. Chrétien LP, Théau J, Ménard P. Visible and thermal infrared remote sensing for the detection of white-tailed deer using an unmanned aerial system.. Wildl. Soc. Bull. 2016;40:181–191.
    doi: 10.1002/wsb.629google scholar: lookup
  18. Deng J. Imagenet: A large-scale hierarchical image database.. in 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255 (IEEE, 2009).
  19. Hermann KL, Chen T, Kornblith S. The origins and prevalence of texture bias in convolutional neural networks.. in 34th Conference on Neural Information Processing System (2020).
  20. Tabak MA, Norouzzadeh MS, Wolfson DW, Newton EJ, Boughton RK, Ivan JS, Odell EA, Newkirk ES, Conrey RY, Stenglein J, Iannarilli F, Erb J, Brook RK, Davis AJ, Lewis J, Walsh DP, Beasley JC, VerCauteren KC, Clune J, Miller RS. Improving the accessibility and transferability of machine learning algorithms for identification of animals in camera trap images: MLWIC2.. Ecol Evol 2020 Oct;10(19):10374-10383.
    doi: 10.1002/ece3.6692pmc: PMC7548173pubmed: 33072266google scholar: lookup
  21. Liu Y, Liu S, Wang Z. A general framework for image fusion based on multi-scale transform and sparse representation.. Inf. Fusion. 2015;24:147–164.
    doi: 10.1016/j.inffus.2014.09.004google scholar: lookup
  22. Gonzalez RC, Eddins SL, Woods RE. Digital Image Publishing Using MATLAB.. Prentice Hall; 2004.
  23. Pajares G, De La Cruz JM. A wavelet-based image fusion tutorial.. Pattern Recognit. 2004;37:1855–1872.
    doi: 10.1016/j.patcog.2004.03.010google scholar: lookup
  24. Druzhkov PN, Kustikova VD. A survey of deep learning methods and software tools for image classification and object detection.. Pattern Recognit. Image Anal. 2016;26:9–15.
    doi: 10.1134/S1054661816010065google scholar: lookup
  25. Jin X. A survey of infrared and visual image fusion methods.. Infrared Phys. Technol. 2017;85:478–501.
    doi: 10.1016/j.infrared.2017.07.010google scholar: lookup
  26. Yadav R, Samir A, Rashed H, Yogamani S, Dahyot R. CNN Based Color and Thermal Image Fusion for Object Detection in Automated Driving.. (2020).
  27. Meena SD, Agilandeeswari L. Smart animal detection and counting framework for monitoring livestock in an autonomous unmanned ground vehicle using restricted supervised learning and image fusion.. in Neural Processing Letters. Vol. 53 (Springer, 2021).
  28. Chrétien L-P, Théau J, Ménard P. Wildlife multispecies remote sensing using visible and thermal infrared imagery acquired from an unmanned aerial vehicle (UAV). Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2015;40:241–248.
    doi: 10.5194/isprsarchives-XL-1-W4-241-2015google scholar: lookup
  29. Bulanon DM, Burks TF, Alchanatis V. Image fusion of visible and thermal images for fruit detection.. Biosyst. Eng. 2009;103:12–22.
    doi: 10.1016/j.biosystemseng.2009.02.009google scholar: lookup
  30. Liu F, Seipel S. Infrared-visible image registration for augmented reality-based thermographic building diagnostics.. Vis. Eng. 2015;3:1–15.
    doi: 10.1186/s40327-015-0028-0google scholar: lookup
  31. Maurya L, Mahapatra P, Chawla D. A registration strategy from scale adjustment to fine for thermal-visible face images.. Infrared Phys. Technol. 2022;120:104001.
    doi: 10.1016/j.infrared.2021.104001google scholar: lookup
  32. Xu R, Lin H, Lu K, Cao L, Liu Y. A forest fire detection system based on ensemble learning.. Forests. 2021;12:217.
    doi: 10.3390/f12020217google scholar: lookup
  33. Zhu L, Geng X, Li Z, Liu C. Improving YOLOv5 with attention mechanism for detecting boulders from planetary images.. Remote Sens. 2021;13:3776.
    doi: 10.3390/rs13183776google scholar: lookup
  34. Wang Z, Cui Z, Zhu Y. Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation.. Comput Biol Med 2020 Aug;123:103823.
    doi: 10.1016/j.compbiomed.2020.103823pubmed: 32658780google scholar: lookup
  35. Qu G, Zhang D, Yan P. Information measure for performance of image fusion.. Electron. Lett. 2002;38:1.
    doi: 10.1049/el:20020212google scholar: lookup
  36. Xydeas CS, Petrovic V. Objective image fusion performance measure.. Electron. Lett. 2000;36:308–309.
    doi: 10.1049/el:20000267google scholar: lookup
  37. Petrović VS, Xydeas CS. Gradient-based multiresolution image fusion.. IEEE Trans Image Process 2004 Feb;13(2):228-37.
    doi: 10.1109/TIP.2004.823821pubmed: 15376943google scholar: lookup
  38. Szeliski R. Computer Vision: Algorithms and Applications.. (Springer, 2022).
  39. Chabot D, Francis CM. Computer-automated bird detection and counts in high-resolution aerial images: A review.. J. F. Ornithol. 2016;87:343–359.
    doi: 10.1111/jofo.12171google scholar: lookup
  40. Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP. On large-batch training for deep learning: Generalization gap and sharp minima.. in 5th International Conference on Learning Representations (OpenReview.net, 2017).
  41. Swanson A, Kosmala M, Lintott C, Simpson R, Smith A, Packer C. Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna.. Sci Data 2015;2:150026.
    doi: 10.1038/sdata.2015.26pmc: PMC4460915pubmed: 26097743google scholar: lookup
  42. Tabak MA. Machine learning to classify animal species in camera trap images: Applications in ecology.. Methods Ecol. Evol. 2019;10:585–590.
    doi: 10.1111/2041-210X.13120google scholar: lookup
  43. Patterson C, Koski W, Pace P, McLuckie B, Bird DM. Evaluation of an unmanned aircraft system for detecting surrogate caribou targets in labrador1.. J. Unmanned Veh. Syst. 2015;4:53–69.
    doi: 10.1139/juvs-2015-0014google scholar: lookup
  44. Jones LR. Controllable factors affecting accuracy and precision of human identification of animals from drone imagery.. Ecosphere (2023) (in review).
  45. Brack IV, Kindel A, Oliveira LFB. Detection errors in wildlife abundance estimates from unmanned aerial systems (UAS) surveys: Synthesis, solutions, and challenges.. Methods Ecol. Evol. 2018;9:1864–1873.
    doi: 10.1111/2041-210X.13026google scholar: lookup
  46. Bal P, Lyons D, Shuter A. A new ectotherm 3D tracking and behavior analytics system using a depth-based approach with color validation, with preliminary data on Kihansi spray toad (Nectophrynoides asperginis) activity.. Herpetol. Rev. 2020;51:37.

Citations

This article has been cited 0 times.
Subscribe to our newsletter
Join 99,357 horse owners like you who receive our email updates on horse health and nutrition.