Skip to main content
SpringerLink
Log in

A deep learning approach to detect and identify live freshwater macroinvertebrates

  • Published:
Aquatic Ecology Aims and scope Submit manuscript

Abstract

The study of macroinvertebrates using computer vision is in its infancy and still faces multiple challenges including destructive sampling, low signal-to-noise ratios, and the complexity to choose a model algorithm among multiple existing ones. In order to deal with those challenges, we propose here a new framework, dubbed 'MacroNet,’ for the monitoring, i.e., detection and identification at the morphospecies level, of live aquatic macroinvertebrates. This framework is based on an enhanced RetinaNet model. Pre-processing steps are suggested to enhance the characterization propriety of the original algorithm. The images are split into fixed-size tiles to better detect and identify small macroinvertebrates. The tiles are then fed as an input to the model, and the resulting bounding box is assembled. We have optimized the anchor boxes generation process for high detection performance using the k-medoid algorithm. In order to enhance the localization accuracy of the original RetinaNet model, the complete intersection over union loss has been integrated as a regression loss to replace the standard loss (a smooth l1 norm). Experimental results show that MacroNet outperforms the original RetinaNet model on our database and can achieve on average 74.93% average precision (AP), depending on the taxon identity. In our database, taxa were identified at various taxonomic levels, from species to order. Overall, the proposed framework offers promising results for the non-lethal and cost-efficient monitoring of live freshwater macroinvertebrates.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

All data and code are available upon request.

References

  • Ärje J, Melvad C, Jeppesen MR et al (2020) Automatic image-based identification and biomass estimation of invertebrates. Methods Ecol Evol 11(8):922–931

    Article  Google Scholar 

  • Beery S, Morris D, Yang S et al (2019) Efficient pipeline for automating species id in new camera trap projects. Biodivers Inf Sci Stand 3:e37222. https://doi.org/10.3897/biss.3.37222

    Article  Google Scholar 

  • Birk S, Bonne W, Borja A et al (2012) Three hundred ways to assess Europe’s surface waters: an almost complete overview of biological methods to implement the water framework directive. Ecol Ind 18:31–41

    Article  Google Scholar 

  • Bjerge K, Nielsen JB, Sepstrup MV et al (2021) An automated light trap to monitor moths (lepidoptera) using computer vision-based tracking and deep learning. Sensors 21(2):343

    Article  PubMed  PubMed Central  Google Scholar 

  • Bonneau M, Vayssade JA, Troupe W et al (2020) Outdoor animal tracking combining neural network and time-lapse cameras. Comput Electron Agric 168(105):150

    Google Scholar 

  • Borowiec ML, Dikow RB, Frandsen PB et al (2022) Deep learning as a tool for ecology and evolution. Methods Ecol Evol 13(8):1640–1660

    Article  Google Scholar 

  • Carranza-Rojas J, Joly A, Goëau H, et al (2018) Automated identification of herbarium specimens at different taxonomic levels. Multimed Tools Appl Environ Biodivers Inform 151–167

  • Christensen JH, Mogensen LV, Ravn O (2020) Deep learning based segmentation of fish in noisy forward looking mbes images. IFAC-PapersOnLine 53(2):14546–14551

    Article  Google Scholar 

  • Dharmarajan A, Velmurugan T (2013) Applications of partition based clustering algorithms: a survey. In: 2013 IEEE international conference on computational intelligence and computing research (ICCIC), pp 1–5

  • Ding W, Taylor G (2016) Automatic moth detection from trap images for pest management. Comput Electron Agric 123:17–28

    Article  Google Scholar 

  • Ditria EM, Connolly RM, Jinks EL et al (2021) Annotated video footage for automated identification and counting of fish in unconstrained seagrass habitats. Front Mar Sci 8(629):485

    Google Scholar 

  • Eichinski P, Alexander C, Roe P, et al (2022) A convolutional neural network bird species recognizer built from little data by iteratively training, detecting, and labeling. Front Ecol Evol 133

  • Farley SS, Dawson A, Goring SJ et al (2018) Situating ecology as a big-data science: current advances, challenges, and solutions. Bioscience 68(8):563–576

    Article  Google Scholar 

  • Fernandez Garcia G, Corpetti T, Nevoux M et al (2023) Acousticia, a deep neural network for multi-species fish detection using multiple models of acoustic cameras. Aquat Ecol 2023:1–13

    Google Scholar 

  • Ferreira AC, Silva LR, Renna F et al (2020) Deep learning-based methods for individual recognition in small birds. Methods Ecol Evol 11(9):1072–1085

    Article  Google Scholar 

  • French G, Mackiewicz M, Fisher M, et al (2018) Jellymonitor: Automated detection of jellyfish in sonar images using neural networks. In: 2018 14th IEEE international conference on signal processing (ICSP). IEEE, pp 406–412

  • Girshick R, Donahue J, Darrell T, et al (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  • Gorsky G, Ohman MD, Picheral M et al (2010) Digital zooplankton image analysis using the zooscan integrated system. J Plankton Res 32(3):285–303

    Article  Google Scholar 

  • Guo H (2017) Big data drives the development of earth science

  • He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • Høye TT, Ärje J, Bjerge K et al (2021) Deep learning and computer vision will transform entomology. Proc Natl Acad Sci 118(2):e2002545117

    Article  PubMed  PubMed Central  Google Scholar 

  • Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6(1):1–54

    Article  Google Scholar 

  • Joutsijoki H (2013) An application of one-vs-one method in automated taxa identification of macroinvertebrates. In: 2013 Fourth global congress on intelligent systems. IEEE, pp 125–130

  • Joutsijoki H, Juhola M (2017) A comparison of classification methods in automated taxa identification of benthic macroinvertebrates. Int J Data Sci 2(4):273–300

    Article  Google Scholar 

  • Joutsijoki H, Meissner K, Gabbouj M et al (2014) Evaluating the performance of artificial neural networks for the classification of freshwater benthic macroinvertebrates. Eco Inform 20:1–12

    Article  Google Scholar 

  • Júnior TDC, Rieder R, Di Domênico JR et al (2022) Insectcv: a system for insect detection in the lab from trap images. Eco Inform 67(101):516

    Google Scholar 

  • Kiranyaz S, Ince T, Pulkkinen J et al (2011) Classification and retrieval on macroinvertebrate image databases. Comput Biol Med 41(7):463–472

    Article  PubMed  Google Scholar 

  • Le Quinio A, De Oliveira E, Girard A et al (2023) Automatic detection, identification and counting of anguilliform fish using in situ acoustic camera data: Development of a cross-camera morphological analysis approach. PLoS ONE 18(2):e0273588

    Article  PubMed  PubMed Central  Google Scholar 

  • Levin SA (2013) Encyclopedia of biodiversity. Elsevier Science, Amsterdam

    Google Scholar 

  • Lin TY, Dollár P, Girshick R, et al (2017a) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125

  • Lin TY, Goyal P, Girshick R, et al (2017b) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

  • Lytle DA, Martínez-Muñoz G, Zhang W et al (2010) Automated processing and identification of benthic invertebrate samples. J N Am Benthol Soc 29(3):867–874

    Article  Google Scholar 

  • Marburg A, Bigham K (2016) Deep learning for benthic fauna identification. In: OCEANS 2016 MTS/IEEE monterey. IEEE, pp 1–5

  • Mayo M, Watson AT (2007) Automatic species identification of live moths. Knowl-Based Syst 20(2):195–202

    Article  Google Scholar 

  • Miele V, Dussert G, Spataro B et al (2021) Revisiting animal photo-identification using deep metric learning and network analysis. Methods Ecol Evol 12(5):863–873

    Article  Google Scholar 

  • Mittal S, Srivastava S, Jayanth JP (2022) A survey of deep learning techniques for underwater image classification. IEEE Trans Neural Netw Learn Syst

  • Mondy CP, Villeneuve B, Archaimbault V et al (2012) A new macroinvertebrate-based multimetric index (i2m2) to evaluate ecological quality of French Wadeable streams fulfilling the wfd demands: a taxonomical and trait approach. Ecol Ind 18:452–467

    Article  Google Scholar 

  • Mustafa WA, Kader MMMA (2018) A review of histogram equalization techniques in image enhancement application. J Phys Conf Ser 2018:012026

    Article  Google Scholar 

  • Neubeck A, Van Gool L (2006) Efficient non-maximum suppression. In: 18th international conference on pattern recognition (ICPR’06). IEEE, pp 850–855

  • Niu SQ, Knouft JH (2017) Hydrologic characteristics, food resource abundance, and spatial variation in stream assemblages. Ecohydrology 10(1):e1770

    Article  Google Scholar 

  • Norouzzadeh MS, Nguyen A, Kosmala M et al (2018) Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. Proc Natl Acad Sci 115(25):E5716–E5725

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Oksuz K, Cam BC, Kalkan S et al (2020) Imbalance problems in object detection: A review. IEEE Trans Pattern Anal Mach Intell 43(10):3388–3415

    Article  Google Scholar 

  • Ozge Unel F, Ozkalayci BO, Cigla C (2019) The power of tiling for small object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 0–0

  • Park J, Kim DI, Choi B et al (2020) Classification and morphological analysis of vector mosquitoes using deep convolutional neural networks. Sci Rep 10(1):1012

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Plastiras G, Kyrkou C, Theocharides T (2018) Efficient convnet-based object detection for unmanned aerial vehicles by selective tile processing. In: Proceedings of the 12th international conference on distributed smart cameras, pp 1–6

  • Raitoharju J, Riabchenko E, Ahmad I et al (2018) Benchmark database for fine-grained image classification of benthic macroinvertebrates. Image Vis Comput 78:73–83

    Article  Google Scholar 

  • Raphael A, Dubinsky Z, Iluz D et al (2020) Neural network recognition of marine benthos and corals. Diversity 12(1):29

    Article  Google Scholar 

  • Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271

  • Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767

  • Redmon J, Divvala S, Girshick R, et al (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  • Ren S, He K, Girshick R, et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28

  • Rezatofighi H, Tsoi N, Gwak J, et al (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 658–666

  • Rigoudy N, Benyoub A, Besnard A, et al (2022) The deepfaune initiative: a collaborative effort towards the automatic identification of French fauna in camera-trap images. bioRxiv pp 2022–03

  • Ružička V, Franchetti F (2018) Fast and accurate object detection in high resolution 4k and 8k video using gpus. In: 2018 IEEE high performance extreme computing conference (HPEC). IEEE, pp 1–7

  • Shokri M, Cozzoli F, Ciotti M et al (2021) A new approach to assessing the space use behavior of macroinvertebrates by automated video tracking. Ecol Evol 11(7):3004–3014

    Article  PubMed  PubMed Central  Google Scholar 

  • Singh R, Mumbarekar V (2022) Neural network model approach for automated benthic animal identification. ICT Express 8(4):640–645

    Article  Google Scholar 

  • Sohrab F, Raitoharju J (2020) Boosting rare benthic macroinvertebrates taxa identification with one-class classification. In: 2020 IEEE symposium series on computational intelligence (SSCI), IEEE, pp 928–933

  • Sun M, Yang X, Xie Y (2020) Deep learning in aquaculture: a review. J Comput 31(1):294–319

    Google Scholar 

  • Tarling P, Cantor M, Clapés A et al (2022) Deep learning with self-supervision and uncertainty regularization to count fish in underwater images. PLoS ONE 17(5):e0267759

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Tuia D, Kellenberger B, Beery S et al (2022) Perspectives in machine learning for wildlife conservation. Nat Commun 13(1):792

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Wallace JB, Webster JR (1996) The role of macroinvertebrates in stream ecosystem function. Annu Rev Entomol 41(1):115–139

    Article  CAS  PubMed  Google Scholar 

  • Wang Q, Ma Y, Zhao K et al (2020) A comprehensive survey of loss functions in machine learning. Ann Data Sci 2020:1–26

    Google Scholar 

  • Wang X, Song J (2021) Iciou: Improved loss based on complete intersection over union for bounding box regression. IEEE Access 9:105686–105695

    Article  Google Scholar 

  • Wei Y, Duan Y, An D (2022) Monitoring fish using imaging sonar: capacity, challenges and future perspective. Fish Fish 23(6):1347–1370

    Article  Google Scholar 

  • Weinstein BG (2018) A computer vision for animal ecology. J Anim Ecol 87(3):533–545

    Article  PubMed  Google Scholar 

  • Yang Z, Wang X, Wu J, et al (2022) Edgeduet: Tiling small object detection for edge assisted autonomous mobile vision. IEEE/ACM Trans Netw

  • Ying Z, Li G, Ren Y, et al (2017) A new image contrast enhancement algorithm using exposure fusion framework. In: International Conference on Computer Analysis of Images and Patterns, Springer, pp 36–46

  • Yu J, Jiang Y, Wang Z, et al (2016) Unitbox: an advanced object detection network. In: Proceedings of the 24th ACM international conference on Multimedia, pp 516–520

  • Zang X, Yin T, Hou Z et al (2021) Deep learning for automated detection and identification of migrating American EEL Anguilla rostrata from imaging sonar data. Remote Sens 13(14):2671

    Article  Google Scholar 

  • Zheng H, Wang R, Yu Z et al (2017) Automatic plankton image classification combining multiple view features via multiple kernel learning. BMC Bioinform 18:1–18

    Article  CAS  Google Scholar 

  • Zheng Z, Wang P, Liu W, et al (2020) Distance-iou loss: Faster and better learning for bounding box regression. In: Proceedings of the AAAI conference on artificial intelligence, pp 12993–13000

Download references

Acknowledgements

We are deeply thankful to our colleagues from U3E/PEARL experimental facility (Yoann Bennevault, Bernard Joseph, Maïra Coke, and Antoine Gallard) and from DECOD (Eric Edeline, Caroline Gorzerino, Damien Fourcy, Marc Collinet, Eric Petit, Marie-Agnès Coutellec, and Julie Coudreuse) for their help with the mesocosm experiment. Particular thanks to Alan Bazin for his substantial help with the data acquisition and expertise on macroinvertebrates.

Funding

This study was funded by Rennes Métropole through an 'Allocation d’Installation Scientifique’ program (21C0693) and by INRAE via a 'Appel à Projet d’Intégration’ funding opportunity (EB10) and internal overheads. SJa received part of his postdoctoral salary from the Région Bretagne (SAD19021-00057661).

Author information

Authors and Affiliations

  1. ESE, Ecology and Ecosystems Health, INRAE, Agrocampus Ouest, 35042, Rennes, France

    Sami Jaballah

  2. UMR DECOD (Ecosystem Dynamics and Sustainability), INRAE, Institut Agro, IFREMER, Rennes, France

    Guglielmo Fernandez Garcia, François Martignac, Jean-Marc Roussel & Olivier Dézerald

  3. UMR 6554 LETG, CNRS, Université de Rennes 2, 35000, Rennes, France

    Guglielmo Fernandez Garcia

  4. UMR 1349 IGEPP, INRAE, 35653, Le Rheu, France

    Nicolas Parisey & Stéphane Jumel

Authors
  1. Sami Jaballah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guglielmo Fernandez Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. François Martignac
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicolas Parisey
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stéphane Jumel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jean-Marc Roussel
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Olivier Dézerald
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SJa developed the framework. OD collected the data and produced most of the database with help from SJa. GFG and NP provided ideas concerning the framework. OD, FM, and JMR led discussions around ecological aspects of this study. OD and SJu developed the image acquisition protocol. OD and JMR obtained the funding that supported this study. SJa and OD wrote the first drafts. All authors edited earlier versions of this manuscript and approved its final version.

Corresponding author

Correspondence to Sami Jaballah.

Ethics declarations

Conflict of interest

The authors have no competing interests of any kind.

Additional information

Handling Editor: Y. S. Park.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jaballah, S., Garcia, G.F., Martignac, F. et al. A deep learning approach to detect and identify live freshwater macroinvertebrates. Aquat Ecol 57, 933–949 (2023). https://doi.org/10.1007/s10452-023-10053-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10452-023-10053-7

Keywords

Search

Navigation