Skip to main content
SpringerLink
Log in

The limitations of differentiable architecture search

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

In this paper, we will provide a detailed explanation of the limitations behind differentiable architecture search (DARTS). Algorithms based on the DARTS paradigm tend to converge towards degenerate solutions. A degenerate solution corresponds to an architecture with a shallow graph containing mainly skip connections. We have identified 6 sources of errors that could explain this phenomenon. Some of these errors can only be partially eliminated. Therefore, we will propose an innovative solution to remove degenerate solutions from the search space. We will demonstrate the validity of our approach through experiments conducted on the CIFAR10 and CIFAR100 databases. Our code is available at the following link: https://scm.univ-tours.fr/projetspublics/lifat/darts_ibpria_sparcity

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

The CIFAR-10 and CIFAR-100 databases can be obtained using the following link: https://www.cs.toronto.edu/%C2%A0kriz/cifar.html

References

  1. Ang A, Ma J, Liu N, Huang K, Wang Y (2021) Fast projection onto the capped simplex with applications to sparse regression in bioinformatics. Neural Inf Process Syst (NeurIPS)

  2. Balles L, Hennig P (2018) Dissecting adam: the sign, magnitude and variance of stochastic gradients. In: International conference on machine learning (PMLR)

  3. Choe H, Na B, Mok J, Yoon S (2021) Variance-stationary differentiable NAS. In: British machine vision conference (BMVC)

  4. Franceschi L, Donini M, Frasconi P, Pontil M (2017) Forward and reverse gradient-based hyperparameter optimization. In: International conference on machine learning (ICML)

  5. Fu J, Luo H, Feng J, Low KH, Chua TS (2016) DrMAD: distilling reverse-mode automatic differentiation for optimizing hyperparameters of deep neural networks. In: International joint conference on artificial intelligence (IJCAI)

  6. Gu YC, Wang LJ, Liu Y, Yang Y, Wu YH, Lu SP, Cheng MM (2021) DOTS: decoupling operation and topology in differentiable architecture search. In: Conference on computer vision and pattern recognition (CVPR)

  7. He C, Ye H, Shen L, Zhang T (2020) MiLeNAS: efficient neural architecture search via mixed-level reformulation. In: Conference on computer vision and pattern recognition (CVPR)

  8. Hong W, Li G, Zhang W, Tang R, Wang Y, Li Z, Yu Y (2020) DropNAS: grouped operation dropout for differentiable architecture search. In: International joint conference on artificial intelligence (IJCAI)

  9. Hou P, Jin Y, Chen Y (2021) Single-DARTS: towards stable architecture search. In: IEEE/CVF international conference on computer vision workshops (ICCVW)

  10. Kendall MG (1938) A new measure of rank correlation. Biometrika

  11. Kingma DP, Ba JL (2015) ADAM: a method for stochastic optimization. In: International conference on learning representations (ICLR)

  12. Lacharme G, Cardot H, Lenté C, Monmarché N (2023) DARTS with degeneracy correction. In: Iberian conference on pattern recognition and image analysis (IbPRIA)

  13. Lee HB, Lee H, Shin J, Yang E, Hospedales TM, Hwang SJ (2022) online hyperparameter meta-learning with hypergradient distillation. In: International conference on learning representations (ICLR)

  14. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A (2018) Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res

  15. Li G, Qian G, Delgadillo IC, Muller M, Thabet A, Ghanem B (2020) SGAS: sequential greedy architecture search. In: Conference on computer vision and pattern recognition (CVPR)

  16. Liang H, Zhang S, Sun J, He X, Huang W, Zhuang K, Li Z (2021) DARTS+: improved differentiable architecture search with early stopping. http://arxiv.org/abs/1909.06035

  17. Lin M, Wang P, Sun Z, Chen H, Sun X, Qian Q, Li H, Jin R (2021) Zen-NAS: a zero-shot NAS for high-performance image recognition. In: International conference on computer vision (ICCV)

  18. Liu H, Simonyan K, Yang Y (2019) Darts: differentiable architecture search. In: International conference on learning representations (ICLR)

  19. Lorraine J, Vicol P, Duvenaud D (2020) Optimizing millions of hyperparameters by implicit differentiation

  20. Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: International conference on learning representations (ICLR)

  21. Luketina J, Berglund M, Greff K, Raiko T (2016) Scalable gradient-based tuning of continuous regularization hyperparameters. In: International conference on machine learning (ICML)

  22. Metz L, Maheswaranathan N, Sun R, Daniel Freeman C, Poole B, Sohl-Dickstein J (2020) Using a thousand optimization tasks to learn hyperparameter search strategies Neural Inf Process Syst (NeurIPS)

  23. Real E, Aggarwal A, Huang Y, Le QV (2019) Regularized evolution for image classifier architecture search. In: Conference on artificial intelligence (AAAI)

  24. Vicol P, Lorraine JP, Pedregosa F, Duvenaud D, Grosse RB (2022) On implicit bias in overparameterized bilevel optimization. In: International conference on machine learning (ICML)

  25. Vicol P, Metz L, Sohl-Dickstein J (2021) Persistent unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies

  26. Wei T, Wang C, Rui Y, Chen CW (2016) Network morphism. In: Proceedings of machine learning research (PMLR)

  27. Wu Y, Ren M, Liao R, Grosse R (2018) Understanding short-horizon bias in stochastic meta-optimizations. In: International conference on learning representations (ICLR)

  28. Yibo Y, Hongyang L, Shan Y, Fei W, Chen Q, Zhouchen L (2020) ISTA-NAS: efficient and consistent neural architecture search by sparse coding. In: Proceedings of the 34th international conference on neural information processing systems (NeurIPS)

  29. Zhang M, Su S, Pan S, Chang X, Abbasnejad E, Haffari R (2021) iDARTS: differentiable architecture search with stochastic implicit gradients. In: International conference on machine learning (ICML)

  30. Zhou P, Xiong C, Socher R, Hoi SCH (2020) Theory-inspired path-regularized differential network architecture search. Neural Inf Process Syst (NeurIPS)

  31. Zoph B, Vasudevan V, Shlens J, Le QV (2019) Learning transferable architectures for scalable image recognition. In: Conference on computer vision and pattern recognition (CVPR)

Download references

Funding

Region Centre Val de Loire.

Author information

Authors and Affiliations

  1. Université de Tours, LIFAT, 64 Av. Jean Portalis, 37200, Tours, France

    Lacharme Guillaume, Cardot Hubert, Lente Christophe & Monmarche Nicolas

Authors
  1. Lacharme Guillaume
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cardot Hubert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lente Christophe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Monmarche Nicolas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lacharme Guillaume.

Ethics declarations

Ethical approval

No ethical approval.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Settings experimentations

Appendix A: Settings experimentations

figure c
figure d

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guillaume, L., Hubert, C., Christophe, L. et al. The limitations of differentiable architecture search. Pattern Anal Applic 27, 40 (2024). https://doi.org/10.1007/s10044-024-01260-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10044-024-01260-5

Keywords

Search

Navigation