Skip to main content
SpringerLink
Log in

On vulnerabilities in EVT-based timing analysis: an experimental investigation on a multi-core architecture

  • Published:
Design Automation for Embedded Systems Aims and scope Submit manuscript

Abstract

Hardware architectures based on multiple cores, cache memory and branch prediction usually preclude the application of classical methods for determining execution time bounds for real-time tasks. As such bounds are fundamental in the designing of real-time systems, Measurement-Based Probabilistic Timing Analysis has been employed. A common choice is the derivation of probabilistic Worst-Case Execution Time via the use of Extreme Value Theory, a branch of statistics used to estimate the probability of rare events that are more extreme than observations. However, pWCET estimations are usually reported in a controlled or simulated environment. In this paper we apply MBPTA in a real multi-core platform, namely Raspberry Pi 3B, taking into consideration possible interference due to operating system and concurrent activities. The results indicate that although EVT is effective, it does not always produce adequate models and coherent pWCET estimations. As MBPTA is primarily called for when classical methods are not applicable, as it is the case for the studied platform, the results reported in this paper highlight risks and vulnerabilities when applying MBPTA-EVT for pWCET inference.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Available at https://cran.r-project.org/.

  2. Further information at https://cran.r-project.org/web/packages/extRemes/extRemes.pdf.

References

  1. Abella J, Hardy D, Puaut I, Quinones E, Cazorla FJ (2014) On the comparison of deterministic and probabilistic WCET estimation techniques. In: Proceedings - Euromicro conference on real-time systems, pp 266–275

  2. Abella J, Quiñones E, Wartel F, Vardanega T, Cazorla FJ (2014) Heart of gold: making the improbable happen to increase confidence in MBPTA. In: 2014 26th Euromicro conference on real-time systems. IEEE, pp 255–265

  3. Arcaro LF, Palma Silva K, Silva De Oliveira R (2018) On the reliability and tightness of GP and Exponential models for probabilistic WCET estimation. ACM Trans Des Autom Electron Syst (TODAES) 23(3):1–27

    Article  Google Scholar 

  4. Berezovskyi K, Santinelli L, Bletsas K, Tovar E (2014) WCET measurement-based and extreme value theory characterisation of CUDA kernels. In: Proceedings of the 22nd international conference on real-time networks and systems, pp 279–288

  5. Bernat G, Burns A, Newby M (2005) Probabilistic timing analysis: an approach using copulas. J Embed Comput 1(2):179–194

    Google Scholar 

  6. Boyerinas BM (2016) Determining the statistical power of the kolmogorov-smirnov and anderson-darling goodness-of-fit tests via monte carlo simulation. Technical report, Center of Naval Analyses Arlington United States

  7. Burns A, Edgar S (2000) Predicting computation time for advanced processor architectures. In: Proceedings - Euromicro conference on real-time systems, pp 89–96

  8. Cazorla FJ, Vardanega T, Quiñones E, Abella J (2013) Upper-bounding program execution time with extreme value theory. In: 13th international workshop on worst-case execution time analysis. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  9. Cazorla FJ, Kosmidis L, Mezzetti E, Hernandez C, Abella J, Vardanega T (2019) Probabilistic Worst-Wase timing analysis: taxonomy and Somprehensive survey. ACM Comput Surv (CSUR) 52(1):1–35

    Article  Google Scholar 

  10. Coles S (2001) An introduction to statistical modeling of extreme values. Springer, London

    Book  MATH  Google Scholar 

  11. Cristina Soares Garção T (2017) Avaliação empírica do risco de mercado: estimação do Value-at-risk pela Teoria dos Valores Extremos. PhD thesis

  12. Cucu-Grosjean L, Santinelli L, Houston M, Lo C, Vardanega T, Kosmidis L, Abella J, Mezzetti E, Quiñones E, Cazorla F J (2012) Measurement-based probabilistic timing analysis for multi-path programs. In: Euromicro conference on real-time systems, pp 91–101

  13. Davis R, Cucu-Grosjean L (2019) A survey of probabilistic timing analysis techniques for real-time systems. LITES Leibniz Trans Embed Syst 6(1):1–53

    Google Scholar 

  14. Dietrich D, De Haan L, Husler J (2002) testing extreme value conditions. Extremes 5(1):71

    Article  MathSciNet  MATH  Google Scholar 

  15. Drees H, De Haan L, Li D (2006) Approximations to the tail empirical distribution function with application to testing extreme value conditions. J Stat Plan Inference 136:3498–3538

    Article  MathSciNet  MATH  Google Scholar 

  16. Efron B, Tibshirani RJ (1994) An introduction to the Bootstrap. CRC Press, Florida

    Book  MATH  Google Scholar 

  17. Garrido M, Diebolt J (2007) The ET Test, a goodness-of-fit test for the distribution tail. In: Methodology, practice and inference, second international conference on mathematical methods in reliability, pp 427–430

  18. Griffin D, Burns A (2010) Realism in statistical analysis of worst case execution times. OpenAccess Ser Inform 15:44–53

    Google Scholar 

  19. Guet F, Santinelli L, Morio J (2017) On the representativity of execution time measurements: studying dependence and multi-mode tasks. In: 17th international workshop on worst-case execution time analysis (WCET 2017). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  20. Gustafsson J, Betts A, Ermedahl A, Lisper B (2010) The Mälardalen WCET benchmarks: past, present and future. In: 10th international workshop on worst-case execution time analysis (WCET 2010). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  21. Hansen J, Hissam S, Moreno GA (2009) Statistical-based WCET estimation and validation. OpenAccess Ser Inform 10:1–11

    Google Scholar 

  22. Heckmann R, Langenbach M, Thesing S, Wilhelm R (2003) The influence of processor architecture on the design and the results of WCET tools. IEEE, vol 91, pp 1038–1054

  23. Hosking J (1990) L-moments: analysis and estimation of distributions using linear combinations of order statistics. J R Stat Soc Ser B (Methodol) 52:105–124

    MathSciNet  MATH  Google Scholar 

  24. Huang WK, Stein ML, McInerney DJ, Sun S, Moyer EJ (2016) Estimating changes in temperature extremes from millennial-scale climate simulations using generalized extreme value (GEV) distributions. Adv Stat Climatol Meteorol Oceanogr 2(1):79–103

    Article  Google Scholar 

  25. Hüsler J, Li D (2006) On testing extreme value conditions. Extremes. 9:69–86

    Article  MathSciNet  MATH  Google Scholar 

  26. Jamile V, Lima G (2022) Possible risks with EVT-based timing analysis: an experimental study on a multi-core platform. In: 2022 IEEE XII Brazilian symposium on computing systems engineering (SBESC). IEEE, pp 1–8

  27. Jiménez Gil S, Bate I, Lima G, Santinelli L, Gogonel A, Cucu-Grosjean L (2017) Open challenges for probabilistic measurement-based worst-case execution time. IEEE Embed Syst Lett 9(3):69–72

    Article  Google Scholar 

  28. Leadbetter MR (1991) On a basis for ‘Peaks over Threshold’ modeling. Stat Probab Lett 12(4):357–362

    Article  MathSciNet  MATH  Google Scholar 

  29. Leonetti P, Chokami A (2022) The maximum domain of attraction of multivariate extreme value distributions is small. Electron Commun Probab 27

  30. Lima G, Dias D, Barros E (2016) Extreme value theory for estimating task execution time bounds: a careful look. In: Proceedings - Euromicro conference on real-time systems, pp 200–211

  31. Lu Y, Nolte T, Bate I, Cucu-Grosjean L (2011) A new way about using statistical analysis of worst-case execution times. ACM SIGBED Rev 8(3):11–14

    Article  Google Scholar 

  32. Maxim C, Gogonel A, Maxim D, Cucu-Grosjean L (2013) HAL Id: hal-00766063 estimation of probabilistic minimum inter-arrival times using extreme value theory

  33. Maxim D, Soboczenski F, Bate I, Tovar E (2015) Study of the reliability of statistical timing analysis for real-time systems. In: Proceedings of the 23rd international conference on real time and networks systems, pp 55–64

  34. Melani A, Noulard E, Santinelli L (2013) Learning from probabilities: dependences within real-time systems. In: 2013 IEEE 18th conference on emerging technologies & factory automation (ETFA). IEEE, pp 1–8

  35. Nélis V, Meumeu Yomsi P, Pinho LM, Bernat G (2014) Another look at the pWCET estimation problem

  36. Özari Ç, Eren Ö, Saygin H (2019) A new methodology for the block maxima approach in selecting the optimal block size. Techn Gaz 26(5):1292–1296

    Google Scholar 

  37. Palma Silva K, Arcaro LF, Silva De Oliveira R (2017) On using GEV or Gumbel models when applying EVT for probabilistic WCET estimation. In: 2017 IEEE real-time systems symposium (RTSS). IEEE, pp 220–230

  38. Reghenzani F, Massari G, Fornaciari W (2018) The Misconception of exponential tail upper-bounding in probabilistic real time. IEEE Embed Syst Lett 11(3):77–80

    Article  Google Scholar 

  39. Reghenzani F, Massari G, Fornaciari W (2020) Probabilistic-WCET reliability: on the experimental validation of EVT hypotheses. Microprocess Microsyst 77:103135

    Article  Google Scholar 

  40. Ross Leadbetter M, Lindgren G, Rootzén H (1978) Conditions for the convergence in distribution of maxima of stationary normal processes. Stoch Process Appl 8(2):131–139

    Article  MathSciNet  MATH  Google Scholar 

  41. Santinelli L, Guo Z (2017) On the criticality of probabilistic worst-case execution time models. In: International symposium on dependable software engineering: theories, tools, and applications. Springer, pp 59–74

  42. Santinelli L, Morio J, Dufour G, Jacquemart D (2014) On the sustainability of the extreme value theory for WCET estimation. In: 14th international workshop on worst-case execution time analysis. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  43. Silva D, Caeiro F, Oliveira M (2018) Modelação estatística de valores extremos: probabilidades de excedência, quantis extremais, limite superior do suporte e níveis de retorno no lançamento do disco do atletismo. Rev Estat Univ Fed Ouro Preto 7

  44. Spearing H, Tawn J, Irons D, Paulden T, Bennett G (2021) Ranking, and other properties, of elite swimmers using extreme value theory. J R Stat Soc Ser A (Stat Soc) 184(1):368–395

    Article  MathSciNet  Google Scholar 

  45. Tadeu A, Lima G (2021) On the selection of relevant hardware events for explaining execution time behavior. In: 2021 IEEE XI Brazilian symposium on computing systems engineering (SBESC). IEEE, pp 1–8

  46. The LINUX man-pages (2022) https://man7.org/linux/man-pages/man2/perf_event_open.2.html, 2021. Accessed from May 2022

  47. Vilardell S, Serra I, Mezzetti E, Abella J, Cazorla F, Castillo J (2022) Using Markov’s inequality with power-Of-k function for probabilistic WCET estimation. In: 34th Euromicro conference on real-time systems (ECRTS 2022), Vol 231 pp 20:1–20:24, https://drops.dagstuhl.de/opus/volltexte/2022/16337

  48. Weaver VM, McKee SA (2008) Can hardware performance counters be trusted?. In: 2008 IEEE international symposium on workload characterization. IEEE, pp 141–150

  49. Wilhelm R, Engblom J, Ermedahl A, Holsti N, Thesing S, Whalley D, Bernat G, Ferdinand C, Heckmann R, Mitra T, Mueller F, Puaut I, Puschner P, Staschulat J, Stenström P (2008) The worst-case execution-time problem-overview of methods and survey of tools. ACM Trans Embed Comput Syst 7(3):53

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Dr. Verônica Lima and MSc. Tadeu Nogueira for their support.

Funding

This work have been partially funded by CAPES (Brazil), Grant no. 001, by the Inria-UFBA Associated Teams Program under the Kepler project, by the FR ANRT Joint CIFRE Inria and StatInf, Grant CIFRE no. \(1072\_/2020\) and by BPI France under the FR PSPC-regions - 2021 STARTREC project. The authors declare they have no financial interests, neither do they have any affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript. All data used in the research can be made available under request.

Author information

Authors and Affiliations

  1. Computer Science Institute, UFBA, Avenida Milton Santos, s/n - Campus de Ondina, PAF 2, Salvador, Bahia, 40.170-110, Brazil

    Jamile Vasconcelos & George Lima

  2. INRIA, 2 Rue Simone IFF, 75012, Paris, France

    Marwan Wehaiba El Khazen & Liliana Cucu-Grosjean

  3. StatInf, 2 Rue Simone IFF, 75012, Paris, France

    Marwan Wehaiba El Khazen & Adriana Gogonel

Authors
  1. Jamile Vasconcelos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marwan Wehaiba El Khazen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adriana Gogonel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liliana Cucu-Grosjean
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JV and GL conceived the study and the original manuscript. JV performed the experiment. GL supervised the project. MW verified the analytical methods, implemented and applied the Drees and Dietrich tests. AG and LC helped supervise the project. All authors discussed the results and reviewed the manuscript.

Corresponding authors

Correspondence to Jamile Vasconcelos or George Lima.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is an extended version of the conference paper that appeared in DOI: https://doi.org/10.1109/SBESC56799.2022 [26].

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vasconcelos, J., Lima, G., Wehaiba El Khazen, M. et al. On vulnerabilities in EVT-based timing analysis: an experimental investigation on a multi-core architecture. Des Autom Embed Syst (2023). https://doi.org/10.1007/s10617-023-09277-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10617-023-09277-5

Keywords

Search

Navigation