Statistics, philosophy, and health: the SMAC 2021 webconference

2.1 Do you think that statisticians should learn to rely on deterministic models?

We open with the contribution of Professor Antoine Chambaz, professor of statistics at Université de Paris, a member of MAP5 (UMR CNRS 8145) and the head of its Statistics team. He is also the director of FP2M (FR CNRS 2036), the Parisian Federation of Mathematical Modelling. Professor Chambaz holds a PhD in mathematics (2003) from the Paris-Sud University. His main research interest is in theoretical, computational and applied statistics, and in causality. In particular, he contributes to the development and application of the targeted learning methodology, with a focus on applications to medicine and precision medicine, and studies problems at the intersection of statistics and machine learning. Professor Chambaz coedits The International Journal of Biostatistics. Professor Chambaz’s video can be seen at this link.

To answer questions on a phenomenon of interest, the statistician defines a statistical model M for the law P of the experiment consisting of observing the phenomenon, and learns some problem-specific, relevant features of P. In parallel, countless deterministic models have been developed to model natural phenomena. What do I call a “deterministic model”? It is a mathematical model obtained by combining elementary blocks, each block built based on the fundamental laws of physics (e.g., conservation of mass-energy, linear momentum). The model provides a (possibly multi-scale) description of a phenomenon under the form of equations involving quantities that are observed or not. Physico-chemical, physiological, environmental parameters drive the regime of the model. Such deterministic models lend themselves to being combined in sophisticated “hierarchies of models”. Here are my questions to my esteemed colleagues: do you also think, as I do, that statisticians should learn to rely on deterministic models? If so, how?

2.2 What do statistical tests really say?

What follows is the contribution of Professor Bruno Falissard. After some initial training in mathematics and fundamental physics, Professor Falissard engaged in medical studies and specialized in Child and Adolescent Psychiatry in 1991. He completed a PhD in biostatistics and was an assistant professor in child and adolescent psychiatry in 1996–1997, an associate professor in biostatistics at University Paris-Nord in 1997–2002, and a full professor in biostatistics at University Paris-Sud from 2002. Professor Falissard is the head of the Center of Epidemiology and Population Health and co-author of more than 400 papers. His areas of research are in methodology and epistemology of medical research, in particular in psychiatry. Professor Falissard’s video can be seen at this link.

Between 1920 and 1930, the decade that can be considered as the origin of the formalization of statistical tests, a drama took place. Among the characters, there was a “father” (Karl Pearson), a “son” (Egon Pearson), a “genius” (Ronald Fisher) and a “mathematician” (Jerzy Neyman). This drama could appear at first glance as a ridiculous family dispute: is “p” or “alpha” the cornerstone of statistical inference? It is in fact much more than that, and a hundred years after the statistical community is still burdened with very simple questions: is acceptance of a null hypothesis possible? Are one-sided tests conceivable? Or, on the contrary, are statistical tests always one-sided? Is there really a difference between “p = 0.048” and “p = 0.052”? What did R. Fisher and J. Neyman mean, exactly, when claiming that the former dealt with inductive inference and the latter with inductive behaviour?

2.3 Is the field of data science ready to solve the world’s problems?

We now turn to the contribution of Professor Michael R. Kosorok. Professor Kosorok is a W.R. Kenan, Jr. Distinguished Professor of Biostatistics and Professor of Statistics and Operations Research. He is the 2019 ASA Gottfried E. Noether Senior Scholar Award recipient, and is a Fellow of the American Association for the Advancement of Science, the American Statistical Association, and the Institute of Mathematical Statistics. His video can be seen at this link.

The field of data science, an amalgamation of many disciplines, including the foundational fields of computational sciences, informatics, mathematics, and statistics, is crucially important in the quest to improve the world through wise use of data and information. Each of the foundational constituent disciplines brings crucial perspectives and capabilities to the table, but are we flexible and deep enough in the way we are thinking about the foundations of data science? Are we working sufficiently well with applied domain experts and key stakeholders in society to maximize benefit and minimize harm? Are we inclusive enough in how we collaborate with application domain experts?

The focus of the field of statistics is inference, which, generally speaking, refers to the question of how conclusions from data will have meaning in the real world. Statistics has a long tradition of models and frameworks for answering these questions; however, we are at the cross-roads of a potential major transition in the field. The essence of this question at the cross-roads is whether we are willing to sufficiently adapt as a field to rise to the occasion that presents itself. These questions include the following four: (1) can we expand our theoretical foundations, methods and applications to be sufficiently broad to handle all data and analytic situations which may present themselves? (2) Are we ready to better teach and train the world on the fundamentals of inference, inferential intuition, and practical quantitative reasoning, including teaching other data scientists, domain science experts, policy makers, and society at large? (3) Are we prepared to work with new sources of data and information, including qualitative, non-stochastically generated samples, and data acquired in real time, for example? (4) Are we willing to take on collaborative leadership to a greater degree, as needed? Similar questions arise in the other constituent foundational data science domains, and we all need to proactively prepare for the coming explosion in data science and its applications.

2.4 Why can’t we cure more patients who have advanced cancer?

We now have the contribution of Dr Elizabeth Krakow. Dr Krakow is an assistant professor in the Clinical Research Division at Fred Hutchinson Cancer Research Center and in Medical Oncology at the University of Washington. Her clinical practice is focused on blood and bone marrow transplantation. She conducts clinical trials of immunologic therapies to prevent and treat leukemia relapse after such transplantations. Her video, taking the format of a conversation with Dr Jerald Radich also of the Fred Hutchinson Cancer Research Center, can be seen at this link.

Biologists have developed “evolutionary” models of cancer progression based on relationships among competing and cooperating malignant clones, the patient’s immune system, treatments administered, and tissue microenvironments. They have characterized basic mechanisms of tumour progression under the pressure exerted by anti-neoplastic treatments, such as: Darwinian selection pressure, culling-the-herd (where all malignant clones are reduced proportionately, only to regrow once treatment is stopped), and stochastic emergence of new mutations. However, there is a gap when applying these models in clinical practice. We are limited by the cost of single cell sequencing and by sampling bias introduced when a small volume of cancerous tissue is excised at a single time-point (because serial, large-volume tissue biopsies are usually too invasive, and the cancer continues to change after the biopsy is taken). Therefore, when looking at a given patient, we usually do not know which of these models applies to his or her cancer progression. Using less costly bulk tumour sequencing on sub-optimally sized tumour samples, how can we infer which phylogenetic branches of clonal evolution are being traced by a specific patient’s cancer? How can we use these biological models to guide treatment selection?

A second question arises from the following challenge: clinical trials are generally designed by frequentists, but then we, as clinicians, try to be Bayesian. We set up some prior probability of survival for the patient based on our experience, tumour staging, etc. and then, based on the clinical trial results, we estimate how the probability of survival will change if we choose treatment A or treatment B. But how can we make the necessary leap from trial-derived statistical models to patient-specific prediction? Moreover, for many cancer-associated mutations or combinations of mutations, the population frequency is so low that it is impossible to launch a traditional randomized treatment trial. The tremendous inter-patient variability in drug metabolism and immunologic responses adds further complexity. Can society build the infrastructure to collect and fairly disseminate big, yet granular and accurate, data sets that might support new clinical study designs that take advantage of our molecular-level understanding of cancer evolution and that are better suited to developing and validating highly personalized adaptive treatment strategies?

2.5 Four questions to ask in scrutinizing accounts of statistical inference

Next, we have the contribution of Professor Deborah Mayo. Professor Mayo is Professor Emerita in the Department of Philosophy at Virginia Tech. She is author of Error and the Growth of Experimental Knowledge (1996, Chicago) which won the 1998 Lakatos Prize. Her most recent book is Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars ([1], CUP). She blogs at errorstatistics.com. Her video contribution can be seen at this link.

The main source of today’s crisis in replication is that high-powered methods make it too easy to uncover impressive looking findings, even if they are false. If little or nothing has been done to uncover flaws in inferring a claim, we do not have evidence for it. It has not passed any kind of a severe test. So the first question to ask in scrutinizing a statistical account is: does it permit violations of this minimal requirement for evidence? A second question concerns presuppositions about the role of probability in inference. In frequentist error statistics, probability is intended to assess and control the probabilities of misleading interpretations of data. To those who assume probability should provide degrees of belief or probability in hypotheses, a p-value appears to be irrelevant, or necessarily misconstrued. But that is to presuppose a rival philosophy of inference akin to Carnap insisting Popper’s falsificationist become a confirmationist. So the second question to ask is: does an account enable falsifying claims at least statistically? We often hear that it is just too easy to obtain small p-values. Yet replication attempts find it is difficult to get small p-values when they have preregistered hypotheses. What this shows is that the problem is not p-values, but failing to adjust their values for cherry picking, multiple testing, and other biasing selection effects. A third question is: is the uncertainty measure directly altered by biasing selection effects, as we think they should be? The same data-dredged hypothesis can occur in methods put forward as alternatives to p-values: likelihood ratios, Bayes’ factors, and others, except now we lose control of the grounds to criticize inferences for flouting error control. Finally, even those who criticize p-values will employ them, at least if they care to test the assumptions of their statistical model – an altogether important task. So a fourth question is: is the account capable of testing its own assumptions? So, if someone is selling you an account or a reform, where the answer to any of these questions is no, you may wish to hold off buying it.

2.6 Being wrong about randomisation

The next contribution was that of Professor Stephen Senn. Professor Senn is a medical statistician. He has worked in England, Scotland, Switzerland, Luxembourg, and France as an academic and as a research scientist, and also in the pharmaceutical industry. His research interests are statistical methods in drug development. Professor Senn’s video contribution can be seen at this link.

You sometimes hear the following criticism of randomisation: whether or not some factors are balanced as a result of random allocation, since there are indefinitely many factors, some of them must be imbalanced. Therefore, the analysis of randomised clinical trials is not valid. There may be good arguments against randomisation but this is a really bad one. For me, it is a litmus test statement. If you believe it, I know that you do not understand randomization. I also strongly suspect that you do not understand the basics of statistical analysis. And I am absolutely confident that you have never tried to simulate the problem that you claim exists. Doing is a cure for stupid speculation.

To show how wrong it is, let me give you a statement that is true and that is almost the opposite of this. If we knew that every possible prognostic factor were balanced, then the standard analysis of clinical trials would be wrong. The critics of randomisation have picked up the problem from the wrong end. They have started by assuming that the goal is perfect estimation. But perfect estimation cannot accept any imperfections. Therefore, we must always worry about things that might be imbalanced. It is clear that randomisation cannot balance everything. Therefore, randomisation cannot be trusted. If you believe that this is the way to start the argument, let me ask you a question. Do you think that anything useful can be said about the probability that a fair die will show three sixes if rolled three times? After all, there are indefinitely many factors that could affect the outcome.

The correct way to start is to accept that perfect estimation is impossible. Setting probabilistic bounds on how imperfect your estimates are is possible and useful. It is this goal that randomisation supports. Standard statistical methods make an allowance for random imbalance. If we knew that every possible prognostic factor were balanced, then the standard analysis of clinical trials would be wrong.

2.7 The need for realistic statistical models and utilization of sieve MLE (highly adaptive lasso)

Finally, the videos concluded with the contribution of Professor Mark van der Laan. Professor van der Laan is the Jiann-Ping Hsu/Karl E. Peace Professor of Biostatistics and Statistics at the University of California, Berkeley. He has made contributions to survival analysis, semiparametric statistics, multiple testing, censored data and causal inference. He also developed the targeted maximum likelihood methodology and general theory for super-learning. He is a founding editor of the Journal of Causal Inference and The International Journal of Biostatistics. He has authored four books on targeted learning, censored data, and multiple testing, authored over 350 publications, and graduated 55 PhD students. He is the recipient of the Mortimer Spiegelman Award (2004), the COPSS Presidents’ Award (2005), and the van Dantzig Award (2005). Professor van der Laan’s video can be viewed at this link.

An important question raised is whether our field is willing to adopt the formulation of statistical estimation problems in terms of (1) realistic statistical models for the data distribution, and (2) estimands as defined as mappings from these possible data distributions into the answer to question of interest (or approximation thereof). The emphasis is on the statistical model being realistic so that it only incorporates assumptions about the data generating experiment that are known to be true of at minimal can be well defended. As a result of accepting this translation of the real world into a statistical estimation problem, is one then willing to define an a priori defined estimator, possibly based on outcome blind data, and corresponding statistical inference, augmented possibly with a sensitivity analysis with respect to non-testable assumptions that enhance the interpretation of the estimand of the data distribution. We often refer to this as the roadmap for statistical learning and causal inference [2, 3].

The second issue is related to this so-called formulation of a statistical estimation problem in the sense that it requires utilization of machine learning algorithms that are able to learn the true target functions of the data distribution at a fast enough rate. We conjecture that good robust machine learning algorithms should aim to be sieve maximum likelihood behaving estimators so that they solve a rich set of score equations, but also generate scores that are embedded in a well-behaving Donsker class. In particular, a more recent nonparametric maximum likelihood estimator terms the Highly Adaptive Lasso MLE (HAL-MLE) is contrasted to other machine learning algorithms [4, 5]. It is noted that HAL-MLE indeed solves a class of score equations that approximates the space of all score equations as sample size increases, and, in addition, it converges at a n ^−1/3 rate to the true target function, up till log n-factors. As a result it has been shown to result in efficient plug-in estimators of a large class of target features of the data distribution [6]. The key properties which makes this HAL-MLE so robust are, in particular, that it solves a rich set of score equations and that the linear span of these score equations remains embedded in a nice Donsker class with excellent entropy integral. We suggest that this perspective on benchmarking machine learning algorithms would move the field forward in the right direction by focusing on algorithms that satisfy these key fundamental properties that drive their statistical behaviour.