2. Akaike and predictive accuracy

Suppose you have a body of data concerning two quantities, X and Y. You know X and Y are linearly related and you wish to find the particular linear function that captures their relation. There is a standard solution to this problem on which there is consensus among statisticians. You must find the linear function with maximum likelihood relative to the data, where the likelihood of a hypothesis H given data D, L(H, D), is defined as the probability (or probability density) of obtaining D conditional on H being true; that is, L(H, D) = ^df P(D|H).

What if the inference problem involves choosing between members of different models, for example between the family of linear functions and the family of parabolic functions? (Hereafter, I’ll refer to families of hypotheses as ‘models,’ and I’ll reserve the term ‘hypothesis’ for particular members of models.) Here you cannot simply maximize likelihood. If you do, you will almost always choose a parabolic function (or generally the most complex model), even if the true relation is linear. This is due to the existence of noise in data. The more complex a model is, the more freedom it has to fit the noise, and, therefore, the more likely it is to fit the noise instead of the main pattern in the data. This is called ‘overfitting’ the data. Akaike’s framework offers a solution for how to avoid overfitting if one’s aim is to estimate (and maximize) the predictive accuracies of models—or equivalently to estimate (and minimize) their relative KL divergences from truth.

A few notational conventions. I will denote the truth by f. By the truth, I mean the true distribution of observed values. If the object of inquiry is the relation between two variables, X and Y, whose (unknown) true relation is y = D(x) with an observational error, E, f is captured by the equation, y = D(x) + E. Footnote ⁹ Notice that whether or not D(x) is deterministic, f is always probabilistic because E is probabilistic. Consider a model, F(θ), defined over a parameter space Θ, (θ ∈ Θ). For example, if F is the family of linear functions with a normally distributed error with the unknown variance, σ, F can be characterized by F:{y = ax + b + N(0,σ), a,b∈R, σ∈R⁺}. Here the parameter space of F is three dimensional. Since F is understood as a family of probability functions over various possible values of X and Y, we can associate a likelihood function to it, F: L(θ) = L(θ,z) = g(z|θ), where g(z|θ) is the probability density of obtaining the data set z = {(x ₁, x ₁), (x ₂, y ₂), … ,(x_n, y_n)} conditional on θ being the true parameter value. The value of θ for which L(θ) is maximized is called the Maximum Likelihood Estimate (MLE) of F and I denote it by $ \hat{\unicode{x03B8}} $ .

Equation (1) defines the KL divergence of a fully specified probability distribution from another. The KL divergence of a model from the truth is defined as the average (with respect to data) KL divergence of the MLE of the model. That is,

(2) $$ \left\{D\mathrm{KL}\left(f\;\Big\Vert F\left(\unicode{x03B8} \right)\right)\right\}= df\;E\boldsymbol{y}\left\{D\mathrm{KL}\left(f\;\Big\Vert F\left(\hat{\unicode{x03B8}}\left(\mathbf{y}\right)\right)\right)\right\}=E\boldsymbol{y}E\boldsymbol{x}\left[\log \left(f\left(\mathbf{x}\right)\right)\right]\hbox{-} E\boldsymbol{y}E\boldsymbol{x}\left[\log \left(g\left(\mathbf{x}|\hat{\unicode{x03B8}}\left(\mathbf{y}\right)\right)\right)\right]. $$

Here g(x| $ \hat{\unicode{x03B8}} $ (y)) is the likelihood (relative to data set x) of the MLE of F with respect to the data y. The last equality holds because of (1) and the fact that log(f/g) = log(f) – log(g). E _y E _x [log(f(x))] is an unknown constant that depends solely on f. When we are interested in comparing contender models, this term can be ignored, since it is equal for all models. Therefore, a quantity of interest is:

(3) $$ \mathrm{A}(F)= df\;E\mathbf{y}E\mathbf{x}\left[\log \left(g\left(\mathbf{x}|\hat{\unicode{x03B8}}\left(\mathbf{y}\right)\right)\right)\right]. $$

Malcolm Forster and Elliott Sober have called this quantity the predictive accuracy of F. Footnote ¹⁰ Suppose you obtain data, y , generated by f and use it to determine $ \hat{\unicode{x03B8}} $ (y). Then you obtain a new data set, x , and measure the fit of $ \hat{\unicode{x03B8}} $ (y) relative to x, where fit is measured in terms of the logarithm of likelihood (hereafter, log-likelihood). The predictive accuracy of F is the average value—with respect to both x and y—of this log-likelihood. The predictive accuracy of f is more than any other hypothesis or model. And the KL-closer to f a model is, the better its predictions are.

The goal in the Akaikean framework is to estimate predictive accuracy, which, by equations (2) and (3), is equivalent to the KL divergences of the model from truth modulo a constant, which is equal to the unknown predictive accuracy of f. Akaike offered the Akaike Information Criterion (AIC) of model F as an estimator of its predictive accuracy.

(4) $$ \mathrm{AIC}\left(F,\mathbf{y}\right)= df\;\mathrm{logL}\left(\hat{\unicode{x03B8}}\left(\mathbf{y}\right)\right)\hbox{-} k $$

Here logL( $ \hat{\unicode{x03B8}} $ (y)) is the log-likelihood of F’s MLE and k is its number of adjustable parameters. Akaike showed that if certain rather nonrestrictive conditions are met, AIC(F) is an unbiased estimator of A(F) for large data sets. An estimator is unbiased just in case its average value equals the value it estimates. That is,

(5) $$ E\mathbf{y}\left[\mathrm{AIC}\left(F,\mathbf{y}\right)\right]=\mathrm{A}(F) $$

Footnote ¹¹

Equation (5) captures Akaike’s main result.

3. Precisifying the realist thesis

Three features of the empirical success of scientific theories are pertinent to NMA. First, our best theories have sometimes fit large bodies of data very precisely. Second, such theories are not exceedingly complicated— they do not achieve such marvelous fit with data by making increasingly complicated (or ad hoc) modifications. Third, and importantly, scientific theories have been able to predict surprising facts that have nothing to do with the body of observations for the accommodation of which the theories were originally formulated. For example, Fresnel’s wave theory of light entailed that if an opaque disk is placed in front of a point source of light, there will be a bright spot at the center of the disk’s shadow. That such a bright spot exists wasn’t part of the evidence for the accommodation of which Fresnel designed his theory—indeed it was pointed out as an “absurd” consequence of the theory by Poisson. And this makes the theory especially well-confirmed by the observation of the bright spot. Or so it is argued by proponents of predictivism: the thesis that ceteris paribus, successful predictions provide stronger evidence for a theory than accommodations.

I believe a very strong version of predictivism can be established using Akaike’s results. Accordingly, other things being equal, the model that predicts a body of data has a few units higher AIC score than the model that accommodates that same body of data. (A difference of a few units of AIC amounts to a significant difference in estimated predictive accuracy.)Footnote ¹² However, since I’ll argue that NMA cannot establish anything stronger than RKL, even if such a strong version of predictivism is true, establishing the truth of this version of predictivism is unnecessary for my argument in the present essay. The reader who maintains a weaker version of predictivism (or rejects it altogether) should find no problem with my assumption of this version of predictivism here.

Remarkably Successful theories are simple and fit the data excellently. Moreover, they have had successful novel predictions. Therefore, they enjoy an amazing balance of goodness-of-fit and simplicity: we can be very confident of their predictive accuracies. What explains this? RKL. By equations (2) and (3), if a theory is KL-close to truth, its predictive accuracy is close to that of f—the highest possible predictive accuracy.Footnote ¹³ If T is KL-close to f, the expected (meaning, average) accuracy of T’s predictions is high. Thus, it is reasonable to expect T to have a track-record of empirical success.

The above explanation might appear tautological, but it’s not. RKL concerns a general tendency of a theory. Remarkable Success concerns its actual history. Compare: that the coin I am tossing is fair explains the observation that in the 1,000 times I tossed it, the ratio of heads to tails was nearly 1. However, admittedly the above explanation doesn’t offer much by way of explaining Remarkable Success. Surely, we’d want a deeper explanation of Remarkable Success. I offer such an explanation by offering an explanation for RKL itself in section 5.

Before ending this section, I’d like to point out that RKL is the weakest sufficient explanation of Remarkable Success in the sense that if RKL is false, then Remarkable Success is an improbable coincidence. This comes directly from the fact that if T is not KL-close to f, its predictions aren’t likely to be close to data generated by f. ^{Footnote 14}

4. Is RKL realist enough?

5. The realist’s rejoinder

Recall that one of NMA’s premises was Uniqueness: that the Realist Thesis offers the only plausible explanation for Remarkable Success. Having RKL as a potential explanation for Remarkable Success, the problem for the realist is that she cannot easily appeal to NMA to establish a stronger version of the Realist Thesis. She has two options. She can argue that a stronger version is a better explanation of Remarkable Success than RKL. Or, she can argue that RKL itself calls for an explanation and that explanation must appeal to a stronger thesis. I won’t discuss the first option in detail. As I argued in section 3, if the proposed explanation doesn’t entail RKL, then Remarkable Success becomes an improbable coincidence. This makes the first option unlikely to work. But no more could be said about this without knowing the exact content of such a rival explanation.

However, the second option appears promising. Indeed, this is a rather typical argumentative strategy for the realist. For example, Arthur Fine argues that Remarkable Success can be explained by the instrumental reliability of our theories, where ‘instrumental reliability’ is understood as a kind of “capacity” to produce good predictions.Footnote ¹⁹ In response, Stathis Psillos writes,

A virtually identical argument can be offered against RKL. What explains the fact that our theories are remarkably KL-close to truth other than its fundamental commitments being approximately true? RKL itself cries for explanation.

Fair enough. I owe the realist an explanation for RKL and I think I have one. Our theories are KL-close to truth because the methodology of science involves (among other things) replacing theories that are less simpleFootnote ²⁰ and less close to the extant data by those that are more so. Akaike’s results tell us that if we follow this methodology, we will continually get KL-closer to truth. Thus, if there are theories in our radar that are meaningfully KL-close to truth, we will sooner or later choose them. Nothing about the truth (or near truth, whatever that means) of fundamental/ontological aspects of our theories is required.Footnote ²¹

Here the main idea is that scientific methodology involves (among other things) balancing considerations of goodness-of-fit with those of simplicity. To the extent that this is the case, scientific methodology jibes with the Akaikean recommendation. Given Akaike’s results, it follows that subsequent theories are typically KL-closer to truth than their predecessors.

Two points must be mentioned here. First, since RKL concerns only the predictive elements of the theory, the above explanation is satisfactory only if considerations of fit and simplicity of the predictive elements of the theory play an important role in theory choice. Fortunately, the fit of the predictive elements and the fit of the theory are the same thing—the theory is connected to data via its predictive elements. However, simplicity considerations are not exhausted by those of predictive elements. For example, although other things being equal, theories with more parsimonious ontologies have simpler predictive elements (a linear function of one variable is simpler than a linear function of two variables), the ontological parsimony of the theory need not be reflected in the simplicity of its predictive elements. Second, considerations of fit and simplicity ground many other considerations that are important in theory choice. Earlier I claimed that the evidential significance of predictions is grounded in simplicity-favoring considerations. Forster and Sober (Reference Forster and Sober1994) argue that unification is similarly grounded.Footnote ²² This explanation of RKL works if the totality of such considerations plays a major role in theory choice. I think they do.

Objection: the Akaikean framework recommends optimizing a very particular balance between goodness-of-fit and simplicity. Thus, to show that the methodology is responsive to the same considerations doesn’t establish that scientific methodology and the Akaikean recommendation are in line. They must assign these considerations the same relative weights. This objection, although strictly speaking true, overlooks an important fact. Scientific inferences, at least insofar as our best theories are concerned, are different from typical curve-fitting or model selection problems in an important respect: they are based on very large data. When data size increases, the first term in AIC (logL(L(M))) is proportional to n, the number of data points. (Because with large n, the average per datum log-likelihood is almost constant for different values of n.) However, the penalty term for complexity remains constant. Therefore, considerations of goodness-of-fit dominate those of simplicity. In other words, when data size is large, there is a lexical ordering between the two considerations: simplicity-favoring considerations are relevant only when rival theories have (or are supposed to have) equally good fit with the data.

Although it goes well beyond the scope of the present essay to show this conclusively, I think scientific methodology does give simplicity a secondary status. The most famous historical example in which simplicity-favoring considerations supposedly played a major role is the case of Ptolemaic versus Copernican astronomy. In the seventeenth century, when Copernicanism became dominant in scientific circles, there existed a relatively massive body of observational data on the motions of the planets. (Notice that every single precise-enough observation is a datum here.) The two astronomical theories were about equally fit with those observations. Thus, the relative simplicity of the Copernican system could play a decisive role in theory choice.Footnote ²³

Examples like this are suggestive but are no proof for my claim. The best argument I can offer (still not a proof) is this. Scientists often adopt a theory despite its inability to accommodate part of observational data in part because it is simple or has other epistemic merits. But they do so believing (justifiably or not) that the discrepancy will be sorted out. That is, they don’t accept the “errors” in the deliverances of the theory in a permanent fashion. They temporarily allow for those “anomalies” because they are confident in the theory’s ability eventually to accommodate them or for the recalcitrant data to turn out to be misleading evidence. If there was an in-principle competition between simplicity and goodness-of-fit, it would have been an acceptable argument that the anomaly exists and is not going to go away, but this epistemic vice is counterbalanced by the theory’s other merits, like simplicity. This doesn’t appear to be a typically acceptable argumentative strategy in science, especially when it comes to our best theories.

The above argument might lead to a misunderstanding. The idea that scientific methodology is roughly in accord with the Akaikean recommendation might appear to suggest a clear defense of methodological instrumentalism (that science aims at instrumental success). The goal in the Akaikean framework is maximizing predictive accuracy—an undoubtedly instrumentalist goal. Thus, to suggest that the practice of science is in accord with the Akaikean recommendation might seem an endorsement of methodological instrumentalism. However, I have argued that scientific methodology jibes with the Akaikean recommendation only insofar as it gives a secondary status to simplicity when data size is large. Interestingly, we have another framework based on Bayesian Information Criterion (BIC) (defined by BIC(F)=^df logL(L(F))) – ( $ \frac{k}{2} $ )log(n), where n is the number of data points) that can be interpreted in ways that are harmonious with methodological realism,Footnote ²⁴ but similarly assigns a secondary status to simplicity for large data. (When n is large, the first term, which is proportional to n, dominates the second term, which is proportional to log(n)).Footnote ²⁵ Thus, my observation that simplicity has a secondary status in scientific methodology is compatible with both realist and instrumentalist frameworks in model selection.

This ends my argument for the claim that successive scientific theories are typically KL-closer to truth than their predecessors. This is the main idea in my explanation for RKL, but two more ideas are needed. (A) That there aren’t too many theories in our theory pool, so that we could always get KL-closer but never close enough in any meaningful way. And more importantly, (B) that some of our candidate theories for explaining the entirety (or a big part) of the data are meaningfully KL-close to truth to begin with.Footnote ²⁶ Neither fact is trivial, but we can be fairly confident about both. We have direct access to the number of theories taken seriously by scientists. Although the number of candidate theories (both at any given time and collectively over time) isn’t meager, it’s not unmanageable either. So A is true. B is more interesting and more difficult to establish. Indeed, I think B is neither true of all sciences, nor always about any particular science. Sometimes we don’t have any good candidate theory that can satisfactorily fit the existing data without unacceptable complexity. In fact, the hitherto falsity of B for certain scientific disciplines, like psychology, might be the reason why they arguably haven’t attained the status of mature science. However, when it comes to scientific areas to which our best scientific theories belong, we have direct evidence that some theories in our radar are KL-close to truth, because we have direct evidence that our theory itself is such.

The realist might justifiably ask for an explanation for A and B themselves. Why is it that we are able, at least in certain areas of inquiry, to formulate theories that are KL-close to truth? I think this is a fair question, but one that is unlikely, even prima facie, to help the realist’s position. Consider the set of all the relatively simple models describing functional dependencies between the “natural” properties of a system (properties that are describable by predicates like green and blue, as opposed to grue and bleen) and call it M(S), where S is the system under study. We can think of M(S) as the set of our candidate models for S prior to consulting the data at any given point in the development of a science. (It’s a mistake to think of M(S) as a fixed set determined a priori. More about this momentarily.) This is because we hardly ever consider hypotheses that are either formulated in terms of functions of great complexity (like the family of polynomials of degree 1000) or in terms of other, “nonnatural” properties like grue. M(S) is huge but it’s not unmanageably so, especially given the fact that data typically weeds out an absolute majority of its members. Thus, the fact that M(S) is the set of our candidate models explains A. B can be explained by the fact that there are members of M(S) that are KL-close to truth (given an error distribution). But what explains this fact?

The realist might argue that only the (near) truth of the fundamental/ontological aspects of our theories can explain this. Richard Boyd has offered such an argument. Accordingly, all aspects of scientific method—including which properties of a system are deemed “natural” or the kind of functional dependencies that are considered “simple”are theory-dependent. Therefore, the judgment to include a given function in M(S) is itself theory dependent. Boyd adds,

I agree that the determination of the members of M(S) (among other aspects of scientific methodology) is a theory-dependent judgement. However, I think the move from this idea to “a thoroughgoingly realistic explanation” isn’t warranted. Take the concept of ‘inertial mass’ as it was introduced to natural philosophy by Newton. Why were he and other natural philosophers who came after him convinced that mass is a “natural” property of a physical system? Because they observed that simple hypotheses formulated in terms of mass can fit the extant data nicely. That is, the fact that scientific methodology jibes with the Akaikean recommendation explains the fact that the development of M(S) over time leads to the inclusion of new hypotheses that are KL-closer to truth than their predecessors.

This is not (yet) a full explanation of the fact that some members of M(S) are meaningfully KL-close to truth. We could be getting closer (in adding new members to M(S)) but not be meaningfully close. Thus, the remaining question is: Why is it that we are able to formulate theories that are meaningfully KL-close to truth at all? I don’t know the answer, but I can think of only two initially promising ones. (I) Nothing. It is just a brute fact that there are members of M(S) that are KL-close to truth. This might not be as bad an idea as it initially appears. As I argued earlier, being KL-close to truth is not a particularly exacting requirement. Therefore, that our most fundamental epistemic tendencies (e.g., the concepts in terms of which we tend to think) are such that in some areas of inquiry, there are members of M(S) that are KL-close to truth might not be a particularly impressive epistemic condition to need an explanation. (II) It is evolutionarily beneficial to think in terms of concepts that are such that simple hypotheses formulated in their terms are KL-close to truth. Notice, however, that although constructing theories that are KL-close to truth (or instrumentally reliable) might have evolutionary benefits, forming ontologically/fundamentally true theories has no further evolutionary benefit. (That is, having ontologically/fundamentally true beliefs has no evolutionary benefit conditional on the belief being instrumentally reliable.) Thus, if this explanation is indeed true, it doesn’t favor the realist’s position.Footnote ²⁷

Is there any realist-friendly explanation for the fact that some of the members of M(S) are KL-close to truth? The only thing that comes to my mind is something like Descartes’s appeal to God’s benevolence in creating us (which justifies his clear-and-distinct-idea criterion). According to that explanation, we tend to think about concepts that enable us to discover deep truths, which then entail predictive elements that are KL-close to truth. Regardless of what one thinks about this epistemological view, I think even Descartes would have probably agreed that it is particularly unconvincing to argue for it by way of explaining the Remarkable Success of science. I cannot think of any plausible realist rejoinders at this point.

Therefore, I conclude that the strongest version of the Realist Thesis that NMA can plausibly establish is RKL and, in some instances, its extensions like RKL⁺.

Perhaps there is a deeper lesson we can learn from NMA’s inability to establish a strong version of realism, independently of the particular nature of RKL and its explanation. In many academic disciplines, it is part of common wisdom that there ain’t no such thing as a free lunch. Philosophers are by no means strangers to this idea. It is a notorious fact about deductive reasoning that it doesn’t discover anything that’s not already contained in the premises. That is why there’s been more attention in recent decades to various nondeductive forms of reasoning, both as objects of study and as forms of reasoning within philosophy. Granted that such nondeductive forms of reasoning are ampliative—they add something to the premises—we should still be wary of arguments that purport to give us something big for nothing. But that is exactly what NMA purports to give us. It begins with the instrumental success of science and purports to establish the (near) truth of ontological or fundamental commitments of scientific theories. Even if we didn’t have a well worked out answer to the realist’s challenge for explaining Remarkable Success, this big leap in NMA would have been sufficient ground for concern. Indeed, I find it particularly problematic that NMA is insensitive to the particular content and the prima facie metaphysical plausibility of the theory’s ontology—instrumental success is supposed to be enough.

A useful concept in studying a physical system is that of its ‘degrees of freedom.’ When the number of constraints on the system are less than its degrees of freedom, its state is indeterminable. Perhaps a similar notion is useful in epistemology. When all you know about a theory is that it is predictively successful, there is no way you can decide the truth of particular parts of it. More information is needed to settle the issue one way or the other. And in all likelihood, the issue is going to be settled differently for different theories, even if they are all predictively successful. This is compatible with the plausible idea that the predictive success of a theory provides evidence for its fundamental commitments. But having evidence for something is different from having a (fallible) proof for it.