On Exponential-time Hypotheses, Derandomization, and Circuit Lower Bounds

1.2.1 Background: Uniform Hardness vs. Randomness.

Intuitively, using “hardness-to-randomness” results, we expect that a strong lower bound such as rETH would imply a strong derandomization result. When starting from lower bounds for non-uniform circuits, and aiming to deduce worst-case derandomization, smooth tradeoffs that yield such results are well-known (see, e.g., [34, 44, 50, 54, 57]) The key problem, however, is that the long line of works that starts from hardness for uniform algorithms (and aims to deduce average-case derandomization) did not yield such smooth tradeoffs so far (see [6, 8, 20, 26, 27, 33, 35, 41, 51, 56]).

Ideally, given an exponential lower bound for uniform probabilistic algorithms (such as \(\mathcal {E}\not\subseteq \mathtt {i.o.}\mathcal {BPTIME}[2^{\epsilon \cdot n}]\)),³ we would like to deduce that there exists a PRG with exponential stretch for uniform circuits, and consequently that \(\mathcal {BPP}=\mathcal {P}\) in “average-case.”⁴ However, prior to the current work, the state-of-the-art (by Trevisan and Vadhan [56]) could at best yield PRGs with sub-exponential stretch (i.e., with seed length \(\mathrm{polylog}(n)\)), even if the hypothesis refers to an exponential lower bound. Moreover, the best currently known PRG only works on infinitely many input lengths.

Previous works bypassed these two obstacles in various indirect ways. Carmosino, Impagliazzo, and Sabin [8] deduced polynomial-time derandomization of \(\mathcal {BPP}\) on all input lengths relying on strong hypotheses from fine-grained complexity (these hypotheses are implied by the “strong” version of rETH, i.e., by \(\mathsf {rSETH}\)). Gutfreund and Vadhan [27] deduced (subexponential-time) derandomization of \(\mathcal {RP}\) on all input lengths, rather than of \(\mathcal {BPP}\) (see details below). Last, a line of works dealing with uniform “hardness-to-randomness” for \(\mathcal {AM}\) (rather than for \(\mathcal {BPP}\)) was able to bypass both obstacles in this context (see, e.g., [26, 41, 51]).⁵

1.2.2 Our Contribution to Uniform Hardness vs. Randomness.

In this work, we tackle both obstacles directly. Loosely speaking, our first main result is that rETH implies the existence of a PRG for uniform circuits with near-exponential stretch, which can be used for average-case derandomization of \(\mathcal {BPP}\) in nearly-polynomial-time. Specifically, the PRG that we construct has seed length \(\tilde{O}(\log (n)\), and the corresponding derandomization runs in time \(2^{\tilde{O}(\log (n))}=n^{\mathrm{loglog}(n)^{O(1)}}\).

Our hardness assumption will in fact be weaker than rETH: It suffices to assume that the Totally Quantified Boolean Formula (\(\mathbf {TQBF}\)) problem cannot be solved by probabilistic algorithms that run in time \(2^{n/\mathrm{polylog}(n)}\) (see Definition 4.6 for a standard definition of \(\mathtt {TQBF}\)). This hypothesis is weaker than rETH, because \(3\text{-}\mathtt {SAT}\) reduces to \(\mathtt {TQBF}\) with a linear overhead in the input length. (Indeed, it is a far weaker hypothesis, since \(\mathtt {TQBF}\) is \(\mathcal {PSPACE}\)-complete, whereas \(3\text{-}\mathtt {SAT}\) is only \(\mathcal {NP}\)-complete.)

The technical statement of Theorem 1.1 is even stronger: For every \(t(n)=n^{\mathrm{polyloglog}(n)}\), the PRG is \((1/t)\)-pseudorandom for every distribution over circuits that can be sampled in time t and with \(O(\log (t))\) bits of advice (see Theorem 4.14 for details).

Theorem 1.1 establishes for the first time that hardness assumptions for \(\mathcal {BPTIME}\) yield a PRG for uniform circuits with seed length as short as \(\tilde{O}(\log (n))\) and running time as small as \(2^{\tilde{O}(\log (n))}\). The proof of this result is based on careful refinements of the proof framework of [33], using new technical tools that we construct. The latter tools significantly refine and strengthen the technical tools that were used by [56] to obtain the previously best uniform hardness-to-randomness tradeoff. For high-level overviews of the proof of Theorem 1.1 (and of the new constructions), see Section 2.1.

Overcoming the “infinitely-often” barrier. The hypothesis in Theorem 1.1 is that any probabilistic algorithm that runs in time \(2^{n/\mathrm{polylog}(n)}\) fails to compute \(\mathtt {TQBF}\) infinitely-often, and the corresponding conclusion is that the PRG “fools” uniform circuits only infinitely-often. (The meaning of “infinitely-often” is “on infinitely many input lengths,” and the meaning of “almost-always” that will be used next is “on all but finitely many input lengths.” Recall that a hypothesis of the form \(L\notin \mathcal {BPTIME}[T]\) only means that every probabilistic time-T algorithm fails to compute L infinitely-often.)

The shortcoming of Theorem 1.1 that the derandomization works only infinitely-often is identical to all previous uniform “hardness-to-randomness” results that used the [33] proof framework.⁶\(^{,}\)⁷ However, known techniques (see, e.g., [27]) can nevertheless be adapted to yield an almost-always PRG that uses \(O(\log (n))\) bits of non-uniform advice (relying on an almost-always lower bound hypothesis).

We are able to significantly improve this: Assuming the “almost-always” version of rETH, we show that \(\mathcal {BPP}\) can be derandomized in average-case and almost-always, using only a triply logarithmic number (i.e., \(O(\mathrm{logloglog}(n))\)) of advice bits. In fact, as in Theorem 1.1, it suffices to assume hardness for \(\mathtt {TQBF}\), rather than for \(3\text{-}\mathtt {SAT}\).

Similarly to Theorem 1.2, the conclusion in Theorem 1.2 can be strengthened so it holds for every distribution \(\mathcal {X}\) samplable in time \(t(n)=n^{\mathrm{polyloglog}(n)}\), and the derandomization succeeds on all but a \((1/t)\)-fraction of the inputs under \(\mathcal {X}\) (rather than only on a \(1-1/n\) fraction).

1.3.1 Background and a Surprising Observation.

The motivating observation for our results in this section is that NETH has an unexpected consequence to the long-standing question of whether worst-case derandomization of \(pr\mathcal {BPP}\) is equivalent to circuit lower bounds against \(\mathcal {E}\). Specifically, recall that two-way implications between derandomization and circuit lower bounds have been gradually developing since the early ’90s (for surveys, see, e.g., [45, 60]), and that it is a long-standing question whether the foregoing implications can be strengthened to show a complete equivalence between the two. One well-known implication of such an equivalence would be that any worst-case derandomization of \(pr\mathcal {BPP}\) necessitates the construction of PRGs that “fool” non-uniform circuits.⁹

Then, being more concrete, the motivating observation for our results in this section is that NETH implies an affirmative answer to the foregoing classical question. In fact, this is not difficult to show, relying on known results (see Section 2.2 for details).

1.3.2 Our Results: Even Very Weak Forms of NETH Suffice for the Equivalence.

Our main contribution is in showing that, loosely speaking, even a very weak form of NETH suffices to answer the question of equivalence in the affirmative, and that this weak form of NETH is in some sense inherent. Specifically, we say that \(L\subseteq \lbrace 0,1\rbrace ^*\) has \(\mathcal {NTIME}[T]\)-uniform circuits if there exists a non-deterministic machine M that gets input \(1^n\), runs in time \(T(n)\), and satisfies the following: For some non-deterministic choices M outputs a single circuit \(C:\lbrace 0,1\rbrace ^{n}\rightarrow \lbrace 0,1\rbrace ^{}\) that decides L on all inputs \(x\in \lbrace 0,1\rbrace ^n\), and whenever M does not output such a circuit, it outputs \(\perp\). We also quantify the size of the output circuit, when this size is smaller than \(T(n)\).

The weak forms of NETH that will suffice to show equivalences between derandomization and circuit lower bounds are of the form “\(\mathcal {E}\) does not have \(\mathcal {NTIME}[T]\)-uniform circuits of size \(S(n)\ll T(n)\),” for values of T and S that will be specified below. In words, this hypothesis rules out a world in which every \(L\in \mathcal {E}\) can be computed by small circuits that can be efficiently produced by a uniform (non-deterministic) machine. Indeed, this hypothesis is weaker than the NETH-style hypothesis \(\mathcal {E}\not\subseteq \mathcal {NTIME}[T]\), and even than the hypothesis \(\mathcal {E}\not\subseteq (\mathcal {NTIME}[T]\cap \mathcal {SIZE}[T])\).¹⁰ The fact that such a weak hypothesis suffices to deduce that derandomization and circuit lower bounds are equivalent can be seen as appealing evidence that the equivalence indeed holds.

Our results refer both to the “low-end” parameter regime, which connects relatively weak circuit lower bounds to relatively slow derandomization algorithms, and to the “high-end” parameter regime, which connects strong circuit lower bounds to fast derandomizatoin algorithms. Showing an equivalence in the former regime will require weaker hypothesis, compared to the latter regime.

Starting with the “low-end” regime, our first result is that if \(\mathcal {E}\) cannot be decided by \(\mathcal {NTIME}[2^{n^{\delta }}]\)-uniform circuits of polynomial size (for some \(\delta \gt 0\)), then derandomization of \(pr\mathcal {BPP}\) in sub-exponential time is equivalent to lower bounds for polynomial-sized circuits against \(\mathcal {EXP}\).

The scaling of Theorem 1.4 to the “high-end” regime us not smooth and uses different proof techniques (see Section 5 for details). Nevertheless, an analogous result holds for the extreme “high-end” setting: Under the stronger hypothesis that \(\mathcal {E}\) cannot be decided by \(\mathcal {NTIME}[2^{\Omega (n)}]\)-uniform circuits, we show that \(pr\mathcal {BPP}=pr\mathcal {P}\) is equivalent to lower bounds for exponential-sized circuits against \(\mathcal {E}\); that is:

(We remind the reader again that circuit lower bounds as in Theorems 1.4 and 1.5 are known to be equivalent to the existence of corresponding PRGs that fool non-uniform circuits [3, 34, 44, 54, 57]. Thus, the hypotheses in these theorems imply that derandomization requires PRGs.)

The very weak version of NETH is inherent (for a stronger conclusion that it yields). Remarkably, as mentioned above, hypotheses such as the ones in Theorems 1.4 and 1.5 actually yield a stronger conclusion and are also necessary for that stronger conclusion. Specifically, the stronger conclusion is that even non-deterministic derandomization of \(pr\mathcal {BPP}\) (such as \(pr\mathcal {BPP}\subseteq pr\mathcal {NSUBEXP}\)) yields circuit lower bounds against \(\mathcal {E}\), which in turn yield PRGs for non-uniform circuits.

Note that in Theorem 1.6 there is a gap between the hypothesis that implies Equation (1.1) and the conclusion from Equation (1.1). Specifically, the hypothesis refers to \(\mathcal {NTIME}[2^{n^{\delta }}]\)-uniform circuits of polynomial size, whereas the conclusion refers to \(\mathcal {NP}\)-uniform circuits. By optimizing the parameters, this gap between sub-exponential and polynomial can be considerably narrowed (see Theorem 5.11).

2.1.1 The Well-structured Function f^ws.

Let us now be more specific about the properties of the well-structured function \(f^{\mathtt {ws}}\) that we need in our proof. Our function \(f^{\mathtt {ws}}\) will satisfy the following:

The last property, which is implicit in many works and was recently made explicit by Goldreich and G. Rothblum [23], asserts the following: There exists a uniform algorithm T that gets as input a circuit \(C:\lbrace 0,1\rbrace ^{n}\rightarrow \lbrace 0,1\rbrace ^{*}\) that agrees with \(f^{\mathtt {ws}}_n\) on at least \(\delta (n)\) of the inputs, and labeled examples \((x,f^{\mathtt {ws}}(x))\) where \(x\in \lbrace 0,1\rbrace ^{n}\) is uniformly chosen, runs in time \(2^{n/\mathrm{poly}\log (n)}\) and with high probability outputs a circuit \(C^{\prime }:\lbrace 0,1\rbrace ^{n}\rightarrow \lbrace 0,1\rbrace ^{*}\) that computes \(f^{\mathtt {ws}}_n\) on all inputs (see Definition 4.2).

(Our construction of \(f^{\mathtt {ws}}\) will also satisfy an additional property, which will only be used in the proof of Theorem 1.2 (i.e., of the “almost-always” version of the result). We will describe this property in the proof outline for Theorem 1.2 below.)

The construction of \(f^{\mathtt {ws}}\). Let us now explain how we construct \(f^{\mathtt {ws}}\). Following Trevisan and Vadhan [56], our \(f^{\mathtt {ws}}\) is an artificial \(\mathcal {PSPACE}\)-complete problem that we carefully construct. Their goal was to construct a \(\mathcal {PSPACE}\)-complete problem that will be simultaneously downward self-reducible and randomly self-reducible. Our goal will be to obtain a construction with stronger completeness and random self-reducibility properties while compromising on a slower downward self-reducibility algorithm (as detailed above). In a gist, we do so by drastically improving the efficiency of parts of their construction; details follow.

The construction in [56] is based on the proof of \(\mathcal {IP}=\mathcal {PSPACE}\) [42, 52]. Recall that the latter proof starts with a given \(3\text{-}\mathtt {SAT}\) formula \(\varphi\), which represents a fully quantified instance for \(\mathtt {TQBF}\) (see Definition 4.6 for the standard definition). The proof then arithmetizes the \(\mathtt {TQBF}\) function on \(\varphi\) by a low-degree polynomial \(P^{(\varphi ,0)}=Q_1\circ Q_2\circ \cdots \circ Q_{\mathrm{poly}(n)} \circ P^{(\varphi)}\), where \(P^{(\varphi)}\) is a standard arithmetization of \(3\text{-}\mathtt {SAT}\), and the \(Q_i\)’s are suitable arithmetic operators (i.e., arithmetizations of the \(\forall\) and of the \(\exists\) operators, as well as an operator that lowers the degree of the intermediary polynomial). Finally, the proof defines a sequence of \(\mathrm{poly}(n)\) polynomials \(P^{(\varphi ,1)},\ldots ,P^{(\varphi ,\mathrm{poly}(n))}\), where for \(i=1,\ldots ,\mathrm{poly}(n)\), the polynomial \(P^{(\varphi ,i)}\) applies one less operator to \(P^{(\varphi)}\), compared to \(P^{(\varphi ,i-1)}\). The crucial observation of [56] is that computing each \(P^{(\varphi ,i)}\) efficiently reduces to computing \(P^{(\varphi ,i-1)}\), and thus this sequence of polynomials already has a property reminiscent to downward self-reducibility (whereas the polynomials are of low degree, and thus compute functions that are random self-reducible).

Loosely speaking, the function from [56] defines, for every integer \(n\in \mathbb {N}\), a corresponding interval \(I_n\) of \(\mathrm{poly}(n)\) input lengths; for simplicity of presentation, let us pretend that this interval is \(I_n=[n,\ldots ,N=\mathrm{poly}(n)]\). At input length \(N=\mathrm{poly}(n)\) the function gets as input a \(3\text{-}\mathtt {SAT}\) formula \(\varphi\) over n variables and outputs \(P^{(\varphi ,0)}\). Then, for \(i\in [\mathrm{poly}(n)]\), at input length \(N-i\), the function gets input \((\varphi ,w)\), where w is a sequence of auxiliary variables, and outputs \(P^{(\varphi ,i)}(w)\). Given the observation mentioned above, this function is downward self-reducible and randomly self-reducible.

Going through their proof (with needed adaptations for our “high-end” parameter setting), we encounter four different polynomial overheads in the input length, when reducing from \(\mathtt {TQBF}\) to their function. The first and obvious one is that inputs of length n are mapped to inputs of length \(N=\mathrm{poly}(n)\), corresponding to the number of rounds in the \(\mathcal {IP}=\mathcal {PSPACE}\) protocol. The other polynomial overheads in the input length come from their reduction of \(\mathtt {TQBF}\) to an intermediate problem that takes both \(\varphi\) and w as part of the input and is still amenable to arithmetization¹³ from the field size that is required for the stronger random self-reducibility property that we need and from the way the \(\mathrm{poly}(n)\) polynomials are combined into a single Boolean function.

The main challenge is to eliminate all of the foregoing overheads simultaneously. Our first main idea is to use an \(\mathcal {IP}=\mathcal {PSPACE}\) protocol with \(\mathrm{polylog}(n)\) rounds instead of \(\mathrm{poly}(n)\) rounds, so the first overhead (i.e., the additive overhead in the input length caused by the number of operators) will be only \(\mathrm{polylog}(n)\) instead of \(\mathrm{poly}(n)\). Indeed, in such a protocol the verification time in each round is high, and therefore our downward self-reducibility algorithm is relatively slow and makes many queries; but we will be able to afford this.

While implementing this idea, we define a different intermediate problem that is both amenable to arithmetization and reducible from \(\mathtt {TQBF}\) in quasilinear time, relying on an efficient Cook-Levin theorem (see Claim 4.7.1); we move to an arithmetic setting that will support the strong random self-reducibility property that we want and arithmetize the intermediate problem in this setting (see Claim 4.7.2); we show how to execute arithmetic operators in a “batch” in this arithmetic setting (see Claim 4.7.3); and we efficiently combine the resulting collection of polynomials into a single Boolean function (see the last part of the proof of Lemma 4.7).

We stress that we are “paying” for all the optimizations above by the fact that the associated algorithms (for downward self-reducibility and for our notion of random self-reducibility) now run in time \(2^{n/\mathrm{polylog}(n)}\), rather than polynomial time; but again, we are able to afford this in our proof.

2.1.2 Instantiating the Reference [33] Proof Framework with the Function f^ws.

Given this construction of \(f^{\mathtt {ws}}\), we now use a variant of the proof framework of Impagliazzo and Wigderson [33], as follows: For simplicity, in this overview, we show how to “fool” polynomial-time distinguishers that do not use advice. (The full technical proof appears in Section 4.2; see the proof of Lemma 4.9.)

Let \(\mathtt {ECC}\) be the Goldreich-Levin [21] (i.e., Hadamard) encoding \(\mathtt {ECC}(f^{\mathtt {ws}})(x,r)=\oplus _{i}f^{\mathtt {ws}}(x)_i\cdot r_i\). Our PRG is the Nisan-Wigderson PRG, instantiated with \(\mathtt {ECC}(f^{\mathtt {ws}})\) as the hard function, and with seed length \(\tilde{O}(\log (n))\). To analyze it, we rely on the well-known “uniform reconstruction” argument of [33] (following [44]), which shows the following: If for input length n there exists a uniform \(\mathrm{poly}(n)\)-time distinguisher A for the PRG, then for input length \(\ell =\widetilde{O}(\log (n))\) there is a weak learner for \(\mathtt {ECC}(f^{\mathtt {ws}})\). That is, there exists an algorithm that gets input \(1^{\ell }\) and oracle access to \(\mathtt {ECC}(f^{\mathtt {ws}})\) on \(\ell\)-bit inputs, runs in time \(\mathrm{poly}(n)\approx 2^{\ell /\mathrm{polylog}(\ell)}\), and outputs a small circuit that agrees with \(\mathtt {ECC}(f^{\mathtt {ws}})\) on approximately \(1/2+1/n^2\approx 1/2+\delta _0(\ell)\) of the \(\ell\)-bit inputs, where \(\delta _0(\ell)=2^{-\ell /\mathrm{polylog}(\ell)}\).

Thus, assuming that there exists a distinguisher for the PRG as above for every \(n\in \mathbb {N}\), we deduce that a weak learner exists for every \(\ell \in \mathbb {N}\). Following the “bootstrapping” idea of [33], we now iteratively construct, for each input length \(i=1,\ldots ,\ell\), a circuit of size \(2^{i/\mathrm{polylog}(i)}\) for \(f^{\mathtt {ws}}_i\). The base case \(i=1\) is trivial. And, in iteration \(i\gt 1\), having already obtained a circuit \(C_{i-1}\) for \(f^{\mathtt {ws}}_{i-1}\), we run the weak learner for \(\mathtt {ECC}(f^{\mathtt {ws}})\) on input length \(2i\) and answer its oracle queries using the downward self-reducibility of \(f^{\mathtt {ws}}\), the circuit \(C_{i-1}\), and the fact that \(\mathtt {ECC}(f^{\mathtt {ws}})_{2i}\) is easily computable given access to \(f^{\mathtt {ws}}_i\).

The weak learner outputs a circuit \(C^{(0)}_i\) of size \(2^{2i/\mathrm{polylog}(2i)}\) that agrees with \(\mathtt {ECC}(f^{\mathtt {ws}})\) on approximately \(1/2+\delta _0(2i)\) of the \(2i\)-bit inputs, and we want to transform it into a circuit that computes \(f^{\mathtt {ws}}\) on all i-bit inputs. To do so, we first use the list-decoding algorithm of Goldreich and Levin [21] to efficiently transform \(C^{(0)}_i\) to a circuit \(C^{(1)}_i\) of similar size that computes \(f^{\mathtt {ws}}\) on a approximately \(\delta (i)=\mathrm{poly}(\delta _0(2i))\) of the i-bit inputs; the algorithm of [21] succeeds only with probability \(\mathrm{poly}(\delta)\), so we run it for \(\mathrm{poly}(1/\delta)\) times and each time test the agreement of the resulting circuit with \(f^{\mathtt {ws}}\), using the circuit, \(C_{i-1}\), and the downward self-reducibility of \(f^{\mathtt {ws}}\).

Our goal now is to transform \(C^{(1)}_i\) into a circuit of similar size that computes \(f^{\mathtt {ws}}\) on all i-bit inputs. Recall that in general, performing such transformations by a uniform algorithm is challenging (intuitively, if the truth-table of \(f^{\mathtt {ws}}\) is a codeword in an error-correcting code, then this task corresponds to uniform list-decoding of a “very corrupt” version of \(f^{\mathtt {ws}}\)). However, in our specific setting, we can produce random labeled samples for \(f^{\mathtt {ws}}\), using its downward self-reducibility and the circuit \(C_{i-1}\). Relying on the sample-aided worst-case to average-case reducibility of \(f^{\mathtt {ws}}\), we can transform \(C^{(1)}_i\) to a circuit \(C_i\) of similar size that computes \(f^{\mathtt {ws}}_i\) on all inputs.

Finally, since \(\mathtt {TQBF}\) is reducible with quasilinear overhead to \(f^{\mathtt {ws}}\), if we can compute \(f^{\mathtt {ws}}\) in time \(2^{n/\mathrm{polylog}(n)}\), then we can compute \(\mathtt {TQBF}\) in such time, a contradiction. This establishes that the generator is indeed pseudorandom, and, since \(f^{\mathtt {ws}}\) is computable in space \(O(\ell)=\tilde{O}(\log (n))\) (and thus in time \(n^{\mathrm{polyloglog}(n)}\)), the pseudorandom generator is also computable in time \(n^{\mathrm{polyloglog}(n)}\).

2.1.3 The “Almost-always” Version: Proof of Theorem 1.2.

We now explain how to adapt the proof above to get an “almost-always” PRG with near-exponential stretch. For starters, we will use a stronger property of \(f^{\mathtt {ws}}\), namely, that it is downward self-reducible in a polylogarithmic number of steps; this means that for every input length \(\ell\) there exists an input length \(\ell _0\ge \ell -\mathrm{polylog}(\ell)\) such that \(f^{\mathtt {ws}}\) is efficiently computable at input length \(\ell _0\) (i.e., \(f^{\mathtt {ws}}_{\ell _0}\) is computable in time \(2^{\ell _0/\mathrm{polylog}(\ell _0)}\) without a “downward” oracle); see Section 4.1.1 for intuition and details about this property.

Now, observe that the transformation of a probabilistic distinguisher A for the PRG to a probabilistic algorithm F that computes \(f^{\mathtt {ws}}\) actually gives a “point-wise” guarantee: For every input length \(n\in \mathbb {N}\), if A distinguishes the PRG on a corresponding set of input lengths \(S_{n}\), then F computes \(f^{\mathtt {ws}}\) correctly at input length \(\ell =\ell (n)=\widetilde{O}(\log (n))\); specifically, we want to use the downward self-reducibility argument for \(f^{\mathtt {ws}}\) at input lengths \(\ell ,\ell -1,\ldots ,\ell _0\), and \(S_n\) is the set of input lengths at which we need a distinguisher for G to obtain a weak learner for \(\mathtt {ECC}(f^{\mathtt {ws}})\) at input lengths \(\ell ,\ell -1,\ldots \ell _0\). Moreover, since \(f^{\mathtt {ws}}\) is downward self-reducible in \(\mathrm{polylog}\) steps, we will only need weak learners at inputs \(\ell ,\ldots ,\ell _0=\ell -\mathrm{polylog}(\ell)\); hence, we can show that \(S_n\) is a set of \(\mathrm{polylog}(\ell)=\mathrm{polyloglog}(n)\) input lengths in the interval \([n,n^2]\) (see Lemma 4.9 for the precise calculation). Taking the contrapositive, if \(f^{\mathtt {ws}}\) cannot be computed by F on almost all \(\ell\)’s, then for every \(n\in \mathbb {N}\) there exists an input length \(m\in S_n\subset [n,n^2]\) such that G fools A at input length m.¹⁴

Our derandomization algorithm gets input \(1^n\) and also gets the “good” input length \(m\in S_{n}\) as non-uniform advice; it then simulates \(G(1^m)\) (i.e., the PRG at input length m) and truncates the output to n bits. (We can indeed show that truncating the output of our PRG preserves its pseudorandomness in a uniform setting; see Proposition 4.12 for details.) The crucial point is that, since \(|S_{n}|=\mathrm{polyloglog}(n)\), the advice length is \(O(\mathrm{logloglog}(n))\). Note, however, that for every potential distinguisher A there exists a different input length \(m\in S_{n}\) such that G is pseudorandom for A on m. Hence, our derandomization algorithm (or, more accurately, its advice) depends on the distinguisher that it wants to “fool.” Thus, for every \(L\in \mathcal {BPP}\) and every efficiently samplable distribution \(\mathcal {X}\) of inputs, there exists a corresponding “almost-always” derandomization algorithm \(D_{\mathcal {X}}\) (see Proposition 4.12).

4.1.1 Well-structured Function: Definition.

Loosely speaking, we will say that a function \(f:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\) is well-structured if it satisfies three properties. The first property, which is not crucial for our proofs but simplifies them a bit, is that f is length-preserving; that is, for every \(x\in \lbrace 0,1\rbrace ^*\) it holds that \(|f(x)|=|x|\).

The second property is a strengthening of the notion of downwards self-reducibility. Recall that a function \(f:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\) is downwards self-reducible if \(f_n\) can be computed by an efficient algorithm that has oracle access to \(f_{n-1}\). First, we quantify the notion of “efficient,” to also allow for a very large running time (e.g., running time \(2^{n/\mathrm{polylog}(n)}\)). Second, we also require that for any \(n\in \mathbb {N}\) there exists an input length m that is not much smaller than n such that \(f_m\) is efficiently computable without any “downward” oracle. That is, intuitively, if we try to compute f on input length n by “iterating downwards” using downward self-reducibility, our “base case” in which the function is efficiently computable is not input length \(O(1)\), but a large input length m that is not much smaller than n. More formally:

The third property that we need is a refinement of the notion of random self-reducibility, which is called sample-aided worst-case to average-case reducibility. This notion was recently made explicit by Goldreich and G. Rothblum [23] and is implicit in many previous results (see, e.g., the references in [23]).

To explain the notion, recall that if a function f is randomly self-reducible, then a circuit \(\widetilde{C}\) that computes f on most of the inputs can be efficiently transformed to a (probabilistic) circuit C that computes f on every input (whp). We want to relax this notion by allowing the efficient algorithm that transforms \(\widetilde{C}\) into C to obtain random labeled samples for f (i.e., inputs of the form \((r,f(r))\) where r is chosen uniformly at random). The main advantage in this relaxation is that we will not need to assume that \(\widetilde{C}\) computes f on most of the inputs but will be satisfied with the weaker assumption that \(\widetilde{C}\) computes f on a tiny fraction of the inputs. Specifically²⁴:

For high-level intuition of why labeled samples can be helpful for worst-case to average-case reductions, and for a proof that if f is a low-degree multivariate polynomial then it is sample-aided worst-case to average-case reducible, see Appendix B.

We are now ready to define well-structured functions. Fixing a parameter \(\delta \gt 0\), a function \(f^{\mathtt {ws}}\) is \(\delta\)-well-structured if it is length-preserving, downward self-reducible in time \(\mathrm{poly}(1/\delta)\), and sample-aided worst-case to \(\delta\)-average case reducible. That is:

In the following definition, we consider reductions from a decision problem \(L\subseteq \lbrace 0,1\rbrace ^*\) to a well-structured function \(f^{\mathtt {ws}}:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\). To formalize this, we consider both a reduction R, which transforms any input x for L to an input \(R(x)\) for \(f^{\mathtt {ws}}\), and a “decision algorithm” D, which translates the non-Boolean result \(f^{\mathtt {ws}}(R(x))\) into a decision of whether or not \(x\in L\).

4.1.2 Overview of Our Construction.

For \(\delta =2^{-n/\mathrm{polylog}(n)}\) and \(s=\mathrm{polylog}(n)\), our goal is to construct a \((\delta ,s)\)-well-structured function \(f^{\mathtt {ws}}:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\) such that \(\mathtt {TQBF}\) reduces to \(f^{\mathtt {ws}}\) in quasilinear time (and thus with quasilinear blow-up). Throughout the section, assume that an n-bit input to \(\mathtt {TQBF}\) is simply a \(3\text{-}\mathtt {SAT}\) formula \(\varphi\) on n variables, and it is assumed that all variables are quantified in order with alternating quantifiers (e.g., \(\forall w_1 \exists w_2 \forall w_3 ... \varphi (w_1,\ldots ,w_n)\); see Definition 4.6).

Our starting point is the well-known construction of Trevisan and Vadhan [56], which (loosely speaking) transforms the protocol underlying the \(\mathcal {IP}=\mathcal {PSPACE}\) proof into a computational problem \(L_{TV}:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\).²⁵ They required that \(L_{TV}\) will meet the weaker requirements (compared to our requirements) of being downward self-reducible and randomly self-reducible, where the latter means reducible from being worst-case computable to being computable on, say, .99 of the inputs.

Before describing our new construction, let us first review the original construction of \(L_{TV}\). For every \(n\in \mathbb {N}\), fix a corresponding interval \(I_n=[N_0,N_1]\) of \(r(n)=\mathrm{poly}(n)\) input lengths. The input to \(L_{TV}\) at any input length in \(I_n\) (disregarding necessary padding) is a pair \((\varphi ,w)\in \mathbb {F}^{2n}\), where \(\mathbb {F}\) is a sufficiently large field. (The field size is chosen such that both P and related polynomials that are described below will be of low degree.) If \((\varphi ,w)\in \lbrace 0,1\rbrace ^{2n}\), then we think of \(\varphi\) as representing a \(3\text{-}\mathtt {SAT}\) formula and of w as representing an assignment. At input length \(N_0\), we define \(L_{TV}(\varphi ,w)=P(\varphi ,w)\), where \(P(\varphi ,x)\) is a low-degree arithmetized version of the Boolean function \((\varphi ,w)\mapsto \varphi (w)\).

Now, recall that the \(\mathcal {IP}=\mathcal {PSPACE}\) protocol defines three arithmetic operators on polynomials (two quantification operators and a linearization operator). Then, at input length \(N_0+i\), the problem \(L_{TV}\) is recursively defined by applying one of the three arithmetic operators on the polynomial from the previous input length \(N_0+i-1\).²⁶ Observe that computing \(L_{TV}\) at input length \(N_0+i\) corresponds to the residual computational problem that the verifier faces at the \((r-i)^{th}\) round of the \(\mathcal {IP}=\mathcal {PSPACE}\) protocol when instantiated for formula \(\varphi\) and with \(r=r(n)\) rounds. Indeed, at the largest input length \(N_1=N_0+r(n)\) the polynomial \(L_{TV}\) is simply a low-degree arithmetized version of the function that decides whether or not \(\varphi \in \mathtt {TQBF}\) (regardless of w); thus, \(\mathtt {TQBF}\) can be reduced to \(L_{TV}\) by mapping \(\varphi \in \lbrace 0,1\rbrace ^n\) to \((\varphi ,1^n)\in \mathbb {F}^{2n}\) and adding padding to get the input to be of length \(N_1=\mathrm{poly}(n)\). Note that \(L_{TV}\) is indeed both downward self-reducible (since for each operator O and polynomial P, we can compute \(O(P)(\varphi ,w)\) in polynomial-time with two oracle queries to P), and randomly self-reducible (since the polynomials have low degree.)

Let us now define our \(f^{\mathtt {ws}}:\lbrace 0,1\rbrace ^*\rightarrow \lbrace 0,1\rbrace ^*\), which replaces their \(L_{TV}\), and highlight what is different in our setting. Recall that our main goal is to construct the well-structured function \(f^{\mathtt {ws}}\) such that \(\mathtt {TQBF}\) is reducible to \(f^{\mathtt {ws}}\) with only quasilinear overhead in the input length (i.e., we need to avoid polynomial overheads), while keeping the running time of all operations (i.e., of the algorithms for downward self-reducibility and for sample-aided worst-case to rare-case reducibility) to be at most \(2^{n/\mathrm{polylog}(n)}\).

The first issue, which is relatively easy to handle, is the number of bits that we use to represent an (arithmetized) input \((\varphi ,w)\) for \(f^{\mathtt {ws}}\). Recall that we want \(f^{\mathtt {ws}}\) to be worst-case to \(\delta\)-average-case reducible for a tiny \(\delta =2^{-n/\mathrm{polylog}(n)}\); thus, \(f^{\mathtt {ws}}\) will involve computing polynomials over a field of large size \(|\mathbb {F}|\ge \mathrm{poly}(1/\delta)\). Using the approach of [56], we would need \(2n\cdot \log (|\mathbb {F}|)=\tilde{\Omega }(n^2)\) bits to represent \((\varphi ,w)\), and thus the reduction from \(\mathtt {TQBF}\) to \(f^{\mathtt {ws}}\) would incur a polynomial overhead. This is easily solvable by considering a “low-degree extension” instead of their “multilinear extension”: To represent an input \((\varphi ,w)\in \lbrace 0,1\rbrace ^{2n}\) to \(f^{\mathtt {ws}}\), we will use few elements in a very large field. Specifically, we will use \(\ell =\mathrm{polylog}(n)\) variables (i.e., the polynomial will be \(\mathbb {F}^{2\ell }\rightarrow \mathbb {F}\)) such that each variable “provides” \(O(n/\mathrm{polylog}(n))\) bits of information.

A second problem is constructing a low-degree arithmetization \(P(\varphi ,w)\) of the Boolean function that evaluates \(\varphi\) at w. In [56], they solve this by first reducing \(\mathtt {TQBF}\) to an intermediate problem \(\mathtt {TQBF}^{\prime }\) that is amenable to such low-degree arithmetization; however, their reduction incurs a quadratic blow-up in the input length, which we cannot afford in our setting. To overcome this, we reduce \(\mathtt {TQBF}\) to another intermediate problem, denoted \(\mathtt {TQBF^{loc}}\), which is amenable to low-degree arithmetization, such that the reduction incurs only a quasilinear blow-up in the input length. (Loosely speaking, we define \(\mathtt {TQBF^{loc}}\) by applying a very efficient Cook-Levin reduction to the Turing machine that gets input \((\varphi ,w)\) and outputs \(\varphi (w)\); see Claim 4.7.1 for precise details.) We then carefully arithmetize \(\mathtt {TQBF^{loc}}\), while “paying” for this efficient arithmetization by the fact that computing the corresponding polynomial now takes time \(\exp (n/\ell)=\mathrm{poly}(1/\delta)\), instead of \(\mathrm{poly}(n)\) time as in [56] (see Claim 4.7.2).

Third, the number of polynomials in the construction of \(L_{TV}\) (i.e., the size of the interval \(I_n\)) is \(r(n)=\mathrm{poly}(n)\), corresponding to the number of rounds in the \(\mathcal {IP}=\mathcal {PSPACE}\) protocol. This poses a problem for us, since the reduction from \(\mathtt {TQBF}\) maps an input of length n is to an input of length \(N_1\ge \mathrm{poly}(n)\). We solve this problem by “shrinking” the number of polynomials to be polylogarithmic, using an approach similar to an \(\mathcal {IP}=\mathcal {PSPACE}\) protocol with only \(\mathrm{polylog}(n)\) rounds and a verifier that runs in time \(2^{n/\mathrm{polylog}(n)}\): Intuitively, at each input length, we define \(f^{\mathtt {ws}}\) by simultaneously applying \(O(\log (1/\delta))\) operators (rather than a single operator) to the polynomial that corresponds to the previous input length. Indeed, as one might expect, this increases the running time of the downward self-reducibility algorithm to \(\mathrm{poly}(1/\delta)\), but we can afford this. Implementing this approach requires some care, since multiple operators will be applied to a single variable (which represents many bits of information), and since the linearization operator needs to be replaced by a “degree-lowering operation” (that will reduce the individual degree of a variable to be \(\mathrm{poly}(1/\delta)\)); see Claim 4.7.3 for details.

Last, we also want our function to be downward self-reducible in \(\mathrm{polylog}(n)\) steps (i.e., after \(\mathrm{polylog}(n)\) “downward” steps, the function at the now-smaller input length is computable in time \(\mathrm{poly}(1/\delta)\) without an oracle). This follows by noting that the length of each interval \(I_n\) is now polylogarithmic, and that at the “bottom” input length the function \(f^{\mathtt {ws}}\) simply computes the arithmetized version of \(\mathtt {TQBF^{loc}}\), which (as mentioned above) is computable in time \(\mathrm{poly}(1/\delta)\).

The complexity of \(f^{\mathtt {ws}}\). For our derandomization result, it suffices to prove that \(f^{\mathtt {ws}}\) is computable in time \(2^{\tilde{O}(n)}\), rather than in linear space. (This is because our derandomization algorithm enumerates over all choices for a seed of length \(\tilde{O}(n)\) and computes the Nisan-Wigderson generator on each choice, with \(f^{\mathtt {ws}}\) as the hard function.) However, analogously to Trevisan and Vadhan [56], we prove the stronger statement that \(f^{\mathtt {ws}}\) is computable in linear space.²⁷ This stronger property may be of independent interest and in particular may be used in future work for constructions of PRGs that work in small space. (See Remark 4.13 for further details.)

4.1.3 The Construction Itself.

We consider the standard “totally quantified” variant of the Quantified Boolean Formula (\(\mathbf {QBF}\)) problem, called Totally Quantified Boolean Formula (\(\mathbf {TQBF}\)). In this version, the quantifiers do not appear as part of the input, and we assume that all the variables are quantified and that the quantifiers alternate according to the index of the variable (i.e., \(x_i\) is quantified by \(\exists\) if i is odd and otherwise quantified by \(\forall\)).

Recall that a formula \(\varphi\) that is represented by n bits actually has less than n input variables, since the representation length of an m-bit formula is \(O(m\cdot \log (m))\). Thus, an n-bit \(\varphi\) actually has at most \(n/O(\log (n))\) variable. In Definition 4.6, we assume for simplicity (and to avoid cumbersome notation) that \(\varphi\) has precisely n input variables, but some of these are dummy variables that are ignored.²⁸

Recall that \(\mathtt {QBF}\), in which the quantifiers are part of the input, is reducible in linear time to \(\mathtt {TQBF}\) from Definition 4.6 (by renaming variables and adding dummy variables).

The main result in this section is a construction of a well-structured function \(f^{\mathtt {ws}}\) such that \(\mathtt {TQBF}\) can be reduced to \(f^{\mathtt {ws}}\) with only quasilinear blow-up. This construction is detailed in the following lemma:

5.1.1 Solving (1,1/3)CAPP using Many Untrusted CAPP Algorithms.

Recall that in the problem \((\alpha ,\beta)\text{-}\mathsf {CAPP}\), we get as input a description of a circuit, and our goal is to distinguish between circuits with acceptance probability at least \(\alpha \gt 0\) and circuits with acceptance probability at most \(\beta \gt 0\); we also denote \(\mathsf {CAPP}=(2/3,1/3)\text{-}\mathsf {CAPP}\) (see Definition 3.1). Assume that we want to solve \(\mathsf {CAPP}\) on an input circuit C of description length n and that we are guaranteed that an algorithm A solves \(\mathsf {CAPP}\) on some input length (unknown to us) in the interval \([n,S(n)]\) for some function S. This problem arises, for example, if we assume that \(pr\mathcal {BPP}\subset \mathtt {i.o.}pr\mathcal {NP}\) (which implies that \(\mathsf {CAPP}\in \mathtt {i.o.}pr\mathcal {NP}\)) and want to derandomize \(\mathcal {MA}\) infinitely-often. (This is because when the \(\mathcal {MA}\) verifier gets an input of length m, the derandomization of the verifier corresponds to a \(\mathsf {CAPP}\) problem on some input length \(n=m^k\), but we are not guaranteed that the \(\mathsf {CAPP}\) algorithm works on input length n.³⁸) How can we solve this problem?

If we invoke the algorithm A on each input length in the interval \([n,S(n)]\), while feeding it C as input each time (i.e., C is padded up to the appropriate length), then we obtain a variety of answers, and it is not clear a priori how we can distinguish the correct answer from possibly misleading ones. In this section, we show a solution for this problem in the setting where we only need to solve \(\mathsf {CAPP}\) with one-sided error and when A solves a problem in \(pr\mathcal {BPP}\) that slightly generalizes \(\mathsf {CAPP}\). Intuitively, since we only need to solve \((1,1/3)\text{-}\mathsf {CAPP}\), it will be possible to prove to us that C is not a YES instance (i.e., that C does not accept all of its inputs); and, since A solves a problem that slightly generalizes \(\mathsf {CAPP}\), we will be able to modify it to an algorithm that is able to provide such a proof when C is not a YES instance. Details follow.

We first define the aforementioned variation of \((\alpha ,\beta)\text{-}\mathsf {CAPP}\), denoted \(\mathsf {pCAPP}\) (for “parametrized \(\mathsf {CAPP}\)”), in which \(\alpha\) and \(\beta\) are specified as part of the input.

Note that if \(\ell (v)=O(\log (S(v)))\), then \(\mathsf {pCAPP}[S,\ell ]\in pr\mathcal {BPP}\). (This is since we can uniformly sample \(\epsilon ^{-2}\) inputs for C, where \(\epsilon =\beta -\alpha \ge 1/\mathrm{poly}(S(v))\), and estimate \(\Pr _x[C(x)=1]\) with accuracy \((\alpha -\beta)/2\), with high probability.) We now show that solving \((1,1/3)\text{-}\mathsf {CAPP}\) for circuits of size \(S(n)\) infinitely-often reduces to solving \(\mathsf {pCAPP}\) infinitely-often (i.e., on an arbitrary infinite set of input lengths).

5.1.2 A Strengthened Karp-Lipton Style Result for the “Low-end” Setting.

To prove our first strengthening of [3], let \(L\in \mathcal {EXP}\), and note that by our assumption \(L\in \mathcal {P}/\mathrm{poly}\). Consider an \(\mathcal {MA}\) verifier V that gets input \(1^n\), guesses a circuit \(C_L:\lbrace 0,1\rbrace ^{n}\rightarrow \lbrace 0,1\rbrace ^{}\), and tries to decide if \(C_L\) correctly computes \(L_n=L\cap \lbrace 0,1\rbrace ^{n}\). The key observation is that, since this decision problem (of deciding whether or not a given n-bit circuit computes \(L_n\)) is in \(\mathcal {EXP}\), we can apply the original Karp-Lipton style result of [3] to it. The latter result implies that there exists an \(\mathcal {MA}\) verifier M that decides whether or not \(C_L\) computes \(L_n\) correctly. Our verifier V guesses \(C_L\) and a witness for M, simulates M, and if M confirms that \(C_L\) computes \(L_n\), then V outputs \(C_L\).

We will derandomize the foregoing \(\mathcal {MA}\) verifier in one of two ways: The first relies on a hypothesis of the form \(pr\mathcal {BPP}\subseteq pr\mathcal {NSUBEXP}\), which immediately implies that \(\mathcal {MA}\subseteq \mathcal {NSUBEXP}\). The second relies on a hypothesis of the form \(pr\mathcal {BPP}\subset \mathtt {i.o.}pr\mathcal {SUBEXP}\); in this case, we derandomize the \(\mathcal {MA}\) verifier infinitely-often, relying on the fact that the \(\mathcal {MA}\) verifier can be assumed to have perfect completeness [18] and on Lemma 5.5 (which was presented in Section 5.1.1). Note that in both cases, the running time of the resulting non-deterministic machine is sub-exponential, but the size of the output circuit \(C_L\) is nevertheless still polynomial.

The following statement and proof generalize the above, using parametrized “collapse” and derandomization hypotheses. Specifically, if we assume that \(\mathcal {E}\subset \mathcal {SIZE}[S]\) and that \(pr\mathcal {BPP}\) can be derandomized in time T, then we deduce that \(\mathcal {E}\) has \(\mathcal {NTIME}[T^{\prime }]\) uniform circuits of size \(S(n)\), where \(T^{\prime }(n)\approx T(S(S(n)))\).

Note that, in the proof of Proposition 5.6, we did not use the fact that \(L^{\mathtt {nice}}\) is randomly self-reducible, but only the facts that \(L^{\mathtt {nice}}\) is complete for \(\mathcal {E}\) under linear-time reductions (such that all n-bit inputs are mapped to \(n^{\prime }\)-bit inputs, for \(n^{\prime }=O(n)\)) and that it has an instance checker with query length \(\ell (n)=O(n)\).

5.1.3 A Strengthened Karp-Lipton Style Result for the “High-end” Setting.

The result presented next asserts that if \(\mathcal {E}\in \mathcal {SIZE}[S]\) and \(pr\mathcal {BPP}\) can be derandomized in time T, then \(\mathcal {E}\) has \(\mathcal {NTIME}[T^{\prime }]\) uniform circuits (with a trivial size bound of \(T^{\prime }(n)\)), where \(T^{\prime }\approx T(S(n))\). The main difference between this result and the result presented in Section 5.1.3, other than the differences in parameters, is that for this result, we will need to assume that \(pr\mathcal {BPP}\) can be derandomized deterministically, rather than only non-deterministically.

Let us briefly describe the proof idea. We construct a circuit for an \(\mathcal {E}\)-complete problem \(L^{\mathtt {nice}}\) that has an instance checker and that is randomly self-reducible (see Section 3.5 for definitions and details). We guess a circuit \(C^{L^{\mathtt {nice}}}\) for \(L^{\mathtt {nice}}\), which exists by our “collapse” hypothesis, and randomly check whether or not this circuit “convinces” the instance checker on almost all inputs; if it does, then we instantiate the instance checker with \(C^{L^{\mathtt {nice}}}\) as an oracle, to obtain a “corrupt” version of \(L^{\mathtt {nice}}\), denoted \(\tilde{L}\). We then construct a probabilistic circuit \(C^{\prime }\) that decides \(L^{\mathtt {nice}}\), with high probability, using the random self-reducibility of \(L^{\mathtt {nice}}\) and oracle access to \(\tilde{L}\).

Now, under the hypothesis \(pr\mathcal {BPP}\subseteq pr\mathcal {DTIME}[T]\), we can derandomize the two probabilistic steps in the foregoing construction. Specifically, we derandomize the probabilistic verification that the circuit \(C^{L^{\mathtt {nice}}}\) “convinces” the instance checker on almost all inputs, and we also derandomize the probabilistic circuit itself (i.e., we actually output a deterministic circuit that constructs the probabilistic circuit \(C^{\prime }\) and applies a deterministic \(\mathsf {CAPP}\) algorithm to \(C^{\prime }\)). Details follow.

Note that the actual hypothesis of Proposition 5.7 is weaker than the hypothesis \(pr\mathcal {BPP}\in pr\mathcal {DTIME}[T]\), since we only require an algorithm for \(\mathsf {CAPP}\) for large circuits (i.e., for v-bit circuits of size \(\mathrm{poly}(v)\cdot S(v)\)).