Contextual Ranking and Selection with Gaussian Processes and Optimal Computing Budget Allocation

5.1 GP-C-OCBA

In classical R&S literature, optimal computing budget allocation (OCBA) [4] is a popular approach for maximizing the PCS asymptotically with independent Gaussian noise. Reference [9] later extended the OCBA approach [4, 5] to the contextual R&S problem, where the author derives the Karush-Kuhn-Tucker (KKT) conditions for maximizing the rate function (Equation (1)), and proposes an idealized sampling policy that iteratively realizes the KKT conditions. The idealized policy derived by Reference [9] (see Section III C of the article for derivation) relies on \(F(k, c)\), \(\sigma ^2(k, c)\), and \(\pi ^*(c)\), which are not known in practice, as well as \(\hat{p}(k, c)\), which denotes the fraction of total samples allocated to \((k, c)\) so far and is different than \(p(k, c)\) used in the derivation to denote the idealized asymptotic allocation rate. For a practical algorithm, Reference [9] replaces \(F(k, c)\) and \(\sigma ^2(k, c)\) with the sample mean and variance, respectively, and \(\pi ^*(c)\) with the corresponding estimate to define the C-OCBA policy.

As shown in Theorem 4.1, using the GP model yields the same rate function as using the sample mean predictors in Reference [9], so we take a similar approach and use the posterior mean \(\mu _n(k, c)\) and the posterior variance \(\Sigma _n(k, c)\), which are our estimates based on n observations collected so far, in place of \(F(k, c)\) and \(\sigma ^2(k, c) / \hat{p}(k, c)\) in the idealized policy. The resulting GP-C-OCBA policy is presented in Algorithm 1.

Our experiments (in Section 7) show that using the GP model with the GP-C-OCBA sampling strategy leads to significantly higher contextual PCS using the same sampling budget, thanks to the improvements in the posterior inference from using a statistical model that leverages the hidden correlation structure in the reward function, i.e., correlation between reward functions under different contexts. An additional benefit of our approach over Reference [9] is in its applicability when the initial sampling budget is too small to draw multiple samples from each alternative. Using normal random variables to model each alternative-context pair requires a small number of samples from each pair for the initial estimate of the variance, which may limit the applicability of the algorithm when the sampling budget is limited. The GP prior, however, can be trained using very few samples for each alternative, rather than each alternative-context pair, thus the modified algorithm can be used even with a limited sampling budget.

A final remark about the GP-C-OCBA is that in the extreme case where there is only a single context, the sampling policy is equivalent to the OCBA-2 algorithm presented in Reference [22], which has been shown to achieve the optimal convergence rate in the standard R&S setting. Thus, GP-C-OCBA can be viewed as a principled extension of a rate optimal R&S algorithm to the contextual R&S setting.

5.2 Integrated Knowledge Gradient

On a related note, another applicable method for the contextual R&S problem is the IKG algorithm, which has been developed for the closely related problem of contextual Bayesian optimization. In this section, we adapt the IKG algorithm to our setting, and compare it with GP-C-OCBA. IKG offers a strong benchmark for our method, since it is based on the same GP model and has demonstrated superior sampling efficiency in prior work [26, 27].

Knowledge Gradient (KG) [11] is a value-of-information type policy that was originally proposed for the R&S problem and later expanded to global optimization of black-box functions. It is well known for its superior sampling efficiency, which comes at a significant computational cost. For a given context \(c^{\prime }\), we can write the KG factor, which measures the expected improvement in value of the maximizer for context \(c^{\prime }\) from adding an additional sample at \((k, c)\), as \(\begin{equation*} \text{KG}(k, c; c^{\prime }) = \mathbb {E}_n[\max _{k^{\prime } \in \mathcal {K}} \mu _{n+1}(k^{\prime }, c^{\prime }) \mid (k^{n+1}, c^{n+1}) = (k, c)] - \max _{k^{\prime } \in \mathcal {K}} \mu _n(k^{\prime }, c^{\prime }). \end{equation*}\) In the classical R&S setting, where c and \(c^{\prime }\) are redundant (i.e., there is only a single context), the KG policy operates by evaluating the alternative \(k^* = \arg \max _k \text{KG}(k, c; c)\). To extend this to the contextual Bayesian optimization problem, References [8, 26, 27] each study an integrated (or summed) version of KG, under slightly different problem settings, where either the context space or both alternative-context spaces are continuous. The main differences between these three works are in how they approximate and optimize the integrated KG factor in their respective problem settings. For our problem setting, these approaches are equivalent, and we refer to the sampling policy as IKG. We use IKG as a benchmark to evaluate the sampling efficiency of our proposed algorithm.

The IKG factor is simply a weighted sum of KG factors corresponding to each context. It measures the weighted sum of the improvement in value of maximizers, and is written as \(\begin{equation*} \text{IKG}(k, c) = \sum _{c^{\prime } \in \mathcal {C}} \text{KG}(k, c; c^{\prime }) w(c^{\prime }). \end{equation*}\) At each iteration, the IKG policy samples the alternative-context pair that maximizes the IKG factor, \((\tilde{k}^*, \tilde{c}^*) = \arg \max _{k \in \mathcal {K}, c \in \mathcal {C}} \text{IKG}(k, c)\). The IKG policy is summarized in Algorithm 2.

The main difficulty with using the IKG policy is its computational cost. In the finite alternative-context setting that we are working with, the \(\text{KG}(k; c)\) can be computed exactly using Algorithm 1 from [11], which has a cost of \(\mathcal {O}(K \log K)\) for any pair \((k, c)\), given \(\mu _n(\cdot , \cdot)\) and \(\Sigma _n(\cdot , \cdot)\). This translates to an \(\mathcal {O}(|\mathcal {C}| K \log K)\) cost for calculating the IKG factor for a given \((k, c)\). In total, to find the next pair to sample using IKG costs \(\mathcal {O}(|\mathcal {C}|^2 K^2 \log K)\) for calculating the IKG factors, and an additional \(\mathcal {O}(n_k^3 + |\mathcal {C}|^2 n_k + |\mathcal {C}|n_k^2)\) (when the unchanged posterior terms are re-used between iterations) to calculate posterior mean and covariance matrices, where \(n_k\) denotes the total number of samples allocated to alternative k.

However, the cost of GP-C-OCBA is dominated by the cost of calculating the posterior mean and variance for each alternative-context pair, which has a total cost of \(\mathcal {O}(n_k^3 + |\mathcal {C}| n_k^2)\) (when the unchanged posterior terms are re-used between iterations). Note that we avoid the \(|\mathcal {C}|^2 n_k\) term, since our algorithm only requires the posterior variance, as well as the \(\mathcal {O}(|\mathcal {C}|^2 K^2 \log K)\) cost of IKG calculations. This puts GP-C-OCBA at a significant advantage in terms of computational complexity.

6.1 Consistency of GP-C-OCBA

We prove the result under a set of mild assumptions.

The first three points are common assumptions from the literature that simplify the analysis. The known observation noise level is needed for exact GP inference, and the assumption of a fixed prior eliminates the need to study the hyper-parameters in the analysis. In practice, the form of the GP prior is kept fixed, however, the hyper-parameters of the GP are periodically updated to better fit the data, which is typically done via maximum likelihood estimation. Though showing this rigorously for an adaptive sampling policy is a daunting task, the hyper-parameters of the GP prior typically converge to some final value and remain fixed after that. The last point is merely technical, since in practice it would not make sense to consider two contexts that are assumed to be perfectly correlated. The following lemma shows that this implies that \(\Sigma _n(\mathcal {C}, \mathcal {C}; k)\) must remain strictly positive definite. Note that since the posterior variance is a deterministic function of the sampling decisions (i.e., it is independent of the observations) the following statement holds for any sequence of sampling decisions.

The proofs of this and the following statements are included in the online supplement. The posterior variance of a GP model at a given point decreases monotonically as we add more samples to the model. Thus, for a given alternative-context, the posterior variance in the limit is 0 if that pair gets sampled infinitely often, and is a strictly positive value otherwise. The lemma below shows this rigorously.

The intuitive interpretation of Lemma 6.3 is that, despite modeling correlations between contexts, we cannot drive the uncertainty about the reward function of an alternative-context to zero, unless we collect infinitely many observations for that pair. This rules out the pathological cases where the statistical model treats a reward estimate as certain purely based on observations of other alternative-context pairs.

We will build on these results to show that GP-C-OCBA policy samples each alternative-context pair infinitely often. We will first establish that if a given context gets sampled infinitely often, all alternative-context pairs corresponding to that context must get sampled infinitely often.

If we only had a single context, then Lemma 6.4 would be sufficient to prove the consistency of the algorithm. However, with multiple contexts, we need to ensure that a strict subset of contexts cannot consume all observations after a certain iteration. The following theorem extends Lemma 6.4 to show that, as the budget goes to infinity, all alternative-context pairs are sampled infinitely often by GP-C-OCBA.

Allocating infinitely many samples to each alternative-context pair would have been sufficient to prove consistency if we did not model any correlation between contexts. However, we are not aware of any general consistency results regarding the GP posterior mean, so we need to ensure that the predictions converge to the true reward function. The following lemma establishes this result.

Coupled with Theorem 6.5, Lemma 6.6 shows that using GP-C-OCBA to allocate the observation budget, the posterior mean converges almost surely to the true reward for all alternative-context pairs. Thus, it follows that the predicted best alternative must almost surely converge to the correct best alternative for all contexts. The following corollary summarizes the result.

6.2 Convergence to the Optimal Allocation Ratio and the Convergence Rate

The results of the previous subsection show that, given enough samples, GP-C-OCBA is guaranteed to identify the optimal policy, \(\pi ^*(c)\). However, these are asymptotic results, and they do not provide any insight into how fast this convergence occurs. Under certain conditions on the prior covariance matrix, we can show that the allocation ratio produced by GP-C-OCBA converges to the optimal static allocation ratio, which is obtained by maximizing the rate function given in Theorem 4.1.

Let \(\begin{equation*} \eta (k, c) = \frac{(F(\pi ^*(c), c) - F(k, c))^2}{\sigma ^2(\pi ^*(c), c) / p^*(\pi ^*(c), c) + \sigma ^2(k, c) / p^*(k, c)} \end{equation*}\) and \(\begin{equation*} \hat{\eta }_n(k, c) = \frac{(F(\pi ^*(c), c) - F(k, c))^2}{\sigma ^2(\pi ^*(c), c) / \hat{p}_n(\pi ^*(c), c) + \sigma ^2(k, c) / \hat{p}_n(k, c)}. \end{equation*}\) The work of Reference [9] shows that the optimal static allocation ratio, i.e., the allocation ratio maximizing the large deviations rate function, satisfies the following conditions.

The following result states that the allocation ratio produced by GP-C-OCBA converges to \(p^*(k, c)\), i.e., that GP-C-OCBA asymptotically obtains the optimal static allocation ratio.

The proof is borrowed from the proof of Theorem 4 in Reference [9], where the authors proposed a similar algorithm as GP-C-OCBA. The difference between the two algorithms is that in Reference [9] they used sample mean and sample variance to estimate the true expected performance and observation noise variance, whereas we replace the sample mean with the posterior mean \(\mu _n(k,c)\) and replace the sample variance with \(N_n(k,c) \Sigma _n(k,c)\) for alternative k and context c. The validity of the proof relies on the convergence of the two posterior estimators. While the Gaussian posterior mean \(\mu _n(k,c)\) is guaranteed to converge to the true value \(F(k,c)\) almost surely as long as \(N_n(k,c)\) goes to infinity (see Reference [11]), the convergence of \(\Sigma _n(C,C;k)\) is more subtle, since the different components converge at different rates. This may be of independent interest for sequential sampling with GP model. We summarize the convergence result of posterior covariance matrix in Lemma 6.10.

Lemma 6.10 is a stronger result of Lemma 6.3, since as \(N_n(k,c)\) tends to infinity, Lemma 6.10 tells us that \(\Sigma _n(k,c)\) converges to 0 at the rate \(\frac{1}{N_n(k,c)}\) as well as the limit it converges to after dividing the rate. With the convergence of posterior estimators, we have the following Lemma 6.11, which guarantees that all the budget ratio assigned to any context-alternative \(\hat{p}_n(k,c)\) is positive in the long run.

Although Theorem 6.9 shows that the sample allocation ratio of GP-C-OCBA converges to the optimal allocation ratio, this, by itself, does not say much about the actual convergence rate of the contextual PCS. In the following, we extend the convergence rate result presented by Reference [22] for the OCBA algorithm to the contextual setting. We show that the contextual PFS converges to 0 at the optimal exponential rate. To the best of our knowledge, this is the first convergence rate result in the literature for a contextual R&S algorithm.

Proof. Recall from the proof of Theorem 4.1 that \(PFS^n_{\sim }\) can be lower and upper bounded by a constant (1 and \(|\mathcal {C}|(|\mathcal {K}| - 1)\), respectively) multiple of \(\begin{equation*} \max _{c \in \mathcal {C}}\max _{k \ne \pi ^*(c)} P(\mu _n(\pi ^*(c), c) \lt \mu _n(k, c)). \end{equation*}\) With the prior as specified, the posterior mean is equal to the sample mean. In which case [3], \(\begin{equation*} P(\mu _n(\pi ^*(c), c) \lt \mu _n(k, c)) = \Phi \left(- \sqrt {\hat{\eta }_n(k, c) n} \right), \end{equation*}\) where \(\Phi (\cdot)\) denotes the cumulative distribution function of the standard Gaussian distribution. Since for \(x \lt 0\), \(\Phi (x) \doteq \exp (- x^2 / 2)\) [17], we get \(\begin{equation*} P(\mu _n(\pi ^*(c), c) \lt \mu _n(k, c)) \doteq \exp \left(- \hat{\eta }_n(k, c) n / 2 \right). \end{equation*}\) By Theorem 6.9, \(\hat{p}_n(k, c) \rightarrow p^*(k, c)\), which implies that \(\hat{\eta }_n(k, c) \rightarrow \eta (k, c)\) as \(n \rightarrow \infty\). Thus, putting it all together, we get \(\begin{equation*} \qquad \qquad \qquad \;\;\;\qquad \qquad PFS^n_{\sim } \doteq \exp \left(- \left(\min _{c \in \mathcal {C}} \min _{k \ne \pi ^*(c)} \eta (k, c)\right)n / 2 \right).\qquad \qquad \qquad \qquad \qquad \square \end{equation*}\)

Note that Theorem 6.9 is an extension of the result originally proved for C-OCBA [9] and the convergence rate result in Theorem 6.12 also applies to C-OCBA under the assumption of known observation noise. This can be interpreted as saying that GP-C-OCBA and C-OCBA are asymptotically equivalent. A natural question in this case is whether GP-C-OCBA can be shown to have an advantage over C-OCBA under a finite budget setting. Though it is difficult to show this rigorously, we can argue for it in an informal way. By modeling the performance of an alternative-context with a Gaussian process, we implicitly assume that the contexts that are similar will have similar performances, with the particular measure of similarity being learned from the data. As a result, for a given alternative-context, by learning about the contexts that are similar to this context, we can refine our model of the contexts without having to evaluate the context itself. This is similar to the linearity assumptions that we see in the literature, e.g., in Reference [29]. In their case, there is an explicit assumption of linearity, which, if holds true, allows for evaluating only the extreme contexts to determine the policy for all contexts. Due to the assumed linearity, the performance of any given context can be estimated by solving the regression equations. There are many cases where linearity provides a good approximation, but there are many others where it does not hold true at all. Our modeling of similarity, as opposed to linearity, works in a similar way. It does not eliminate the need to evaluate all contexts. However, by allowing inference on one context’s performance via other similar contexts’ evaluations, it reduces the number of evaluations needed, leading to a better finite time performance. An example of this was shown in Figure 1, where we saw that the GP posterior mean produced predictions that resembled the true rewards much more closely, while the sample mean predictor required many more samples to refine the estimates.

6.3 Estimation for Contexts Outside \(\mathcal {C}\)

In this section, we briefly discuss if a context \(c^{\prime } \not\in \mathcal {C}\) is given, how we estimate \(F(k,c^{\prime })\) and what statistical guarantee we can say about this estimate. As discussed in Section 3, we can use the same GP model by incorporating the point \(c^{\prime }\) in the context set. Then, we can calculate the posterior estimator for any alternative-context pair \((k,c^{\prime })\) in the same way as \(\begin{equation*} \mu _n(k,c^{\prime }) = \mu _0(k, c^{\prime }) + \Sigma _0(c^{\prime }, D_n(k); k) A_n^{-1}(k) (Y_n(k) - \mu _0(D_n(k)))^{\top } \end{equation*}\) and \(\begin{equation*} \Sigma _n(c^{\prime }, c^{\prime }; k) = \Sigma _0(c^{\prime }, c^{\prime }; k) - \Sigma _0(c^{\prime }, D_n(k); k) A_n^{-1}(k) \Sigma _0(D_n(k), c^{\prime }; k). \end{equation*}\) Since no simulation is run under context \(c^{\prime }\), \(\mu _n(k,c^{\prime })\) cannot converge to \(F(k,c^{\prime })\). This can also be reflected in the posterior variance \(\Sigma _n(c^{\prime },c^{\prime };k)\), which represents the confidence level for reward estimation. The following result quantifies such confidence level in the asymptotic sense. It is a byproduct from the proof of Lemma 6.10.

The expression on the right hand side provides us with some intuition. Let \(Z(C^{\prime })\) be a multivariate normal random vector following \(\mathcal {N}(\mu _0(k, \mathcal {C^{\prime }}),\Sigma _0(\mathcal {C^{\prime }},\mathcal {C^{\prime }};k))\) , where \(\mathcal {C^{\prime }} = \mathcal {C} \bigcup \lbrace c^{\prime }\rbrace\). Then, the posterior distribution for \((k,c^{\prime })\) converges to a normal distribution with variance being the same as that of the conditional distribution \(Z(c^{\prime }) | Z(\mathcal {C})\). This is because as \(n \rightarrow \infty\), we gather sufficient information for each \((k,c)\) with \(c\in \mathcal {C}\). The posterior distribution for \((k,\mathcal {C})\) will concentrate on the true value \(F(k,\mathcal {C})\), and hence the posterior distribution for \((k,c^{\prime })\) converges to the marginal distribution of \(Z(c^{\prime }) | Z(\mathcal {C})\) as \(Z(\mathcal {C}) = F(k,\mathcal {C})\). In particular, if \(c^{\prime }\in \mathcal {C}\), then we have \(\Sigma _n(c^{\prime },c^{\prime };k)\) converges to 0, which coincides with Lemma 6.2 and Lemma 6.10. If \(c^{\prime }\not\in \mathcal {C}\) and \(\Sigma _0(\mathcal {C^{\prime }},\mathcal {C^{\prime }};k)\) is positive definite, then \(\Sigma _n(c^{\prime },c^{\prime };k)\) cannot converge to 0, meaning the uncertainty level remains positive, and the value of this uncertainty level is determined by the prior correlation, which usually depends on the distance from \(c^{\prime }\) to \(\mathcal {C}\).

7.1 Synthetic Test Functions

For the experiments, we generate the true rewards, \(F(k, c)\), by evaluating common global optimization test functions on randomly drawn points from the function domain. We use the first dimension of the function input for the alternatives, i.e., each alternative corresponds to a fixed value of \(x_1\), and spread the alternatives evenly across the corresponding domain. The remaining input dimensions are used for the contexts, and thus contexts are \(d-1\) dimensional vectors for a d dimensional test function. Put together, this corresponds to \(F(k, c) = f(x_k, x_c)\) where \(x_k\) and \(x_c\) are fixed realizations of 1 and \(d-1\) dimensional uniform random variables, respectively. The rewards are observed with additive Gaussian noise with standard deviation set as \(\frac{f_{max} - f_{min}}{100 / 3}\), where \(f_{max}\) and \(f_{min}\) are estimated using 1,000 samples drawn uniformly at random from the function domain. We use the following functions in our experiments:

7.2 Results of Synthetic Test Functions

The experiment results are plotted in Figure 2. We ran each experiment for 2,000 iterations, except for IKG in Hartmann and Cosine8 functions, which were run for 1,000 iterations due to their excessive cost. The plots show the empirical contextual PCS, estimated using 100 replications. The first four plots compare all algorithms. However, the last two only compare GP-C-OCBA, LEVI and IKG, due to small initial budget preventing drawing of multiple samples from each alternative-context pair, which is necessary to form an initial estimate of sample mean and variance that is used by DSCO and C-OCBA, as well as the difficulty of setting up the extreme design for the two-stage algorithms. In all four of the experiments comparing all algorithms, we see that the algorithms using the GP models achieve the highest contextual PCS, with the exception of LEVI, which performs poorly on worst-case PCS. This demonstrates the benefit of using a statistical model that leverages the hidden correlation structure in the reward function, and also highlights the importance of using a sampling strategy that provides a good coverage of the search space. In particular, for the worst-case PCS, we see that the DSCO and C-OCBA perform significantly worse in both problems, with the performance of TS and TS+ also diminishing significantly. For TS and TS+, this clearly demonstrates the weakness of using a linear model when the underlying function is non-linear, which leads to 0 worst-case PCS in the Griewank function. Similarly, the poor performance of DSCO and C-OCBA in the worst-case PCS is explained by a total of 2,400 samples being far from sufficient to form reliable estimates for 200 alternative-context pairs using an independent statistical model. On a related note, although a comparison with opportunity cost as the performance measure is beyond the scope of this article, we include the result for opportunity cost in Appendix B. Both IKG and LEVI were originally designed to minimize opportunity cost, and it is not surprising that they outperform GP-C-OCBA when transitioning to the performance measure of opportunity cost. It is worthwhile to use different methods for different performance measures.

Fig. 2.

View Figure

Although in some experiments GP-C-OCBA is slightly trailing behind IKG, which attributes to its asymptotic property, we see that GP-C-OCBA is highly competitive against IKG, while having significantly smaller computational complexity. As a cheaper alternative to IKG, LEVI provides good performance on two of the problems, while falling behind in others, particularly when considering the worst-case PCS. The wall-clock times for the experiments are reported in Table 1. We see that even in the experiments with smallest number of alternatives and contexts, the IKG algorithm takes about 5 times as long to run, with the ratio increasing significantly to about 75 times as we move to larger experiments. The run time of LEVI is slightly longer than that of GP-C-OCBA, which makes it a good alternative in settings where it shows good performance. The reported run times are for 1,000 iterations of a full experiment, and include the cost of fitting the GP model, which is identical for all algorithms. It is worth noting that the cost of evaluating the test functions in these experiments is insignificant compared to the running time. As the cost of function evaluation increases, the results would look nicer for IKG, though it would still remain the more expensive alternative.

Overall, the experiments show that GP-C-OCBA is highly competitive in terms of sampling efficiency and computational cost. We believe that this makes GP-C-OCBA an attractive option for any practitioner that is faced with a contextual R&S problem.

7.3 Extension to Continuous Context Spaces

In this article, we have focused on the setting of discrete context spaces and developed the GP-C-OCBA algorithm under this assumption. However, there are applications where a user might be interested in using our approach with a continuous context set. In this section, we describe a simple extension of GP-C-OCBA to support continuous contexts, and present results comparing it with random sampling and LEVI on several synthetic test problems. We use the same problems from Section 7.2 with the Expected PCS objective. For each problem, we use the first input dimension for the alternatives with the same number of alternatives as before, and use the remaining input dimensions as the continuous context set.

As defined in Algorithm 1, GP-C-OCBA cannot be used with continuous contexts. The presence of an infinite number of contexts will lead to \(\hat{p}(k, c)\) evaluating to zero with probability one, preventing us from making a meaningful comparison in step 5. To overcome this issue, we use kernel density estimation, defined as \(\hat{p}(k, c) = \sum _{i=1}^{n} e^{-\Vert c - c_i\Vert _2} 1\lbrace k = k_i\rbrace\), which values each observation from a context proportional to its proximity to the context under consideration. With this modification, the rest of the algorithm is followed as is. For optimizing both GP-C-OCBA (minimizing \(\zeta (k, c)\)) and LEVI in this finite alternative - continuous context space, we follow a two step approach. First, we solve the continuous optimization problem \(\min _c [\min _{k \ne \pi ^n(c)} \zeta (k, c)]\) (and the corresponding maximization problem for LEVI) to find the context, then we find the corresponding alternative by fixing the context.

The results are shown in Figure 3 with the run times reported in Table 2. Of the four problems, we see LEVI and GP-C-OCBA performing quite similarly on the first two, and each one performing better than the other in one of the remaining problems. Overall, we see the PCS curves flattening out towards the end of the experiments, which is explained by both algorithms getting greedy and allocating a large portion of the budget on the predicted best alternatives. In addition, for the higher dimensional experiments, the expected PCS values are rather small, which highlights the difficulty of learning the best alternative across a high dimensional context space. Except for the Branin experiment, LEVI is considerably more expensive to run than GP-C-OCBA, which again highlights GP-C-OCBA as a low cost, sample effective algorithm. It is also worth noting that the extension presented in this section is a very simple extension of GP-C-OCBA to continuous contexts. It is quite possible that there are other extensions that work much better, however a theoretically grounded study of such extensions is beyond the scope of this work.

Fig. 3.

View Figure