Juan Ungredda
Skip Abstract Section
Abstract
Many real-world optimisation problems such as hyperparameter tuning in machine learning or simulation-based optimisation can be formulated as expensive-to-evaluate black-box functions. A popular approach to tackle such problems is Bayesian optimisation, which builds a response surface model based on the data collected so far, and uses the mean and uncertainty predicted by the model to decide what information to collect next. In this article, we propose a generalisation of the well-known Knowledge Gradient acquisition function that allows it to handle constraints. We empirically compare the new algorithm with four other state-of-the-art constrained Bayesian optimisation algorithms and demonstrate its superior performance. We also prove theoretical convergence in the infinite budget limit.
- . 2012. Kernels for vector-valued functions: A review. Foundations and Trends® in Machine Learning 4, 3 (March 2012), 195–266.
DOI: Google Scholar
Digital Library
- . 2016. Simulation optimization: A review of algorithms and applications. Annals of Operations Research 240, 1 (2016), 351–380.Google ScholarCross Ref
- . 2017. Constraint handing in efficient global optimization. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 673–680.Google ScholarDigital Library
- . 2020. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems 33. http://arxiv.org/abs/1910.06403Google Scholar
- . 2016. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. arXiv abs/1602.04450 (2016).Google Scholar
- . 2023. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. Machine Learning 112 (2023), 3713–3747.Google Scholar
- . 2019. Sequential model based optimization of partially defined functions under unknown constraints. Journal of Global Optimization 79 (2019), 281–303.Google Scholar
- . 2021. A new knowledge gradient-based method for constrained Bayesian optimization. arXiv:2101.08743 [cs.LG] (2021).Google Scholar
- . 2011. Probability and Stochastics. Graduate Texts in Mathematics, Vol. 261. Springer.Google ScholarCross Ref
- . 2021. Scalable constrained Bayesian optimization. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, Arindam Banerjee and Kenji Fukumizu (Eds.). Proceedings of Machine Learning Research, Vol. 130. PMLR, 730–738. https://proceedings.mlr.press/v130/eriksson21a.htmlGoogle Scholar
- . 2008. Engineering Design via Surrogate Modelling. Wiley.Google ScholarCross Ref
- . 2018. A tutorial on Bayesian optimization. arXiv:1807.02811 [stat.ML] (2008).Google Scholar
- . 2014. Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Machine Learning (ICML’14). 3.Google Scholar
- . 2016. Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics 58, 1 (2016), 1–11.
DOI: Google ScholarCross Ref - . 2016. A general framework for constrained Bayesian optimization using information-based search. Journal of Machine Learning Research 17, 1 (
Jan. 2016), 5549–5601. Google ScholarDigital Library - . 2014. Predictive entropy search for efficient global optimization of black-box functions. In Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 1 (NIPS’14). MIT Press, Cambridge, MA, 918–926.Google Scholar
- . 1998. Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13 (
Jan. 1998), 455–492.DOI: Google ScholarDigital Library - . 2007. Most likely heteroscedastic Gaussian process regression. In Proceedings of the 24th International Conference on Machine Learning(ICML’07). ACM, 393–400.
DOI: Google ScholarDigital Library - . 2021. Constrained Optimization in Simulation: Efficient Global Optimization and Karush-Kuhn-Tucker Conditions. CentER Discussion Paper Series No. 2021-031. CentER, Cenber for Economic Research.
DOI: Google ScholarCross Ref - . 2017. Lookahead Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS’17). 1888–1898. Google ScholarDigital Library
- . 2017. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis 14, 2 (2017), 495–519.
DOI: Google ScholarCross Ref - . 2018. Continuous multi-task Bayesian optimisation with correlation. European Journal of Operational Research 270, 3 (2018), 1074–1085.Google ScholarCross Ref
- . 2020. Practical Bayesian optimization of objectives with conditioning variables. arXiv:2002.09996 [stat.ML] (2020).Google Scholar
- . 2019. Bayesian simulation optimization with common random numbers. In Proceedings of the Winter Simulation Conference(WSC’19). IEEE, 3492–3503. Google ScholarDigital Library
- . 2022. Coupling and selecting constraints in Bayesian optimization under uncertainties. arXiv:2204.00527 (2022).
DOI: Google ScholarCross Ref - . 2014. A stepwise uncertainty reduction approach to constrained global optimization. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Samuel Kaski and Jukka Corander (Eds.). Proceedings of Machine Learning Research, Vol. 33. PMLR, 787–795. http://proceedings.mlr.press/v33/picheny14.htmlGoogle Scholar
- . 2016. Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian. In Advances in Neural Information Processing Systems 29, , , , , and (Eds.). Curran Associates, 1435–1443. http://papers.nips.cc/paper/6439-bayesian-optimization-under-mixed-constraints-with-a-slack-variable-augmented-lagrangian.pdfGoogle Scholar
- . 2013. A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization 48 (2013), 607–626.
DOI: Google ScholarDigital Library - . 2017. Multi-information source optimization. In Advances in Neural Information Processing Systems, , , , , , , and (Eds.), Vol. 30. Curran Associates, 1–11. https://proceedings.neurips.cc/paper/2017/file/df1f1d20ee86704251795841e6a9405a-Paper.pdfGoogle Scholar
- . 2006. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.Google ScholarDigital Library
- . 2002. Flexibility and Efficiency Enhancements for Constrained Global Design Optimization with Kriging Approximations. Ph. D. Dissertation. University of Michigan.Google Scholar
- . 1998. Global versus local search in constrained optimization of computer models. IMS Lecture Notes Monogram Series 34 (1998), 11–25.
DOI: Google ScholarCross Ref - . 2011. The correlated knowledge gradient for simulation optimization of continuous parameters using Gaussian process regression. SIAM Journal on Optimization 21, 3 (2011), 996–1026.
DOI: Google ScholarCross Ref - . 2016. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104, 1 (2016), 148–175.Google ScholarCross Ref
- . 2017. Discretization-free knowledge gradient methods for Bayesian optimization. arXiv:1707.06541 [stat.ML] (2017).Google Scholar
- . 2021. Constrained two-step look-ahead Bayesian optimization. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS’21). 1–13.Google Scholar
Index Terms
- Bayesian Optimisation for Constrained Problems
Recommendations
How Bayesian should Bayesian optimisation be?
GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference CompanionBayesian optimisation (BO) uses probabilistic surrogate models - usually Gaussian processes (GPs) - for the optimisation of expensive black-box functions. At each BO iteration, the GP hyperparameters are fit to previously-evaluated data by maximising ...
Are we forgetting about compositional optimisers in Bayesian optimisation?
Bayesian optimisation presents a sample-efficient methodology for global optimisation. Within this framework, a crucial performance-determining subroutine is the maximisation of the acquisition function, a task complicated by the fact that acquisition ...
Bayesian Optimisation vs. Input Uncertainty Reduction
Simulators often require calibration inputs estimated from real-world data, and the estimate can significantly affect simulation output. Particularly when performing simulation optimisation to find an optimal solution, the uncertainty in the inputs ...
Comments