# Optimization over machine learning surrogates

## Summary

Many interesting optimization problems combine both data-driven surrogates and mechanistic models. For example, consider a supply chain application where a transshipment model, i.e. a linear program, represents the flow of material and a data-driven model, e.g. a neural network, represents demand. The resulting optimization problem requires simultaneously optimizing over both (i) equations representing the supply chain and (ii) the already-trained neural network.

Different than classic machine learning, the neural network weights and biases are now held constant. Instead, we use the inputs x and outputs y of the neural network as decision variables:

Other instances of this problem arise in (i) black-box optimization over tree-based models with input constraints, (ii) engineering applications where machine learning models replace complicated constraints or act as surrogates, or (iii) maximizing a neural acquisition function.

## What contributions have we made?

• Optimization over gradient-boosted trees with additional constraints (Mistry et al., 2021). Our custom branch-and-bound strategy explicitly uses the patterns already in the trees.
• Partition-based formulations (Kronqvist et al., 2021, Tsay et al., 2021). Many optimization formulations are written with either a big-M or a convex hull formulation. Prior literature often observes that the big-M formulation is too light and the convex hull formulation is too heavy. Our Goldilocks-style solution makes the optimization formulation just right for the application.
• Tree ensemble kernels for Bayesian optimization (Thebelt et al., 2022). We use the kernel interpretation of tree ensembles as a Gaussian Process prior to obtain model variance estimates and develop an optimization formulation for the acquisition function.

## Software

### OMLT: Optimization & machine learning toolkit

OMLT is a Python package for representing machine learning models (neural networks and gradient-boosted trees) within the Pyomo optimization environment (Ceccon, Jalving, et al., 2022):

Here is an introduction to OMLT and a discussion of switching optimization formulations in OMLT:

### ENTMOOT: Multiobjective black-box optimization using gradient-boosted trees

In collaboration with researchers from BASF, Alexander Thebelt developed ENTMOOT to handle tree-based models in Bayesian optimization (Thebelt et al., 2021). We use Gurobi as the underlying solver to optimize over both the gradient-boosted trees and the input constraints.

Here is an introduction to ENTMOOT and a discussion of extending ENTMOOT to the multi-objective setting (Thebelt et al., 2022):