Lagrange multiplier methods for constrained optimisation

[MarkBlyth]

Feature description

Gradient-free optimisers wouldn't be expected to work out the box for problems with equality constraints. A sensible way around this would be to convert the constrained problem to an unconstrained problem with the Lagrange multipliers method, which then allows any of the PyBOP optimisers (gradient-based or metaheuristic) to be used.

Motivation

When parameterising empirical models, it's often useful to enforce some forms of equality constraint, as some parameters can be explicitly constrained from prior knowledge. Lagrange multipliers would then allow these constraints to work alongside all the PyBOP optimisers.

Possible implementation

Maybe this could sit as a wrapper around a cost function?

Additional context

No response