Observer gurobi refactor #3698

michaelbynum · 2025-08-12T21:42:42Z

Summary/Motivation:

This PR refactors the gurobi interfaces to achieve two goals:

reduce duplicate code
use the new contrib.observer package

Changes proposed in this PR:

create a directory for gurobi now that there are 4 files
create a base class for all of the interfaces that use gurobipy, making the direct and persistent interfaces more consistent (e.g., tc_map, postsolve, solve, mipstart)
modify the persistent interface to use the contrib.observer package
add mipstart functionality to gurobi direct and gurobi minlp
add a minimum gurobipy version to the direct interface (addMConstr was not added until version 9)
simplified some of the persistent interface

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…into observer_gurobi_refactor

codecov · 2025-08-13T03:36:34Z

Codecov Report

❌ Patch coverage is 90.82840% with 93 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.20%. Comparing base (13facca) to head (2ab061a).

Files with missing lines	Patch %	Lines
...contrib/solver/solvers/gurobi/gurobi_persistent.py	89.26%	83 Missing ⚠️
...ontrib/solver/solvers/gurobi/gurobi_direct_base.py	94.44%	8 Missing ⚠️
...omo/contrib/solver/solvers/gurobi/gurobi_direct.py	97.05%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3698      +/-   ##
==========================================
- Coverage   89.40%   86.20%   -3.21%     
==========================================
  Files         909      911       +2     
  Lines      105541   105552      +11     
==========================================
- Hits        94364    90996    -3368     
- Misses      11177    14556    +3379

Flag	Coverage Δ
builders	`29.14% <22.09%> (+0.03%)`	⬆️
default	`86.04% <90.82%> (?)`
expensive	`35.79% <22.09%> (?)`
linux	`?`
linux_other	`?`
osx	`?`
win	`?`
win_other	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…into observer_gurobi_refactor

…iscrete_vars'

jsiirola

Some initial comments and questions. I haven't gotten through the persistent interface yet. I have lots of design questions there -- it might be helpful to discuss offline / at a dev call.

pyomo/contrib/solver/plugins.py

jsiirola · 2025-11-12T18:28:19Z

pyomo/contrib/solver/solvers/gurobi/__init__.py

+from .gurobi_direct import GurobiDirect
+from .gurobi_persistent import GurobiPersistent
+from .gurobi_direct_minlp import GurobiDirectMINLP


We have had issues with relative imports; it would probably be best to convert these to absolute imports...

Oh, I think this has to be a relative import. This is in the __init__.py file.

jsiirola · 2025-11-12T18:56:34Z

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

+        GurobiDirectBase._gurobipy_env = None
+
+    @staticmethod
+    def env():


We should provide a mechanism for users to construct their own Env and pass it to us. This is necessary for thing like working with a trial license.

Suggested change

def env():

def env(env=None):

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

jsiirola · 2025-11-14T06:11:50Z

pyomo/contrib/solver/solvers/gurobi/gurobi_direct.py

+        self._gurobi_vars = None
+        self._pyomo_vars = None


Why are these being stored on the solver interface? Direct is supposed to be non-persistent, and adding this seems to violate that principle.

This seems to only be needed for _pyomo_gurobi_var_iter(), which is in turn only needed by _mipstart. I propose that both the attributes and _pyomo_gurobi)var_iter() be removed, and instead _mipstart be updated to accept the SolutionLoader as an argument. As the SolutionLoader is responsible for remembering the mapping between Pyomo and Gurobi, that feels like the natural place to query to get the mapping within _mipstart.

I like that suggestion. Will do.

Is this going to work for callbacks too? They have the same issue, that they need this state (and actually more) to work... So I think there may be a bigger design question lurking here: For example, they need the gurobipy model to be somewhere they can get to: In persistent, that is currently also sitting on the solver interface.

(I was originally thinking that this kind of state could be added in a contextmanager kind of style, and removed before solve is over, regardless of what happens. But that certainly might not be the right answer.)

Excellent points, @emma58. I think this is worth further discussion.

jsiirola · 2025-11-14T06:26:30Z

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

+        )
+
+
+def _load_suboptimal_mip_solution(solver_model, var_map, vars_to_load, solution_number):


Should this (and other loaders) be updated so that instead of var_map, it accepts an iterable of (pyomo_var, gurobi_var) tuples? That way interfaces that do not generate the mapping can still make use of this method?

I think I have to iterate over it twice...

jsiirola · 2025-11-14T06:29:13Z

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

+    original_solution_number = solver_model.getParamInfo('SolutionNumber')[2]
+    solver_model.setParam('SolutionNumber', solution_number)
+    gurobi_vars_to_load = [var_map[v] for v in vars_to_load]
+    vals = solver_model.getAttr("Xn", gurobi_vars_to_load)
+    res = ComponentMap(zip(vars_to_load, vals))
+    solver_model.setParam('SolutionNumber', original_solution_number)


These implementations all seem to be geared around "partial loads", and are actually quite inefficient for "full loads". As I would expect that the vast majority of users / use cases will actually be doing a full load, I would suggest redesigning these methods so that full loads are efficient, and partial loads are efficient if the number of objects to load is small.

What makes you say this implementation is inefficient?

At least in the case of the direct interface, we already have the list of gurobi vars and pyomo vars in the order in which they were given to Gurobi, so if you are getting everything you could get away with:

return ComponentMap(zip(pyomo_vars, solver_model.getAttr('Xn')))

and avoid iterating over things twice, and forming an extra list, and performing N COmponentMap lookups (plus unknown lookups within Gurobi).

Let me do some profiling and see if this even matters.

A good model to use in profiling is @emma58's 400M var LP.

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

jsiirola · 2025-11-14T06:40:42Z

pyomo/contrib/solver/solvers/gurobi/gurobi_persistent.py

+    def invalidate(self):
+        self._valid = False


Doesn't the Direct interface also need to worry about invalidating the last loader before the next solve? Does this really need to be a specialization for the persistent interface?

The direct interface solution loader should still be valid after subsequent solves, I think???

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py

michaelbynum and others added 16 commits August 9, 2025 07:20

refactoring gurobi interfaces

070811d

Merge branch 'main' into observer_gurobi_refactor

27a3a14

revert_gurobi_persistent

d70dbb5

refactoring gurobi interfaces

5b1d3f9

refactoring gurobi interfaces

4818130

refactoring gurobi interfaces

7998fda

refactoring gurobi interfaces

909be88

bugs

862c387

refactoring gurobi interfaces

8f7a61e

remove unused imports

92fa4f5

run black

8a9fc46

update solution loader

7249b19

Merge branch 'main' into observer_gurobi_refactor

25c48e7

Merge remote-tracking branch 'origin/main' into observer_gurobi_refactor

df56887

run black

275d848

Merge remote-tracking branch 'michaelbynum/observer_gurobi_refactor' …

cfa8633

…into observer_gurobi_refactor

michaelbynum added 5 commits August 13, 2025 07:23

dont free gurobi models twice

1788ff3

merge observer

873f176

forgot to inherit from PersistentSolverBase

a43a38b

bug

e76baae

bug

c2a0177

blnicho added this to Pyomo 6.9.4 Release Aug 19, 2025

github-project-automation bot moved this to Todo in Pyomo 6.9.4 Release Aug 19, 2025

Merge branch 'main' into observer_gurobi_refactor

2c7208f

mrmundt removed this from Pyomo 6.9.4 Release Aug 26, 2025

michaelbynum mentioned this pull request Sep 17, 2025

Separate solver_available and license_available #3730

Closed

michaelbynum added 3 commits October 2, 2025 07:16

merge in main and observer

19fecf7

Merge remote-tracking branch 'michaelbynum/observer_gurobi_refactor' …

31e7e97

…into observer_gurobi_refactor

run black

576a217

emma58 mentioned this pull request Oct 28, 2025

contrib.solver: Gurobi warm start option #3774

Closed

michaelbynum added 4 commits October 28, 2025 10:56

merge observer into observer_gurobi_refactor

6ccbaef

directory for all gurobi interfaces

cf000a1

merge main into observer_gurobi_refactor

9abb4bf

clean up gurobi interfaces

7b20095

pyomo-autotest removed the AT: STALE label Nov 1, 2025

michaelbynum and others added 11 commits November 1, 2025 11:38

Merge branch 'observer' into observer_gurobi_refactor

43a864f

update gurobi persistent to use observer

5073ba0

gurobi refactor: bugs

5e280dc

run black

8bff218

typo

d9dc14d

contrib.solvers: bug in gurobi refactor

1c26ae3

contrib.solver: update tests

a4858ca

run black

122511b

Changing the config option name for 'use_mipstart' to be 'warmstart_d…

71963f1

…iscrete_vars'

Adding tests for Gurobi warmstarts in all the interfaces

8712884

Merge branch 'main' into observer_gurobi_refactor

8a729c0

blnicho added this to Pyomo 6.10 Nov 11, 2025

github-project-automation bot moved this to Todo in Pyomo 6.10 Nov 11, 2025

revert modification to ipopt interface

7069f89

michaelbynum mentioned this pull request Nov 12, 2025

Model Observer #3695

Merged

michaelbynum changed the title ~~[Depends on #3695] Observer gurobi refactor~~ Observer gurobi refactor Nov 12, 2025

Merge branch 'main' into observer_gurobi_refactor

93bb118

michaelbynum requested a review from emma58 November 12, 2025 17:07

Merge branch 'main' into observer_gurobi_refactor

7724678

jsiirola reviewed Nov 14, 2025

View reviewed changes

michaelbynum commented Nov 14, 2025

View reviewed changes

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py Show resolved Hide resolved

michaelbynum commented Nov 14, 2025

View reviewed changes

pyomo/contrib/solver/solvers/gurobi/gurobi_direct_base.py Outdated Show resolved Hide resolved

michaelbynum added 2 commits November 16, 2025 07:12

contrib.solver.gurobi: better handling of temporary config options

9bfad14

fix error

2ab061a

		)


		def _load_suboptimal_mip_solution(solver_model, var_map, vars_to_load, solution_number):

Observer gurobi refactor #3698

Are you sure you want to change the base?

Observer gurobi refactor #3698

Conversation

michaelbynum commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

codecov bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jsiirola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emma58 Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

michaelbynum commented Aug 12, 2025 •

edited

Loading

codecov bot commented Aug 13, 2025 •

edited

Loading

emma58 Nov 14, 2025 •

edited

Loading