Fixing inconsistent results when using seed in theta_est_leaveNout

[sscini]

Fixes # .

When using the reactor_design leave_N_out example in parmest, the results from multiple runs of the example vary a lot, with random seed defined.

Summary/Motivation:

Using a random seed should allow for reproducible results. Currently, the results vary in the leave_N_out reactor design example (pyomo/pyomo/contrib/parmest/examples/reactor_design/leaveNout_example.py) for parmest even with an active random seed. I included a temporary .py file (pyomo/pyomo/contrib/parmest/examples/reactor_design/mwe_bootstrap_error_SSE.py) with the minimum working example showing the results vary with multiple runs.

Changes proposed in this PR:

• Modify leaveNout_bootstrap_test to produce consistent outputs

TODO before converting from draft:

• Inform pyomo team
• Implement solution
• Testing and documentation

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[sscini]

Added minimum working example as temp .py file. Terminal results for multiple runs shown below (plotting code commented out for output simplicity).

(parmest-dev-mac) scini@DOWLINGLAB-2 reactor_design % python mwe_leaveNout_error_seed.py
79946.41368293496
k1 0.866704
k2 1.591101
k3 0.000168
dtype: float64
k1 k2 k3
0 0.865192 1.588109 0.000167
1 0.865927 1.590812 0.000168
2 0.865887 1.592281 0.000167
3 0.866543 1.600056 0.000168
4 0.865621 1.596912 0.000168
1.0
(parmest-dev-mac) scini@DOWLINGLAB-2 reactor_design % python mwe_leaveNout_error_seed.py
75370.65092521983
k1 0.875008
k2 1.651461
k3 0.000168
dtype: float64
k1 k2 k3
0 0.872583 1.652644 0.000168
1 0.876466 1.647144 0.000169
2 0.874011 1.653098 0.000169
3 0.873361 1.655661 0.000168
4 0.876720 1.647979 0.000169
1.0

[sscini]

Good afternoon @adowling2 @mrmundt @blnicho

I found an issue in one of the reactor_design examples for parmest, where the output to the leaveNout_example is not reproducible even with the same random seed defined. This does not extend to the bootstrap example, once a random seed is added in the bootstrap. I included minimum working examples, the leaveNout_example and bootstrap_example with very minor modifications (described in comments), in .py files within my branch (/pyomo/pyomo/contrib/parmest/examples/reactor_design/mwe_bootstrap_error_seed.py and mwe_leaveNout_error_seed.py), and shared the output above.

I believe I found the issue, and added as a comment in the parmest.py file.

Please let me know if this is a known issue or a misunderstanding on my end. I will be attending the next Developers' meeting on Tuesday, and would be happy to discuss more or answer questions.

[blnicho]

@sscini thanks for reporting this, it looks like a bug. Your fix in parmest.py makes sense. Please add the seed option to the existing examples instead of duplicating them.