DNMY: [SOF] Begin schema for stochastic file format

[odow]

This PR is an exploratory look at how we could build a multistage stochastic file format out of nested MathOptFormat models (#47).

Based on the formulation outlined here: http://www.optimization-online.org/DB_HTML/2018/11/6914.html

Biggest question is to which modifications (#3) are incorporated into StochOptFormat, or whether they are natively supported in MOF.

cc @joaquimg @frapac @blegat

[frapac]

Thanks Oscar, that's a step forward!
I like this approach. I agree that we lack a common standard in stochastic programming.
Some questions ahead:

• Do we plan to support multidimensional probability laws directly? That is, by writing:

        "exhaustive_noises": [{
            "probability": [0.5, 0.5],
            "modifications": [{
                "head": "NewSet",
                "constraint": "volume_balance",
                "new_set": {
                    "head": "EqualTo",
                    "values": [0.0; 1.0] 
                }

• How to interpret outgoing arcs whose probabilities do not sum to one? Should we interpret them automatically by adding an empty arc to null or by rescaling the probabilities to one?
• Do you have any idea on how to serialize SMOF in binary file?
• I remember that we had a lot of discussions with @blegat and @leclere about the description of stochastic problems. We concluded that the separation of noises in nodes and in arcs arises some difficulties. For instance, how could we relate node's probabilities with arcs probabilities? Could they be correlated?
• How could we distinct hazard-decision noises from decision-hazard ones?

Anyway, I should have a deeper look at your article, I may find some answer to my questions.

I think that a next step could be to describe Shapiro's Brazilian problem with SMOF.

Feel free to ask if you need any help!

[frapac]

Anyway, I am looking forward a usable benchmark for stochastic programming implemented with SMOF files!

[odow]

Do we plan to support multidimensional probability laws directly? That is, by writing:

We represent multi-dimensional random variables by their cartesian product. The modifications field is an array with an arbitrary number of modifications for each realization from the cartesian space.

How to interpret outgoing arcs whose probabilities do not sum to one? Should we interpret them automatically by adding an empty arc to null or by rescaling the probabilities to one?

Empty arc to null. This is the interpretation of an infinite horizon problem, or one with discounts between stages.

Do you have any idea on how to serialize SMOF in binary file?

This is a bigger MOF problem (#6). If we find a solution for that, we find a solution for this.

We concluded that the separation of noises in nodes and in arcs arises some difficulties. For instance, how could we relate node's probabilities with arcs probabilities? Could they be correlated?

The arc probabilities are independent of the probabilities within a node. The paper describes the motivation for this. It is a core, required feature of the format.

How could we distinct hazard-decision noises from decision-hazard ones?

I'm not sure about this. Probably with a field in each model that lists the non-anticipative variables:

"nonanticipative_variables": ["u", "y"]

Anyway, I should have a deeper look at your article, I may find some answer to my questions.

Please do.

I think that a next step could be to describe Shapiro's Brazilian problem with SMOF.

Do you have a copy?

My next step is to make a reader/writer for Kokako.jl.

[frapac]

Vincent has a copy of Shapiro's problem.
I am currently refactoring the code I wrote during my PhD to release it open-source. I will try to implement a SMOF writer to export the problems I studied in a shareable format.

The formulation behind dual SDDP challenges a lot or own way to formulate stochastic problems (as we cannot decompose local stage problems across the different realizations of the noises). I will try to figure out how to formulate dual stochastic problem with the format you are currently proposing.

[odow]

I will try to implement a SMOF writer to export the problems I studied in a shareable format.

I wouldn't rush. This will probably change a bit.

The formulation behind dual SDDP challenges a lot or own way to formulate stochastic problems (as we cannot decompose local stage problems across the different realizations of the noises).

I think we have to distinguish between formulating the model, and solving the dual problem.

You can write down the decomposed primal version of the problem which is "the model." Your solver should take this formulation and transform it into something that is more amenable to your solution process.

[leclere]

Hello ! Seems to be a good job, I'll need a deeper look to make more comments. For Shapiro's data unfortunately I cannot give them to you, but I can look for the name of the Brazilian guy who could release them. Best,

On Mon, 4 Mar 2019 at 22:58 François Pacaud ***@***.***> wrote: Vincent has a copy of Shapiro's problem. I am currently refactoring the code I wrote during my PhD to release it open-source. I will try to implement a SMOF writer to export the problems I studied in a shareable format. The formulation behind dual SDDP challenges a lot or own way to formulate stochastic problems (as we cannot decompose local stage problems across the different realizations of the noises). I will try to figure out how to formulate dual stochastic problem with the format you are currently proposing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJCk3VpnTZcIDaXJRZAJaSDwJo6tMuWdks5vTZbrgaJpZM4bbYW2> .

-- Vincent Leclère

[odow]

We also need MultirowChange.

[odow]

Closing and moving development to https://github.com/odow/StochOptFormat