Need to make server "params" param more precise so GUI can be formed

[jbfaden]

For years we've been talking about sending information for creating a GUI to define the "parameters" string for the Das2Server.

When Das2 was created, thick Java clients where the only Das2Server clients, so the client author didn't need a specification of the parameters. This was either the same person writing the data reader and the Java GUI, or two of us might exchange emails about how the parameters would be specified.

With Autoplot, a GUI provides a tool where the scientist can browse from available datasets, and when these are arranged nicely, this is effective. However, when the reader has special parameters, the scientist must look at the DSDF's param_nn properties and manually form this string for themselves. This is error-prone and tedious.

This ticket is to specify how these param_nn controls (or a new control) can be introduced to make the specification more precise so that a GUI can be created by the client. For example, with the Juno mission we often have controls like

param_02 = 'XMAS | If specified only output the Xmas flag'

which could be implemented as a checkbox with the string "only output the Xmas flag". (Note I did some interpretation there to remove "If specified".) More complex specifications would allow for a drop list of options, integer and float options, and strings with regular expressions. What other controls have we used?

This should also have a fall-back. For example, the Autoplot GUI will still have its "parameters" area, but I imagine it would have a button to press to enter a modal GUI, and the last text area of this GUI would be all the parameters it could not recognize. It would be ideal if this effectively supports existing readers for the Juno mission, but correctly and completely supports future readers.

[cpiker]

In general, parameter handling is a sophisticated thing. For das v2 stuff I wouldn't try to handle the general case. For example Cassini Survey is insane, but the majority of the others could be handled with few rules that handle the following three cases:

1. Parameter is present or absent
2. Parameter is present or absent, and has a string substitution pattern
3. Parameter is present or absent, has a substitution pattern, and there are constraints on the substituted values.

In keeping with the general feel of DSDFs which aren't sophisticated, an expanded form for param_XX values in the DSDF could be:

param_name | description | substitution_pattern | value_constraint

where all of description, substitution_pattern, and value_constraint are optional. Here's some examples:

param_00 = 'Bx'                                        ; Case 1, without description
param_00 = 'Bx | Output only search coil X axis data'  ; Case 1 with description

param_01 = 'fft_len | Time points in each DFT | --dftlen @' ; Case 2
param_01 = 'fft_len | | --dftlen @'                         ; Case 2, without description

param_02 = 'fft_len | Time points in each DFT | --dftlen @ | range: 128 to 6144'  ; Case 3 range

param_00 = 'sensor | Output WBR data for sensor | @ | set: Bx By Bz Ex Eu Ev Ew' ; Case 3 enumeration

From the above, it follows that case 1 is just case 3 with an implied substitution pattern of @ and an implied constraint of set: Bx

In the simple patterns above @ is where you poke in the value. Of course regex is the way to go for programmer interfaces, but since DSDFs are for less technical users, avoiding regex seems a fit better to me.

In general I'd avoid typing data, most non-programmers don't really distinguish between data types and data values. For the value_constraints I'd vote that ranges are always floats (that the server may truncate to integers), and sets are always strings.

[wilkinsd]

The description in Chris' comment makes sense to me. Definitely keep it as simple as possible for non-programmers.

[jbfaden]

I agree. The problem with regular expressions is that besides being somewhat exclusive, they aren't completely consistent between implementations.

[cpiker]

It would help me out on the server side if one additional keyword was added to the recognized DSDF set.

paramSeparator = ';'  ;  Using semicolons as the separator between multiple params sent to the server
paramSeparator = ' '  ;  ditto, but spaces instead

By default we use spaces now, but internally it's easier on the server if some non-whitespace token could also be specified. I presume the default would continue to be space whenever this keyword is not present. For consistency between whitespace separators and non-whitespace, the server-side interpretation rule could be that multiple separators always collapse into one. Thus there are no fixed "fields" in the overall HTTP GET params= value

[jbfaden]

I just got bit by the bug where spaces are converted to "+" in the URL, and then the Das2Server doesn't convert it back to a space. This would help.

[jbfaden]

(Not related to this ticket, but for future missions we should really avoid parameters with pluses in them like +5V. These cause lots of problems because URLs should encode spaces as pluses and pluses as %2B, but we have inconsistent logic.

[cpiker]

Yea, sorry about that. You shouldn't have to deal with that client side. I'd enter an issue in das2py-server and tell everyone else it's not Autoplot's fault.

[jbfaden]

I was able to work around it. It looks like if I use a space in the getDataSet call it works okay. It's odd that it would affect what is sent to the server.

[jbfaden]

I suggest I work up a document on the wiki, and we can go from there. Perhaps this is agreeable: https://github.com/das-developers/das2docs/wiki/developer.das2server.params . I use the "developer" prefix so that people do not think that this is a completed feature.

[cpiker]

Thanks for kicking this off. I've added a formal write up here for comments:
Structured Sub-Values for Params. Unless there's a better location, this issue would be a good place to post comments on the proposed formalization of params= sub-values.

Since this repository has typically been of interest only to developers, I didn't use the "developer" prefix, though I maybe should have added the word proposal to the title.

[jbfaden]

Chris this looks fairly straight-forward. Here are questions I have:

When a range is specified, is the upper bound inclusive? I assume it is since this is an fft you are talking about, but the documentation should state this.

I find the @ business a bit confusing. Is the goal to allow the name of the parameter to change?

param_04 = 'median_filter | Take the median value of each set of N '+$
           'points prior to the FFT | --median @ | option: 3 5'

I think this would be a droplist with the options "3" or "5", and this would be labelled "median_filter". In Autoplot's Jython, which I see as similar to this, I might make the labelled box be:

median_filter, Take the median value of each set of N points prior to the FFT
[drop list with "", 3, and 5]

Is the string "median_filter" to be seen by the scientist?

In Autoplot's Jython, this would be:

getParam('median_filter', '', 'Take the median value of each set of N points prior to the FFT', [ '', '3', '5' ] )

Also, when "--median 3" is sent, and paramSep = ';', then would this be --median;3;... and then presumably the reader or server is going to break on ";"?

[cpiker]

When a range is specified, is the upper bound inclusive? I assume it is since this is an fft you are talking about, but the documentation should state this.

Agreed and a very good point. To stick with the rest of das2 I think it should be an exclusive upper bound. Since the intended audience of the document is developers (as is most of das2doc so far) that shouldn't be an issue.

I find the @ business a bit confusing. Is the goal to allow the name of the parameter to change?

Thanks for pointing this out. The @ symbol is very important, it's what gets replaced by the user-supplied value.
imagine a reader with a command line parameter such as:

--dftlen=@

If the user enters 512 into a GUI to represent the desired DFT length, then that value replaces the @ symbol when transmitted to the server, like this:

--dftlen=512

param_04 = 'median_filter | Take the median value of each set of N '+$
           'points prior to the FFT | --median @ | option: 3 5'

I think this would be a droplist with the options "3" or "5", and this would be labelled "median_filter".

So long as the user can also select neither 3 nor 5 (aka there is a null option) that would work.
There needs to be a way for the GUI to signal that the sub-value is not sent at all.

median_filter, Take the median value of each set of N points prior to the FFT
[drop list with "", 3, and 5]

Is the string "median_filter" to be seen by the scientist?

That's kind of up to you as the GUI developer. The description string `Take the median value of each set of N
points prior to the FFT is the intended control label provided by the DSDF author, but there's no reason you couldn't show the parameter name as well if it helps.

Also, when "--median 3" is sent, and paramSep = ';', then would this be --median;3;... and then presumably the reader or server is going to break on ";"?

Actually that wouldn't parse when sent to the server. The entire sub-value is --median 3 which would be URL encoded as either

--median+3
--median%203

Only if another sub-value were provided for params= would a semicolon ; be needed.

Thanks for reading over this carefully, really appreciate the feed-back.

[cpiker]

Based on @jbfaden feedback, I've updated the proposal so that it:

1. uses more concise section definitions that better explain the intent of each section
2. better explains the role of @
3. better relates sub-value definitions to GUI controls
4. better demonstrates fall-back sub-value handling for bad definitions

[jbfaden]

I think we should probably have positional arguments as well. Is it the case these have never been used? Suppose:

reader.sh arg1 arg2

where arg1 and arg2 are can both be xxx or yyy. If you have

reader.sh xxx yyy

this is different than

reader.sh yyy xxx

I would assume that the order is identified by the position in the dsdf (e.g. param_00 and param_01). The parser will have to detect this. Also to consider is:

param_00 = 'first | either xxx or yyy | @'
param_01 = 'second | either xxx or yyy | @'
param_02 = 'third | either xxx or yyy | @ | option: xxx yyy
param_03 = 'fourth | either xxx or yyy or both | @ | set: , xxx yyy

[jbfaden]

I've written an Autoplot script to experiment with this: https://github.com/autoplot/dev/blob/master/demos/2023/20231115/demoDas2Gui.jy

This should be pointed at either a raw dsdf file or at the Das2Server to get the dsdf response. This does not support the custom separator, but just varies that the spec will work.

[cpiker]

I'm not aware of any readers that required extra parameters to be in a certain order, but that doesn't mean there aren't any. Typically what has been done is that if the same reader has an order based parameter that's used to differentiate data sets and the order-based parameters are hard-coded in the dsdf.

Classic examples of this are the ephemeris readers that can handle different coordinate systems. The underlying binary is the same for:

• Juno/Ephemeris/EuropaCoRotational
• Juno/Ephemeris/GanymedeCoRotational
• Juno/Ephemeris/Geocentric
• Juno/Ephemeris/Heliocentric
• Juno/Ephemeris/IoCoRotational
• Juno/Ephemeris/Jovicentric

and a hard coded positional parameter is used to distinguish the outputs, but that's intentionally hidden from client programs.

It may be worth stating clearly in the spec that sub-values of param= can be placed in any order.

[jbfaden]

Regarding the script mentioned a minute ago, note there's a bug in Autoplot which prevents the droplist use to pick a URL. I'll be fixing this, but if you want you should be able to make a local copy of the script and edit it.

[cpiker]

I've updated the wiki to clearly specify that order shouldn't affect the output of the server.

Please let me know if you think that order should matter. I can be convinced ether way, though order dependent parameters may add complications on the server side.