Allow PG to process external data

[pstaabp]

This has been discussed at various points and want to flesh out some ideas before starting on coding.

For a PG problem, create the ability to use external data. The following would be a possible (simplified) example:

DOCUMENT();
loadMacros(‘PGstandard.pl’)

$data = getExternalStorage(‘data1’);

$x =  … # pull out a column of data
$ans = mean($x); 

TEXT(beginproblem());
BEGIN_PGML

[ display a table with the data] 

Find the mean of the above data: [___]{$ans}

END_PGML
ENDDOCUMENT();

In processing this problem there would need to be a way to detect that the data stored in data1 (which would be stored in a database via webwork2 or webwork3) and then that data needs to be fed to the problem. Therefore some type of preprocessing would need to occur.

A possibility would be to add a block like:

BEGIN_PREPROCESS
$data = getExternalStorage('data1');
END_PREPROCESS

and if such a block exists that PG needs to request that data before processing the problem. I realize that for such problems, this would require two exchanges between webwork2 and pg to properly render the problem the first time. If the requested data is passed to the problem, then it should properly render.

Also other related ideas that have been discussed include

• the external data can scoped to various levels (global, courses, problemsets, usersets, users)
• We will need to provide methods as well to put/write data to be used. At a course/problem set level this could be in webwork2/3, but for usersets, methods in PG would be helpful. One use case would be for a student to use the first problem in a set to put their dataset in. Subsequent problems in the set would ask questions about the dataset. A way to handle this could be a POSTPROCESS block to write to the database.
• There has been talk about how to passing data from one PG problem to the next in a dataset. It seems like a structure like this could be used.
• More robust problem sets could be built in which the same values are used for the problems in the set or a JITAR-like set that uses the same values.

[dlglin]

If you want to avoid preprocessing there could be a (possibly empty) data structure that is passed to all PG problems. The getExternalStorage('data1') call would then just need to retrieve data1 from this data structure. If the data isn't present then it would spit an error, but this wouldn't require multiple exchanges between pg and the front end.

We also discussed problems having default values if the external data is not available. I see cases where one would want to do so (e.g. if a parameter is set by the UI then use that value, otherwise generate a value), and other cases where you would want to spit an error (e.g. "using your data set from the previous question...").

[pstaabp]

If you want to avoid preprocessing there could be a (possibly empty) data structure that is passed to all PG problems.

This is a great idea, however if there is a lot of external data, my guess is that we would need to pass everything in the scope of the problem to PG. Since PG is basically going to run as a separate service, I wonder if there is an issue passing too much data.