At the CLI, compile modules next to their source rather than to a different location

[blerner]

(migrated from https://github.com/orgs/brownplt/teams/pyret/discussions/10)

Idea: let's avoid a whole bunch of nonsense about modules interacting between Pyret and JS by putting compiled JS files alongside the source. This allows us to compile imports to require in JS, and to use require (and tools that parse it) to handle transitive paths that go Pyret -> JS -> Pyret.

Here's the case that I want to handle well that this will be great for:

Consider a JS module that

• Imports a Pyret module
• Imports a JS module by relative path
• Is itself imported by a Pyret module

I view this as a fairly common case, since we may want to:

• Define types as a Pyret structure
• Define fast, core implementations in JS that use those types, and is reasonably organized using a few raw JS modules
• Write some code in Pyret that imports the fast JS representations and also uses those types, and then re-exports functions and types to other code

The issue with the current system Pyret's CLI supports is that the programmer has to explicitly list all of the JS files that need to be copied over into the standalone, and there's no getting around it because of relative path issues. Consider:

  image-structs.arr
  -----------------

  data Color: rgba(...) | red | green | blue end

  image-lib.js
  --------------
  { requires: [{type: "pyret-file", path: "image-structs.arr"}],
    nativeRequires: ["image-helpers.js"],
    theModule: function(runtime, imageStructs, imageHelpers) {
        ...
    }
  }

  image-helpers.js
  ----------------
  define(["fast-rgba-calculator.js", "some-fancy-canvas-polyfill.js"], function(rgbaCalc, canvas) {
    ...
    return {
      makeSquare: ...,
      makeCircle: ...
    };
  });


  seam-carving.arr
  ----------------
  import js-file("image-lib.js") as I

Consider what's required to (a) traverse this module graph and (b) relocate it to a built directory:

1. Do the Pyret module traversal, starting from seam-carving.arr:

   • Find the image-lib.js dependency and parse it
   • Find the image-structs.arr dependency and parse it
   • Check that the image-helpers.js file exists and copy it over
   • Do a JS/REQUIREJS module traversal to find fast-rgba-calculator, some-fancy-canvas-polyfill

   This last step is the major, major pain. For all the things our module traversal visits, we have total and unambiguous information about how to copy things over, can hash a URI, etc. But once we drop into JS-land, we either have to mandate an extremely strict format, OR we have to handle whatever JS/node allow for finding and importing modules.

As a result, we don't do this, instead:

2. In the current compiler, we make the programmer manually list all the JS files that they want to copy over, and these get copied into the standalone at the end.

   This is a big pain because the standard library has a bunch of these, as well, and without doing a JS module traversal we cannot know which ones are actually needed. As a result, we end up with the entirety of the Pyret standard library copied into many standalones. This is a really annoying burden on compiler development and always trips up new contributors, because by its nature it's hard to give a good error when a module isn't included, since we don't traverse to find them.

   The same thing will happen with user-defined code, where they will need to manually track a manifest of the individual JS files they need, hampering refactoring of JS code into modules.

So, instead of doing our current approach of having a separate compiled-dir, I think we should just emit code alongside existing files (exceptions below). This avoids the need to ever copy anything to a new place, which immediately is fraught because we could copy too much or too little and need dep-finding at the JS level to get it right. Instead, relative includes will be left relative and we avoid a number of issues.

The other benefit is that the resulting code can be processed by existing JS tools that now about require(), by inspecting only the files ending in .js

In the new scheme, the above example would look like:

  image-structs.arr
  -----------------

  data Color: rgba(...) | red | green | blue end

  image-lib.arr.static.json # this needs to be specified for files imported into Pyret
  --------------
  { pyretRequires: [{type: "pyret-file", path: "image-structs.arr"}], }
  
  image-lib.arr.js
  --------------
  // The programmer is responsible for putting this require in their source and
  // duplicating it in .static.json. If they had this include with no
  // .static.json pyretRequires defined, they'd be hosed
  var IS = require("image-structs.arr.js");

  var IH = require("image-helpers.js"); // This is just a normal require


  image-helpers.js
  ----------------
  // Now this is just a plain JS file
  var rgbaCalc = require("fast-rgba-calculator.js");
  var canvas = require("some-fancy-canvas-polyfill.js");
  module.exports = {
      makeSquare: ...,
      makeCircle: ...
    };


  seam-carving.arr
  ----------------
  # This potentially doesn't even need to know that this is a JS file, but I'm
  # including the js-file dependency for clarity.
  import js-file("image-lib.arr.js") as I

After compilation, this directory would hold:

  image-structs.arr         # unchanged
  image-structs.arr.js      # containing compiled code for the Color datatype
  image-structs.arr.json    # containing types for the Color constructors

  image-lib.arr.json        # unchanged
  image-lib.arr.js          # unchanged

  image-helpers.js          # unchanged
  fast-rgba...              # unchanged
  some-fancy...             # unchanged

  seam-carving.arr          # unchanged
  seam-carving.arr.js       # containing compiled code for seam carving
                            # INCLUDING     "I = require('image-lib.arr.js');"
  seam-carving.arr.json     # containing types and deps information for seam-carving

This is related to a flag that Bucklescript provides:

https://bucklescript.github.io/bucklescript/Manual.html#_in_source_build_support_since_1_9_0

Exceptions:

• We may still want the compiled-dir to hold any more exotically-located modules, like shared-gdrive imports, that we may want to support from the CLI.
• It could be reasonable to still store URI-hashed types and other static information in the compiled-dir, including source locations or other error-reporting data that a JS-based dependency tool wouldn't care about