Compiling Lua

[ahgamut]

I thought it might be fun to compile a portable version of a language which has C source code.
I picked Lua, and I eventually got it to compile and run on Windows/Linux.

I thought some of the compilation errors might be useful info:

• The following functions/macros in the Lua source clashed names with those in cosmopolitan.h: panic, reverse and isempty.
• strcoll, tanh, sinh, and cosh are unimplemented (leads to linker error)
• UCHAR_MAX in limits.h of the C stdlib is not defined even though SCHAR_MAX is
• Maybe similar to printf doubles with %lf gives nan #58, but floats don't print in the portable Lua console

[jart]

Outstanding! You successfully built Lua as a 336kb Actually Portable Executable using Cosmopolitan! Now we know exactly what features are missing and which things need to change. I intend to get each one of these things fixed. This issue will be updated live as that happens.

[ahgamut]

I forked the Lua Github mirror and created a cosmopolitan branch from Lua 5.4.2.

https://github.com/ahgamut/lua/tree/cosmopolitan

Change COSMO_LIBDIR in the makefile to the appropriate folder containing cosmopolitan.h and friends, and run make.

Just for fun, the git diff due to the above changes (excluding the README and dummy files) is:

23	8	makefile
18	18	ltable.c
6	6	lgc.c
5	5	lapi.c
4	4	lauxlib.c
4	3	lvm.c
3	1	loslib.c
3	0	lmathlib.c # filled in sinh, cosh, tanh
2	2	lvm.h
2	2	ltests.c
2	2	liolib.c
1	1	luaconf.h
1	1	lua.c
1	1	lobject.h
1	1	loadlib.c
1	1	llex.c
1	1	ldo.c
 17 files changed, 78 insertions(+), 57 deletions(-)

[ingenieroariel]

Passing by to ask a somewhat related question, what would need to be changed in order to compile lua as a regular elf with cosmopolitan instead of an ape?

On nixos it gets written to a read only location and the binary does not run, I assume it is related to the "self updating headers":

Failed to execute process '/nix/store/0bxwk7hyscxlc0a6fr3wlrzjba4bcp02-lua-unstable/bin/lua'. Reason:
exec: unknown error (errno was 8)
The file '/nix/store/0bxwk7hyscxlc0a6fr3wlrzjba4bcp02-lua-unstable/bin/lua' is marked as an executable but could not be run by the operating system.

with import <nixpkgs> {};

let
  cosmopolitan = import ./cosmopolitan.nix ;
in stdenv.mkDerivation rec {
  pname = "lua";
  version = "unstable";

  src = fetchFromGitHub {
    owner = "ahgamut";
    repo = "lua";
    rev = "aa20d02dffc4f96d673d30e7378c0c02a6df13f8";
    hash = "sha256-GdXFk4AYE8EP886r7tQk+//rTm4sFMu0x2cVyCyrFgM=";
  };

  nativeBuildInputs = [ makeWrapper cosmopolitan gcc ];


  preBuild = ''
    makeFlagsArray=(SHELL=/bin/sh "MKDIR=mkdir -p" "COSMO_LIBDIR=${cosmopolitan}/lib")
  '';

  installPhase = ''
    runHook preInstall
    mkdir -p $out/{bin,lib/include}
    make
    cp lua.exe $out/bin/lua
    runHook postInstall
  '';
}

with import <nixpkgs> {};


stdenv.mkDerivation rec {
  pname = "cosmopolitan";
  version = "unstable";

  src = fetchFromGitHub {
    owner = "jart";
    repo = "cosmopolitan";
    rev = "cdc54ea1fd86253ced49dd24d45660cadac1d018";
    hash = "sha256-vIDNWaM2+ztwx9hknl5+3ENkx9FxH7/VQ8uUW/sckyg=";
  };

  nativeBuildInputs = [ makeWrapper ];

  preBuild = ''
    makeFlagsArray=(SHELL=/bin/sh "MKDIR=mkdir -p")
  '';

  installPhase = ''
    runHook preInstall
    mkdir -p $out/{bin,lib/include}
    install o/cosmopolitan.h $out/lib/include
    install o/cosmopolitan.h $out/lib
    install o/cosmopolitan.a o/libc/crt/crt.o o/ape/ape.{o,lds} $out/lib
    makeWrapper ${gcc}/bin/gcc $out/bin/cosmoc --add-flags "-O -static -nostdlib -nostdinc -fno-pie -no-pie -mno-red-zone -fuse-ld=bfd -Wl,-T,$out/lib/ape.lds -include $out/lib/include/cosmopolitan.h $out/lib/{crt.o,ape.o,cosmopolitan.a}"
    runHook postInstall
  '';
}

[benwbooth]

Wouldn't you just need to run lua first as part of the nix install? Maybe lua --version or something?

[mnemnion]

Strikes me that low-hanging fruit would be to 'namespace' all the macros and internal functions in cosmopolitan, so that you would have cosmo_panic instead of just panic.

I haven't played around with cosmopolitan/APE (yet) so this could be completely off base.

[jart]

Passing by to ask a somewhat related question, what would need to be changed in order to compile lua as a regular elf with cosmopolitan instead of an ape? @ingenieroariel

Try running make CPPFLAGS=-DSUPPORT_VECTOR=113 which disables Windows + Metal + XNU support. By turning off those three systems, the APE bootloader will choose to produce an ELF binary. See https://justine.lol/cosmopolitan/howfat.html and please note that this functionality got added to Cosmopolitan after the 0.2 release so you'll need to be sure you're using the latest nightly until I can cut a 0.3 release. https://justine.lol/cosmopolitan/download.html

[jart]

@ingenieroariel It's also worth mentioning that the Cosmopolitan Makefile has similarly strict build-time requirements. The way we're currently approaching this problem is by using a wrapper program that creates atomic copies when it's asked to launch a .com binary. Since it uses vfork() and copy_file_range() it's just the fastest thing imaginable. Probably 10x faster than doing the same thing using a shell script.

cosmopolitan/tool/build/compile.c

Lines 523 to 543 in 839e2f4

	/*
	* create temporary copy when launching APE binaries
	* and we help FindDebugBinary to find debug symbols
	*/
	if (!IsWindows() && endswith(cmd, ".com")) {
	comdbg = xasprintf("%s.dbg", cmd);
	cachedcmd = xasprintf("o/%s", cmd);
	if (fileexists(comdbg)) {
	AddEnv(xasprintf("COMDBG=%s", comdbg));
	}
	if (FileExistsAndIsNewerThan(cachedcmd, cmd)) {
	cmd = cachedcmd;
	} else {
	if (startswith(cmd, "o/")) {
	cachedcmd = NULL;
	}
	originalcmd = cmd;
	cmd = xasprintf("%s.tmp.%d", originalcmd, getpid());
	copyfile(originalcmd, cmd, 0);
	}
	}

[jart]

@ahgamut I believe I've addressed everything now in the previous three linked changes. Please take a look and let me know if there's anything else you need. Thank you again for your contribution. The community is very excited about the work we're doing.

[ahgamut]

I have listed out the errors that I have seen/fixed during Lua compilation.
(Unsure which of these are within the scope of Cosmopolitan).

I am using the cosmopolitan branch in my fork of the Lua Github mirror:

• Lua v5.4.2 source
  • added dummy headers
  • modified makefile
  • minimal modifications to source files
• crt.o from the zip file and rest of cosmopolitan compiled from the latest commit.

Errors from earlier that do not appear now:

• UCHAR_MAX is available now
• no issues with setlocale/LC_ALL
• no issues with BUFSIZ/frame size errors
• sinh, cosh, tanh are now available (no linker error)
• no naming clashes for panic, reverse, isempty
• dummy headers for entire C stdlib avoid any #include problems
• created the following dummy headers (I could just comment out the includes in the source)
  • sys/wait.h (lauxlib.c)
  • sys/types.h (liolib.c)
  • unistd.h (loslib.c, lua.c)

Errors that still appear / require adding a fix to the Lua source code for successful compilation:

• strcoll is missing (linker error)
• localeconv()->decimal_point is not available, so just using .
• blocked out dlfcn.h from loadlib.c (it requires dlopen)
• ldo.c expects _setjmp/_longjmp if the LUA_USE_POSIX flag is enabled (I disabled this)
• LUA_USE_POSIX also requires the following functions in liolib.c

#if defined(LUA_USE_POSIX)
#define l_getc(f)		getc_unlocked(f)
#define l_lockfile(f)		flockfile(f) // linker missing the two below functions
#define l_unlockfile(f)		funlockfile(f)
#else
#define l_getc(f)		getc(f)
#define l_lockfile(f)		((void)0)
#define l_unlockfile(f)		((void)0)
#endif

With the above minor changes, lua.exe compiles successfully.
Later I will try some of the tests provided alongside the Lua source.

[jart]

Anything you need can potentially be in scope, so long as it isn't cross-compiling SDL DLLs for ARM with MSVC that run on BSD. Also I totally overlooked strcoll. Thanks for the checkboxes. That makes it much easier to digest. Reopening.