Initial gazelle extension implementation

Gazelle language extension for erlang. Should be considered
experimental. Breaking changes may still be made in the extension
without a major version bump to rules_erlang.

Supports BUILD file generation for hex, rebar & "bare" erlang
apps (erlang sources in the typical layout). Not all hex and rebar
information is imported - notably rebar "profiles" are not yet handled.

The extension generates a "fine-grained" build. In "classic"
rules_erlang, the erlang_app/test_erlang_app macros compile all beam
files for a given application in one or two invocations of `erlc`. The
gazelle extension instead generates a rule per beam file, resulting in
an invocation of `erlc` for each. This improves caching, though is
probably somewhat slower on the first compilation of an app. It also
allows apps that require more than 2 compilation "phases" to work
directly. As this results in many more rules in the BUILD file, the
`erlang_generate_beam_files_macro` directive can be used to instead
declare most of rules in an `app.bzl` file imported by the BUILD file.

The extension also provides some support for the gazelle
"update-repos" command. The -from_file option can be used with
rebar.lock files. Can also be run with references to hex.pm or
github.com to import dependencies directly. This is non-recursive,
though transitive deps are printed for convenience. This is deliberate
design decision, as bazel prior to version 6 with bzlmod does not have
a built-in mechanism for handling of transitive deps.

The extension generates a `moduleindex.yaml` file in the repo root,
which is used as an index of which modules are contained in a given
erlang application. This allows calls at runtime of a given module to
be mapped to the appropriate dependency. It's updated by both the
default fix/update commands, as well as update-repos, so it's
recommended to check it into VCS.

A number of gazelle directives are supported to help customize BUILD
file generation:

- erlang_behaviour_source_lib
- erlang_exclude_when_rule_of_kind_exists
- erlang_generate_beam_files_macro
- erlang_always_generate_test_beam_files
- erlang_skip_rules
- erlang_apps_dirs
- erlang_app_dep
- erlang_app_dep_ignore
- erlang_app_extra_app
- erlang_no_tests

Other changes:

Some new versions of existing rules are added, such as
`erlang_bytecode2` and `assert_suites2`.

erlang_bytecode2 compiles with transitive deps

since gazelle generates minimal deps, we need to include the
transitive deps so that, for instance, headers include headers of deps
are present.

For compilation, we actually don't need all the files from a dep, just
public headers and compiled behaviours. In fact, only the transient
headers are needed, not the transient behaviours. So this could be
optimized.

erlang_build rule produces tars, not directories - this simpilies the
"installation" of internal erlang builds in other rules

assert_suites2 macro does not require passing the list of existing
rules explicitly

self referencing include_lib directives are supported

The -hidden flag is used for ct_run

Author: HoloRin

.bazelversion

@@ -1 +1 @@
-latest
+last_rc

.github/workflows/test.yaml

@@ -29,11 +29,13 @@ jobs:
           --noexperimental_enable_bzlmod \
           --color=yes
     - name: RESOVLE TEST LOGS PATH
+      if: always()
       working-directory: test
       run: |
         echo "::set-output name=LOGS_PATH::$(readlink -f bazel-testlogs)"
       id: resolve-test-logs-path
     - name: CAPTURE TEST LOGS
+      if: always()
       uses: actions/upload-artifact@v3
       with:
         name: bazel-testlogs-${{matrix.otp}}
@@ -58,6 +60,7 @@ jobs:
       shell: bash
       run: |
         cat << EOF >> user.bazelrc
+          startup --output_user_root=C:/tmp
           startup --windows_enable_symlinks
           build --enable_runfiles
           build --color=yes
@@ -74,17 +77,20 @@ jobs:
         set ERLANG_HOME=%ERL_PATH:\bin\erl.exe=%
 
         bazelisk test //... --noexperimental_enable_bzlmod
-    #! https://github.com/actions/upload-artifact/issues/240
-    #! - name: RESOVLE TEST LOGS PATH
-    #!   run: |
-    #!     Set-Variable -Name logs_path -Value (Get-Item bazel-testlogs).Target
-    #!     Write-Output "::set-output name=LOGS_PATH::$logs_path"
-    #!   id: resolve-test-logs-path
-    #! - name: CAPTURE TEST LOGS
-    #!   uses: actions/upload-artifact@v3
-    #!   with:
-    #!     name: bazel-testlogs-${{matrix.otp}}
-    #!     path: ${{ steps.resolve-test-logs-path.outputs.LOGS_PATH }}
+    - name: RESOVLE TEST LOGS PATH
+      if: always()
+      working-directory: test
+      shell: bash
+      run: |
+        LOGS_TAR=$PWD/logs.tar
+        cd bazel-testlogs
+        tar -cf $LOGS_TAR *
+    - name: CAPTURE TEST LOGS
+      if: always()
+      uses: actions/upload-artifact@v3
+      with:
+        name: bazel-testlogs-windows-${{matrix.otp}}
+        path: test/logs.tar
   test-bzlmod:
     runs-on: ubuntu-20.04
     strategy:
@@ -105,11 +111,13 @@ jobs:
       run: |
         bazelisk test //...
     - name: RESOVLE TEST LOGS PATH
+      if: always()
       working-directory: test
       run: |
         echo "::set-output name=LOGS_PATH::$(readlink -f bazel-testlogs)"
       id: resolve-test-logs-path
     - name: CAPTURE TEST LOGS
+      if: always()
       uses: actions/upload-artifact@v3
       with:
         name: bazel-testlogs-bzlmod-${{matrix.otp}}
@@ -133,6 +141,7 @@ jobs:
       shell: bash
       run: |
         cat << EOF >> user.bazelrc
+          startup --output_user_root=C:/tmp
           startup --windows_enable_symlinks
           build --enable_runfiles
           build --color=yes
@@ -143,6 +152,20 @@ jobs:
       shell: cmd
       run: |
         bazelisk test //...
+    - name: RESOVLE TEST LOGS PATH
+      if: always()
+      working-directory: test
+      shell: bash
+      run: |
+        LOGS_TAR=$PWD/logs.tar
+        cd bazel-testlogs
+        tar -cf $LOGS_TAR *
+    - name: CAPTURE TEST LOGS
+      if: always()
+      uses: actions/upload-artifact@v3
+      with:
+        name: bazel-testlogs-bzlmod-windows-${{matrix.otp}}
+        path: test/logs.tar
   test-bzlmod-internal-erlang:
     runs-on: ubuntu-20.04
     steps:
@@ -170,14 +193,16 @@ jobs:
           --toolchain_resolution_debug="@rules_erlang.*" \
           --verbose_failures
     - name: RESOVLE TEST LOGS PATH
+      if: always()
       working-directory: test
       run: |
         echo "::set-output name=LOGS_PATH::$(readlink -f bazel-testlogs)"
       id: resolve-test-logs-path
     - name: CAPTURE TEST LOGS
+      if: always()
       uses: actions/upload-artifact@v3
       with:
-        name: bazel-testlogs-bzlmod-${{matrix.otp}}
+        name: bazel-testlogs-bzlmod-internal-erlang
         path: ${{ steps.resolve-test-logs-path.outputs.LOGS_PATH }}/*
   test-host-erlang-change-detected:
     runs-on: ubuntu-20.04
@@ -212,3 +237,31 @@ jobs:
         if [[ "${{ steps.before.outputs.otp }}" == "${{ steps.after.outputs.otp }}" ]]; then
           echo "Fail"
         fi
+  test-gazelle-extension:
+    runs-on: ubuntu-20.04
+    steps:
+    - name: CHECKOUT
+      uses: actions/checkout@v2
+    - name: CONFIGURE ERLANG
+      uses: erlef/setup-beam@v1
+      with:
+        otp-version: 25
+    - name: TEST GAZELLE PACKAGE
+      run: |
+        cat << EOF >> user.bazelrc
+          build --color=yes
+        EOF
+
+        bazelisk test //gazelle:all \
+          || bazelisk test //gazelle:all
+    - name: RESOVLE TEST LOGS PATH
+      if: always()
+      run: |
+        echo "::set-output name=LOGS_PATH::$(readlink -f bazel-testlogs)"
+      id: resolve-test-logs-path
+    - name: CAPTURE TEST LOGS
+      if: always()
+      uses: actions/upload-artifact@v2
+      with:
+        name: bazel-testlogs-gazelle
+        path: ${{ steps.resolve-test-logs-path.outputs.LOGS_PATH }}/*

BUILD.bazel

@@ -1,7 +1,5 @@
-load(
-    "@bazel_skylib//rules:common_settings.bzl",
-    "string_flag",
-)
+load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
+load("@bazel_gazelle//:def.bzl", "gazelle")
 
 config_setting(
     name = "debug_build",
@@ -33,3 +31,19 @@ string_flag(
     build_setting_default = "b",
     visibility = ["//visibility:public"],
 )
+
+# Gazelle configuration options.
+# See https://github.com/bazelbuild/bazel-gazelle#running-gazelle-with-bazel
+# gazelle:prefix github.com/rabbitmq/rules_erlang
+# gazelle:exclude bazel-out
+gazelle(name = "gazelle")
+
+gazelle(
+    name = "update_go_deps",
+    args = [
+        "-from_file=go.mod",
+        "-to_macro=gazelle/deps.bzl%gazelle_deps",
+        "-prune",
+    ],
+    command = "update-repos",
+)

MODULE.bazel

@@ -4,13 +4,45 @@ module(
     version = "3.8.5",
 )
 
+bazel_dep(
+    name = "rules_go",
+    repo_name = "io_bazel_rules_go",
+    version = "0.37.0",
+)
 bazel_dep(
     name = "rules_pkg",
     version = "0.5.1",
 )
 bazel_dep(
     name = "bazel_skylib",
-    version = "1.2.0",
+    version = "1.3.0",
+)
+bazel_dep(
+    name = "gazelle",
+    repo_name = "bazel_gazelle",
+    version = "0.28.0",
+)
+
+# The union of go deps from bazel-gazelle and this repo
+go_deps = use_extension("@bazel_gazelle//:extensions.bzl", "go_deps")
+
+go_deps.from_file(go_mod = "@rules_erlang//:go.mod")
+
+use_repo(
+    go_deps,
+    "com_github_bazelbuild_buildtools",
+    "com_github_bmatcuk_doublestar_v4",
+    "com_github_fsnotify_fsnotify",
+    "com_github_emirpasic_gods",
+    "com_github_google_go_cmp",
+    "com_github_pelletier_go_toml",
+    "com_github_pmezard_go_difflib",
+    "in_gopkg_yaml_v2",
+    "org_golang_x_mod",
+    "org_golang_x_sync",
+    "org_golang_x_tools",
+    # Used internally by the go_deps module extension.
+    "bazel_gazelle_go_repository_directives",
 )
 
 erlang_package = use_extension(

WORKSPACE.bazel

@@ -1,37 +1,22 @@
-load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+workspace(name = "rules_erlang")
 
-http_archive(
-    name = "rules_pkg",
-    sha256 = "038f1caa773a7e35b3663865ffb003169c6a71dc995e39bf4815792f385d837d",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/rules_pkg/releases/download/0.4.0/rules_pkg-0.4.0.tar.gz",
-        "https://github.com/bazelbuild/rules_pkg/releases/download/0.4.0/rules_pkg-0.4.0.tar.gz",
-    ],
-)
+load(":internal_deps.bzl", "rules_erlang_internal_deps")
 
-load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")
+rules_erlang_internal_deps()
 
-rules_pkg_dependencies()
+load(":internal_setup.bzl", "rules_erlang_internal_setup")
 
-http_archive(
-    name = "bazel_skylib",
-    sha256 = "af87959afe497dc8dfd4c6cb66e1279cb98ccc84284619ebfec27d9c09a903de",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.2.0/bazel-skylib-1.2.0.tar.gz",
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.2.0/bazel-skylib-1.2.0.tar.gz",
-    ],
-)
+rules_erlang_internal_setup()
 
-load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
+load("//gazelle:deps.bzl", "gazelle_deps")
 
-bazel_skylib_workspace()
+# gazelle:repository_macro gazelle/deps.bzl%gazelle_deps
+gazelle_deps()
 
-load(":rules_erlang.bzl", "erlang_config", "rules_erlang_dependencies")
+load(":rules_erlang.bzl", "erlang_config")
 
 erlang_config(rules_erlang_workspace = "@")
 
-rules_erlang_dependencies()
-
 load("@erlang_config//:defaults.bzl", "register_defaults")
 
 register_defaults()

app_file.bzl

@@ -3,9 +3,14 @@ load(
     _app_file = "app_file",
 )
 
-def app_file(**kwargs):
+def app_file(
+        app_name = "",
+        dest = "ebin",
+        **kwargs):
     _app_file(
         app_file_tool = Label("//tools/app_file_tool:app_file_tool"),
+        app_name = app_name,
+        out = "%s/%s.app" % (dest, app_name),
         private_stamp_detect = select({
             Label("//private:private_stamp_detect"): True,
             "//conditions:default": False,

app_file2.bzl

@@ -0,0 +1,14 @@
+load(
+    "//private:app_file.bzl",
+    _app_file = "app_file",
+)
+
+def app_file(**kwargs):
+    _app_file(
+        app_file_tool = Label("//tools/app_file_tool:app_file_tool"),
+        private_stamp_detect = select({
+            Label("//private:private_stamp_detect"): True,
+            "//conditions:default": False,
+        }),
+        **kwargs
+    )

ct.bzl

@@ -4,8 +4,8 @@ load(
 )
 load(
     "//private:ct.bzl",
-    "ct_test",
     _code_paths = "code_paths",
+    _ct_test = "ct_test",
     _sanitize_sname = "sanitize_sname",
 )
 load(
@@ -51,7 +51,8 @@ def ct_suite(
     ct_suite_variant(
         name = name,
         suite_name = suite_name,
-        deps = deps + runtime_deps,
+        deps = deps,
+        runtime_deps = runtime_deps,
         **kwargs
     )
 
@@ -70,7 +71,7 @@ def ct_suite_variant(
 
     data_dir_files = native.glob(["test/{}_data/**/*".format(suite_name)])
 
-    ct_test(
+    _ct_test(
         shard_suite = Label("@rules_erlang//tools/shard_suite:shard_suite"),
         name = name,
         suite_name = suite_name,
@@ -86,10 +87,42 @@ def ct_suite_variant(
 
     return suite_name
 
+def ct_test(
+        name = None,
+        suite_name = None,
+        compiled_suites = None,
+        deps = [":test_erlang_app"],
+        shard_suite = Label("@rules_erlang//tools/shard_suite:shard_suite"),
+        **kwargs):
+    if suite_name == None or suite_name == "":
+        suite_name = name
+    if compiled_suites == None:
+        compiled_suites = [":{}_beam_files".format(suite_name)]
+    _ct_test(
+        name = name,
+        suite_name = suite_name,
+        compiled_suites = compiled_suites,
+        deps = deps,
+        shard_suite = shard_suite,
+        is_windows = select({
+            "@bazel_tools//src/conditions:host_windows": True,
+            "//conditions:default": False,
+        }),
+        **kwargs
+    )
+
 def assert_suites(suite_names, suite_files = None):
     if suite_files == None:
         suite_files = native.glob(["test/**/*_SUITE.erl"])
     for f in suite_files:
         sn = f.rpartition("/")[-1].replace(".erl", "")
         if not sn in suite_names:
-            fail("A bazel rule has not been defined for {} (expected {} in {}".format(f, sn, suite_names))
+            fail("A bazel rule has not been defined for {} (expected {} in {})".format(f, sn, suite_names))
+
+def assert_suites2(suite_files = None):
+    if suite_files == None:
+        suite_files = native.glob(["test/**/*_SUITE.erl"])
+    for f in suite_files:
+        sn = f.rpartition("/")[-1].replace(".erl", "")
+        if not sn in native.existing_rules():
+            fail("A bazel rule has not been defined for {}".format(f))