Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a job that runsmake full-source-dist, but mark it as "allow fail" for now #252

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
62 changes: 39 additions & 23 deletions pipelines/main/launch_unsigned_jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,45 @@
# something about the privileged steps.

steps:
- group: "Check"
notify:
- github_commit_status:
context: "Check"
steps:
- label: "Launch check jobs"
plugins:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
commands: |
# We launch the whitespace check first, because we want it to finish quickly.
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/doctest.yml
# buildkite-agent pipeline upload .buildkite/pipelines/main/misc/fullsrcdist.yml # TODO: uncomment this line

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/pdf_docs/build_pdf_docs.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
agents:
queue: "julia"
os: "linux"
- group: "Full Source Dist" # TODO: delete this line
steps: # TODO: delete this line
- label: "Launch fullsrcdist jobs" # TODO: delete this line
plugins: # TODO: delete this line
- JuliaCI/external-buildkite#v1: # TODO: delete this line
version: "./.buildkite-external-version" # TODO: delete this line
repo_url: "https://github.com/JuliaCI/julia-buildkite" # TODO: delete this line
commands: | # TODO: delete this line
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/fullsrcdist.yml # TODO: delete this line
agents: # TODO: delete this line
queue: "julia" # TODO: delete this line
os: "linux" # TODO: delete this line
- group: "Build"
notify:
- github_commit_status:
Expand Down Expand Up @@ -44,29 +83,6 @@ steps:
agents:
queue: "julia"
os: "linux"
- group: "Check"
notify:
- github_commit_status:
context: "Check"
steps:
- label: "Launch check jobs"
plugins:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
commands: |
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/doctest.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/pdf_docs/build_pdf_docs.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
agents:
queue: "julia"
os: "linux"
- group: "Test"
notify:
- github_commit_status:
Expand Down
33 changes: 33 additions & 0 deletions pipelines/main/misc/fullsrcdist.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
steps:
- group: "Full Source Dist" # TODO: delete this line
# - group: "Check" # TODO: uncomment this line
steps:
- label: "fullsrcdist"
key: "fullsrcdist"
soft_fail: true # TODO: delete this line
plugins:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- JuliaCI/julia#v1:
# Drop default "registries" directory, so it is not persisted from execution to execution
persist_depot_dirs: packages,artifacts,compiled
version: '1.6'
- staticfloat/sandbox#v1:
rootfs_url: https://github.com/JuliaCI/rootfs-images/releases/download/v4.8/package_linux.x86_64.tar.gz
rootfs_treehash: "2a058481b567f0e91b9aa3ce4ad4f09e6419355a"
workspaces:
- "/cache/repos:/cache/repos"
- JuliaCI/julia#v1:
# Drop default "registries" directory, so it is not persisted from execution to execution
persist_depot_dirs: packages,artifacts,compiled
version: '1.6'
timeout_in_minutes: 10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think full-source-dist may take more than 10 minutes... I've never timed it but anecdotally it feels like an eternity. It does a source build of all dependencies then Julia then all documentation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, it actually builds Julia? I thought it just downloads the source code of the dependencies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah haha. It builds Julia so that it can use the built binary to build the documentation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😬

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. A full from-source build of the dependencies and Julia takes more than 30 minutes on our fastest CI machines. See e.g. https://buildkite.com/julialang/julia-master-scheduled/builds/159#01847a53-374d-441c-90cc-c4dcfef6ae22

Maybe we should move the full-source-dist job to the once-daily scheduled pipeline. That's where the existing USE_BINARYBUILDER=0 job lives.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm now downloading a pre-built Julia, but it seems to insist on building everything (including LLVM) from source.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ararslan What's the difference between make full-source-dist and make full-source-dist USE_BINARYBUILDER=0? Shouldn't they both produce the same output? Presumably we shouldn't have to build LLVM from source just to be able to build the docs and download the source tarballs of all dependencies, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

USE_BINARYBUILDER=0 ensures that the source code of the dependencies is included in the resulting tarball alongside Julia's source code. It's meant to be platform-agnostic and buildable without network access, which IIUC isn't possible with the default JLL-based dependencies.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but shouldn't make full-source-dist do that regardless of the value of USE_BINARYBUILDER?

I guess what I'm getting at is, do we need to actually do a full from-source build of LLVM etc just to build this tarball? AFAICT, the tarball needs the following contents:

  1. Source code of Julia
  2. Source code of all of Julia's dependencies
  3. HTML documentation

It seems to me that we can use a regular Julia build to build the HTML docs, and then download the source code of all the dependencies, and then stick everything together.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We definitely could do that, the Makefile just isn't set up that way

commands: |
make full-source-dist USE_BINARYBUILDER=0
DilumAluthge marked this conversation as resolved.
Show resolved Hide resolved
agents:
queue: "julia"
# Only run on `sandbox.jl`-capable machines (not `docker`-isolated ones) since we need nestable sandboxing
sandbox_capable: "true"
os: "linux"
arch: "x86_64"