Skip to content

Commit

Permalink
Build engines without Docker (#170)
Browse files Browse the repository at this point in the history
* Build Wasmtime engine using a `build.rs` script

The current mechanism for building engines using Docker has several
issues: it is slow (Cargo must re-download all required crates), it is
too Linux-specific, and it does not play well with workflows that
attempt to build Sightglass in Docker (i.e., docker-in-docker is
tricky). This change provides a `build.rs` script to build both the
engine shared library and a `.build-info` metadata file: the engine
shared library is built using the same steps the `Dockerfile` used and
the `.build-info` file attempts to capture some of the environmental
information that Docker fixed but is now system-dependent (e.g., tool
versions for reproducing the build).

* Remove CLI commands--`build-engine`, `clean`

If we no longer build Sightglass engines using Docker (see previous
commit) and store them in a central place (the Sightglass cache
directory), then it no longer makes sense to have the `build-engine` and
`clean` commands available in the CLI. This change removes these
commands and associated dead code.

Alternately, we could remove Docker but not the cache directory; by
default, the Wasmtime engine `build.rs` script could place built files
there. The difficulty with this is that we previously used the `dirs`
crate to reliably find the `data_dir()` within the user's home
directory. It is difficult to use crates from a standalone script like
`build.rs` and without a reliable way to access the user's home
directory, it seemed easiest to just copy the built files to
`engines/wasmtime`.

* ci: use the `build.rs` script

Instead of using the `build-engine` command (now removed) or manually
building the engine library, use the `build.rs` script throughout the CI
workflow files.

* fix: check that alternate engine exists
  • Loading branch information
abrown authored May 24, 2022
1 parent 58f7bc2 commit cd8c11b
Show file tree
Hide file tree
Showing 16 changed files with 314 additions and 477 deletions.
32 changes: 7 additions & 25 deletions .github/workflows/sightglass.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,37 +41,19 @@ jobs:
with:
toolchain: nightly

- name: Download Cached Wasmtime Bench API
- name: Download Cached Wasmtime engine
uses: actions/cache@v2
id: wasmtime-cache
with:
path: wasmtime/target/release/*wasmtime_bench_api.*
path: engines/wasmtime/*
key: wasmtime-${{ runner.os }}-${{ env.WASMTIME_COMMIT }}

- name: Clone Wasmtime
- name: Build Wasmtime engine
if: steps.wasmtime-cache.outputs.cache-hit != 'true'
uses: actions/checkout@v2
with:
repository: "bytecodealliance/wasmtime"
ref: ${{ env.WASMTIME_COMMIT }}
submodules: recursive
path: wasmtime

- name: Build Wasmtime Bench API
if: steps.wasmtime-cache.outputs.cache-hit != 'true'
run: cargo build --manifest-path wasmtime/crates/bench-api/Cargo.toml --release

- name: Configure Wasmtime Bench API (linux)
if: runner.os == 'Linux'
run: echo "SIGHTGLASS_TEST_ENGINE=$(pwd)/wasmtime/target/release/libwasmtime_bench_api.so" >> $GITHUB_ENV

- name: Configure Wasmtime Bench API (macos)
if: runner.os == 'macOS'
run: echo "SIGHTGLASS_TEST_ENGINE=$(pwd)/wasmtime/target/release/libwasmtime_bench_api.dylib" >> $GITHUB_ENV

- name: Configure Wasmtime Bench API (windows)
if: runner.os == 'Windows'
run: echo "SIGHTGLASS_TEST_ENGINE=$(pwd)\\wasmtime\\target\\release\\wasmtime_bench_api.dll" >> $env:GITHUB_ENV
working-directory: ./engines/wasmtime
run: |
rustc build.rs
./build
- name: Build all
run: cargo +nightly build --verbose --all
Expand Down
15 changes: 11 additions & 4 deletions benchmarks-next/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,20 @@
set -e
PROJECT_DIR=$(dirname "$0" | xargs dirname)
SIGHTGLASS="cargo +nightly run --bin sightglass-cli --"
ENGINE=$PROJECT_DIR/engines/wasmtime/libengine.so
export RUST_LOG=debug

$SIGHTGLASS build-engine wasmtime
# If an engine is not available, build it.
if [[ ! -f $ENGINE ]]; then
pushd $PROJECT_DIR/engines/wasmtime
rustc build.rs
./build
popd
fi

# Benchmark each Wasm file.
for BENCH_FILE in $(find $PROJECT_DIR/benchmarks-next -name benchmark.wasm); do
BENCH_DIR=$(dirname $BENCH_FILE)
BENCH_NAME=$(basename $BENCH_DIR)

# Run the Wasm benchmark.
$SIGHTGLASS benchmark --engine wasmtime --processes 1 --iterations-per-process 3 --working-dir $BENCH_DIR $BENCH_FILE
$SIGHTGLASS benchmark --engine $ENGINE --processes 1 --iterations-per-process 3 --working-dir $BENCH_DIR $BENCH_FILE
done
38 changes: 0 additions & 38 deletions crates/build/src/artifact.rs

This file was deleted.

193 changes: 0 additions & 193 deletions crates/build/src/engine.rs

This file was deleted.

19 changes: 0 additions & 19 deletions crates/build/src/git.rs

This file was deleted.

50 changes: 8 additions & 42 deletions crates/build/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,49 +1,15 @@
mod artifact;
mod docker;
mod engine;
mod git;
mod metadata;
mod wasm;

use anyhow::Context;
use std::path::PathBuf;

pub use artifact::Artifact;
pub use docker::{DockerBuildArgs, Dockerfile};
pub use engine::*;
pub use git::GitLocation;
pub use metadata::{BuildInfo, BuildSystem, GitSource};
pub use wasm::WasmBenchmark;

/// Get the local directory where sightglass stores cached data.
pub fn sightglass_data_dir() -> anyhow::Result<PathBuf> {
let mut p = dirs::data_local_dir().ok_or_else(|| {
anyhow::anyhow!(
"missing an application data folder for storing sightglass engines; e.g. \
/home/.../.local/share, C:\\Users\\...\\AppData\\Local"
)
})?;
p.push("sightglass");
Ok(p)
}

/// Clean up and remove any cached data.
pub fn clean() -> anyhow::Result<()> {
let sightglass_data_dir = sightglass_data_dir()?;
if !sightglass_data_dir.is_dir() {
return Ok(());
}

log::info!(
"Recursively removing sightglass cached data in {}",
sightglass_data_dir.display()
);
std::fs::remove_dir_all(&sightglass_data_dir).with_context(|| {
format!(
"failed to recursively remove {}",
sightglass_data_dir.display(),
)
})?;

Ok(())
/// Calculate the library name for a sightglass library on the target operating system: e.g.
/// `engine.dll`, `libengine.so`.
pub fn get_engine_filename() -> String {
format!(
"{}engine{}",
std::env::consts::DLL_PREFIX,
std::env::consts::DLL_SUFFIX
)
}
Loading

0 comments on commit cd8c11b

Please sign in to comment.