Add support for complete out of tree build

[gonzoua]

Description

This pull requests adds support for the complet out-of-tree builds. Although the out-of-tree builds were supported by the build utility for some time, the BaseTools part still created files in-tree. This patchset addresses this blocker by:

• Moving the BaseTools interim files and binaries to $WORKSPACE directory
• Making BaseTool/Tests use system's tmp directory for its interim files

Having full out-of-tree builds is convenient because it allows integrating EDK2 builds in a system that enforces read-only access to the codebase to prevent cross-target build contamination.

• Breaking change?
  • Breaking change - Does this PR cause a break in build or boot behavior?
  • Examples: Does it add a new library class or move a module to a different repo.
• Impacts security?
  • Security - Does this PR have a direct security impact?
  • Examples: Crypto algorithm change or buffer overflow fix.
• Includes tests?
  • Tests - Does this PR include any explicit test code?
  • Examples: Unit tests or integration tests.

How This Was Tested

Normal build env test

host$ docker run --rm -ti -v "${HOME}":"${HOME}" \
    -e EDK2_DOCKER_USER_HOME="${HOME}" 
    -v $(pwd):/host -w /host \
    --name edk2-ubuntu-dev
edk2-ubuntu-dev$ source edksetup.sh
edk2-ubuntu-dev$ make -j -C BaseTools
edk2-ubuntu-dev$ export PACKAGES_PATH=$(pwd)
edk2-ubuntu-dev$ build -t GCC5 -a AARCH64 -p ArmVirtPkg/ArmVirtQemu.dsc

Read-only codebase builds test

host$ docker run --rm -ti -v "${HOME}":"${HOME}" \
    -e EDK2_DOCKER_USER_HOME="${HOME}" 
    -v $(pwd):/host:ro -w /host \
    --name edk2-ubuntu-dev
edk2-ubuntu-dev$ export WORKSPACE=/tmp/workspace
edk2-ubuntu-dev$ mkdir -p $WORKSPACE/Conf
edk2-ubuntu-dev$ export EDK_TOOLS_PATH=$(pwd)/BaseTools
edk2-ubuntu-dev$ source edksetup.sh
edk2-ubuntu-dev$ make -j -C BaseTools
edk2-ubuntu-dev$ export PACKAGES_PATH=$(pwd)
edk2-ubuntu-dev$ build -t GCC5 -a AARCH64 -p ArmVirtPkg/ArmVirtQemu.dsc

Integration Instructions

N/A

[niruiyu]

If I understand correctly, you are fixing the BaseTools binary build issue which generates the obj/exe files under BaseTools directory. I like it very much.

Can you explain a bit more about where the obj/exe files will be with your patch?

[gonzoua]

The executables are placed in $WORKSPACE/BaseTools/bin and the obj files are placed in the mirrored dir structure in $WORKSPACE, for instance obj files for BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.c is $WORKSPACE/BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.o

[leiflindholm]

This does not look obviously related to the described change?
If it is general cleanup to make the end result tidier, could it be broken out into a separate commit?

[gonzoua]

It's general cleanup, added as a separate commit.

[niruiyu]

The executables are placed in $WORKSPACE/BaseTools/bin and the obj files are placed in the mirrored dir structure in $WORKSPACE, for instance obj files for BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.c is $WORKSPACE/BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.o

I thought usually $WORKSPACE points to the project root directory. Then your approach that places the intermediate files in $WORKSPACE might not work. Why not place these intermediate files in Build directory?

[gonzoua]

The executables are placed in $WORKSPACE/BaseTools/bin and the obj files are placed in the mirrored dir structure in $WORKSPACE, for instance obj files for BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.c is $WORKSPACE/BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.o

I thought usually $WORKSPACE points to the project root directory. Then your approach that places the intermediate files in $WORKSPACE might not work. Why not place these intermediate files in Build directory?

Could you please elaborate? My understanding is WORKSPACE is where the main (non-BaseTools) build places its files, it can be /tmp/workspace for out-of-tree build or it can point to the edk2 root dir. If WORKSPACE points to the project root, it by definition should be writable, and the behavior I described is the same as the one that currently exists.

[niruiyu]

The executables are placed in $WORKSPACE/BaseTools/bin and the obj files are placed in the mirrored dir structure in $WORKSPACE, for instance obj files for BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.c is $WORKSPACE/BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.o

I thought usually $WORKSPACE points to the project root directory. Then your approach that places the intermediate files in $WORKSPACE might not work. Why not place these intermediate files in Build directory?

Could you please elaborate? My understanding is WORKSPACE is where the main (non-BaseTools) build places its files, it can be /tmp/workspace for out-of-tree build or it can point to the edk2 root dir. If WORKSPACE points to the project root, it by definition should be writable, and the behavior I described is the same as the one that currently exists.

You can open edksetup.bat. The logic there assumes the WORKSPACE points to the parent folder of "BaseTools". It's the root of the source code, not the temp directory that holds output files.

[gonzoua]

The executables are placed in $WORKSPACE/BaseTools/bin and the obj files are placed in the mirrored dir structure in $WORKSPACE, for instance obj files for BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.c is $WORKSPACE/BaseTools/Source/C/LzmaCompress/Sdk/C/LzmaDec.o

I thought usually $WORKSPACE points to the project root directory. Then your approach that places the intermediate files in $WORKSPACE might not work. Why not place these intermediate files in Build directory?

Could you please elaborate? My understanding is WORKSPACE is where the main (non-BaseTools) build places its files, it can be /tmp/workspace for out-of-tree build or it can point to the edk2 root dir. If WORKSPACE points to the project root, it by definition should be writable, and the behavior I described is the same as the one that currently exists.

You can open edksetup.bat. The logic there assumes the WORKSPACE points to the parent folder of "BaseTools". It's the root of the source code, not the temp directory that holds output files.

Right, these are default values if neither WORKSPACE nor EDK_TOOLS_PATH are defined. But it's not a strict requirement, as far as I understand. They can be specified them to different locations. For instance, section 4.1.1 Development Environments sets WORKSPACE to one level up from the edk2 root and uses PACKAGES_PATH to set search path.

Section "Read-only codebase builds test" in the PR's description also demonstrates that this approach works.

[ardbiesheuvel]

I personally always use a workspace that is separate from the edk2/ directory, by setting WORKSPACE to a different directory.

I guess using $WORKSPACE/Build rather than $WORKSPACE for the BaseTools would make this feature useful even in the case where WORKSPACE points to the main repo, so perhaps this is something to consider. (I.e., you can clean the working area by deleting everything under Build/)