Adjustments for FreeBSD

[joperator]

Although not yet officially supported, there's still a need for Azure Pipelines Agent for FreeBSD, as described here.

This PR contains adjustments to build and run Azure Pipelines Agent on FreeBSD:

• Most of the adjustments in the .cs and .sh files are made to account for FreeBSD where Linux is already accounted for.
• I've added a DaemonControlManager.cs for FreeBSD, according to the control managers for other platforms.
• The adjustment in Common.props is already supported by dotnet CLI, but not by Visual Studio, causing all projects that import Common.props to fail to load with error : Invalid static method invocation syntax: "[System.Runtime.InteropServices.OSPlatform]::FreeBSD". Method 'System.Runtime.InteropServices.OSPlatform.FreeBSD' not found.
  This should be fixed by Microsoft at some point.
• I've added daemon.svc.sh.template and vsts.agent.daemon.template to run Azure Pipelines Agent as a daemon on FreeBSD, since systemd is not supported on this platform.

With this PR you should be able to build Azure Pipelines Agent on FreeBSD using the following workflow:

• Install dependencies:
  pkg install bash curl git icu libinotify libunwind node
• In src/dev.sh change DOTNETSDK_VERSION to the version of the .NET SDK that you're using on FreeBSD.
  The .NET SDK must be able to create self-contained apps for FreeBSD.
• In src/Common.props change RuntimeFrameworkVersion to the version of the NuGet package Microsoft.NETCore.App.Runtime.freebsd-x64 that is available to you.
• Add the package source (can also be a local feed) containing Microsoft.NETCore.App.Runtime.freebsd-x64.*.*.*.nupkg to all NuGet.Config files in the repo.
• Customize and run the following script from the root of the repo:

#!/bin/sh

script_path="$(cd "$(dirname "${0}")" && pwd)"
sdk_version="ENTER_DOTNETSDK_VERSION_HERE"
sdk_path="${script_path}/_dotnetsdk"
version_path="${sdk_path}/${sdk_version}"

if [ ! -f /bin/bash ]; then
    ln -s /usr/local/bin/bash /bin/bash
fi

if [ -d "${sdk_path}" ]; then
    rm -r "${sdk_path}"
fi

mkdir "${sdk_path}"
ln -s ENTER_DOTNETSDK_LOCATION_HERE "${version_path}"
echo "${sdk_version}" > "${version_path}/.${sdk_version}"

src_path="${script_path}/src"

cd "${src_path}"
"${src_path}/dev.sh" layout Release
"${src_path}/dev.sh" build Release
"${src_path}/dev.sh" test Release

• The agent binaries should now be available under _layout/freebsd-x64.

[ghost]

All CLA requirements met.

[Thefrank]

Third times a charm? I know that @wfurt and @jasonpugsley have both tried to get support added in with no luck. Anyways...

This does build after ignoring errors for CA2000 and CA1711. I did not use the above script however.
Tests pass too.
I really should've turned off CA before building instead of just ignoring the CA#### errors

[davidchisnall]

Awesome to see some progress here! There are a lot of places that are 'if we're on Linux, or FreeBSD, or macOS', which really mean 'if we're on some POSIX platform'. It's generally a good idea when writing portable software to start with the lowest common denominator and then add specialisations. It would be great to factor these out into an isPosix check that is set for everything non-Windows. We can then special case thing for specific systems as required. For example, the case-sensitive filesystem check should be 'is POSIX and not macOS'. That way, the next person who comes along and wants to add support for NetBSD / OpenBSD / Solaris / Haiku / whatever just has to deal with the places where their platform diverges from POSIX, rather than cluttering up the code further.

[arrowd]

Awesome to see some progress here! There are a lot of places that are 'if we're on Linux, or FreeBSD, or macOS', which really mean 'if we're on some POSIX platform'.

I'd prefer "Unix-like platform" or something like that. After spending years of work in FreeBSD ports I learned that POSIX is actually a pretty strict term, which shouldn't be used so lightly. In the Linux world there are APIs that are claiming to be POSIX-compliant, but they actually have various subtle specifics that do not match FreeBSD.

I believe, the checks should use either if(FreeBSD), if(POSIX) or if(UnixLike) depending on the context.

[Thefrank]

Without getting too into depth about this, if we want POSIX compliance we are going to have to define which revision of POSIX. The newest is, uh, this one? Which revision should be explicitly named somewhere too

While is(UnixLike) might be preferred as an ease of coding item I think is(POSIX) would be better for a shipping product. HOWEVER, is(POSIX) for everything that non-windows might require a large refactor. is(UnixLike) would require a growing number of exceptions (e.g., FreeBSD does not ship with sudo or bash and does not use systemd, most but not all Linux distros ship with systemd, MacOS enjoys doing its own thing) but allows for an easier drop-in addition of a platform (like this pull request).

[joperator]

I agree with both of you, @arrowd and @Thefrank:
As you've already noted, there are a lot of places where a new platform needs to be added. But instead of checking the platform, how about checking capabilities that are managed in a central location? I'm thinking of the following in a new Capabilities.cs file:

public static class Capabilities
{
    public static bool IsCaseSensitive
    {
        get => RuntimeInformation.IsOSPlatform(OSPlatform.Linux) || RuntimeInformation.IsOSPlatform(OSPlatform.FreeBSD);
    }

    public  static bool SupportsSystemD
    {
        get => RuntimeInformation.IsOSPlatform(OSPlatform.Linux);
    }

    ...
}

We already know capabilities in another context from Azure Pipelines Agent, so why not use this concept in source code as well?
Imagine you want to build a cross-platform .NET binary and you have a Windows, Linux, and FreeBSD agent. Then it doesn't matter what OS the chosen agent has, only whether it is capable of building a cross-platform .NET binary, because the created binaries will run on all these OSes.
In azure-pipelines-agent source code, it often also doesn't really matter whether it runs on Windows, Linux, FreeBSD, Unix-like systems, or whatever, since usually only the capabilities (case sensitivity, systemd, sudo, bash, ...) of the current system are important. Having them all in a single file would make adding new platforms much easier, because then you would only have to check for each capability whether the new platform has it or not, without having to edit multiple files across the repo.

However, this sounds like a useful but also big change that would blow up this PR. Maybe we can keep an eye on this in the future and create a separate PR for it?

[Thefrank]

Any update on this?
What else is this missing to get it merged in?

[joperator]

Everything is ready from my side. Do you have this PR in view, @davidchisnall and @wfurt?

[davidchisnall]

All of my comments are addressed and I'm happy but I'm not a maintainer on (or even contributor to) this repo, just a bystander with a vested interest in seeing Azure Pipelines support FreeBSD.

[Thefrank]

its been about 5 months. I think this is good for another poke.

[Thefrank]

@joperator
oh! I have actually run into a bug when trying to use DownloadPipelineArtifact@2

The item it was processing:

      - task: DownloadPipelineArtifact@2
        inputs:
          buildType: 'current'
          artifactName: 'RuntimePackages'
          itemPattern: 'Microsoft.*.freebsd-x64.*.nupkg'
          targetPath: '$(Build.SourcesDirectory)/runtime/artifacts/packages/Release/Shipping'

The Error:

2021-10-12T06:25:01.5244949Z Verbose, [Downloading 0DBF76F48E2CBE205EC9D01B90AF1C54877975EEBD5CCB944702A157D4FEC1FE02 to /usr/home/azureagent/vsts/_work/5/s/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Host.freebsd-x64.6.0.0-rc.1.21451.13.symbols.nupkg.] Try 1/3, non-retryable exception caught. Throwing. Details:
2021-10-12T06:25:01.5247410Z No LastRequestResponse on exception InvalidOperationException: Invalid Operating System. System.InvalidOperationException: Invalid Operating System.
2021-10-12T06:25:01.5248966Z    at Microsoft.VisualStudio.Services.Content.Common.FileLink.CreateHardLinkInternal(String existingFileName, String newFileName)
2021-10-12T06:25:01.5250265Z    at Microsoft.VisualStudio.Services.Content.Common.FileLink.CreateHardLink(String existingFileName, String newFileName)
2021-10-12T06:25:01.5251804Z    at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.DownloadToFileAsync(DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
2021-10-12T06:25:01.5253663Z    at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
2021-10-12T06:25:01.5256732Z    at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupIdentifier dedupId, String fullPath, UInt64 fileSize, GetDedupAsyncFunc dedupFetcher, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
2021-10-12T06:25:01.5258643Z    at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.<>c__DisplayClass25_2.<<DownloadAsyncWithManifestPath>b__9>d.MoveNext()
2021-10-12T06:25:01.5260972Z --- End of stack trace from previous location ---
2021-10-12T06:25:01.5262149Z    at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper.<>c__DisplayClass11_0.<<CreateTaskWrapper>b__0>d.MoveNext()
2021-10-12T06:25:01.5264326Z --- End of stack trace from previous location ---
2021-10-12T06:25:01.5265421Z    at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)

[joperator]

@Thefrank
I am not very familiar with the DownloadPipelineArtifact task as I only use Azure DevOps Server on-premises and therefore have always used the DownloadBuildArtifacts task. When I use the DownloadPipelineArtifact task, I get the following error on Windows, Linux and FreeBSD, which is why I can't reproduce the actual problem:

##[error]Pipeline Artifact Task is not supported in on-premises. Please use Build Artifact Task instead.

However, your exception does not come directly from the Azure Pipelines Agent itself, but from the referenced vss-api-netcore package, which contains Microsoft.VisualStudio.Services.Content.Common.dll. I followed the stack trace of your exception and have found the root cause of the problem in the Microsoft.VisualStudio.Services.Content.Common.FileLink.CreateHardLinkInternal(String existingFileName, String newFileName) method, which only checks for Windows, OSX and Linux and throws the exception you described for all other platforms. Since the vss-api-netcore package does not seem to be open source, it's up to Microsoft to fix this and release an updated version of the package to support FreeBSD now.

cc: @wfurt

[wfurt]

yah. I think the relevant code lives outside of this repo. I'm yet to trace the real source.

[Thefrank]

DownloadBuildArtifacts throws the same error for me. Likely the same place (looks like you found it in azure-pipelines-task-lib)

[joperator]

Yes, but keep in mind that these are two errors in two different places:

• Invalid Operating System is in vss-api-netcore.
• Platform not supported: freebsd is in azure-pipelines-task-lib, which will be fixed by my PR 799.

[Thefrank]

Patching Microsoft.VisualStudio.Services.Content.Common.dll and replacing it with the one the agent uses fixes the issue for me.

The file looks like it comes from Microsoft Azure DevOps Server. Closed source :(

[joperator]

Any updates on fixing this issue in Microsoft.VisualStudio.Services.Content.Common.dll, @wfurt?

[Thefrank]

Thanks for maintaining this PR!

Also: Happy 2yr anniversary to this PR!

[sec]

Getting close to 3yr :)

[joperator]

Getting close to 3yr :)

@sec
Yes, maybe I'll bake a birthday cake for it next week. 🎂

With the lang/dotnet port for FreeBSD in hand, perhaps someone is more willing to merge this and microsoft/azure-pipelines-task-lib#799 before the big party starts?
Maybe @wfurt will add them to his saga? 😉

[Thefrank]

New tests that need skipping as they are also skipped on osx and linux:

azure-pipelines-agent/src/Test/L0/Worker/Handlers/ProcessHandlerL0.cs

Line 27 in abbe484

public async void ProcessHandlerV2_BasicExecution()

azure-pipelines-agent/src/Test/L0/Worker/Handlers/ProcessHandlerL0.cs

Line 129 in abbe484

public async void ProcessHandlerV2_Validation_passes()

azure-pipelines-agent/src/Test/L0/Worker/Handlers/ProcessHandlerL0.cs

Line 73 in abbe484

public async void ProcessHandlerV2_FileExecution()

azure-pipelines-agent/src/Test/L0/Util/ProcessUtilL0.cs

Line 60 in abbe484

public void Test_GetParentProcessId_ViaInterop()

azure-pipelines-agent/src/Test/L0/Util/ProcessUtilL0.cs

Line 80 in abbe484

public void Test_GetParentProcess_ViaInterop()

This should give a clean test.

[joperator]

@Thefrank Thanks for your advice.
These tests and some Git tests, which are also skipped on OSX and Linux, are now skipped on FreeBSD as well.

[Thefrank]

You might want to add export PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
into src/Misc/layoutbin/vsts.agent.daemon.template
as the behavior of daemon now matches the documentation.

[arrowd]

The /usr/local part should not be hardcoded but computed from sysctl:

LOCALBASE=$(sysctl -n user.localbase)
linkExternalTool "${LOCALBASE}/bin/node" node/bin node

[joperator]

The /usr/local part is now computed from sysctl.

[joperator]

You might want to add export PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin" into src/Misc/layoutbin/vsts.agent.daemon.template as the behavior of daemon now matches the documentation.

@Thefrank
It's not entirely clear to me what the advantage of exporting the PATH here is. What was the previous behavior of daemon?

[Thefrank]

@joperator oops! my mistake was saying daemon not service >.<

export PATH is not a requirement for Azure Pipelines to run, but it is helpful.

SHORT: if the /usr/local/bin /usr/local/sbin are not in PATH. Programs that use PATH to find other programs (e.g., a build system that also uses python) will fail. service does not include these.

(too?) LONG:

This was only discovered with FreeBSD 14.0 as earlier versions seem to have behavior inconsistent with the documentation and code.

https://man.freebsd.org/cgi/man.cgi?query=service&apropos=0&sektion=8&manpath=FreeBSD+14.1-RELEASE+and+Ports&arch=default&format=html#ENVIRONMENT

https://github.com/freebsd/freebsd-src/blob/10e31f0946d820d53adc58b7d013b969e4a9a8ed/usr.sbin/service/service.sh#L177

if anything from /usr/local/bin or /usr/local/sbin is needed and the application uses PATH to try and find it, it will fail.
This hits the *arr dotNET applications when they look for binaries and jellyfin when trying to find ffmpeg. For Azure Pipelines, this will impact its ability to use build systems that require things like python that are installed to /usr/local and found via PATH.

Under FreeBSD 13 these programs had no issues despite the documentation stating that these additional local items would not be on PATH.

Bug from jellyfin: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277161

AFAIK This (FreeBSD 14.0) version is the correct behavior as it matches the code and documentation.

[joperator]

@Thefrank
If I understand correctly, /usr/local/bin and /usr/local/sbin are only for services not in PATH. If so, it looks like this doesn't matter for the Azure Pipelines Agent as the PATH variable is already exported in runsvc.sh. The .path file used there is created by env.sh, which is called by config.sh during agent setup, which usually isn't done by a service.

[littleq0903]

I am trying to get this to build and run recently, where would be the feed containing Microsoft.NETCore.App.Runtime.freebsd-x64...*.nupkg

thanks in advance

[joperator]

@littleq0903
Well, there isn't an official feed that contains those required NuGet packages. However, you can download them here. Alternatively, you can download a .NET SDK here or install the dotnet package for FreeBSD, which should contain the required NuGet package contents in their packs directory.

[sec]

I also try to maintain up-to-date private feed you can use