Building under Manjaro Linux

[obones]

This ticket is here to discuss what I had to do to get a build working on my Manjaro Linux, on top of the changes that I already submitted as pull requests.
I am not that confident about the changes which is why I want to discuss them here first.

Initial run gives this error:

GitVersion: Process returned an error (exit code 1)

This is fixed by editing Tools\Cake\GitVersion.CommandLine.4.0.0\tools\LibGit2Sharp.dll.config and replacing the os="linux" line with this one:

<dllmap os="linux" dll="git2-15e1193" target="libgit2.so" />

I got a hint of this by looking at this message but I'm not sure this can be fixed by CCNet itself

Then, I had to modify ccnet.build at line 177 because it says mono here. However, doing this, it tries to run a binary called mono from the project folder. The exact error message is this:

'/path/to/checkout/CruiseControl.Net/project/mono' failed to start

I thus replaced mono with ${msbuild.file} just like the line above, but it failed a bit later with msbuild complaining that it was receiving multiple projects to build.
This comes form the fact that there is this line as the first argument for the execution:

<arg value="msbuild.file" if="${platform::is-unix()}"/>

Well, I believe it was intended to be ${msbuild.file} instead of just msbuild.file but in any case, it's not required at all as it gets interpreted as the name of a project to build.
After having removed that arg line, I was able to run the build.sh script to completion.

Any thoughts/opinions on this are welcome

[obones]

Ok, I went further and tried running the tests.
At first, I received this error:

System.Runtime.Remoting.RemotingException : Tcp transport error

Looking around, I found this issue: OmniSharp/omnisharp-roslyn#1548

And sure enough, adding --inprocess to the arguments for nunit at line 266 of ccnet.build allowed the tests to run to completion.

Now, on to figure out why 184 of them failed (91 failures, 93 errors)

[savornicesei]

I'm running latest openSUSE Tumbleweed, 20210113 snapshot with kernel 5.10.5-1 and mono:

Mono JIT compiler version 6.12.0.107 (tarball Mon Dec 28 10:00:47 UTC 2020)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

I have mono-complete pattern installed and I'm running ./build.sh --target=build

Issue 1: NSIS issue from PR #301
Encountered. Fixed by changes in PR #301

Issue 2: GitVersion: Process returned an error (exit code 1)
Not encountered

Issue 3 <arg value="msbuild.file" if="${platform::is-unix()}"/>
You are correct. it was supposed to be ${msbuild.file} because the full command should be mono /usr/lib/mono/msbuild/15.0/bin/MSBuild.dll ccnet.sln

Removing the <arg value="msbuild.file" if="${platform::is-unix()}"/> line gives the following error:

build:

     [echo] Build configuration: Build
     [echo] Source Directory: /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project
     [echo] Build Directory: /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/Build
     [echo] MSBuild path: mono
     [exec] Cannot open assembly '/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/ccnet.sln': File does not contain a valid CIL image.

BUILD FAILED - 0 non-fatal error(s), 1 warning(s)

/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/ccnet.build(180,10):
External Program Failed: mono (return code was 2)

Removing the basedir="${src.dir}" started the build process just to fail a little later with

     [echo] Build configuration: Build
     [echo] Source Directory: /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project
     [echo] Build Directory: /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/Build
     [echo] MSBuild path: mono
     [exec] Microsoft (R) Build Engine version 16.0.0.0 for Mono
     [exec] Copyright (C) Microsoft Corporation. All rights reserved.
     [exec] 
     [exec] Build started 16.01.2021 01:17:48.
     [exec] Project "/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/ccnet.sln" on node 1 (Clean;Build target(s)).
     [exec] ValidateSolutionConfiguration:
     [exec]   Building solution configuration "Build|Any CPU".
     [exec] Project "/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/ccnet.sln" (1) is building "/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/PowerShell/PowerShell.csproj" (2) on node 1 (Clean target(s)).
     [exec] /usr/lib/mono/msbuild/15.0/bin/Microsoft.CSharp.CurrentVersion.targets(331,5): error MSB4019: The imported project "/usr/lib/mono/msbuild/15.0/bin/Roslyn/Microsoft.CSharp.Core.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk. [/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/PowerShell/PowerShell.csproj]
     [exec] Done Building Project "/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/PowerShell/PowerShell.csproj" (Clean target(s)) -- FAILED.

This is a problem with RoslynTargetsPath which seems to point to /usr/lib/mono/msbuild/15.0/bin/Roslyn on my system.
It can be fixed by either making a soft link to /usr/lib/mono/msbuild/Current/bin/Roslyn or by sending the right path in MSBuild command line if running on linux:

<arg value="/v:n" /> <!-- verbosity levels: q[uiet], m[inimal], n[ormal] (default), d[etailed], and diag[nostic] -->
<arg value="/p:RoslynTargetsPath=${path::combine(directory::get-parent-directory(msbuild.file), '../../Current/bin/Roslyn')}" if="${platform::is-unix()}" />

Either ways seems to give the same horror:

/usr/lib/mono/msbuild/15.0/bin/Roslyn/Microsoft.Managed.Core.targets(60,27): error MSB4184: The expression ""../CommonAssemblyInfo.cs".GetPathsOfAllDirectoriesAbove()" cannot be evaluated. Method 'System.String.GetPathsOfAllDirectoriesAbove' not found. [/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/project/WebDashboard/WebDashboard.csproj]
     [exec] 

There are some small additional adjustments to be done here: echo lines from line 170 to 174 should be moved at line 178 and last echo line should be <echo message="MSBuild.path: ${msbuild.file}" />

[obones]

Removing the line gives the following error

Thing is, this arg must be removed when ${msbuild.file} is also replaced to be the value returned by vswhere which on my system is /usr/bin/msbuild and automatically calls mono with the appropriate DLL. So in this case, there is no need to call mono directly.
On top of this, there does not seem to be a search in $PATH for the executable, so using just mono leads to it being searched for in the current folder. And because it is not there, I get the error mentioned above.

Clearly, it appears that depending on the distribution, some things are available and some others are not.

[savornicesei]

Hmmm...
It would not be the first time things are packaged different in openSUSE (thinking at libICU). Seems I need to test the changes on several distros. I'll check them out in the following days.

[savornicesei]

@obones
framework::get-msbuild-path(vswhere.executable)
can't return /usr/bin/msbuild since it checks hardcoded paths in /usr/lib (check Tools/ccnet-utils.inc.build - which follows the same logic as NuGet client in detecting MSBuild on linux).
If it's not too much trouble for you could you paste the script into a console app and check how it returns /usr/bin/msbuild?

L.E: Seems like my mono environment was broken. I was using the official mono-project mono packages and at some moment in time I switched to the openSUSE-provided ones without switching and upgrading msbuild, msbuild-libhostfxr and msbuild-sdkresolver to the ones provided by Mono:Factory.

Issue 3 from previous comment works by removing basedir="${src.dir}"

I'm thinking that perhaps we should probe first for msbuild in $PATH and if it does not exist then we call mono MSBuild.dll ccnet.sln
What do you think?

The tests fail for me (on a local branch containing your PRs #300, #304, #305, #306) with the following error:

/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/ccnet.build(245,18):
The fileset specified is empty after scanning '/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/Tools/NUnit' for: NAnt.Core.StringCollectionWithGoodToString:
nunit.framework.dll

[obones]

Well, ok, here is the beginning of the log when building:

 [echo] MSBuild path: /files/download/Linux/CruiseControl.Net/Build
 [echo] MSBuild exe: /usr/lib/mono/4.5/../msbuild/15.0/bin/MSBuild.dll
 [exec] Starting '/usr/lib/mono/4.5/../msbuild/15.0/bin/MSBuild.dll (/files/download/Linux/CruiseControl.Net/project/ccnet.sln /t:Clean;Build /p:TargetFrameworkVersion=4.5;Configuration=Build /p:WarningsNotAsErrors=619 /p:SHFBROOT=/files/download/Linux/CruiseControl.Net/Tools/EWSoftware.SHFB.2019.11.17/tools /p:DISABLE_COM=True /p:DISABLE_JSCRIPT=True /p:MONO_IOMAP=all /p:MONO_PATH=/files/download/Linux/CruiseControl.Net/lib)' in '/files/download/Linux/CruiseControl.Net/project'
 [exec] Microsoft (R) Build Engine version 16.8.0-ci-20621-01 pour Mono
 [exec] Copyright (C) Microsoft Corporation. Tous droits réservés.

So clearly, I was wrong about /usr/bin/msbuld it's just that the DLL can be started all by itself, there is no need to put mono in front of it. How and why this works is a mystery to me.

But there is one thing for sure, using just mono as the command line does not work at all because it looks for a mono executable in the current folder, and there is none to be found. The log in that case is this:

     [echo] MSBuild path: /files/download/Linux/CruiseControl.Net/Build
     [echo] MSBuild exe: mono
     [exec] Starting '/files/download/Linux/CruiseControl.Net/project/mono (/files/download/Linux/CruiseControl.Net/project/ccnet.sln /t:Clean;Build /p:TargetFrameworkVersion=4.5;Configuration=Build /p:WarningsNotAsErrors=619 /p:SHFBROOT=/files/download/Linux/CruiseControl.Net/Tools/EWSoftware.SHFB.2019.11.17/tools /p:DISABLE_COM=True /p:DISABLE_JSCRIPT=True /p:MONO_IOMAP=all /p:MONO_PATH=/files/download/Linux/CruiseControl.Net/lib)' in '/files/download/Linux/CruiseControl.Net/project'

BUILD FAILED

/files/download/Linux/CruiseControl.Net/ccnet.build(182,10):
'/files/download/Linux/CruiseControl.Net/project/mono' failed to start.
    ApplicationName='/files/download/Linux/CruiseControl.Net/project/mono', CommandLine='/files/download/Linux/CruiseControl.Net/project/ccnet.sln /t:Clean;Build /p:TargetFrameworkVersion=4.5;Configuration=Build /p:WarningsNotAsErrors=619 /p:SHFBROOT=/files/download/Linux/CruiseControl.Net/Tools/EWSoftware.SHFB.2019.11.17/tools /p:DISABLE_COM=True /p:DISABLE_JSCRIPT=True /p:MONO_IOMAP=all /p:MONO_PATH=/files/download/Linux/CruiseControl.Net/lib', CurrentDirectory='/files/download/Linux/CruiseControl.Net/project', Native error= Cannot find the specified file

This is why I just use the ${msbuild.file} value directly.

[savornicesei]

So just removing the basedir attribute from exec does not work for you?

[obones]

Hum... I did not understand your suggestion, but now it makes sense. Here is what I have:

    <property name="msbuild.exe" value="mono" if="${platform::is-unix()}" />
    <exec program="${msbuild.exe}" workingdir="${src.dir}" verbose="${nant.verbosity}">
        <arg value="${msbuild.file}" if="${platform::is-unix()}"/>

And this works fine indeed.

The tests fail for me (on a local branch containing your PRs #300, #304, #305, #306) with the following error:

There is this comment at the top of these actions:

Hack to copy all required files into the UnitTest dictionary until xbuild on Mono 2.5 is fixed

So I have disabled this entire section and that allowed to run the build script.

[obones]

Just so you know, with today's latest changes in #304 and #305, I have the following result:

 [exec] Test Run Summary
 [exec]   Overall result: Warning
 [exec]   Test Count: 2994, Passed: 2902, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 92
 [exec]     Skipped Tests - Ignored: 69, Explicit: 6, Other: 17
 [exec]   Start time: 2021-01-23 18:06:39Z
 [exec]     End time: 2021-01-23 18:08:01Z
 [exec]     Duration: 82.401 seconds

So all is good, the tests are passing!

[savornicesei]

Reopening this since running ./build.sh --target=run-tests is unsuccesful on my openSUSE (20210215 snapshot):
`[exec] => ThoughtWorks.CruiseControl.UnitTests.Core.Tasks.NDependTaskTests.ExecuteWillCreateANewDirectory
[exec] [NonParallelWorker:INFO] Task output:
[exec] => ThoughtWorks.CruiseControl.UnitTests.Core.Tasks.NDependTaskTests.RunningOnExistingDirectoryChecksFilesAndCopiesWhenNewer
[exec] [NonParallelWorker:INFO] Task output:
[exec]
[exec] Errors, Failures and Warnings
[exec]
[exec] 1) Error :
[exec] System.Runtime.Remoting.RemotingException : Tcp transport error.
[exec] ----> System.Runtime.Remoting.RemotingException : Connection closed
[exec] --RemotingException
[exec] Tcp transport error.
[exec]
[exec]
[exec] Server stack trace:
[exec] at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x00017] in <17677762ab73446d9bcbff2e572bbfd2>:0
[exec] at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x0005e] in <17677762ab73446d9bcbff2e572bbfd2>:0
[exec] at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg) [0x00066] in <17677762ab73446d9bcbff2e572bbfd2>:0
[exec]
[exec] Exception rethrown at [0]:
[exec] at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke (System.Runtime.Remoting.Proxies.RealProxy rp, System.Runtime.Remoting.Messaging.IMessage msg, System.Exception& exc, System.Object[]& out_args) [0x0014d] in <5e0e0d8da8ee40bd894ec0077d8958ce>:0
[exec] at (wrapper managed-to-native) System.Object.__icall_wrapper_mono_remoting_wrapper(intptr,intptr)
[exec] at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent.Run(NUnit.Engine.ITestEventListener,NUnit.Engine.TestFilter)
[exec] at NUnit.Engine.Runners.ProcessRunner.RunTests (NUnit.Engine.ITestEventListener listener, NUnit.Engine.TestFilter filter) [0x00025] in :0
[exec] --
[exec] RemotingException
[exec] Connection closed
[exec] at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.StreamRead (System.IO.Stream networkStream, System.Byte[] buffer, System.Int32 count) [0x00011] in <17677762ab73446d9bcbff2e572bbfd2>:0
[exec] at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x00000] in <17677762ab73446d9bcbff2e572bbfd2>:0
[exec]
[exec] Test Run Summary
[exec] Overall result: Failed
[exec] Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
[exec] Start time: 2021-02-18 07:07:04Z
[exec] End time: 2021-02-18 07:07:46Z
[exec] Duration: 41.677 seconds
[exec]
[exec] Results (nunit3) saved as /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/BuildMetrics/nunit-result.xml
[exec]
[exec] NUnit.Engine.NUnitEngineUnloadException : Agent Process was terminated successfully after error.
[exec] ----> System.Runtime.Remoting.RemotingException : Connection refused

BUILD FAILED

/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/ccnet.build(267,10):
External Program Failed: /data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/Tools/NUnit.ConsoleRunner.3.10.0/tools/nunit3-console.exe (return code was 156)

Total time: 64.5 seconds.

Nant: runTests target exit code: 1
An error occurred when executing task 'run-tests'.
`

[obones]

[exec] 1) Error :
[exec] System.Runtime.Remoting.RemotingException : Tcp transport error.
[exec] ----> System.Runtime.Remoting.RemotingException : Connection closed
[exec] --RemotingException
[exec] Tcp transport error.

Could it be a transient error?
It's very strange that TCP transport would fail just for one test.

[savornicesei]

It seems to be a nunit agent error.
Passing --inprocess to nunit console allowed it to finish running the CC tests. It reported 15 failed tests from which 4 are related to Windows registry (Core.Util.RegistryTest.cs):

Expected: <ThoughtWorks.CruiseControl.Core.CruiseControlException>
[exec] But was: <System.Security.SecurityException: No access to the given key ---> >System.UnauthorizedAccessException: Access to the path '/etc/mono/registry/LocalMachine' is denied. ---> System.IO.IOException: Permission denied

I think we should mark all tests in RegistryTest as unavailable on linux.

[obones]

Well, I excluded the only one that as causing me issues, but maybe the entire test suite should be excluded indeed.

What are the other 11 failing tests?

[savornicesei]

@obones
Besides the registry ones, several failed because of hostname (fixed by editing the \etc\hosts file) and 5 are BitKeeper tests:

System.ArgumentException : Item has already been added. Key in dictionary: 'PAGER'  Key being added: 'PAGER'
     [exec] TearDown : Moq.MockException : Mock<ProcessExecutor:22>:
     [exec] This mock failed verification due to the following:
     [exec] 
     [exec]    ProcessExecutor executor => executor.Execute(FileName: [C:\Program Files\BitKeeper\bk.exe] -- Arguments: [changes -R] -- WorkingDirectory: [/data/Proiecte/OSS/CruiseControl.NET/CruiseControl.NET/Build/UnitTests/source] -- StandardInputContent: [] -- Timeout: [600000]):
     [exec]    This setup was not matched.
     [exec]   at System.Collections.Hashtable.Insert (System.Object key, System.Object nvalue, System.Boolean add) [0x001b6] in <5e0e0d8da8ee40bd894ec0077d8958ce>:0 
     [exec]   at System.Collections.Hashtable.Add (System.Object key, System.Object value) [0x00000] in <5e0e0d8da8ee40bd894ec0077d8958ce>:0 
     [exec]   at System.Collections.Specialized.CaseSensitiveStringDictionary.Add (System.String key, System.String value) [0x0000e] in <83dd749384734033afca92f4cfac782c>:0 
     [exec]   at ThoughtWorks.CruiseControl.Core.Sourcecontrol.BitKeeper.BitKeeper.NewProcessInfo (System.String args, ThoughtWorks.CruiseControl.Core.IIntegrationResult result) [0x00024] in <12662c0a498740488db62ae2613a3585>:0 
     [exec]   at ThoughtWorks.CruiseControl.Core.Sourcecontrol.BitKeeper.BitKeeper.GetModifications (ThoughtWorks.CruiseControl.Core.IIntegrationResult from, ThoughtWorks.CruiseControl.Core.IIntegrationResult to) [0x00008] in <12662c0a498740488db62ae2613a3585>:0 
     [exec]   at ThoughtWorks.CruiseControl.UnitTests.Core.Sourcecontrol.Bitkeeper.BitKeeperTest.VerifyGetModificationsCommandNonVerbose () [0x00029] in <bdf3ea100b484e02a4272164684d0a4a>:0 
     [exec]   at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
     [exec]   at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <5e0e0d8da8ee40bd894ec0077d8958ce>:0 
     [exec] --TearDown
     [exec]   at Moq.Mock.Verify () [0x00012] in <d623253e7ec945729823df3f9cdd649f>:0 
     [exec]   at ThoughtWorks.CruiseControl.UnitTests.Core.ProcessExecutorTestFixtureBase.Verify () [0x00000] in <bdf3ea100b484e02a4272164684d0a4a>:0 
     [exec]   at ThoughtWorks.CruiseControl.UnitTests.Core.Sourcecontrol.Bitkeeper.BitKeeperTest.VerifyAll () [0x00000] in <bdf3ea100b484e02a4272164684d0a4a>:0 
     [exec]   at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
     [exec]   at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <5e0e0d8da8ee40bd894ec0077d8958ce>:0 

It should not fail even if the BitKeeper path is a Windows path. Not sure about that Key in collection error since I cannot build the ccnet.sln due to jscript and speech refs - but there is a way to ignore them using compilation flags and a Directory.Build.props - I just have to remember how I did it 😀

[savornicesei]

As a side note, there's a linux version of BitKeeper.

[obones]

I believe the "setup was not matched" is because of the previous exception that prevents the code that would use the mock from ever running.