[WIP] Provide Support for Linux #36

RLovelett · 2018-02-02T13:58:04Z

Infraestructure

Update the build tools and CI

Code

Ogra Error Message

/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:95:128: warning: redundant same-type constraint 'Wrapped.Iterator.Element' == 'Wrapped.Element'
extension Optional where Wrapped: Collection & ExpressibleByArrayLiteral, Wrapped.Element: Encodable, Wrapped.Iterator.Element == Wrapped.Element {
                                                                                                                               ^
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:95:33: note: previous same-type constraint 'Wrapped.Element' == 'Wrapped.Iterator.Element' implied here
extension Optional where Wrapped: Collection & ExpressibleByArrayLiteral, Wrapped.Element: Encodable, Wrapped.Iterator.Element == Wrapped.Element {
                                ^
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:89:102: warning: redundant conformance constraint 'Self.Element': 'Encodable'
extension Collection where Self: ExpressibleByArrayLiteral, Self.Element: Encodable, Iterator.Element: Encodable {
                                                                                                     ^
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:89:73: note: conformance constraint 'Self.Element': 'Encodable' written here
extension Collection where Self: ExpressibleByArrayLiteral, Self.Element: Encodable, Iterator.Element: Encodable {
                                                                        ^
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:36:18: error: cannot convert value of type 'Int' to type 'NSNumber' in coercion
                return .number(self as NSNumber)
                               ^~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:42:18: error: cannot convert value of type 'Double' to type 'NSNumber' in coercion
                return .number(self as NSNumber)
                               ^~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:48:18: error: cannot convert value of type 'Float' to type 'NSNumber' in coercion
                return .number(self as NSNumber)
                               ^~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:54:18: error: cannot convert value of type 'UInt' to type 'NSNumber' in coercion
                return .number(self as NSNumber)
                               ^~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:109:29: error: cannot convert value of type 'Int' to type 'NSNumber' in coercion
        return .number(self.rawValue as NSNumber)
                       ~~~~~^~~~~~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:115:29: error: cannot convert value of type 'Double' to type 'NSNumber' in coercion
        return .number(self.rawValue as NSNumber)
                       ~~~~~^~~~~~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:121:29: error: cannot convert value of type 'Float' to type 'NSNumber' in coercion
        return .number(self.rawValue as NSNumber)
                       ~~~~~^~~~~~~~
/home/rlovelett/Source/langserver-swift/.build/checkouts/Ogra.git-4459770976581387768/Sources/Encodable.swift:127:29: error: cannot convert value of type 'UInt' to type 'NSNumber' in coercion
        return .number(self.rawValue as NSNumber)
                       ~~~~~^~~~~~~~
error: terminated(1): /usr/bin/swift-build-tool -f /home/rlovelett/Source/langserver-swift/.build/debug.yaml main output:

Fix Ogra (Fixed in Update to require Swift 4.1 and resolve warnings and deprecations #38)
Update source code to Swift 4.1 (Fixed in Update to require Swift 4.1 and resolve warnings and deprecations #38)
Update SwiftPM to Swift 4.1 (Fixed in Update to require Swift 4.1 and resolve warnings and deprecations #38)
Cross-platform logging
Get PR that fixes SR-5773 merged into corelibs-libdispatch
Ensure all tests are passing

Documentation

Update the README

felix91gr · 2018-04-07T04:34:56Z

What does waitForDataInBackgroundAndNotify do? Is it related to File Events?

felix91gr · 2018-04-10T05:22:40Z

No, it is actually a part of Foundation and it’s not currently implemented. We’ll have to do something about it, unless the corelibs team surprises us with a patch that implements that.

RLovelett · 2018-04-26T23:12:28Z

swiftlang/swift-corelibs-foundation#1427

The author should stop being lazy and finish the requested feedback. 😏

felix91gr · 2018-04-26T23:14:51Z

Omg, I didn't realise you were working on that! Yes, I agree 😜 but also I think his pace might be different. We should not be pressuring him :)

felix91gr · 2018-05-23T18:02:20Z

How're we doing logging? Because tracelog is an option to have cross-platform logging :)

felix91gr · 2018-05-23T18:06:04Z

PS: I edited the original to-do list so that it's easier to follow, how's it look?

RLovelett · 2018-05-23T20:34:20Z

PS: I edited the original to-do list so that it's easier to follow, how's it look?

LGTM 👍

How're we doing logging? Because tracelog is an option to have cross-platform logging :)

So I will be honest and say I haven't put too much thought into logging. So I'll try and stop this from becoming a stream of consciousness.

My two platforms I daily drive are macOS and Fedora. On macOS I want the logging to be done using Unified logging or os_log and on Fedora I want the logging to be available inside the systemd journal, e.g., journalctl. So does that limit or select the loggers we can use? I have no idea. Do we need to do that on the first iteration of logging? Probably not.

The thing that we absolutely cannot do is write to stdout. That is how the LSP and VSCode talk to one another. So if tracelog outputs to stdout then we'll need a different solution.

Other than that, I think its probably best to just try some out and see what happens.

keegancsmith · 2018-05-23T20:39:47Z

LSP also defines window/logMessage

felix91gr · 2018-05-23T20:40:10Z

LGTM 👍

Awesome :)

The thing that we absolutely cannot do is write to stdout. That is how the LSP and VSCode talk to one another. So if tracelog outputs to stdout then we'll need a different solution.

That seems reasonable.

So looking at TraceLog's Readme (under Configuration), I think we can definitely avoid writing to stdout and possibly configure exactly where do we want it to go depending on the OS:

Log output can be configured globally using the LOG_ALL variable, by TAG name using the LOG_TAG_ variable pattern, and/or by a TAG prefix by using the LOG_PREFIX_ variable pattern.

felix91gr · 2018-05-23T20:44:53Z

LSP also defines window/logMessage

Ohh, that is very interesting. Maybe we can pipe the logs to that eventually, or some of them at least (the ones relevant to have on the client's side)

Is this the kind of logging you meant in the to-do list, Ryan? (I know very little about the LSP so I couldn't say)

RLovelett · 2018-05-23T20:53:21Z

LSP also defines window/logMessage

Absolutely I think that's a good one too. Traditionally I didn't use that one because when I was first starting this thing I really wanted to be able to log the raw bytes of the protocol. So if you were doing that over the logMessage API it would be a runaway feedback loop. 😄

So for a long time I was just logging every new message into a different file.

Eventually this became too noisy and I fell back to os_log.

This struck a nice balance between being able to see all the low-level logs from the server and also giving an easy filter mechanism.

I still think that window/logMessage has value. Though right now the challenge is what are messages that are for developers of the LSP and what are user facing log messages.

Also, if you look at the main run loop. It is pretty poorly structured to handle injecting out of band messages. The main run loop is basically designed to receive a request from the client and respond. It is not really designed to just say something without being prompted first. That's a design flaw by me.

Is this the kind of logging you meant in the to-do list, Ryan? (I know very little about the LSP so I couldn't say)

No the logging I want to fix in the todo list is this.

felix91gr · 2018-05-25T04:20:56Z

Also, if you look at the main run loop. It is pretty poorly structured to handle injecting out of band messages. The main run loop is basically designed to receive a request from the client and respond. It is not really designed to just say something without being prompted first. That's a design flaw by me.

It's not like this was another one amongst the tens of LSP implementations for Swift. This was the first one. I wouldn't blame you for having a design mistake in something that was not only entirely new, but also very much urgently needed.

Maybe the design could be reworked after everything is cross-platform and stable.

Is this the kind of logging you meant in the to-do list, Ryan? (I know very little about the LSP so I couldn't say)

No the logging I want to fix in the todo list is this.

I see. I'll read the docs of TraceLog in more detail, I think it could work for this.

felix91gr · 2018-05-26T00:04:02Z

Update on TraceLog:

It can do what we'd like, that is,

My two platforms I daily drive are macOS and Fedora. On macOS I want the logging to be done using Unified logging or os_log and on Fedora I want the logging to be available inside the systemd journal, e.g., journalctl.

Is very much possible :D

Tony thinks this use case is so good that he wants to help directly by implementing it himself ❤️❤️❤️

RLovelett · 2018-05-27T17:10:00Z

Okay. I've gotten it "working". 🎉

I had to kludge a bunch of patches and compile my own copy of Swift in order to get it to work. But it does talk to SourceKit and then back to VSCode. So it's a good proof of concept.

Still many hours left to go but I have made significant progress.

felix91gr · 2018-05-27T17:18:25Z

OMG! :D

YOU CAN DOOO EEEET ✊

...how did you get past the Foundation problems? Are you sneakily using your corelibs PR? :)

RLovelett · 2018-05-27T17:29:37Z

...how did you get past the Foundation problems? Are you sneakily using your corelibs PR? :)

Yup. I've compiled my own version of Swift 4.2. I'm using using my patch(es) and a few others that I've not published yet. I'm compiling swift, swift-stdlib and libdispatch in debug mode so I can actually use lldb against each of those components.

I've actually refactored the run loop to not use the old patch needed for Foundation. It now just uses DispatchIO directly. Which actually, I think, makes it more performant.

The bad news is that now it depends on this patch. So I just traded one issue in corelibs for another I suppose. 🤷‍♂️ 🤦‍♂️

felix91gr · 2018-05-27T17:47:02Z

I'm compiling swift, swift-stdlib and libdispatch in debug mode. So I can actually use lldb against each of those components.

Wow. Having fiddled a little bit with compiling the toolchain, I know that's a ton of work. 🙇‍♂️ thank you

I've actually refactored the run loop to not use the old patch needed for Foundation. It now just uses DispatchIO directly. Which actually, I think, makes it more performant.

The bad news is that now it depends on this patch. So I just traded one issue in corelibs for another I suppose. 🤷‍♂️ 🤦‍♂️

Well, that patch looks certainly much, much simpler. So that's great! It should be merged soonish, I'd say.

The performance gains are great as well. Good job! ^^

felix91gr · 2018-05-28T04:41:18Z

I've updated the to-do list to reflect your new PR 👍

… name was different, now they are the same. There are 3 FIXMEs in the `LinuxMain.swift` file. Those are related to test files that make the test suite crash.

Added guard statements to replace forced unwraps that would otherwise crash Linux when running certain tests. Now that those tests are safe, I've enabled them on Linux. Now test coverage is complete on Linux.

RLovelett · 2018-06-06T00:54:50Z

@felix91gr the cross platform logging wasn't as straight forward as we might have hoped I take it.

Swift on Linux is not for the faint of heart. 😏 🤣

felix91gr · 2018-06-06T01:03:53Z

the cross platform logging wasn't as straight forward as we might have hoped I take it.

Hahaha, I know right? Ah well. I'm gonna see if I can persuade somebody who's at WWDC s.t. they can get a word in for us for expanding the os_log API.

Swift on Linux is not for the faint of heart.

Last year it was oh so much worse 🤣 but it's getting better! And thanks to people like you and @vknabel, among many others, we can push it forward even faster :). IMO the most important part of a project like Swift is the community behind it, and we have a really welcoming one, with very kind people all around.

I've also seen great disposition on the corelibs-foundation team. They really want to complete the library. That will be a huge push for cross-platform compatibility - more or less half of the projects I've been helping to port, rely on unimplemented pieces of the library.

RLovelett · 2018-06-06T01:30:44Z

And thanks to people like you and @vknabel, among many others, we can push it forward even faster

I don't know if I have said or not but I've hit a road block since I cannot actually debug the server on Linux the same way I do on macOS. I'm now learning how LLDB works and trying (so far unsuccessfully) to implement the capability on Linux. 🤞

IMO the most important part of a project like Swift is the community behind it, and we have a really welcoming one, with very kind people all around.

Could not agree more.

more or less half of the projects I've been helping to port, rely on unimplemented pieces of the library

That aligns with my experience too.

Keep up the great work!

felix91gr · 2018-06-06T02:15:47Z

I don't know if I have said or not but I've hit a road block since I cannot actually debug the server on Linux the same way I do on macOS. I'm now learning how LLDB works and trying (so far unsuccessfully) to implement the capability on Linux. 🤞

I've seen your quest for LLDB at the forums. I'm crossing my fingers for you as well!
If I got it correctly, LLDB support at the langserver level means that you can do step-by-step debugging right there in the IDE, like in Visual Studio or Eclipse?

Keep up the great work!

You too! ^^

patilarpith · 2018-08-05T12:24:15Z

Following

tonystone · 2018-09-11T01:27:20Z

@RLovelett do you need assistance adding TraceLog and AdaptiveWriter for "Cross-platform logging"? I should have some time shortly and would be happy to assist when I do.

RLovelett · 2018-09-14T13:42:58Z

Absolutely I'd love the assistance. @felix91gr was the one for the original "vision" of how to do this feature but certainly I can help answer questions. If you have a vision for it by all means move forward with it.

tonystone · 2018-09-14T16:47:27Z

@RLovelett Ok, I suggest that TraceLog is added in a branch off of master in a separate PR, then pulled into here once merged with master. That would isolate out the changes and allow easier review and testing. TraceLog is already cross-platform. The new AdaptiveWriter can be added in that branch which will also satisfy the requirement in this PR/branch. The AdaptiveWriter writes to os_log (Unified Logging) on Darwin and SD journal on Linux so we'd be all set. We'd be able to remove all the #if os() statements currently used for logging. Thoughts?

RLovelett · 2018-09-14T17:10:33Z

Totally agree. That is a great idea. Especially love the idea of it being done directly off of master. Easier to review and move forward.

tonystone · 2018-09-14T20:14:42Z

Great, as soon as I free up a bit, I'll get setup with your dev environment, create the branch and get started.

RLovelett mentioned this pull request Apr 2, 2018

Linux version #18

Open

RLovelett force-pushed the fedora-linux-swift-4.1 branch from c911707 to 13e9ece Compare April 3, 2018 15:29

felix91gr mentioned this pull request May 25, 2018

Question about system logs tonystone/tracelog#44

Closed

RLovelett force-pushed the fedora-linux-swift-4.1 branch from 66483d7 to 38c5442 Compare May 28, 2018 00:35

RLovelett self-assigned this May 28, 2018

felix91gr added the enhancement label May 28, 2018

RLovelett force-pushed the fedora-linux-swift-4.1 branch 3 times, most recently from c1ba9bc to 1a743f4 Compare May 29, 2018 00:39

RLovelett and others added 3 commits May 28, 2018 22:49

A better fixture I can test with

68c6c74

Handle this for Linux

2f578a3

Extended test coverage to Linux. Renamed a test file because its test…

6941bfd

… name was different, now they are the same. There are 3 FIXMEs in the `LinuxMain.swift` file. Those are related to test files that make the test suite crash.

felix91gr and others added 4 commits May 28, 2018 22:49

Added guard statements and XCTFail declarations. Full Linux coverage.

4a13e87

Added guard statements to replace forced unwraps that would otherwise crash Linux when running certain tests. Now that those tests are safe, I've enabled them on Linux. Now test coverage is complete on Linux.

A module for logging

e1dc9dc

WIP

b9179a4

Needed to prevent the system from crashing

268526d

RLovelett force-pushed the fedora-linux-swift-4.1 branch from 1a743f4 to 268526d Compare May 29, 2018 02:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Provide Support for Linux #36

[WIP] Provide Support for Linux #36

RLovelett commented Feb 2, 2018 •

edited

Loading

felix91gr commented Apr 7, 2018

felix91gr commented Apr 10, 2018

RLovelett commented Apr 26, 2018

felix91gr commented Apr 26, 2018

felix91gr commented May 23, 2018

felix91gr commented May 23, 2018

RLovelett commented May 23, 2018

keegancsmith commented May 23, 2018

felix91gr commented May 23, 2018

felix91gr commented May 23, 2018

RLovelett commented May 23, 2018

felix91gr commented May 25, 2018

felix91gr commented May 26, 2018 •

edited

Loading

RLovelett commented May 27, 2018 •

edited

Loading

felix91gr commented May 27, 2018

RLovelett commented May 27, 2018 •

edited

Loading

felix91gr commented May 27, 2018 •

edited

Loading

felix91gr commented May 28, 2018

RLovelett commented Jun 6, 2018

felix91gr commented Jun 6, 2018

RLovelett commented Jun 6, 2018

felix91gr commented Jun 6, 2018

patilarpith commented Aug 5, 2018

tonystone commented Sep 11, 2018

RLovelett commented Sep 14, 2018

tonystone commented Sep 14, 2018

RLovelett commented Sep 14, 2018

tonystone commented Sep 14, 2018

[WIP] Provide Support for Linux #36

Are you sure you want to change the base?

[WIP] Provide Support for Linux #36

Conversation

RLovelett commented Feb 2, 2018 • edited Loading

Infraestructure

Code

Documentation

felix91gr commented Apr 7, 2018

felix91gr commented Apr 10, 2018

RLovelett commented Apr 26, 2018

felix91gr commented Apr 26, 2018

felix91gr commented May 23, 2018

felix91gr commented May 23, 2018

RLovelett commented May 23, 2018

keegancsmith commented May 23, 2018

felix91gr commented May 23, 2018

felix91gr commented May 23, 2018

RLovelett commented May 23, 2018

felix91gr commented May 25, 2018

felix91gr commented May 26, 2018 • edited Loading

RLovelett commented May 27, 2018 • edited Loading

felix91gr commented May 27, 2018

RLovelett commented May 27, 2018 • edited Loading

felix91gr commented May 27, 2018 • edited Loading

felix91gr commented May 28, 2018

RLovelett commented Jun 6, 2018

felix91gr commented Jun 6, 2018

RLovelett commented Jun 6, 2018

felix91gr commented Jun 6, 2018

patilarpith commented Aug 5, 2018

tonystone commented Sep 11, 2018

RLovelett commented Sep 14, 2018

tonystone commented Sep 14, 2018

RLovelett commented Sep 14, 2018

tonystone commented Sep 14, 2018

RLovelett commented Feb 2, 2018 •

edited

Loading

felix91gr commented May 26, 2018 •

edited

Loading

RLovelett commented May 27, 2018 •

edited

Loading

RLovelett commented May 27, 2018 •

edited

Loading

felix91gr commented May 27, 2018 •

edited

Loading