Deserialization works in program but fails in interactive

[jgardella]

I've found a strange issue where deserialization of a type works fine in my program, but fails when I run the same serialization in interactive. In interactive I get the following exception:

Couldn't parse config, error=JSON Path: . Failed to serialize, must be one of following types: record, map, array, list, tuple, union. Type is: Configuration.

I haven't been able to create a minimal example of the issue, but I've made a branch on the affected project to demonstrate the issue here.

To see the issue:

1. Place a debug breakpoint here. This is where we try to parse the Configuration type.
2. Run the Lint project test. This will try to lint the provided project, eventually hitting the parseConfig breakpoint above, when it tries to parse the default configuration file. It should succeed.
3. Run the FSharp.Json.Error.fsx file in interactive debug. You should get the exception above, when it tries to parse the same default configuration.

Through this I was able to confirm that the two functions seem to be receiving the same input.

[vsapronov]

I can take a look later this week. Thank you.

…

On Wed, May 15, 2019, 4:47 PM Jason Gardella ***@***.***> wrote: I've found a strange issue where deserialization of a type works fine in my program, but fails when I run the same serialization in interactive. In interactive I get the following exception: Couldn't parse config, error=JSON Path: . Failed to serialize, must be one of following types: record, map, array, list, tuple, union. Type is: Configuration. I haven't been able to create a minimal example of the issue, but I've made a branch on the affected project to demonstrate the issue here <https://github.com/jgardella/FSharpLint/tree/fsharpjson-bug>. To see the issue: 1. Place a debug breakpoint here <https://github.com/jgardella/FSharpLint/blob/master/src/FSharpLint.Core/Application/ConfigurationManager.fs#L461>. This is where we try to parse the Configuration type <https://github.com/jgardella/FSharpLint/blob/fsharpjson-bug/src/FSharpLint.Core/Application/ConfigurationManager.fs#L440> . 2. Run the Lint project test <https://github.com/jgardella/FSharpLint/blob/fsharpjson-bug/tests/FSharpLint.FunctionalTest/TestApi.fs#L61>. This will try to lint the provided project, eventually hitting the parseConfig breakpoint above, when it tries to parse the default configuration file. It should succeed. 3. Run the FSharp.Json.Error.fsx file <https://github.com/jgardella/FSharpLint/blob/fsharpjson-bug/tests/FSharpLint.FunctionalTest/FSharp.Json.Error.fsx> in interactive debug. You should get the exception above, when it tries to parse the same default configuration. Through this I was able to confirm that the two functions seem to be receiving the same input. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#25>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGYLNXDGKGKFG72SMJZSNDPVQO7PANCNFSM4HNECIEA> .

[jgardella]

Hi @vsapronov, any luck with this?

[vsapronov]

@jgardella I was able to reproduce the issue. I can't get yet why it happens. There is one little detail: it happens even without debug mode in FSI. I keep investigating this.

[vsapronov]

@jgardella The issue basically boils down to the fact that your Configuration type is not recognized as a record type from FSharp.Json.Error.fsx script. If I put this line into your script: FSharpType.IsRecord typeof<Configuration> I would expect it to return true however it returns false.

These are questions I haven't answered yet:

1. Why reflection works so strangely (wrongly) from FSI?
2. Is this behavior common or is it specific to types from FSharpLint libraries for whatever reason?

If you can help with these, particularly (2) it would be great.

[jgardella]

I'm actually not really concerned with this not working in FSI (although it is strange); that's just how I was able to reproduce the issue. The actual issue which is affecting me is that I'm getting this error when I try to use the parseConfig function from another project, referencing FSharpLint.Core.

Maybe the issue is that the Configuration type has a member?

[vsapronov]

Well, please try this from you other project referencing FSharpLint.Core: FSharpType.IsRecord typeof<Configuration>. What it returns to you? If reflection can't see that Configuration type is a record then FSharp.Json neither will work.

I don't think it's related to a member. In my local testing I removed that member (was concerned about it too).

My investigation shows also following:
This line typeof<Configuration>.GetCustomAttributes(typeof<CompilationMappingAttribute>, false) returns zero attributes.
The CompilationMappingAttribute is set by compiler to link CLI generated code to initial F# structures (like record type). This is what FSharpType.IsRecord is using to understand if the type is a record.
Though there's a CompilationMappingAttribute in the collection of custom attributes on the type typeof<CompilationMappingAttribute> when I stop by in debugger I see it.

I suspect that maybe it's caused by FSharp.Core versions discrepancies. Like reflection is looking for CompilationMappingAttribute from one assembly, however the type (Configuration) has CompilationMappingAttribute from different version of the assembly. There was some similar issue logged about FSharpType.IsUnion function, it looks like people solved it with binding redirects. This partially confirms the idea about different FSharp.Core versions: dotnet/fsharp#3225

I'm not sure how much more time I will have to investigate this. I would appreciate if you can follow the traces yourself and share your findings here. Again the main question is why FSharpType.IsRecord typeof<Configuration> does not return true?

[vsapronov]

@jgardella Now I'm pretty sure it's related to FSharp.Core versions. Here's how I diagnosed this. This is the code that I have put into FSharp.Json.Error.fsx:

#r @"bin\Debug\net461\FSharpLint.Core.dll"
//#r @"bin\Debug\net461\FSharp.Json.dll"

//open FSharpLint.Framework
open FSharpLint.Application.ConfigurationManager
//open FSharp.Json

open Microsoft.FSharp.Reflection

type Configuration2 = {
    ignoreFiles : string [] option
    formatting : FormattingConfig option
    conventions : ConventionsConfig option
    typography : TypographyConfig option
    hints : HintConfig option
}

let type1 = typeof<Configuration>
let type2 = typeof<Configuration2>

let attrs1 = type1.GetCustomAttributes(false)
let attrs2 = type2.GetCustomAttributes(false)

let attr1Type = attrs1.[0].GetType()
let attr2Type = attrs2.[0].GetType()

The Configuration2 type is the same as Configuration record type but declared in the fsx script (same assembly as script itslef, though outside of FSharpLint.Core assembly). The custom attribute with index 0 is CompilationMappingAttribute. It is set in both Configuration and Configuration2 types. However for Configuration type it's attribute came from FSharp.Core 4.6.0.0 and for Configuration1 it came from FSharp.Core 4.4.3.0. From runtime persepctive these are different types. Hence the reflection will work for one type but not for another one. I think binding redirects will solve the issue. Redirect to higher version of FSharp.Core in your project and you should be fine as they are backward compatible on the API level.

Let me know how it goes...

[vsapronov]

If you need help to understand more why you have got this issue I can stop by some day at Hoboken. Or just setup FSharp.Json office hours for Thor team at Jet.com :)

[jgardella]

Thanks for looking into this! I tried the binding redirects as you suggested, but the fsx file I created to show the issue is still failing (here).

[vsapronov]

There's no such thing as binding redirects for fsx files. The fsx files are compiled into separate assembly which is then executed.
You have done: specified binding redirects for every FSharpLint project (Console, Core and all tests). This does not affect the assembly which is created from fsx file when you run it.
Again: from my knowledge there are no assembly binding redirects for FSI and fsx files. So to really check bindings you need to test it in the project that is using FSharpLint.Core. If it still won't help please add your side project (which is really needed for you) into FSharpLint repo branch. Debugging fsx when the fsx is not even needed is not fun.

As a side note: please, make some more effort to investigate the issue. Simple "still failing" is not good enough. I showed you how to get what is version of FSharp.Core is being used on the type. There are more ways to see what libraries are loaded into AppDomain... This information would be helpful.

[jgardella]

Thanks for the info, I wasn't aware of how redirects would interact with fsx files. I'll try to make the change in the affected project and do some additional debugging there.

[jgardella]

Small status update:

First, I think my attempt to reproduce this using .fsx was a bit unhelpful. The actual issue is happening in this project, which is a Visual Studio extension. When I try to run the extension in debug mode in a test VS instance, I get that error when I try to load the config on this line.

I checked, and FSharpType.IsRecord typeof<Configuration> is true before the config is loaded, and the FSharp.Core version on the type is 4.6.0.0. Not sure if something strange is happening with VSIX or the test instance. I will continue investigating.

[nzav]

Any news on this? Having the same issue with F# Azure Functions (runtime 2.0)

[vincenzoml]

Hi all, I had this issue today, and it was caused by the fact that I had made "private" the type I was trying to parse from json. Removing "private" solved the issue. So maybe the exception is caused by an accessibility problem?

[NicoVIII]

I could reproduce what @vincenzoml said. I had working code and then (to not expose an only internally used type) I made it private. Then the code didn't work anymore with the error like mentioned above.
After removing "private" it works again.

[vsapronov]

@NicoVIII and @vincenzoml What do you think could be done from library perspective? May be better exception message pointing towards real problem (private access modifier)?
Also @nzav would you be able to check if private modifier is the real reason of your problem?

Thank you for helping to diagnose this issue.

[NicoVIII]

If there is no way (for now?) to fix this issue, I guess a better exception message which points to the real problem would be nice.
I was confused, when the thrown exception told me, that my record type is not a record type 😄