utf-16 surrogates are not processed properly

[R-maan]

In ion-test equivs/utf8/stringutf8.ion there are utf-16 representation (lines 44 and 50) where they must interpret equal; however, the current implementation fails to process utf-16 values properly in round trip tests.

The root cause might be in tokenizer.go#readString() where we call WriteRune(r). That results in some unknown characters to be returned from the reader, and of course anything done with the string value of that reader can be erroneous, as the data is already malformed.

When we try to read the values for lines 44 and 50 in above file, we get back "0xEF 0xBF 0xBD" for both high surrogate and low surrogate which cannot be translated to a valid utf-8 value later on when we call them in bitstream, hence returning 0xFFFD and that is when the tests fails.

A solution can be passing an encoding parameter to the reader (example in ion-dotnet).