Bug in cleartext unicode message validation

[cryptix]

I tested tremola with a friend, using my old patchwork feed (@p13z..). Their phone stopped replicating my feed on a specific message (39) which happens to contain unicode characters. I was gladly able to glean this exact fact from my local EBT state. I can see how this slipped under the table when testing tremola<>tremola since all the content is neatly hidden inside the ciphertext.

So if this is just intended for Tremola<>Tremola communication, this might be an non-issue actually but I vaguely recall potential interaction with Manyverse where this will definitely be the case (emojis in public posts, etc.).

Having written verification code for the classic JSON format myself, I firmly believe these two instructions here are too simple:

1. tremola/app/src/main/java/nz/scuttlebutt/tremola/ssb/SSBmsgTypes.kt

   Line 78 in bc9cd86

   key = "%" + msg.encodeToByteArray().sha256().toBase64() + ".sha256"

Annoyingly enough, the JS implementation does not use Buffer(msg, "utf8") when hashing a message to a key but "binary" instead, using the internal v8 string representation utf16 but somehow truncates everything but the first byte when turning it into a buffer again. I so far got away with this hack in my implementation, blowing up the string to utf16 and then copying every other byte.

2. tremola/app/src/main/java/nz/scuttlebutt/tremola/ssb/SSBmsgTypes.kt

   Line 85 in bc9cd86

   val msg2 = msg.slice(0 .. msg.indexOf(",\n \"signature\":", msg.length-130)-1) + "\n}"

This can also be problematic before checking the signature, it's (again, sadly) paramount especially to escape all the string fields. Aljoscha made a thorough list here. I'd suggest adopting unit tests over the JSON in https://github.com/fraction/ssb-validation-dataset as it contains a good chunk of the gnarly corner cases. And just FYI here and here is what I replace/escape, but the list could be different for other JSON encoders.