Immediate re-encoding of a decoded value adds extra bytes

[MasterKale]

I've got some CBOR-encoded public key bytes in "CTAP2 canonical CBOR encoding form". I'm finding that I can use cbor-x just fine to decode these bytes into an expected JS value, but when I re-encode the decoded value back to bytes, the length increases!

import * as cborx from 'cbor-x';

const publicKey = Buffer.from(
  'a501020326200121582092245fc7ecf708ca52fe47581528a7efba8a0e12cecc63f2006024999a5feff92258202795d576351f08c95595466e1c947108b14bcd909e2bc160de5e8e2df0081e72',
  'hex',
);

console.log('Raw bytes length:', publicKey.length); // 77

const decoded = cborx.decode(publicKey);
const reEncoded = cborx.encode(decoded);

console.log('Re-encoded bytes length:', reEncoded.length); // 87

For comparison, the cbor library I'm replacing with cbor-x correctly re-encodes back to the original length of 77 bytes:

import cbor from 'cbor';

const publicKey = Buffer.from(
  'a501020326200121582092245fc7ecf708ca52fe47581528a7efba8a0e12cecc63f2006024999a5feff92258202795d576351f08c95595466e1c947108b14bcd909e2bc160de5e8e2df0081e72',
  'hex',
);

console.log('Raw bytes length:', publicKey.length); // 77

const decoded = cbor.decode(publicKey);
const reEncoded = cbor.encode(decoded);

console.log('Re-encoded bytes length:', (reEncoded as Buffer).length); // 77

I tried playing around with cborx.Encoder and some of its options (particularly keyMap, mapsAsObjects, and useRecords) to try and get the re-encoded value down to the original length, but no matter what I tried I could never get 77 bytes back - it was always more.

This has thrown off logic I wrote to shift a pointer ahead the number of bytes of the re-encoded value, because the pointer is moved too far forward:

WebAuthn spec - attestation object

This means a subsequent attempt to decode the "extensions" bytes next to the public key bytes fails to decode the entire value, because the pointer is too far ahead.

What are my options here?

[kriszyp]

This maintains 77 bytes for me:

const encoder = new cborx.Encoder({ mapsAsObjects: false })
const decoded = encoder.decode(publicKey);
const reEncoded = encoder.encode(decoded);

console.log('Re-encoded bytes length:', reEncoded.length); // 77

Does that work for you?

(the reason for this is that JavaScript objects do not preserve the type of the keys and so the keys were getting converted to strings when deserialized as objects, whereas Maps preserve the number type of the keys)

[MasterKale]

Well huh, that worked great! I must not have tried just mapsAsObjects. I also didn't pick up from the docs that I can call .decode() on an instance of Encoder. Thank you very much 🎉

One follow-up, if I do this same experiment with .decodeMultiple() then .encode() a single 82 byte gets added at the start of the bytes containing two Maps because .decodeMultiple() wraps them in an array:

const combined = Buffer.from(
  'a501020326200121582092245fc7ecf708ca52fe47581528a7efba8a0e12cecc63f2006024999a5feff92258202795d576351f08c95595466e1c947108b14bcd909e2bc160de5e8e2df0081e72a16c6465766963655075624b6579a56364706b584da5010203262001215820991aabed9de4271a9edead8806f9dc96d6dccd0c476253a5510489ec8379be5b225820a0973cfdedbb79e27fef4ee7481673fb3312504ddca5434cfd23431d6ad29eda6373696758473045022100efb38074bd15b8c82cf09f87fbc6fb3c7169ea4f1806b7e90937374302345b7a02202b7113040731a0e727d338d48542863ce65880aa79e5ea740ac8ccd94347988e656e6f6e6365406573636f70654100666161677569645000000000000000000000000000000000',
  'hex',
);
const encoder = new cborx.Encoder({ mapsAsObjects: false });

console.log('Raw bytes combined length:', combined.length); // 291

const decodedMultiple = encoder.decodeMultiple(combined);
const reEncodedMultiple = encoder.encode(decodedMultiple);

console.log('lib:cbor-x Re-encoded bytes combined length:', reEncodedMultiple.length); // 292

console.log(reEncodedMultiple.toString('hex'));
// The "82" at the start here is the extra byte, everything else is the same as the hex string I put into `combined` above
// 82a501020326200121582092245fc7ecf708ca52fe47581528a7efba8a0e12cecc63f2006024999a5feff92258202795d576351f08c95595466e1c947108b14bcd909e2bc160de5e8e2df0081e72a16c6465766963655075624b6579a56364706b584da5010203262001215820991aabed9de4271a9edead8806f9dc96d6dccd0c476253a5510489ec8379be5b225820a0973cfdedbb79e27fef4ee7481673fb3312504ddca5434cfd23431d6ad29eda6373696758473045022100efb38074bd15b8c82cf09f87fbc6fb3c7169ea4f1806b7e90937374302345b7a02202b7113040731a0e727d338d48542863ce65880aa79e5ea740ac8ccd94347988e656e6f6e6365406573636f70654100666161677569645000000000000000000000000000000000

Is there some way to prevent this extra byte from getting added?

[MasterKale]

I think I got it, adding tagUint8Array: false to the Encoder options took care of the problem ✌️

[kriszyp]

I thought that you were looking for a way to concatenate two encodings sequentially with encoding them as an array (the 0x82 is the encoding of a two-element array). One thing you could do is Buffer.concat(decodedMultiple.map(item => encoder.encode(item))) to concatenate them. It would also be possible to add a encodeMultiple to encode a set of items without array wrapping.

[MasterKale]

I that you were looking for a way to concatenate two encodings sequentially with encoding them as an array

I was! I'm still feeling my way around working with CBOR, and am still making basic mistakes like re-encoding the array of items out of decodeMultiple 😅

One thing you could do is Buffer.concat(decodedMultiple.map(item => encoder.encode(item))) to concatenate them

This makes a ton of sense in retrospect. I'd have never thought to try this because I was thinking too much like a JS dev, where "sequence of things" equals an array. Thank you for the assist 🙇