About surrogate pair characters.

[kado-junichiro]

May I ask a question about the handling of surrogate pair characters?

In Japan, surrogate pair characters may be included on invoices in the name of a company or individual.
For example, if we include the characters "髙橋 𩸽𠮷" in the send invoice xml, when we check the received invoice, "𩸽𠮷" (surrogate pair characters) were automatically removed and changed "髙橋".

The log shows the following, indicating that the surrogate pair characters were recognized as an invalid character and replaced.
**
level
WARN

loggerName
com.helger.xml.serialize.write.EXMLIncorrectCharacterHandling

message
XML content contains invalid character data (will replace): '株式会社 髙橋 𩸽𠮷商事'. Invalid chars are: 0xd867, 0xde3d, 0xd842, 0xdfb7
**
Is this intentionally removed because surrogate pair characters cannot be handled on the Peppol network?
If you can provide any information, would you let us know?

Regards,
Junichiro Kado

[phax]

That is indeed an error in phase4 (and will similarly impact the SMP btw.). I will look into this and keep you posted. Thanks for bringing that also here to my attention

[kado-junichiro]

Philip-san,

We are continuing to test and the problem has been resolved and nothing is wrong at this time.
Thank you for the modification!

Regards,
Junichiro Kado

[kado-junichiro]

Philip-san,

I'm sorry so often, but let me check just in case.

We are currently releasing our access point with Phase4 1.3.9.
Since the surrogate pair issue has been fixed for 2.0.0, we wanted to update our release version from 1.3.9 to 2.0.0, but we found that it is not easy to update due to the change to Java 11, etc.

The surrogate pair issue needed to be addressed immediately, so for now we will keep the Phase4 version with 1.3.9 and only replace ph-xml to 10.2.4.

We think there is no problem with above deal, because we have confirmed with test that the surrogate pair issue is resolved after only replacing ph-xml to 10.2.4.
Also, we have already cofirmed the consistency between Phase4 1.3.9 and ph-xml 10.2.4 on maven repository web site.

If you think there is a problem with above deal, would you let me know?

Regards,
Junichiro Kado

[phax]

Hi @kado-junichiro - okay, that sounds like a smart approach that should work. Lucky me I created the backport ;-) Unfortunately this also requires a small backport in ph-web that I can create.
As I also need to backport some reporting functionality, there will be an 1.4.4 version that also fixes the surrogate problem directly and you would not need this manual tweaking.

Until then - happy Peppoling :)

[kado-junichiro]

Philip-san,

Thank you for your reply and thank you for the information of 1.4.4.
We are now continuing to investigate the possibility of replacing to 2.1.0, but if it is still difficult, we will consider using 1.4.4.

Happy Peppoling!

Regards,
Junichiro Kado

[phax]

@kado-junichiro phase4 1.4.4 release is available from https://github.com/phax/phase4/releases