Codepage conversion for files with "include" directive in agent.cfg not correct

[Lelik2107]

Your question

Hello everyone! The problem indicated in title. In details:

Using utf8 I need attach file with "tag" value. "Tag" vary and defines by script, when user input some info interactively.
It saved in (tag.inf), attaches to inventory report by "include" directive in agent.cfg file.

When Agent is attaching file the collision. Appears a problem for non-latin chars (strings) - two-three-four byte unicode.

As I learned - durin attachment of tag*.inf, my files with "tag = " line consist with latin and non-latin chars (tag = Л-10GJ - first letter is cyrillic letter "El" (13th in cyrill.alphabet). But Agent reads such chars as Byte-Array, but not Char-Array. Because of it the result is two-bytes sequence instead of cyrillic two-byte chars.

I've attach a little of files (ZIP) with vary codings. Files tag*.inf are files attached by Agent. Report*.* - XML results, logs of Agent. And agent.cfg also.

Codepage for Agent is 65001. Files tag*.inf in codepages - cp866, cp1251, cp65001 (look by names). XML results in UTF8. Environment : OS Windows 10 Pro Russian Edu; GLPI-Agent 1.7.1 x86 (because runs on old systems too); GLPI Server 10.0.7 (Ubuntu 18).

The results of import of these encodings are always different. Result string from one report (cp 65001) is:

[Thu Feb 8 12:40:35 2024][info] New inventory from j229pc02-2024-02-08-12-40-35 for local0 (tag=IVC_MPI_�-10�::)
(Secon letter is "De" - 4th letter)_

**But just in cp65001 i see a right (D0x9B (utf8 letter El) and D0x94 (utf8 letter De) ) two-byte! sequenses - but not utf-8 cyrillic chars!**

It's so strange, but I no see something, may be?

Lets discuss it, it's so serious! There is a solution already?

But I didn't found proper here...

For information on "tag": it contains department code (latin) and room number (cyrillic). Parsed with regexp.

Thanks.

agent-include.zip

[g-bougard]

Hi @Lelik2107

included files from configuration are opened there in the code:

glpi-agent/lib/GLPI/Agent/Config.pm

Line 240 in ea45347

if (!open $handle, '<', $file) {

Actually, we don't try to analyze included file encoding as this is dedicated to configuration and we don't expect exotic encoding in cfg files.

About "tag", this is a configuration string which make sens in GLPI to be used in the rules engine. It permits for example to assign an entity or a location. My guess is you should keep the problem as simple as possible and so you should not use exotic encoding here. I of course understand simplicity could be let me use my natural language. But "tag" has not been developed in that usage in mind.

As I understand, you developed a user interface which generates your "inf" files. You should probably manage to update that interface to set "tag" in a supported way by glpi-agent without using specific encoding: just use some coding conventions that you can analyze in GLPI.

Tell me if I'm wrong, but your real concern is to use these tags for rules, so simplifying them before inclusion should be your first goal, so rules in GLPI won't have to be too complicated.

[Lelik2107]

Hi Guillaume Bougard.

Many thank for your remarks. You suggest that I choose an easy way - to abandon both national encodings and utf-8, as I understand. Perhaps you can't read the Cyrillic symbols in utf-8?

I understand mech of rules as I use tags to spread assets across various entities (these tags I set in Latin - in "Add.info" for entity). But I must edit utf-8 encoded .json inventory file to include location (with cyrillic symbols) of each asset (computers). It's not encouraging.

Let's talk about utf-8 only - it's more understandable and GLPI uses utf-8 (and Latin ISO 8859-1) encoding (stated by GLPI). It's not difficult to get a file with the "tag" parameter with encoding of Cyrillic symbols in utf-8 - two-byte per one symbol (char) (see tag-65001.inf file please).

In fact the .inf file with utf-8 encoding reads by GLPI Agent as a Byte-Array but not a Char-Array in utf-8, where each Cyrillic symbol is encoded with bytes: xD0x90 - xD1x8F (see report-with-tag-65001.xml please). Ones again, please: there is two-byte sequences a little but not Cyrillic symbols there. But file with utf-8 encoding!

Excuse, Here is url to utf-8 codes table for 2-byte symbols. Look at, please. https://www.charset.org/utf-8/2

These symbols are necessary when specifying room numbers in the office (Cyrillic is used in numbering and cannot be refused).

UTF-8 encoding assumes coding of all symbols, the national sets for all languages included. And why does the GLPI Agent read the .inf file as a Byte-Array, without looking at the utf-8 encoded symbols? Isn't it a problem for GLPI team and developers - to make GLPI more convenient for all?

Once again, this is quite serious, since it solves the problems of importing national symbols for countries not write in Latin o have a specific symbols in two-three-four byte utf-8 encodings.

Thank you and please respond, it's important.

[g-bougard]

Hi @Lelik2107

as I said, we are talking about configuration file where the agent doesn't expect any encoding since the first day it was developed. Maybe this is a mistake, but this is just a fact.

Actually, even on the command line, if you're using a tag with some encoding it won't be encoded as you think it should be in the inventory.

After years, you're the first guy to complain about a tag encoding requirement and you're acting as I should give you the solution right now... And I really don't like the authoritarian tone you're trying to take to convince me.

Anyway my purpose is you seem to be the interface developer which generates your inf files. So you are free to adapt your development to use a convention which not rely on some encoding and that's just up to you. I can understand you want the tag being more "readable", but this is not absolutely necessary for the tag purpose: "spread assets across various entities" just as you said.

You don't have to convince me using utf-8 encoding is better. In the future, this will probably happen, at least when the agent will be ported to another language. But sorry this won't happen tomorrow.

Don't be lazy and adapt your own development to generate what is really expected.

[Lelik2107]

In fact the tone of my message seemed you authoritarian is incorrect, I think. Anyway, I'm sorry I upset you.

It's seemed I understood all you wrote.

But.. I'm not a developer... that's why I'm asking this question in essence to you. It's a problem for me to edit an original PHP code it could 'understand' utf-8 chars in inputs data by include directives.

And I need just a recommends to choose a way to solve. In case I could develope a code I wouldnt have written this message.

Thank for reply, excuse ones more.

Goodbye

[g-bougard]

Hi, I understand.

If you're not the developer of the interface which generates the included files, you have to first request his support.