feat: offline cross-endian decoding and golden fixtures#31
Merged
Conversation
Integer reads honor the byte order detected from the file magic, but struct.unpack used native byte order for float and double tracepoint arguments, so they decoded as garbage from foreign-endian trace files. Select the unpack byte order from the detected endianness. Signed-off-by: Jo5ta <jo5ta@mail.de>
Committed .clltk_trace fixtures produced by the real library: 1.2.64 and 1.3.0 on aarch64 (little endian) and 1.3.0 on s390x (big endian, generated under qemu-user emulation). Each contains one tracepoint per argument type with fixed values, so decoder output is deterministic. tests/test_golden.py decodes every fixture with the python decoder and compares the formatted messages - guarding backward compatibility with old file versions and cross-endian decoding. Generator source and instructions live in tests/golden/generator/. Signed-off-by: Jo5ta <jo5ta@mail.de>
FilePart gains a foreign-endian flag: when set, get() and getLimted() byte-swap arithmetic values, and sub-FileParts inherit the flag. getReference() stays raw for byte arrays and native-only live paths. Nothing sets the flag yet - behavior is unchanged until the tracebuffer magic check is wired up to it (foreign-endian files are still rejected there). Groundwork for decoding big-endian s390x trace files on little-endian hosts. Signed-off-by: Jo5ta <jo5ta@mail.de>
Extend the golden set with fixtures generated from git worktrees of the versions where the trace file format actually changed: 1.2.39 (oldest buildable state, predates the 1.2.40 dump-format change) and 1.2.49 (predates the 1.2.50 definition-V2 format). Both decode identically to current files with today's python decoder, pinning backward compatibility across every format boundary in the public history. The generator now adapts the writer to the header layout of the checked-out tree, so future boundary versions can be added the same way. The minor <= 1 layout (32-byte mutex) predates the public repository and cannot be regenerated; it stays covered only by the decoder's version branch. Signed-off-by: Jo5ta <jo5ta@mail.de>
The C++ decoder recognized the big-endian file magic but rejected such files. The byte order is now detected from the magic when the file is opened and flagged on the FilePart, which byte-swaps all reads transparently. Places that read outside the FilePart swap explicitly: the atomic ringbuffer head is loaded by value and swapped, ringbuffer entries carry the flag for pid/tid/timestamp/line extraction, the formatter swaps every argument by its type (including float/double and the string/dump size prefixes), and the meta parser swaps its size and line fields. The 48 bit file offset at the start of each entry body is stored as six writer-order bytes, so a foreign value sits in the upper bits of the swapped read and is shifted instead of masked. With this, trace files from s390x decode correctly on little-endian hosts and vice versa, through both the library and the clltk CLI. The golden tests now decode every fixture through the CLI as well; the big-endian fixture is real s390x output. The byte-order marker in the file magic is now documented in the file specification. Foreign-endian files are static by nature (their producer cannot run on this host), so all read paths simply work; no live/offline split is needed. Signed-off-by: Jo5ta <jo5ta@mail.de>
The ELF reader interpreted section headers, relocation records, symbol entries, discovery pointers, and meta entry sizes in host byte order. Swap them based on the byte order declared in the ELF identification (EI_DATA), so clltk meta works on s390x binaries inspected from little-endian hosts: shared objects through virtual addresses and relocatable objects through relocation records (whose e_type check also needed the swap to even take the relocation path). Committed big-endian s390x ELF objects join the golden fixtures and are exercised by tests. Signed-off-by: Jo5ta <jo5ta@mail.de>
Signed-off-by: Jo5ta <jo5ta@mail.de>
test_golden.py references golden-1.3.0-be-s390x.o, but **/*.o in
.gitignore kept it out of the repo, so CI had no file to decode ('No
meta information found'). It only passed locally because the generated
object lingered in the working tree.
Un-ignore golden object files and add a .gitattributes marking golden
fixtures binary so git never applies EOL/text normalization to them.
Signed-off-by: Jo5ta <jo5ta@mail.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2/7 — based on #30.
Decoders (C++ and python) can now decode trace files and ELF binaries written on the opposite byte order — offline decoding only; writers stay native-endian.
?#$~tracebuffer\0little-endian, byte-reversed for big-endian);FilePartcarries a foreign-endian flag and swaps centrally in its typed reads.EI_DATAfor sections, symbols, relocations, and the tracepoint pointer sections, including relocatable.ofiles.tests/test_golden.pylocks the decoders against all of them, both directions (verified decoding LE fixtures on a big-endian s390x container and vice versa).Version 1.4.0.