[Feature] Export section data as Pattern Data

[rockisch]

What feature would you like to see?

Maybe this is already possible, but I couldn't find a way to do it no matter how hard I tried.

I want to find a way to interpret the data under a std::mem::Section as another struct, and expose it under the 'Pattern Data' menu (or somewhere where I can inspect it's fields).

How will this feature be useful to you and others?

The specific use case I'm thinking of is being able to inspect compressed data inside a packet, but I'm sure there are other uses as well.

Request Type

• I can provide a PoC for this feature or am willing to work on it myself and submit a PR

Additional context?

Here's an example: suppose I have a packet with a header and some content zstd encoded. Here's a python script to generate a sample value:

import pyzstd
with open("packet.pak", "wb") as f:
    header = bytes([1, 10])
    f.write(header)
    data_zstd = pyzstd.compress(bytes([0, 20]))
    f.write(data_zstd)

Ideally, I'd want to see the values 0 and 20 organized into structs on the pattern data menu. Here's the script that I think should be working:

import std.mem;
import hex.dec;

struct InnerPacket {
    u8 flag1;
    u8 value;
};

struct Packet {
    u8 header_flag;
    u8 header_value;
    
    u8 data_zstd[while(!std::mem::eof())];
    std::mem::Section data_sec = std::mem::create_section("data_sec");
    hex::dec::zstd_decompress(data_zstd, data_sec);
    
    InnerPacket data @ 0x00 in data_sec [[export]];
};

Packet packet @ 0x00;

However, even with the [[export]] attribute set, the value still doesn't get exported to the pattern data menu:

[paxcut]

Each section has a hex editor window of its own separate from the main hex editor which is meant for the input file.
The hex editor for the section has a patterns data view of its own that is used to view patterns placed in the section the same way you view patterns placed in the input file.. To open the hex editor of a section click on the icon to the right on the Sections tab.
The [[export]] attribute is used to allow showing local variables in the pattern data view. Section data are not local data so it can't be shown in the main hex editor pattern data view.

[rockisch]

Ah, I see! I knew about the section hex viewer, but didn't notice it exposed variables.

But in that case, I'd like to change the feature request to allow exporting data from a section into either the 'context' it was created (Packet in this case), or as a global variable.

The reasoning is a bit more specific though. I tried to simplify it, but in my usecase, there's a flag indicating whether the data is zstd encoded or not, and my plan was to decode or not based on that flag. data is also an array of InnerPackets. Something like this:

struct Packet {
    u8 is_zstd;
    u8 header_value;
    
    if (is_zstd) {
        u8 data_zstd[while(!std::mem::eof())] [[hidden]];
        std::mem::Section data_sec = std::mem::create_section("data_sec");
        hex::dec::zstd_decompress(data_zstd, data_sec);
        
        InnerPacket data[] @ 0x00 in data_sec [[export]];
    } else {
        InnerPacket data[while(!std::mem::eof())];
    }
};

As it stands, depending on the packet data will either end up under packet, or it will end up under the section viewer, which is kinda confusing.

If there's a way to always expose data consistently somewhere (global, under Packet, separate view) I'd be fine with that.

[rockisch]

Not sure if I should edit the issue description or create a new ticket.

[paxcut]

You have data that's part of the input file, the data can be zstd encoded or not but it is still part of the pattern that reads data from the input. The decoded version of the data is not like the data that's read but not encoded. The only way to have the two show in the same place would be to make a copy of the data that is not encoded and write it to the section without modifying it.

I don't know the exact technical details, but afaik it is not possible to bring section data into the main hex editor without overwriting the input file. Global variables that are not placed are restricted in the ways they can be used. If you try to use them in a way thats illegal you'll get an error stating that you can't use data that is not placed. That means that you can't use section data to create global variables. Another way of saying this is that if global variables could hold section data then there would be no need for sections to exist at all.

[rockisch]

Well, if there was a way to override the file buffer (instead of the file itself) I'd be fine with that as well, but I couldn't find a way in the documentation.

[paxcut]

Maybe virtual files can be helpful here. they can be opened as views. i have trouble loading projects that had views open when saved though so be careful if you want to try using them.
Another suggestion would be to write the decoded file to a new separate file on disk that can be opened and read but it sounds like thats not gonna do much.