Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse flatfile with 0x00 null separator #49

Open
ajhalls opened this issue Oct 26, 2018 · 2 comments
Open

Parse flatfile with 0x00 null separator #49

ajhalls opened this issue Oct 26, 2018 · 2 comments

Comments

@ajhalls
Copy link

ajhalls commented Oct 26, 2018

I have a flatfile that uses fixed width columns, has 0x00 (hex null) separators but from what I can see, that also have null padding that I am trying to query.
Because the incoming flatfile doesn't have column headers, this looks like it would be a great tool to work with, but wasn't sure if it was possible to use a hex separator?

Here is an example of the data where � = null:

Ashlyn         �Vandiver       �Ashlyn         �    �4466743����85344
Bill               �Vandiver       �Bill               �    �4466744����85344

(this binary isn't the same as above, , just a representation)
94 DE 94 DE 86 03 64 00 65 00 D1 42 00 00 4A 6F 61 6E 6E 65 20 20 20 20 20 20 20 20 20 00 41 62 62 65 79 20 20 20 20 20 20 20 20 20 20 00 4A 6F 61 6E 6E
94 DE 94 DE 93 03 64 00 65 00 01 B6 03 00 4E 6F 72 6D 61 6E 20 20 20 20 20 20 20 20 20 00 45 61 74 6F 6E 20 20 20 20 20 20 20 20 20 20 00 4E 6F 72 6D 61
94 DE 94 DE EB 03 64 00 65 00 A9 6E 02 00 44 61 6E 69 65 6C 20 20 20 20 20 20 20 20 20 00 43 61 67 65 20 20 20 20 20 20 20 20 20 20 20 00 44 61 6E 69 65
94 DE 94 DE 85 03 64 00 65 00 D1 07 00 00 4A 75 6C 69 65 20 20 20 20 20 20 20 20 20 20 00 51 75 69 6E 63 79 20 20 20 20 20 20 20 20 20 00 4A 75 6C 69 65
5C D9 94 DE 94 03 64 00 65 00 0A 07 00 00 4D 61 72 79 20 20 20 20 20 20 20 20 20 20 20 00 43 61 75 6C 6B 20 20 20 20 20 20 20 20 20 20 00 4D 61 72 79 20

It looks like strings are padded with spaces, and integer values with null. I started writing some logic to play with trying to convert it to CSV, but I will have to do some magic with datatypes that you have already accomplished since I have byte groups that are the dates and integers stored in little endian.

                
for (int i = 0; i < block.Length; i++) //one fixed width line from flatfile
{                     
    if (i == block.Length)
    {
        line += "\n"; // this is the end of the line
    }

    if (i > 1 && i< block.Length)
    {
        // deal with null padding and create csv
        if (block[i] == 0x00 && block[i]+1 != 0x00)
        {
            line += "','";
        }else if (block[i] == 0x00 && block[i]-1 != 0x00)
        {
            line += "','";
        }
    }

    if (block[i]!=0x00)
    {
        line += (char)block[i]; // it isn't null, so display it
    }
}
@sorrell
Copy link
Owner

sorrell commented Oct 29, 2018

So if I'm reading this right, the question is strictly about using a hex value as the data separator?

@ajhalls
Copy link
Author

ajhalls commented Oct 29, 2018

I believe so, though as I noticed the INT columns may have NULL padding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants