Skipped lines prior to csv data in a file

[jbertovic]

Hey... thanks for taking the initiative in updating xsv to qsv.

Just a question on skipped lines. I have a few csv files that come in with other information in the top n lines of the file before the header and csv data start. I pre-process this using some other program and resave the file before using tools like qsv. Is there a way to do this in qsv?

I could try to add the function similar that was proposed in xsv issue 96 with '--skip-lines' option on input.

[jqnatividad]

Hi @jbertovic , xsv just made our data pipelines much faster and easier to compose, and there were too many low-hanging pending PRs and ideas that we wanted ourselves, so I just scratched the itch. :)

And yes, implementing something like --skip-lines would be a great addition to qsv.

I think the most straightforward, cleanest way to enable it is to take advantage of the underlying comment handling capability of the CSV reader that's currently disabled by default.

Perhaps, a --comment option that's off by default, and if set to an ascii character, turns on comment handling using the specified character, mirroring behavior similar to the --delimiter option?

This should not only handle skipping lines before the header, it should also handle commented records in the rest of the CSV.

WDYT?

[jbertovic]

Hey @jqnatividad. Thanks for the response.

I can take a look at getting a --comment option to work. It will force me to look at the internals a bit which will give me an idea if a --skip-lines is worth implementing and how.

I see that --no-header option is located everywhere. Not sure if the --comment option would be needed everywhere....maybe just on the input subcommand. Thoughts?

[jqnatividad]

We can go for a KISS implementation, so you don't have to introduce a --comment option everywhere.

We can just turn on commenting by default with the default # character. Then it will just be a simple matter of adding it to the ReaderBuilder init (https://github.com/jqnatividad/qsv/blob/master/src/config.rs#L323)

Alternatively, if you want to go further, we can add another environment variable - QSV_COMMENTS with the following values:

• not set (commenting off, current behavior)
• set to an ascii character (turned on, using the specified ascii character)

[jqnatividad]

I just did a quick pass at the KISS implementation and added

.comment(Some(b'#')) to the ReaderBuilder init so its now:

        csv::ReaderBuilder::new()
            .flexible(self.flexible)
            .delimiter(self.delimiter)
            .has_headers(!self.no_headers)
            .quote(self.quote)
            .quoting(self.quoting)
            .escape(self.escape)
            .buffer_capacity(rdr_buffer)
            .comment(Some(b'#'))
            .from_reader(rdr)

and it works for skipping lines in both the header and the data part of the CSV.

If you do a quick PR, I'll merge it ASAP.

[jqnatividad]

For reference, line skipping using comments was functionally implemented by @jbertovic in #113.

He's still looking into implementing a specific --skip-lines option for the input command.

--skip-lines was also added in #266 and is now available starting with qsv v0.41.0.

cc @jbertovic @pkptzx