feat(sparql-anything): chunk a stream of lines, not only a file

[ddeboer]

chunk() took a path, so a caller with work to do before chunking – filtering rows out, adding a column – had to write its result to disk only for chunk() to read it back and write the same bytes again. In geonames-rdf that is about 2.4 GB written and read per weekly run, with peak disk holding a table and its chunks at once.

It now takes a stream of lines as well:

const chunks = await chunk(withCountryColumn(rowsOf('allCountries.txt')), {
  rows: 1_000_000,
  into: 'data/chunks',
  name: 'allCountries',
  extension: '.csv',
});

Anything that yields lines will do: an async generator, or a Readable in string mode.

What a stream has to say for itself

ChunkOptions gains name, and extension loses its default for a stream. Both are required of one, and the overload asks for them:

• name is what the chunks are called after. It defaults to a path input's file name, which is where chunk names came from until now. A stream has none, and a call that leaves it out anyway is rejected with a message rather than producing -0000 files. A name that is a path is rejected too, since it would write chunks outside into.
• extension decides how a chunk is read – SPARQL Anything takes the format from the name – so defaulting it to none for a stream would hand back chunks this package's own converter cannot classify. '' says none, explicitly.

One value, one row

A value carrying a line ending of its own is counted as one row and written as several, so a chunk would hold more than rows says – the number a conversion's memory is sized against (ADR 12). It is now refused, naming the value, and a trailing \r is dropped, which is the normalisation a file already gets from readline.

That check also catches a byte stream. createReadStream() yields buffers that fall wherever the reads did rather than on lines, and a Readable satisfies the type, so without it the records would be cut at a 64 KB boundary and nothing would say so.

Failing

A write that fails now stops the lines being pulled. For a file that was already the case, by closing the reader; a stream is pulled a value at a time, so the loop checks and breaks instead, and a producer that is itself a pipeline is not left reading its own input long after there is anywhere to put it.

Everything else is unchanged: the loop body, removeChunksOf() keying on the name, the refusal of an input with no rows. Chunking a path keeps working exactly as before.

Fix #830