Distribution composed of more than one file, but not packaged

[dr-shorthair]

A Distribution may be composed of multiple files which cannot be used independently, such as a shapefile and its attendant sidecars (index and database files). These might not be packaged into a single distributable artefact, such as a tar or zip archive (see #54 and #259) . So a dataset's distribution, while a single entity, is composed of multiple artefacts. We need to show patterns about how these will appear in a catalog.

[agbeltran]

One particular case for the multiple files of a distribution is that of the checksum files. This is something we want to include in DATS (see datatagsuite/schema#11) and it would be good to have specific vocabulary to refer to them.

[agreiner]

I have a use case where we have log files that we release as datasets. One "log" could contain hundreds of raw files, making it impractical to treat each as a separate dataset. They are often grouped in multiple layers of directories, such as a directory per cabinet, and within that a directory per chassis, and within that a directory per slot, etc. We've been releasing them as a single gzip, but they are getting so big that it's impractical to use a simple http download. (We offer Globus instead.)

[makxdekkers]

This seem to be a completely new requirement. And, given the resource and time constraints, I think it will be hard to incorporate a solution in the current revision of DCAT. I also think that it might be difficult to define a single approach/vocabulary that covers all possible relationships -- even the three examples from @dr-shorthair, @agbeltran and @agreiner seem to require different solutions.
In the current revision, you could do something with dcat:accessURL which you could use to point to a landing page where you link to various files and explain their relationships. The use of dcat:downloadURL is only possible for a single file according to the definition: "The URL of the downloadable file in a given format".
Should we leave this for future work?

[dr-shorthair]

@agreiner I wonder if this is just another case of part-whole, as discussed in #411 (proposed new property dcat:componentDistribution) ?

@makxdekkers Yes - I tend to agree - our docket is quite full. However, perhaps we can recommend the solution agreed for #256 (i.e. use dct:relation ) and leave stronger semantics to another day.

[agreiner]

dcat:componentDistribution seems like it would work for my use case. dct:relation, if restricted to files that have no relationship, would not. The tricky bit is to find a place to use componentDistribution. I don't fancy giving people separate RDF descriptions for each of the hundreds of files in a set. I would really like to have a property of a distribution that is a list of component parts, by relative path, sort of a manifest, or maybe even just a URI for a manifest. But we are creating our own vocabulary for handling log files, so we will invent our own for that if DCAT can't cover it.

[makxdekkers]

@agreiner You can always use dcat:accessURL and link to a page or directory where you files live, with a README file to explain what the files are. Or even use dcat:landingPage on the Dataset description to link to such a page.

[agreiner]

That would certainly work for the purposes of getting things into a catalog in a helpful way. What I've been hoping for is a way to reason about these things using RDF. I realize the main purpose of DCAT is just to let people catalog datasets, but it is so close to being useful for this use case, and there are now several related use cases that seem like they would benefit, it just seems a shame not to seize the opportunity.

[davebrowning]

While its clear that we have quite a bit to do, I'd prefer that we don't absolutely rule a problem area out quite yet. We have a list of high priority requirements referenced here as discussed and agreed at the F2F, and a target date of mid-January for the rec-track work. We've also talked about generating more examples and/or a primer after that date. That plan gives us some flex on how we can talk to this issue - extend timescales of the rec-track work, provide examples and suggestions or leave it for a further iteration. (All subject to agreement within the WG and the broader W3M)

On this specific case, I would agree with @agreiner's comment - it would be good to 'seize the opportunity' - and that it would be great if @dr-shorthair pithy summary "a dataset's distribution, while a single entity, is composed of multiple artefacts" found its way into the recommendation even if just a comment. I don't think it has higher priority than the requirements that we're focussing on now. But I hope we have the luxury of deferring any final inclusion decisions to January.

[smrgeoinfo]

For distributions that are aggregations, pointing to something like an OAI-ORE resource map would be a solution. See also DataOne discussion of packaging for an implementation