-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal to reference DSD from data message via agencyID
and version
#145
Comments
Hi @hoehrmann, the need is not clear:
|
Summary: Details: Right now the requirement in the specification is satisfied by: data:
structures:
- links:
- rel: dataStructure
href: https://example.org/X347.xml There is no way to infer the URN of the DSD. This is valid because only any link to the DSD is required, specifying its URN is not required, and it does not have to be hosted as a SDMX-REST web service (otherwise you could guess the needed URN parts from the URL). This could be addressed by adding that for the dataStructure link the As for validation, take this example https://sdmx.oecd.org/public/rest/v2/data/dataflow/OECD.SDD.NAD.SEEA/DSD_NAT_RES@DF_NAT_RES/1.0/CAN.A.T.LEAD.*.A?. That claims to be a SDMX-JSON 2.0.0 data message. Ignoring the error in the It is probably possible to amend the schema to require that in this specific case there must be one link with As for lookups, the current requirement is satisfied by referencing a provisioning agreement. Even if you are lucky and it references a SDMX-REST end point, you can probably only get the dataflow with a single request. In theory you could use As for persistence, if I knew the URN I could try to look it up elsewhere (e.g., I may have old data and the web server just changed its address) but without it I would have to guess the URN (based on the IDs of the fields). |
Thanks for the clarifications. The intent of the current link requirement is to ensure that the artefact (either DF, DSD or ProvisionAgreement) for which data have been requested can be fully identified from the data message. The choice of the artefact type is not arbitrary but must correspond to the artefact used in the original data request. This was meant with the wording "At least the link to the Data Structure Definition, Dataflow or Data Provision Agreement to which the data relates is required.", but the wording in the field guide can be improved. E.g., if data was requested for a dataflow, then the dataflow identification is required. If data was requested for a dsd, then the dsd identification is required. Also, in order for the full artefact identification to be available immediately, that link requires the usage of 'self' for the relationship and indeed the URN of the artefact, e.g., "href": "https://registry.sdmx.org/ws/rest/dataflow/ECB.DISS/BSI_PUB/1.0",
"rel": "self",
"urn": "urn:sdmx:org.sdmx.infomodel.datastructure.dataflow=ECB.DISS:BSI_PUB(1.0)" This information is sufficient to retrieve all required structure artefacts in one single request. This can be further clarified in the field guide. If a client requires the structure information at a later time than the client is free to extract and store the structure information at the same time than the data. If you need the get just the DSD for a DF, then you could use the references=datastructure parameter. Disabling structure retrieval through references as a 'denial of service' protection seems to me an unreasonable approach. Compared to data extractions, structure messages are usually much smaller. I would conclude, that this ticket specifically requests that the URN of the underlying artefact can be found in a more straightforward way (without looping through the links array), by taking it out of the links array and adding it as a separate structure property (similar to the SDMX-ML data messages) or, e.g., by requiring to position that link as the first value of the links array. For issues you find in the practical SDMX implementation https://sdmx.oecd.org/public/rest/, could you please open tickets in this separate code repository https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-sdmxri-nsi-ws/-/issues/ ? |
I would like to add the following point: in a structure message external structures are referenced like this data:
dataStructures:
- id: EXAMPLE
agencyID: EXAMPLE
version: "1.0.0"
name: Example
isExternalReference: true Using |
Introduction:
The current SDMX-JSON data message specification requires that the
data.structure
object reference the Data Structure Definition (DSD) through a link to the Data Provision Agreement (DPA) or Dataflow (DF). This approach necessitates multiple lookups and parsing URN references to identify the relevant DSD. It also presents challenges when dealing with non-URN references or situations where the referenced web version becomes unavailable.Proposed Improvement:
To enhance clarity, convenience, and validation capabilities, this proposal recommends including the
agencyID
andversion
of the referenced DSD directly within thedata.structure
object of SDMX-JSON messages. This eliminates the need for intermediary references through DPA or DF and streamlines the process of identifying the relevant DSD.Benefits:
agencyID
andversion
is very simple.Implementation:
Modify the
data.structure
object schema to include two additional properties:agencyID
: reflects DSD agencyIDversion
: reflects DSD versionRemove requirement to have a links property.
Conclusion:
This proposal offers a more efficient and reliable approach to referencing DSDs within SDMX-JSON messages. Direct inclusion of
agencyID
andversion
within thedata.structure
object simplifies data access, enhances validation, and ensures persistent DSD references, fostering a more streamlined and robust data exchange experience.Alternative:
Having a
datastructure
URN reference would also be okay, but thenid
becomes redundant.(In doubt, please handle this as a public review comment on SDMX 3.1 once the comment period begins.)
The text was updated successfully, but these errors were encountered: