Skip to content

Commit 6d6a509

Browse files
sekmillersekmiller
authored
Merge pull request #10778 from johannes-darms/feat/10171-versions-smaller-response
Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter
2 parents 3352bb7 + bada794 commit 6d6a509

File tree

6 files changed

+75
-13
lines changed

6 files changed

+75
-13
lines changed

‎doc/release-notes/10171-exlude-metadatablocks.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter,
2+
that specifies whether the metadataBlocks should be listed in the output. It defaults to ``false``, preserving backward
3+
compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks
4+
included can dramatically increase the volume of the output). See also [the guides](https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset), #10778, and #10171.

‎doc/sphinx-guides/source/api/native-api.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1295,6 +1295,8 @@ It returns a list of versions with their metadata, and file list:
12951295
12961296
The optional ``excludeFiles`` parameter specifies whether the files should be listed in the output. It defaults to ``true``, preserving backward compatibility. (Note that for a dataset with a large number of versions and/or files having the files included can dramatically increase the volume of the output). A separate ``/files`` API can be used for listing the files, or a subset thereof in a given version.
12971297

1298+
The optional ``excludeMetadataBlocks`` parameter specifies whether the metadata blocks should be listed in the output. It defaults to ``false``, preserving backward compatibility. (Note that for a dataset with a large number of versions and/or metadata blocks having the metadata blocks included can dramatically increase the volume of the output).
1299+
12981300
The optional ``offset`` and ``limit`` parameters can be used to specify the range of the versions list to be shown. This can be used to paginate through the list in a dataset with a large number of versions.
12991301

13001302

@@ -1319,6 +1321,12 @@ The fully expanded example above (without environment variables) looks like this
13191321
13201322
The optional ``excludeFiles`` parameter specifies whether the files should be listed in the output (defaults to ``true``). Note that a separate ``/files`` API can be used for listing the files, or a subset thereof in a given version.
13211323

1324+
.. code-block:: bash
1325+
1326+
curl "https://demo.dataverse.org/api/datasets/24/versions/1.0?excludeMetadataBlocks=false"
1327+
1328+
The optional ``excludeMetadataBlocks`` parameter specifies whether the metadata blocks should be listed in the output (defaults to ``false``).
1329+
13221330

13231331
By default, deaccessioned dataset versions are not included in the search when applying the :latest or :latest-published identifiers. Additionally, when filtering by a specific version tag, you will get a "not found" error if the version is deaccessioned and you do not enable the ``includeDeaccessioned`` option described below.
13241332

‎src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -421,15 +421,16 @@ public Response useDefaultCitationDate(@Context ContainerRequestContext crc, @Pa
421421
@GET
422422
@AuthRequired
423423
@Path("{id}/versions")
424-
public Response listVersions(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("excludeFiles") Boolean excludeFiles, @QueryParam("limit") Integer limit, @QueryParam("offset") Integer offset) {
424+
public Response listVersions(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("excludeFiles") Boolean excludeFiles,@QueryParam("excludeMetadataBlocks") Boolean excludeMetadataBlocks, @QueryParam("limit") Integer limit, @QueryParam("offset") Integer offset) {
425425

426426
return response( req -> {
427427
Dataset dataset = findDatasetOrDie(id);
428428
Boolean deepLookup = excludeFiles == null ? true : !excludeFiles;
429+
Boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks;
429430

430431
return ok( execCommand( new ListVersionsCommand(req, dataset, offset, limit, deepLookup) )
431432
.stream()
432-
.map( d -> json(d, deepLookup) )
433+
.map( d -> json(d, deepLookup, includeMetadataBlocks) )
433434
.collect(toJsonArray()));
434435
}, getRequestUser(crc));
435436
}
@@ -441,6 +442,7 @@ public Response getVersion(@Context ContainerRequestContext crc,
441442
@PathParam("id") String datasetId,
442443
@PathParam("versionId") String versionId,
443444
@QueryParam("excludeFiles") Boolean excludeFiles,
445+
@QueryParam("excludeMetadataBlocks") Boolean excludeMetadataBlocks,
444446
@QueryParam("includeDeaccessioned") boolean includeDeaccessioned,
445447
@QueryParam("returnOwners") boolean returnOwners,
446448
@Context UriInfo uriInfo,
@@ -466,11 +468,12 @@ public Response getVersion(@Context ContainerRequestContext crc,
466468
if (excludeFiles == null ? true : !excludeFiles) {
467469
requestedDatasetVersion = datasetversionService.findDeep(requestedDatasetVersion.getId());
468470
}
471+
Boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks;
469472

470473
JsonObjectBuilder jsonBuilder = json(requestedDatasetVersion,
471474
null,
472-
excludeFiles == null ? true : !excludeFiles,
473-
returnOwners);
475+
excludeFiles == null ? true : !excludeFiles,
476+
returnOwners, includeMetadataBlocks);
474477
return ok(jsonBuilder);
475478

476479
}, getRequestUser(crc));

‎src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -423,11 +423,17 @@ public static JsonObjectBuilder json(FileDetailsHolder ds) {
423423
}
424424

425425
public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles) {
426-
return json(dsv, null, includeFiles, false);
426+
return json(dsv, null, includeFiles, false,true);
427+
}
428+
public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles, boolean includeMetadataBlocks) {
429+
return json(dsv, null, includeFiles, false, includeMetadataBlocks);
430+
}
431+
public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymizedFieldTypeNamesList,
432+
boolean includeFiles, boolean returnOwners) {
433+
return json( dsv, anonymizedFieldTypeNamesList, includeFiles, returnOwners,true);
427434
}
428-
429435
public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymizedFieldTypeNamesList,
430-
boolean includeFiles, boolean returnOwners) {
436+
boolean includeFiles, boolean returnOwners, boolean includeMetadataBlocks) {
431437
Dataset dataset = dsv.getDataset();
432438
JsonObjectBuilder bld = jsonObjectBuilder()
433439
.add("id", dsv.getId()).add("datasetId", dataset.getId())
@@ -472,11 +478,12 @@ public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymized
472478
.add("sizeOfCollection", dsv.getTermsOfUseAndAccess().getSizeOfCollection())
473479
.add("studyCompletion", dsv.getTermsOfUseAndAccess().getStudyCompletion())
474480
.add("fileAccessRequest", dsv.getTermsOfUseAndAccess().isFileAccessRequest());
475-
476-
bld.add("metadataBlocks", (anonymizedFieldTypeNamesList != null) ?
477-
jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList)
478-
: jsonByBlocks(dsv.getDatasetFields())
479-
);
481+
if(includeMetadataBlocks) {
482+
bld.add("metadataBlocks", (anonymizedFieldTypeNamesList != null) ?
483+
jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList)
484+
: jsonByBlocks(dsv.getDatasetFields())
485+
);
486+
}
480487
if(returnOwners){
481488
bld.add("isPartOf", getOwnersFromDvObject(dataset));
482489
}

‎src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -731,6 +731,42 @@ public void testCreatePublishDestroyDataset() {
731731

732732
}
733733

734+
@Test
735+
public void testHideMetadataBlocksInDatasetVersionsAPI() {
736+
737+
// Create user
738+
String apiToken = UtilIT.createRandomUserGetToken();
739+
740+
// Create user with no permission
741+
String apiTokenNoPerms = UtilIT.createRandomUserGetToken();
742+
743+
// Create Collection
744+
String collectionAlias = UtilIT.createRandomCollectionGetAlias(apiToken);
745+
746+
// Create Dataset
747+
Response createDataset = UtilIT.createRandomDatasetViaNativeApi(collectionAlias, apiToken);
748+
createDataset.then().assertThat()
749+
.statusCode(CREATED.getStatusCode());
750+
751+
Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset);
752+
String datasetPid = JsonPath.from(createDataset.asString()).getString("data.persistentId");
753+
754+
// Now check that the metadata is NOT shown, when we ask the versions api to dos o.
755+
boolean excludeMetadata = true;
756+
Response unpublishedDraft = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_DRAFT, apiToken, true,excludeMetadata, false);
757+
unpublishedDraft.prettyPrint();
758+
unpublishedDraft.then().assertThat()
759+
.statusCode(OK.getStatusCode())
760+
.body("data.metadataBlocks", equalTo(null));
761+
762+
// Now check that the metadata is shown, when we ask the versions api to dos o.
763+
excludeMetadata = false;
764+
unpublishedDraft = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_DRAFT, apiToken,true, excludeMetadata, false);
765+
unpublishedDraft.prettyPrint();
766+
unpublishedDraft.then().assertThat()
767+
.statusCode(OK.getStatusCode())
768+
.body("data.metadataBlocks", notNullValue() );
769+
}
734770
/**
735771
* The apis (/api/datasets/{id}/versions and /api/datasets/{id}/versions/{vid}
736772
* are already called from other RestAssured tests, in this class and also in FilesIT.

‎src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1623,14 +1623,18 @@ static Response getDatasetVersion(String persistentId, String versionNumber, Str
16231623
}
16241624

16251625
static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, boolean excludeFiles, boolean includeDeaccessioned) {
1626+
return getDatasetVersion(persistentId,versionNumber,apiToken,excludeFiles,false,includeDeaccessioned);
1627+
}
1628+
static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, boolean excludeFiles,boolean excludeMetadataBlocks, boolean includeDeaccessioned) {
16261629
return given()
16271630
.header(API_TOKEN_HTTP_HEADER, apiToken)
16281631
.queryParam("includeDeaccessioned", includeDeaccessioned)
16291632
.get("/api/datasets/:persistentId/versions/"
16301633
+ versionNumber
16311634
+ "?persistentId="
16321635
+ persistentId
1633-
+ (excludeFiles ? "&excludeFiles=true" : ""));
1636+
+ (excludeFiles ? "&excludeFiles=true" : "")
1637+
+ (excludeMetadataBlocks ? "&excludeMetadataBlocks=true" : ""));
16341638
}
16351639
static Response compareDatasetVersions(String persistentId, String versionNumber1, String versionNumber2, String apiToken) {
16361640
return given()

0 commit comments

Comments
 (0)