Skip to content

Commit b2c918a

Browse files
GPortasGPortas
committed
Merge branch 'develop' of github.com:IQSS/dataverse into 10943-featured-items
2 parents 3df4723 + 6d6a509 commit b2c918a

File tree

13 files changed

+151
-50
lines changed

13 files changed

+151
-50
lines changed

‎doc/release-notes/10171-exlude-metadatablocks.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter,
2+
that specifies whether the metadataBlocks should be listed in the output. It defaults to ``false``, preserving backward
3+
compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks
4+
included can dramatically increase the volume of the output). See also [the guides](https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset), #10778, and #10171.

‎doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
## Improvement and internationalization of harvest status
2+
3+
Added a harvest status to differentiate a complete harvest with errors (Completed with failures) and without errors (Completed)
4+
Harvest status labels are now internationalized
5+
6+
For more information, see issue [#9294](https://github.com/IQSS/dataverse/issues/9294)

‎doc/sphinx-guides/source/api/native-api.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1295,6 +1295,8 @@ It returns a list of versions with their metadata, and file list:
12951295
12961296
The optional ``excludeFiles`` parameter specifies whether the files should be listed in the output. It defaults to ``true``, preserving backward compatibility. (Note that for a dataset with a large number of versions and/or files having the files included can dramatically increase the volume of the output). A separate ``/files`` API can be used for listing the files, or a subset thereof in a given version.
12971297

1298+
The optional ``excludeMetadataBlocks`` parameter specifies whether the metadata blocks should be listed in the output. It defaults to ``false``, preserving backward compatibility. (Note that for a dataset with a large number of versions and/or metadata blocks having the metadata blocks included can dramatically increase the volume of the output).
1299+
12981300
The optional ``offset`` and ``limit`` parameters can be used to specify the range of the versions list to be shown. This can be used to paginate through the list in a dataset with a large number of versions.
12991301

13001302

@@ -1319,6 +1321,12 @@ The fully expanded example above (without environment variables) looks like this
13191321
13201322
The optional ``excludeFiles`` parameter specifies whether the files should be listed in the output (defaults to ``true``). Note that a separate ``/files`` API can be used for listing the files, or a subset thereof in a given version.
13211323

1324+
.. code-block:: bash
1325+
1326+
curl "https://demo.dataverse.org/api/datasets/24/versions/1.0?excludeMetadataBlocks=false"
1327+
1328+
The optional ``excludeMetadataBlocks`` parameter specifies whether the metadata blocks should be listed in the output (defaults to ``false``).
1329+
13221330

13231331
By default, deaccessioned dataset versions are not included in the search when applying the :latest or :latest-published identifiers. Additionally, when filtering by a specific version tag, you will get a "not found" error if the version is deaccessioned and you do not enable the ``includeDeaccessioned`` option described below.
13241332

‎src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -421,15 +421,16 @@ public Response useDefaultCitationDate(@Context ContainerRequestContext crc, @Pa
421421
@GET
422422
@AuthRequired
423423
@Path("{id}/versions")
424-
public Response listVersions(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("excludeFiles") Boolean excludeFiles, @QueryParam("limit") Integer limit, @QueryParam("offset") Integer offset) {
424+
public Response listVersions(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("excludeFiles") Boolean excludeFiles,@QueryParam("excludeMetadataBlocks") Boolean excludeMetadataBlocks, @QueryParam("limit") Integer limit, @QueryParam("offset") Integer offset) {
425425

426426
return response( req -> {
427427
Dataset dataset = findDatasetOrDie(id);
428428
Boolean deepLookup = excludeFiles == null ? true : !excludeFiles;
429+
Boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks;
429430

430431
return ok( execCommand( new ListVersionsCommand(req, dataset, offset, limit, deepLookup) )
431432
.stream()
432-
.map( d -> json(d, deepLookup) )
433+
.map( d -> json(d, deepLookup, includeMetadataBlocks) )
433434
.collect(toJsonArray()));
434435
}, getRequestUser(crc));
435436
}
@@ -441,6 +442,7 @@ public Response getVersion(@Context ContainerRequestContext crc,
441442
@PathParam("id") String datasetId,
442443
@PathParam("versionId") String versionId,
443444
@QueryParam("excludeFiles") Boolean excludeFiles,
445+
@QueryParam("excludeMetadataBlocks") Boolean excludeMetadataBlocks,
444446
@QueryParam("includeDeaccessioned") boolean includeDeaccessioned,
445447
@QueryParam("returnOwners") boolean returnOwners,
446448
@Context UriInfo uriInfo,
@@ -466,11 +468,12 @@ public Response getVersion(@Context ContainerRequestContext crc,
466468
if (excludeFiles == null ? true : !excludeFiles) {
467469
requestedDatasetVersion = datasetversionService.findDeep(requestedDatasetVersion.getId());
468470
}
471+
Boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks;
469472

470473
JsonObjectBuilder jsonBuilder = json(requestedDatasetVersion,
471474
null,
472-
excludeFiles == null ? true : !excludeFiles,
473-
returnOwners);
475+
excludeFiles == null ? true : !excludeFiles,
476+
returnOwners, includeMetadataBlocks);
474477
return ok(jsonBuilder);
475478

476479
}, getRequestUser(crc));

‎src/main/java/edu/harvard/iq/dataverse/harvest/client/ClientHarvestRun.java

Lines changed: 40 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,10 @@
66
package edu.harvard.iq.dataverse.harvest.client;
77

88
import java.io.Serializable;
9+
import java.util.Arrays;
910
import java.util.Date;
11+
12+
import edu.harvard.iq.dataverse.util.BundleUtil;
1013
import jakarta.persistence.Entity;
1114
import jakarta.persistence.GeneratedValue;
1215
import jakarta.persistence.GenerationType;
@@ -40,13 +43,7 @@ public void setId(Long id) {
4043
this.id = id;
4144
}
4245

43-
public enum RunResultType { SUCCESS, FAILURE, INPROGRESS, INTERRUPTED };
44-
45-
private static String RESULT_LABEL_SUCCESS = "SUCCESS";
46-
private static String RESULT_LABEL_FAILURE = "FAILED";
47-
private static String RESULT_LABEL_INPROGRESS = "IN PROGRESS";
48-
private static String RESULT_DELETE_IN_PROGRESS = "DELETE IN PROGRESS";
49-
private static String RESULT_LABEL_INTERRUPTED = "INTERRUPTED";
46+
public enum RunResultType { COMPLETED, COMPLETED_WITH_FAILURES, FAILURE, IN_PROGRESS, INTERRUPTED }
5047

5148
@ManyToOne
5249
@JoinColumn(nullable = false)
@@ -68,36 +65,43 @@ public RunResultType getResult() {
6865

6966
public String getResultLabel() {
7067
if (harvestingClient != null && harvestingClient.isDeleteInProgress()) {
71-
return RESULT_DELETE_IN_PROGRESS;
68+
return BundleUtil.getStringFromBundle("harvestclients.result.deleteInProgress");
7269
}
73-
74-
if (isSuccess()) {
75-
return RESULT_LABEL_SUCCESS;
70+
71+
if (isCompleted()) {
72+
return BundleUtil.getStringFromBundle("harvestclients.result.completed");
73+
} else if (isCompletedWithFailures()) {
74+
return BundleUtil.getStringFromBundle("harvestclients.result.completedWithFailures");
7675
} else if (isFailed()) {
77-
return RESULT_LABEL_FAILURE;
76+
return BundleUtil.getStringFromBundle("harvestclients.result.failure");
7877
} else if (isInProgress()) {
79-
return RESULT_LABEL_INPROGRESS;
78+
return BundleUtil.getStringFromBundle("harvestclients.result.inProgess");
8079
} else if (isInterrupted()) {
81-
return RESULT_LABEL_INTERRUPTED;
80+
return BundleUtil.getStringFromBundle("harvestclients.result.interrupted");
8281
}
8382
return null;
8483
}
8584

8685
public String getDetailedResultLabel() {
8786
if (harvestingClient != null && harvestingClient.isDeleteInProgress()) {
88-
return RESULT_DELETE_IN_PROGRESS;
87+
return BundleUtil.getStringFromBundle("harvestclients.result.deleteInProgress");
8988
}
90-
if (isSuccess() || isInterrupted()) {
89+
if (isCompleted() || isCompletedWithFailures() || isInterrupted()) {
9190
String resultLabel = getResultLabel();
92-
93-
resultLabel = resultLabel.concat("; "+harvestedDatasetCount+" harvested, ");
94-
resultLabel = resultLabel.concat(deletedDatasetCount+" deleted, ");
95-
resultLabel = resultLabel.concat(failedDatasetCount+" failed.");
91+
92+
String details = BundleUtil.getStringFromBundle("harvestclients.result.details", Arrays.asList(
93+
harvestedDatasetCount.toString(),
94+
deletedDatasetCount.toString(),
95+
failedDatasetCount.toString()
96+
));
97+
if(details != null) {
98+
resultLabel = resultLabel + "; " + details;
99+
}
96100
return resultLabel;
97101
} else if (isFailed()) {
98-
return RESULT_LABEL_FAILURE;
102+
return BundleUtil.getStringFromBundle("harvestclients.result.failure");
99103
} else if (isInProgress()) {
100-
return RESULT_LABEL_INPROGRESS;
104+
return BundleUtil.getStringFromBundle("harvestclients.result.inProgess");
101105
}
102106
return null;
103107
}
@@ -106,12 +110,20 @@ public void setResult(RunResultType harvestResult) {
106110
this.harvestResult = harvestResult;
107111
}
108112

109-
public boolean isSuccess() {
110-
return RunResultType.SUCCESS == harvestResult;
113+
public boolean isCompleted() {
114+
return RunResultType.COMPLETED == harvestResult;
115+
}
116+
117+
public void setCompleted() {
118+
harvestResult = RunResultType.COMPLETED;
119+
}
120+
121+
public boolean isCompletedWithFailures() {
122+
return RunResultType.COMPLETED_WITH_FAILURES == harvestResult;
111123
}
112124

113-
public void setSuccess() {
114-
harvestResult = RunResultType.SUCCESS;
125+
public void setCompletedWithFailures() {
126+
harvestResult = RunResultType.COMPLETED_WITH_FAILURES;
115127
}
116128

117129
public boolean isFailed() {
@@ -123,12 +135,12 @@ public void setFailed() {
123135
}
124136

125137
public boolean isInProgress() {
126-
return RunResultType.INPROGRESS == harvestResult ||
138+
return RunResultType.IN_PROGRESS == harvestResult ||
127139
(harvestResult == null && startTime != null && finishTime == null);
128140
}
129141

130142
public void setInProgress() {
131-
harvestResult = RunResultType.INPROGRESS;
143+
harvestResult = RunResultType.IN_PROGRESS;
132144
}
133145

134146
public boolean isInterrupted() {

‎src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,7 @@ public void doHarvest(DataverseRequest dataverseRequest, Long harvestingClientId
163163

164164
try {
165165
if (harvestingClientConfig.isHarvestingNow()) {
166-
hdLogger.log(Level.SEVERE, "Cannot start harvest, client " + harvestingClientConfig.getName() + " is already harvesting.");
166+
hdLogger.log(Level.SEVERE, String.format("Cannot start harvest, client %s is already harvesting.", harvestingClientConfig.getName()));
167167

168168
} else {
169169
harvestingClientService.resetHarvestInProgress(harvestingClientId);
@@ -176,9 +176,16 @@ public void doHarvest(DataverseRequest dataverseRequest, Long harvestingClientId
176176
} else {
177177
throw new IOException("Unsupported harvest type");
178178
}
179-
harvestingClientService.setHarvestSuccess(harvestingClientId, new Date(), harvestedDatasetIds.size(), failedIdentifiers.size(), deletedIdentifiers.size());
180-
hdLogger.log(Level.INFO, "COMPLETED HARVEST, server=" + harvestingClientConfig.getArchiveUrl() + ", metadataPrefix=" + harvestingClientConfig.getMetadataPrefix());
181-
hdLogger.log(Level.INFO, "Datasets created/updated: " + harvestedDatasetIds.size() + ", datasets deleted: " + deletedIdentifiers.size() + ", datasets failed: " + failedIdentifiers.size());
179+
180+
if (failedIdentifiers.isEmpty()) {
181+
harvestingClientService.setHarvestCompleted(harvestingClientId, new Date(), harvestedDatasetIds.size(), failedIdentifiers.size(), deletedIdentifiers.size());
182+
hdLogger.log(Level.INFO, String.format("\"COMPLETED HARVEST, server=%s, metadataPrefix=%s", harvestingClientConfig.getArchiveUrl(), harvestingClientConfig.getMetadataPrefix()));
183+
} else {
184+
harvestingClientService.setHarvestCompletedWithFailures(harvestingClientId, new Date(), harvestedDatasetIds.size(), failedIdentifiers.size(), deletedIdentifiers.size());
185+
hdLogger.log(Level.INFO, String.format("\"COMPLETED HARVEST WITH FAILURES, server=%s, metadataPrefix=%s", harvestingClientConfig.getArchiveUrl(), harvestingClientConfig.getMetadataPrefix()));
186+
}
187+
188+
hdLogger.log(Level.INFO, String.format("Datasets created/updated: %s, datasets deleted: %s, datasets failed: %s", harvestedDatasetIds.size(), deletedIdentifiers.size(), failedIdentifiers.size()));
182189

183190
}
184191
} catch (StopHarvestException she) {

‎src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ public ClientHarvestRun getLastSuccessfulRun() {
297297
int i = harvestHistory.size() - 1;
298298

299299
while (i > -1) {
300-
if (harvestHistory.get(i).isSuccess()) {
300+
if (harvestHistory.get(i).isCompleted() || harvestHistory.get(i).isCompletedWithFailures()) {
301301
return harvestHistory.get(i);
302302
}
303303
i--;
@@ -314,7 +314,7 @@ ClientHarvestRun getLastNonEmptyRun() {
314314
int i = harvestHistory.size() - 1;
315315

316316
while (i > -1) {
317-
if (harvestHistory.get(i).isSuccess()) {
317+
if (harvestHistory.get(i).isCompleted() || harvestHistory.get(i).isCompletedWithFailures()) {
318318
if (harvestHistory.get(i).getHarvestedDatasetCount().longValue() > 0 ||
319319
harvestHistory.get(i).getDeletedDatasetCount().longValue() > 0) {
320320
return harvestHistory.get(i);

‎src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClientServiceBean.java

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -164,8 +164,13 @@ public void deleteClient(Long clientId) {
164164
}
165165

166166
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
167-
public void setHarvestSuccess(Long hcId, Date currentTime, int harvestedCount, int failedCount, int deletedCount) {
168-
recordHarvestJobStatus(hcId, currentTime, harvestedCount, failedCount, deletedCount, ClientHarvestRun.RunResultType.SUCCESS);
167+
public void setHarvestCompleted(Long hcId, Date currentTime, int harvestedCount, int failedCount, int deletedCount) {
168+
recordHarvestJobStatus(hcId, currentTime, harvestedCount, failedCount, deletedCount, ClientHarvestRun.RunResultType.COMPLETED);
169+
}
170+
171+
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
172+
public void setHarvestCompletedWithFailures(Long hcId, Date currentTime, int harvestedCount, int failedCount, int deletedCount) {
173+
recordHarvestJobStatus(hcId, currentTime, harvestedCount, failedCount, deletedCount, ClientHarvestRun.RunResultType.COMPLETED_WITH_FAILURES);
169174
}
170175

171176
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)

‎src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -423,11 +423,17 @@ public static JsonObjectBuilder json(FileDetailsHolder ds) {
423423
}
424424

425425
public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles) {
426-
return json(dsv, null, includeFiles, false);
426+
return json(dsv, null, includeFiles, false,true);
427+
}
428+
public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles, boolean includeMetadataBlocks) {
429+
return json(dsv, null, includeFiles, false, includeMetadataBlocks);
430+
}
431+
public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymizedFieldTypeNamesList,
432+
boolean includeFiles, boolean returnOwners) {
433+
return json( dsv, anonymizedFieldTypeNamesList, includeFiles, returnOwners,true);
427434
}
428-
429435
public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymizedFieldTypeNamesList,
430-
boolean includeFiles, boolean returnOwners) {
436+
boolean includeFiles, boolean returnOwners, boolean includeMetadataBlocks) {
431437
Dataset dataset = dsv.getDataset();
432438
JsonObjectBuilder bld = jsonObjectBuilder()
433439
.add("id", dsv.getId()).add("datasetId", dataset.getId())
@@ -472,11 +478,12 @@ public static JsonObjectBuilder json(DatasetVersion dsv, List<String> anonymized
472478
.add("sizeOfCollection", dsv.getTermsOfUseAndAccess().getSizeOfCollection())
473479
.add("studyCompletion", dsv.getTermsOfUseAndAccess().getStudyCompletion())
474480
.add("fileAccessRequest", dsv.getTermsOfUseAndAccess().isFileAccessRequest());
475-
476-
bld.add("metadataBlocks", (anonymizedFieldTypeNamesList != null) ?
477-
jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList)
478-
: jsonByBlocks(dsv.getDatasetFields())
479-
);
481+
if(includeMetadataBlocks) {
482+
bld.add("metadataBlocks", (anonymizedFieldTypeNamesList != null) ?
483+
jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList)
484+
: jsonByBlocks(dsv.getDatasetFields())
485+
);
486+
}
480487
if(returnOwners){
481488
bld.add("isPartOf", getOwnersFromDvObject(dataset));
482489
}

‎src/main/java/propertyFiles/Bundle.properties

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -636,6 +636,13 @@ harvestclients.viewEditDialog.archiveDescription.tip=Description of the archival
636636
harvestclients.viewEditDialog.archiveDescription.default.generic=This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data.
637637
harvestclients.viewEditDialog.btn.save=Save Changes
638638
harvestclients.newClientDialog.title.edit=Edit Group {0}
639+
harvestclients.result.completed=Completed
640+
harvestclients.result.completedWithFailures=Completed with failures
641+
harvestclients.result.failure=FAILED
642+
harvestclients.result.inProgess=IN PROGRESS
643+
harvestclients.result.deleteInProgress=DELETE IN PROGRESS
644+
harvestclients.result.interrupted=INTERRUPTED
645+
harvestclients.result.details={0} harvested, {1} deleted, {2} failed.
639646

640647
#harvestset.xhtml
641648
harvestserver.title=Manage Harvesting Server

0 commit comments

Comments
 (0)