diff --git a/doc/release-notes/5.12.1-release-notes.md b/doc/release-notes/5.12.1-release-notes.md new file mode 100644 index 00000000000..aa8896660f3 --- /dev/null +++ b/doc/release-notes/5.12.1-release-notes.md @@ -0,0 +1,115 @@ +# Dataverse Software 5.12.1 + +This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. + +## Release Highlights + +### Bug Fix for "Internal Server Error" When Creating a New Remote Account + +Unfortunately, as of 5.11 new remote users have seen "Internal Server Error" when creating an account (or checking notifications just after creating an account). Remote users are those who log in with institutional (Shibboleth), OAuth (ORCID, GitHub, or Google) or OIDC providers. + +This is a transient error that can be worked around by reloading the browser (or logging out and back in again) but it's obviously a very poor user experience and a bad first impression. This bug is the primary reason we are putting out this patch release. Other features and bug fixes are coming along for the ride. + +### Ability to Disable OAuth Sign Up While Allowing Existing Accounts to Log In + +A new option called `:AllowRemoteAuthSignUp` has been added providing a mechanism for disabling new account signups for specific OAuth2 authentication providers (Orcid, GitHub, Google etc.) while still allowing logins for already-existing accounts using this authentication method. + +See the [Installation Guide](https://guides.dataverse.org/en/5.12.1/installation/config.html#allowremoteauthsignup) for more information on the setting. + +### Production Date Now Used for Harvested Datasets in Addition to Distribution Date (`oai_dc` format) + +Fix the year displayed in citation for harvested dataset, especially for `oai_dc` format. + +For normal datasets, the date used is the "citation date" which is by default the publication date (the first release date) unless you [change it](https://guides.dataverse.org/en/5.12.1/api/native-api.html#set-citation-date-field-type-for-a-dataset). + +However, for a harvested dataset, the distribution date was used instead and this date is not always present in the harvested metadata. + +Now, the production date is used for harvested dataset in addition to distribution date when harvesting with the `oai_dc` format. + +### Publication Date Now Used for Harvested Dataset if Production Date is Not Set (`oai_dc` format) + +For exports and harvesting in `oai_dc` format, if "Production Date" is not set, "Publication Date" is now used instead. This change is reflected in the [Dataverse 4+ Metadata Crosswalk][] linked from the [Appendix][] of the User Guide. + +[Dataverse 4+ Metadata Crosswalk]: https://docs.google.com/spreadsheets/d/10Luzti7svVTVKTA-px27oq3RxCUM-QbiTkm8iMd5C54/edit#gid=1901625433&range=K7 +[Appendix]: https://guides.dataverse.org/en/5.12.1/user/appendix.html + +## Major Use Cases and Infrastructure Enhancements + +Changes and fixes in this release include: + +- Users creating an account by logging in with Shibboleth, OAuth, or OIDC should not see errors. (Issue 9029, PR #9030) +- When harvesting datasets, I want the Production Date if I can't get the Distribution Date (PR #8732) +- When harvesting datasets, I want the Publication Date if I can't get the Production Date (PR #8733) +- As a sysadmin I'd like to disable (temporarily or permanently) sign ups from OAuth providers while allowing existing users to continue to log in from that provider (PR #9112) +- As a C/C++ developer I want to use Dataverse APIs (PR #9070) + +## New DB Settings + +The following DB settings have been added: + +- `:AllowRemoteAuthSignUp` + +See the [Database Settings](https://guides.dataverse.org/en/5.12.1/installation/config.html#database-settings) section of the Guides for more information. + +## Complete List of Changes + +For the complete list of code changes in this release, see the [5.12.1 Milestone](https://github.com/IQSS/dataverse/milestone/106?closed=1) in GitHub. + +For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. + +## Installation + +If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.12.1/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.10/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already. + +## Upgrade Instructions + +Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc. + +0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.12.1. + +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. + +```shell +export PAYARA=/usr/local/payara5 +``` + +(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) + +1\. Undeploy the previous version + +```shell + $PAYARA/bin/asadmin list-applications + $PAYARA/bin/asadmin undeploy dataverse<-version> +``` + +2\. Stop Payara + +```shell + service payara stop + rm -rf $PAYARA/glassfish/domains/domain1/generated +``` + +6\. Start Payara + +```shell + service payara start +``` + +7\. Deploy this version. + +```shell + $PAYARA/bin/asadmin deploy dataverse-5.12.1.war +``` + +8\. Restart payara + +```shell + service payara stop + service payara start +``` + +## Upcoming Versions of Payara + +With the recent release of Payara 6 ([Payara 6.2022.1](https://github.com/payara/Payara/releases/tag/payara-server-6.2022.1) being the first version), the days of free-to-use Payara 5.x Platform Community versions [are numbered](https://blog.payara.fish/whats-new-in-the-november-2022-payara-platform-release). Specifically, Payara's blog post says, "Payara Platform Community 5.2022.4 has been released today as the penultimate Payara 5 Community release." + +Given the end of free-to-use Payara 5 versions, we plan to get the Dataverse software working on Payara 6 (#8305), which will require substantial efforts from the IQSS team and community members, as this also means shifting our app to be a [Jakarta EE 10](https://jakarta.ee/release/10/) application (upgrading from EE 8). We are currently working out the details and will share news as soon as we can. Rest assured we will do our best to provide you with a smooth transition. You can follow along in Issue #8305 and related pull requests and you are, of course, very welcome to participate by testing and otherwise contributing, as always. diff --git a/doc/sphinx-guides/source/_static/docsdataverse_org.css b/doc/sphinx-guides/source/_static/docsdataverse_org.css index e4afe89e217..da4ba06ddd4 100755 --- a/doc/sphinx-guides/source/_static/docsdataverse_org.css +++ b/doc/sphinx-guides/source/_static/docsdataverse_org.css @@ -68,7 +68,7 @@ a.headerlink { #sidebar.bs-sidenav { background-color: #f8d5b8; } -#sidebar.bs-sidenav .nav > li > a:hover, #sidebar.bs-sidenav .nav > li > a:focus { +#sidebar.bs-sidenav .nav > li > a:hover, #sidebar.bs-sidenav .nav > li > a:focus, #sidebar.bs-sidenav .nav > li > a.current { background-color: #fbf4c5; border-right: 1px solid #dbd8e0; text-decoration: none; diff --git a/doc/sphinx-guides/source/_static/util/clear_timer.sh b/doc/sphinx-guides/source/_static/util/clear_timer.sh index 3fcd9e8a387..1d9966e4e07 100755 --- a/doc/sphinx-guides/source/_static/util/clear_timer.sh +++ b/doc/sphinx-guides/source/_static/util/clear_timer.sh @@ -17,7 +17,7 @@ DV_DIR=${PAYARA_DIR}/glassfish/domains/domain1 ${PAYARA_DIR}/bin/asadmin stop-domain rm -rf ${PAYARA_DIR}/${DV_DIR}/generated/ -rm -rf ${PAYARA_DIR}/${DV_DIR}/osgi-cache/felix +rm -rf ${PAYARA_DIR}/${DV_DIR}/osgi-cache/ # restart the domain (also generates a warning if app server is stopped) ${PAYARA_DIR}/bin/asadmin start-domain diff --git a/doc/sphinx-guides/source/admin/dataverses-datasets.rst b/doc/sphinx-guides/source/admin/dataverses-datasets.rst index a961ac0b067..7f32e8c2514 100644 --- a/doc/sphinx-guides/source/admin/dataverses-datasets.rst +++ b/doc/sphinx-guides/source/admin/dataverses-datasets.rst @@ -15,7 +15,7 @@ Dataverse collections have to be empty to delete them. Navigate to the Dataverse Move a Dataverse Collection ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Moves a Dataverse collection whose id is passed to a new Dataverse collection whose id is passed. The Dataverse collection alias also may be used instead of the id. If the moved Dataverse collection has a guestbook, template, metadata block, link, or featured Dataverse collection that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move and remove the association. Only accessible to superusers. :: +Moves a Dataverse collection whose id is passed to an existing Dataverse collection whose id is passed. The Dataverse collection alias also may be used instead of the id. If the moved Dataverse collection has a guestbook, template, metadata block, link, or featured Dataverse collection that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move and remove the association. Only accessible to superusers. :: curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/dataverses/$id/move/$destination-id diff --git a/doc/sphinx-guides/source/admin/harvestserver.rst b/doc/sphinx-guides/source/admin/harvestserver.rst index 4e3f6ac0038..6f4f23fc587 100644 --- a/doc/sphinx-guides/source/admin/harvestserver.rst +++ b/doc/sphinx-guides/source/admin/harvestserver.rst @@ -26,7 +26,7 @@ The email portion of :ref:`systemEmail` will be visible via OAI-PMH (from the "I How does it work? ----------------- -Only the published, unrestricted datasets in your Dataverse installation can +Only the published datasets in your Dataverse installation can be made harvestable. Remote clients normally keep their records in sync through scheduled incremental updates, daily or weekly, thus minimizing the load on your server. Note that it is only the metadata @@ -115,10 +115,10 @@ Some useful examples of search queries to define OAI sets: ``keywordValue:censorship`` -Important: New SOLR schema required! +Important: New Solr schema required! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In order to be able to define OAI sets, your SOLR server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. +In order to be able to define OAI sets, your Solr server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. OAI Set updates --------------- diff --git a/doc/sphinx-guides/source/admin/integrations.rst b/doc/sphinx-guides/source/admin/integrations.rst index f6ca34bf3d4..b29e51b581d 100644 --- a/doc/sphinx-guides/source/admin/integrations.rst +++ b/doc/sphinx-guides/source/admin/integrations.rst @@ -57,7 +57,7 @@ their research results and retain links to imported and exported data. Users can organize their data in "Datasets", which can be exported to a Dataverse installation via the command-line interface (CLI). -Renku dataset documentation: https://renku-python.readthedocs.io/en/latest/reference/commands.html#module-renku.cli.dataset +Renku documentation: https://renku-python.readthedocs.io Flagship deployment of the Renku platform: https://renkulab.io diff --git a/doc/sphinx-guides/source/admin/metadatacustomization.rst b/doc/sphinx-guides/source/admin/metadatacustomization.rst index ff1b265cef7..5f7cf85f714 100644 --- a/doc/sphinx-guides/source/admin/metadatacustomization.rst +++ b/doc/sphinx-guides/source/admin/metadatacustomization.rst @@ -565,7 +565,7 @@ In general, the external vocabulary support mechanism may be a better choice for The specifics of the user interface for entering/selecting a vocabulary term and how that term is then displayed are managed by third-party Javascripts. The initial Javascripts that have been created provide auto-completion, displaying a list of choices that match what the user has typed so far, but other interfaces, such as displaying a tree of options for a hierarchical vocabulary, are possible. Similarly, existing scripts do relatively simple things for displaying a term - showing the term's name in the appropriate language and providing a link to an external URL with more information, but more sophisticated displays are possible. -Scripts supporting use of vocabularies from services supporting the SKOMOS protocol (see https://skosmos.org) and retrieving ORCIDs (from https:/orcid.org) are available https://github.com/gdcc/dataverse-external-vocab-support. (Custom scripts can also be used and community members are encouraged to share new scripts through the dataverse-external-vocab-support repository.) +Scripts supporting use of vocabularies from services supporting the SKOMOS protocol (see https://skosmos.org) and retrieving ORCIDs (from https://orcid.org) are available https://github.com/gdcc/dataverse-external-vocab-support. (Custom scripts can also be used and community members are encouraged to share new scripts through the dataverse-external-vocab-support repository.) Configuration involves specifying which fields are to be mapped, whether free-text entries are allowed, which vocabulary(ies) should be used, what languages those vocabulary(ies) are available in, and several service protocol and service instance specific parameters. These are all defined in the :ref:`:CVocConf <:CVocConf>` setting as a JSON array. Details about the required elements as well as example JSON arrays are available at https://github.com/gdcc/dataverse-external-vocab-support, along with an example metadata block that can be used for testing. diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 5685672eceb..e6f7b588ede 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -1,7 +1,7 @@ Solr Search Index ================= -A Dataverse installation requires Solr to be operational at all times. If you stop Solr, you should see a error about this on the root Dataverse installation page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse installation will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take. +A Dataverse installation requires Solr to be operational at all times. If you stop Solr, you should see an error about this on the root Dataverse installation page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse installation will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take. .. contents:: Contents: :local: @@ -9,7 +9,7 @@ A Dataverse installation requires Solr to be operational at all times. If you st Full Reindex ------------- -There are two ways to perform a full reindex of the Dataverse installation search index. Starting with a "clear" ensures a completely clean index but involves downtime. Reindexing in place doesn't involve downtime but does not ensure a completely clean index. +There are two ways to perform a full reindex of the Dataverse installation search index. Starting with a "clear" ensures a completely clean index but involves downtime. Reindexing in place doesn't involve downtime but does not ensure a completely clean index (e.g. stale entries from destroyed datasets can remain in the index). Clear and Reindex +++++++++++++++++ @@ -22,7 +22,7 @@ Get a list of all database objects that are missing in Solr, and Solr documents ``curl http://localhost:8080/api/admin/index/status`` -Remove all Solr documents that are orphaned (ie not associated with objects in the database): +Remove all Solr documents that are orphaned (i.e. not associated with objects in the database): ``curl http://localhost:8080/api/admin/index/clear-orphans`` @@ -36,7 +36,7 @@ Please note that the moment you issue this command, it will appear to end users Start Async Reindex ~~~~~~~~~~~~~~~~~~~ -Please note that this operation may take hours depending on the amount of data in your system. This known issue is being tracked at https://github.com/IQSS/dataverse/issues/50 +Please note that this operation may take hours depending on the amount of data in your system and whether or not you installation is using full-text indexing. More information on this, as well as some reference times, can be found at https://github.com/IQSS/dataverse/issues/50. ``curl http://localhost:8080/api/admin/index`` @@ -60,7 +60,7 @@ If indexing stops, this command should pick up where it left off based on which Manual Reindexing ----------------- -If you have made manual changes to a dataset in the database or wish to reindex a dataset that solr didn't want to index properly, it is possible to manually reindex Dataverse collections and datasets. +If you have made manual changes to a dataset in the database or wish to reindex a dataset that Solr didn't want to index properly, it is possible to manually reindex Dataverse collections and datasets. Reindexing Dataverse Collections ++++++++++++++++++++++++++++++++ @@ -69,7 +69,7 @@ Dataverse collections must be referenced by database object ID. If you have dire ``select id from dataverse where alias='dataversealias';`` -should work, or you may click the Dataverse Software's "Edit" menu and look for dataverseId= in the URLs produced by the drop-down. Then, to re-index: +should work, or you may click the Dataverse Software's "Edit" menu and look for *dataverseId=* in the URLs produced by the drop-down. Then, to re-index: ``curl http://localhost:8080/api/admin/index/dataverses/135`` @@ -89,7 +89,7 @@ To re-index a dataset by its database ID: Manually Querying Solr ---------------------- -If you suspect something isn't indexed properly in solr, you may bypass the Dataverse installation's web interface and query the command line directly to verify what solr returns: +If you suspect something isn't indexed properly in Solr, you may bypass the Dataverse installation's web interface and query the command line directly to verify what Solr returns: ``curl "http://localhost:8983/solr/collection1/select?q=dsPersistentId:doi:10.15139/S3/HFV0AO"`` diff --git a/doc/sphinx-guides/source/admin/troubleshooting.rst b/doc/sphinx-guides/source/admin/troubleshooting.rst index 79ce98322a8..9f085ba90cd 100644 --- a/doc/sphinx-guides/source/admin/troubleshooting.rst +++ b/doc/sphinx-guides/source/admin/troubleshooting.rst @@ -57,7 +57,7 @@ Ingest is both CPU- and memory-intensive, and depending on your system resources ``/usr/local/payara5/mq/bin/imqcmd -u admin purge dst -t q -n DataverseIngest`` will purge the DataverseIngest queue, and prompt for your confirmation. -Finally, list destinations to verify that the purge was successful:: +Finally, list destinations to verify that the purge was successful: ``/usr/local/payara5/mq/bin/imqcmd -u admin list dst`` diff --git a/doc/sphinx-guides/source/admin/user-administration.rst b/doc/sphinx-guides/source/admin/user-administration.rst index 608a8ab2b72..a21263f6f17 100644 --- a/doc/sphinx-guides/source/admin/user-administration.rst +++ b/doc/sphinx-guides/source/admin/user-administration.rst @@ -57,9 +57,9 @@ See :ref:`deactivate-a-user` Confirm Email ------------- -A Dataverse installation encourages builtin/local users to verify their email address upon signup or email change so that sysadmins can be assured that users can be contacted. +A Dataverse installation encourages builtin/local users to verify their email address upon sign up or email change so that sysadmins can be assured that users can be contacted. -The app will send a standard welcome email with a URL the user can click, which, when activated, will store a ``lastconfirmed`` timestamp in the ``authenticateduser`` table of the database. Any time this is "null" for a user (immediately after signup and/or changing of their Dataverse installation email address), their current email on file is considered to not be verified. The link that is sent expires after a time (the default is 24 hours), but this is configurable by a superuser via the ``:MinutesUntilConfirmEmailTokenExpires`` config option. +The app will send a standard welcome email with a URL the user can click, which, when activated, will store a ``lastconfirmed`` timestamp in the ``authenticateduser`` table of the database. Any time this is "null" for a user (immediately after sign up and/or changing of their Dataverse installation email address), their current email on file is considered to not be verified. The link that is sent expires after a time (the default is 24 hours), but this is configurable by a superuser via the ``:MinutesUntilConfirmEmailTokenExpires`` config option. Should users' URL token expire, they will see a "Verify Email" button on the account information page to send another URL. diff --git a/doc/sphinx-guides/source/api/client-libraries.rst b/doc/sphinx-guides/source/api/client-libraries.rst index 634f03a8125..bf9f658808b 100755 --- a/doc/sphinx-guides/source/api/client-libraries.rst +++ b/doc/sphinx-guides/source/api/client-libraries.rst @@ -1,54 +1,67 @@ Client Libraries ================ -Currently there are client libraries for Python, Javascript, R, Java, and Julia that can be used to develop against Dataverse Software APIs. We use the term "client library" on this page but "Dataverse Software SDK" (software development kit) is another way of describing these resources. They are designed to help developers express Dataverse Software concepts more easily in the languages listed below. For support on any of these client libraries, please consult each project's README. +Listed below are a variety of clienty libraries to help you interact with Dataverse APIs from Python, R, Javascript, etc. -Because a Dataverse installation is a SWORD server, additional client libraries exist for Java, Ruby, and PHP per the :doc:`/api/sword` page. +To get support for any of these client libraries, please consult each project's README. .. contents:: |toctitle| :local: -Python ------- +C/C++ +----- -There are two Python modules for interacting with Dataverse Software APIs. +https://github.com/aeonSolutions/OpenScience-Dataverse-API-C-library is the official C/C++ library for Dataverse APIs. -`pyDataverse `_ primarily allows developers to manage Dataverse collections, datasets and datafiles. Its intention is to help with data migrations and DevOps activities such as testing and configuration management. The module is developed by `Stefan Kasberger `_ from `AUSSDA - The Austrian Social Science Data Archive `_. +This C/C++ library was created and is currently maintained by `Miguel T. `_ To learn how to install and use it, see the project's `wiki page `_. -`dataverse-client-python `_ had its initial release in 2015. `Robert Liebowitz `_ created this library while at the `Center for Open Science (COS) `_ and the COS uses it to integrate the `Open Science Framework (OSF) `_ with a Dataverse installation via an add-on which itself is open source and listed on the :doc:`/api/apps` page. +Java +---- + +https://github.com/IQSS/dataverse-client-java is the official Java library for Dataverse APIs. + +`Richard Adams `_ from `ResearchSpace `_ created and maintains this library. Javascript ---------- -https://github.com/IQSS/dataverse-client-javascript is the official Javascript package for Dataverse Software APIs. It can be found on npm at https://www.npmjs.com/package/js-dataverse +https://github.com/IQSS/dataverse-client-javascript is the official Javascript package for Dataverse APIs. It can be found on npm at https://www.npmjs.com/package/js-dataverse It was created and is maintained by `The Agile Monkeys `_. +Julia +----- + +https://github.com/gaelforget/Dataverse.jl is the official Julia package for Dataverse APIs. It can be found on JuliaHub (https://juliahub.com/ui/Packages/Dataverse/xWAqY/) and leverages pyDataverse to provide an interface to Dataverse's data access API and native API. Dataverse.jl provides a few additional functionalities with documentation (https://gaelforget.github.io/Dataverse.jl/dev/) and a demo notebook (https://gaelforget.github.io/Dataverse.jl/dev/notebook.html). + +It was created and is maintained by `Gael Forget `_. + +PHP +--- + +There is no official PHP library for Dataverse APIs (please :ref:`get in touch ` if you'd like to create one!) but there is a SWORD library written in PHP listed under :ref:`client-libraries` in the :doc:`/api/sword` documentation. + +Python +------ + +There are two Python modules for interacting with Dataverse APIs. + +`pyDataverse `_ primarily allows developers to manage Dataverse collections, datasets and datafiles. Its intention is to help with data migrations and DevOps activities such as testing and configuration management. The module is developed by `Stefan Kasberger `_ from `AUSSDA - The Austrian Social Science Data Archive `_. + +`dataverse-client-python `_ had its initial release in 2015. `Robert Liebowitz `_ created this library while at the `Center for Open Science (COS) `_ and the COS uses it to integrate the `Open Science Framework (OSF) `_ with Dataverse installations via an add-on which itself is open source and listed on the :doc:`/api/apps` page. + R - -https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse Software APIs. The latest release can be installed from `CRAN `_. +https://github.com/IQSS/dataverse-client-r is the official R package for Dataverse APIs. The latest release can be installed from `CRAN `_. The R client can search and download datasets. It is useful when automatically (instead of manually) downloading data files as part of a script. For bulk edit and upload operations, we currently recommend pyDataverse. The package is currently maintained by `Shiro Kuriwaki `_. It was originally created by `Thomas Leeper `_ and then formerly maintained by `Will Beasley `_. -Java ----- - -https://github.com/IQSS/dataverse-client-java is the official Java library for Dataverse Software APIs. - -`Richard Adams `_ from `ResearchSpace `_ created and maintains this library. Ruby ---- -https://github.com/libis/dataverse_api is a Ruby gem for Dataverse Software APIs. It is registered as a library on Rubygems (https://rubygems.org/search?query=dataverse). +https://github.com/libis/dataverse_api is a Ruby gem for Dataverse APIs. It is registered as a library on Rubygems (https://rubygems.org/search?query=dataverse). The gem is created and maintained by the LIBIS team (https://www.libis.be) at the University of Leuven (https://www.kuleuven.be). - -Julia ------ - -https://github.com/gaelforget/Dataverse.jl is the official Julia package for Dataverse Software APIs. It can be found on JuliaHub (https://juliahub.com/ui/Packages/Dataverse/xWAqY/) and leverages pyDataverse to provide an interface to Dataverse's data access API and native API. Dataverse.jl provides a few additional functionalities with documentation (https://gaelforget.github.io/Dataverse.jl/dev/) and a demo notebook (https://gaelforget.github.io/Dataverse.jl/dev/notebook.html). - -It was created and is maintained by `Gael Forget `_. diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index 93e1c36f179..6d68d648cb3 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -2614,7 +2614,7 @@ The fully expanded example above (without environment variables) looks like this curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/files/:persistentId/prov-freeform?persistentId=doi:10.5072/FK2/AAA000" -H "Content-type:application/json" --upload-file provenance.json -See a sample JSON file :download:`file-provenance.json <../_static/api/file-provenance.json>` from http://openprovenance.org (c.f. Huynh, Trung Dong and Moreau, Luc (2014) ProvStore: a public provenance repository. At 5th International Provenance and Annotation Workshop (IPAW'14), Cologne, Germany, 09-13 Jun 2014. pp. 275-277). +See a sample JSON file :download:`file-provenance.json <../_static/api/file-provenance.json>` from https://openprovenance.org (c.f. Huynh, Trung Dong and Moreau, Luc (2014) ProvStore: a public provenance repository. At 5th International Provenance and Annotation Workshop (IPAW'14), Cologne, Germany, 09-13 Jun 2014. pp. 275-277). Delete Provenance JSON for an uploaded file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/sphinx-guides/source/conf.py b/doc/sphinx-guides/source/conf.py index 880ed561720..590eee4bd9d 100755 --- a/doc/sphinx-guides/source/conf.py +++ b/doc/sphinx-guides/source/conf.py @@ -66,9 +66,9 @@ # built documents. # # The short X.Y version. -version = '5.12' +version = '5.12.1' # The full version, including alpha/beta/rc tags. -release = '5.12' +release = '5.12.1' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst b/doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst index 7abae535276..52a6a283e9c 100644 --- a/doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst +++ b/doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst @@ -1,5 +1,8 @@ Dataset Semantic Metadata API ============================= +.. contents:: |toctitle| + :local: + The OAI_ORE metadata export format represents Dataset metadata using json-ld (see the :doc:`/admin/metadataexport` section). As part of an RDA-supported effort to allow import of Datasets exported as Bags with an included OAI_ORE metadata file, an experimental API has been created that provides a json-ld alternative to the v1.0 API calls to get/set/delete Dataset metadata in the :doc:`/api/native-api`. @@ -74,7 +77,7 @@ To delete metadata for a Dataset, send a json-ld representation of the fields to curl -X PUT -H X-Dataverse-key:$API_TOKEN -H 'Content-Type: application/ld+json' -d '{"https://dataverse.org/schema/core#restrictions":"No restrictions"}' "$SERVER_URL/api/datasets/:persistentId/metadata/delete?persistentId=$DATASET_PID" -Note, this example uses the term URI directly rather than adding an '@context' element. You can use either form in any of these API calls. +Note, this example uses the term URI directly rather than adding an ``@context`` element. You can use either form in any of these API calls. You should expect a 200 ("OK") response indicating whether a draft Dataset version was created or an existing draft was updated. diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index 53fc11a5915..55f5f550dd9 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -10,7 +10,7 @@ Introduction See :doc:`version-control` for background on our branching strategy. -The steps below describe making both normal releases and hotfix releases. +The steps below describe making both regular releases and hotfix releases. Write Release Notes ------------------- @@ -43,49 +43,110 @@ Increment the version number to the milestone (e.g. 5.10.1) in the following two - modules/dataverse-parent/pom.xml -> ```` -> ```` (e.g. `pom.xml commit `_) - doc/sphinx-guides/source/conf.py (two places, e.g. `conf.py commit `_) -Add the version being released to the lists in the following two files: +Add the version being released to the lists in the following file: - doc/sphinx-guides/source/versions.rst (e.g. `versions.rst commit `_) Check in the Changes Above into a Release Branch and Merge It ------------------------------------------------------------- -For any ordinary release, make the changes above in the release branch you created, make a pull request, and merge it into the "develop" branch. Like usual, you can safely delete the branch after the merge is complete. +For a regular release, make the changes above in the release branch you created, make a pull request, and merge it into the "develop" branch. Like usual, you can safely delete the branch after the merge is complete. If you are making a hotfix release, make the pull request against the "master" branch. Do not delete the branch after merging because we will later merge it into the "develop" branch to pick up the hotfix. More on this later. -Either way, as usual, you should ensure that all tests are passing. Please note that you might need to bump the version in `jenkins.yml `_ in dataverse-ansible to get the tests to run. +Either way, as usual, you should ensure that all tests are passing. Please note that you will need to bump the version in `jenkins.yml `_ in dataverse-ansible to get the tests to pass. Consider doing this before making the pull request. Alternatively, you can bump jenkins.yml after making the pull request and re-run the Jenkins job to make sure tests pass. Merge "develop" into "master" ----------------------------- -Note: If you are making a hotfix release, the "develop" branch is not involved so you can skip this step. +If this is a regular (non-hotfix) release, create a pull request to merge the "develop" branch into the "master" branch using this "compare" link: https://github.com/IQSS/dataverse/compare/master...develop -The "develop" branch should be merged into "master" before tagging. +Once important tests have passed (compile, unit tests, etc.), merge the pull request. Don't worry about style tests failing such as for shell scripts. + +If this is a hotfix release, skip this whole "merge develop to master" step (the "develop" branch is not involved until later). + +Build the Guides for the Release +-------------------------------- + +Go to https://jenkins.dataverse.org/job/guides.dataverse.org/ and make the following adjustments to the config: + +- Repository URL: ``https://github.com/IQSS/dataverse.git`` +- Branch Specifier (blank for 'any'): ``*/master`` +- ``VERSION`` (under "Build Steps"): ``5.10.1`` (for example) + +Click "Save" then "Build Now". + +Make sure the guides directory appears in the expected location such as https://guides.dataverse.org/en/5.10.1/ + +As described below, we'll soon point the "latest" symlink to that new directory. Create a Draft Release on GitHub -------------------------------- -Create a draft release at https://github.com/IQSS/dataverse/releases/new +Go to https://github.com/IQSS/dataverse/releases/new to start creating a draft release. + +- Under "Choose a tag" you will be creating a new tag. Have it start with a "v" such as ``v5.10.1``. Click "Create new tag on publish". +- Under "Target" go to "Recent Commits" and select the merge commit from when you merged ``develop`` into ``master`` above. This commit will appear in ``/api/info/version`` from a running installation. +- Under "Release title" use the same name as the tag such as ``v5.10.1``. +- In the description, copy and paste the content from the release notes .md file created in the "Write Release Notes" steps above. +- Click "Save draft" because we do not want to publish the release yet. + +At this point you can send around the draft release for any final feedback. Links to the guides for this release should be working now, since you build them above. + +Make corrections to the draft, if necessary. It will be out of sync with the .md file, but that's ok (`#7988 `_ is tracking this). + +Run a Build to Create the War File +---------------------------------- -The "tag version" and "title" should be the number of the milestone with a "v" in front (i.e. v5.10.1). +ssh into the dataverse-internal server and undeploy the current war file. -Copy in the content from the .md file created in the "Write Release Notes" steps above. +Go to https://jenkins.dataverse.org/job/IQSS_Dataverse_Internal/ and make the following adjustments to the config: + +- Repository URL: ``https://github.com/IQSS/dataverse.git`` +- Branch Specifier (blank for 'any'): ``*/master`` +- Execute shell: Update version in filenames to ``dataverse-5.10.1.war`` (for example) + +Click "Save" then "Build Now". + +The build number will appear in ``/api/info/version`` (along with the commit mentioned above) from a running installation (e.g. ``{"version":"5.10.1","build":"907-b844672``). + +Build Installer (dvinstall.zip) +------------------------------- + +ssh into the dataverse-internal server and do the following: + +- In a git checkout of the dataverse source switch to the master branch and pull the latest. +- Copy the war file from the previous step to the ``target`` directory in the root of the repo (create it, if necessary). +- ``cd scripts/installer`` +- ``make`` + +A zip file called ``dvinstall.zip`` should be produced. Make Artifacts Available for Download ------------------------------------- Upload the following artifacts to the draft release you created: -- war file (``mvn package`` from Jenkins) -- installer (``cd scripts/installer && make``) -- other files as needed, such as updated Solr schema and config files +- the war file (e.g. ``dataverse-5.10.1.war``, from above) +- the installer (``dvinstall.zip``, from above) +- other files as needed: + + - updated Solr schema + - metadata block tsv files + - config files Publish the Release ------------------- Click the "Publish release" button. +Update Guides Link +------------------ + +"latest" at https://guides.dataverse.org/en/latest/ is a symlink to the directory with the latest release. That directory (e.g. ``5.10.1``) was put into place by the Jenkins "guides" job described above. + +ssh into the guides server and update the symlink to point to the latest release. + Close Milestone on GitHub and Create a New One ---------------------------------------------- @@ -115,7 +176,7 @@ For Hotfixes, Merge Hotfix Branch into "develop" and Rename SQL Scripts Note: this only applies to hotfixes! -We've merged the hotfix into the "master" branch but now we need the fixes (and version bump) in the "develop" branch. Make a new branch off the hotfix branch and create a pull request against develop. Merge conflicts are possible and this pull request should go through review and QA like normal. Afterwards it's fine to delete this branch and the hotfix brach that was merged into master. +We've merged the hotfix into the "master" branch but now we need the fixes (and version bump) in the "develop" branch. Make a new branch off the hotfix branch and create a pull request against develop. Merge conflicts are possible and this pull request should go through review and QA like normal. Afterwards it's fine to delete this branch and the hotfix branch that was merged into master. Because of the hotfix version, any SQL scripts in "develop" should be renamed (from "5.11.0" to "5.11.1" for example). To read more about our naming conventions for SQL scripts, see :doc:`sql-upgrade-scripts`. diff --git a/doc/sphinx-guides/source/developers/troubleshooting.rst b/doc/sphinx-guides/source/developers/troubleshooting.rst index 0463a68d8c8..832785f9860 100755 --- a/doc/sphinx-guides/source/developers/troubleshooting.rst +++ b/doc/sphinx-guides/source/developers/troubleshooting.rst @@ -41,7 +41,7 @@ This command helps verify what host your domain is using to send mail. Even if i 2. From the left-side panel, select **JavaMail Sessions** 3. You should see one session named **mail/notifyMailSession** -- click on that. -From this window you can modify certain fields of your Dataverse installation's notifyMailSession, which is the JavaMail session for outgoing system email (such as on user signup or data publication). Two of the most important fields we need are: +From this window you can modify certain fields of your Dataverse installation's notifyMailSession, which is the JavaMail session for outgoing system email (such as on user sign up or data publication). Two of the most important fields we need are: - **Mail Host:** The DNS name of the default mail server (e.g. smtp.gmail.com) - **Default User:** The username provided to your Mail Host when you connect to it (e.g. johndoe@gmail.com) diff --git a/doc/sphinx-guides/source/index.rst b/doc/sphinx-guides/source/index.rst index f7e81756e5b..148518d2ce5 100755 --- a/doc/sphinx-guides/source/index.rst +++ b/doc/sphinx-guides/source/index.rst @@ -73,6 +73,4 @@ If you have a **security issue** to report, please email `security@dataverse.org Indices and Tables ------------------ -* :ref:`genindex` -* :ref:`modindex` * :ref:`search` diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index f2de9d5702f..c61bf451eb7 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -478,7 +478,7 @@ named config in the same folder") Console Commands to Set Up Access Configuration ############################################### -Begin by installing the CLI tool `pip `_ to install the +Begin by installing the CLI tool `pip (package installer for Python) `_ to install the `AWS command line interface `_ if you don't have it. First, we'll get our access keys set up. If you already have your access keys configured, skip this step. @@ -800,7 +800,7 @@ Refer to :ref:`:NavbarSupportUrl` for setting to a fully-qualified URL which wil Sign Up ####### -Refer to :ref:`:SignUpUrl` and :ref:`conf-allow-signup` for setting a relative path URL to which users will be sent for signup and for controlling the ability for creating local user accounts. +Refer to :ref:`:SignUpUrl` and :ref:`conf-allow-signup` for setting a relative path URL to which users will be sent for sign up and for controlling the ability for creating local user accounts. Custom Header ^^^^^^^^^^^^^ @@ -1235,8 +1235,8 @@ The :S3ArchiverConfig setting is a JSON object that must include an "s3_bucket_n .. _Archiving API Call: -API Calls -+++++++++ +BagIt Export API Calls +++++++++++++++++++++++ Once this configuration is complete, you, as a user with the *PublishDataset* permission, should be able to use the admin API call to manually submit a DatasetVersion for processing: @@ -1244,25 +1244,29 @@ Once this configuration is complete, you, as a user with the *PublishDataset* pe where: -``{id}`` is the DatasetId (or ``:persistentId`` with the ``?persistentId=""`` parameter), and +``{id}`` is the DatasetId (the database id of the dataset) and ``{version}`` is the friendly version number, e.g. "1.2". -The submitDatasetVersionToArchive API (and the workflow discussed below) attempt to archive the dataset version via an archive specific method. For Chronopolis, a DuraCloud space named for the dataset (it's DOI with ':' and '.' replaced with '-') is created and two files are uploaded to it: a version-specific datacite.xml metadata file and a BagIt bag containing the data and an OAI-ORE map file. (The datacite.xml file, stored outside the Bag as well as inside is intended to aid in discovery while the ORE map file is 'complete', containing all user-entered metadata and is intended as an archival record.) +or in place of the DatasetID, you can use the string ``:persistentId`` as the ``{id}`` and add the DOI/PID as a query parameter like this: ``?persistentId=""``. Here is how the full command would look: + +``curl -X POST -H "X-Dataverse-key: " http://localhost:8080/api/admin/submitDatasetVersionToArchive/:persistentId/{version}?persistentId=""`` + +The submitDatasetVersionToArchive API (and the workflow discussed below) attempt to archive the dataset version via an archive specific method. For Chronopolis, a DuraCloud space named for the dataset (its DOI with ":" and "." replaced with "-", e.g. ``doi-10-5072-fk2-tgbhlb``) is created and two files are uploaded to it: a version-specific datacite.xml metadata file and a BagIt bag containing the data and an OAI-ORE map file. (The datacite.xml file, stored outside the Bag as well as inside, is intended to aid in discovery while the ORE map file is "complete", containing all user-entered metadata and is intended as an archival record.) -In the Chronopolis case, since the transfer from the DuraCloud front-end to archival storage in Chronopolis can take significant time, it is currently up to the admin/curator to submit a 'snap-shot' of the space within DuraCloud and to monitor its successful transfer. Once transfer is complete the space should be deleted, at which point the Dataverse Software API call can be used to submit a Bag for other versions of the same Dataset. (The space is reused, so that archival copies of different Dataset versions correspond to different snapshots of the same DuraCloud space.). +In the Chronopolis case, since the transfer from the DuraCloud front-end to archival storage in Chronopolis can take significant time, it is currently up to the admin/curator to submit a 'snap-shot' of the space within DuraCloud and to monitor its successful transfer. Once transfer is complete the space should be deleted, at which point the Dataverse Software API call can be used to submit a Bag for other versions of the same dataset. (The space is reused, so that archival copies of different dataset versions correspond to different snapshots of the same DuraCloud space.). -A batch version of this admin api call is also available: +A batch version of this admin API call is also available: ``curl -X POST -H "X-Dataverse-key: " 'http://localhost:8080/api/admin/archiveAllUnarchivedDatasetVersions?listonly=true&limit=10&latestonly=true'`` -The archiveAllUnarchivedDatasetVersions call takes 3 optional configuration parameters. +The archiveAllUnarchivedDatasetVersions call takes 3 optional configuration parameters. + * listonly=true will cause the API to list dataset versions that would be archived but will not take any action. -* limit= will limit the number of dataset versions archived in one api call to <= . +* limit= will limit the number of dataset versions archived in one API call to ``<=`` . * latestonly=true will limit archiving to only the latest published versions of datasets instead of archiving all unarchived versions. -Note that because archiving is done asynchronously, the calls above will return OK even if the user does not have the *PublishDataset* permission on the dataset(s) involved. Failures are indocated in the log and the archivalStatus calls in the native api can be used to check the status as well. - +Note that because archiving is done asynchronously, the calls above will return OK even if the user does not have the *PublishDataset* permission on the dataset(s) involved. Failures are indicated in the log and the archivalStatus calls in the native API can be used to check the status as well. PostPublication Workflow ++++++++++++++++++++++++ @@ -1376,30 +1380,37 @@ When changing values these values with ``asadmin``, you'll need to delete the ol It's also possible to change these values by stopping Payara, editing ``payara5/glassfish/domains/domain1/config/domain.xml``, and restarting Payara. +.. _dataverse.fqdn: + dataverse.fqdn ++++++++++++++ -If the Dataverse installation has multiple DNS names, this option specifies the one to be used as the "official" host name. For example, you may want to have dataverse.example.edu, and not the less appealing server-123.socsci.example.edu to appear exclusively in all the registered global identifiers, Data Deposit API records, etc. +If the Dataverse installation has multiple DNS names, this option specifies the one to be used as the "official" hostname. For example, you may want to have ``dataverse.example.edu``, and not the less appealing ``server-123.example.edu`` to appear exclusively in all the registered global identifiers, etc. The password reset feature requires ``dataverse.fqdn`` to be configured. -.. note:: - - Do note that whenever the system needs to form a service URL, by default, it will be formed with ``https://`` and port 443. I.e., - ``https://{dataverse.fqdn}/`` - If that does not suit your setup, you can define an additional option, ``dataverse.siteUrl``, explained below. +Configuring ``dataverse.fqdn`` is not enough. Read on for the importance of also setting ``dataverse.siteUrl``. .. _dataverse.siteUrl: dataverse.siteUrl +++++++++++++++++ -.. note:: +``dataverse.siteUrl`` is used to configure the URL for your Dataverse installation that you plan to advertise to your users. As explained in the :ref:`installation ` docs, this setting is critical for the correct operation of your installation. + +For example, your site URL could be https://dataverse.example.edu + +That is, even though the server might also be available at uglier URLs such as https://server-123.example.edu the site URL is the "official" URL. + +The ``dataverse.siteUrl`` JVM option can be configured by following the procedure under :ref:`jvm-options` or by editing ``domain.xml`` directly. You can specify the protocol, host, and port number. Your ``domain.xml`` file could look like this, for example: + +``-Ddataverse.siteUrl=https://dataverse.example.edu`` + +Note that it's also possible to use the ``dataverse.fqdn`` as a variable, if you wish. Here's an example of this as well as a custom port (which is usually not necessary): + +``-Ddataverse.siteUrl=https://${dataverse.fqdn}:444`` - and specify the protocol and port number you would prefer to be used to advertise the URL for your Dataverse installation. - For example, configured in domain.xml: - ``-Ddataverse.fqdn=dataverse.example.edu`` - ``-Ddataverse.siteUrl=http://${dataverse.fqdn}:8080`` +We are absolutely aware that it's confusing to have both ``dataverse.fqdn`` and ``dataverse.siteUrl``. https://github.com/IQSS/dataverse/issues/6636 is about resolving this confusion. dataverse.files.directory +++++++++++++++++++++++++ @@ -2186,7 +2197,7 @@ If ``:SolrFullTextIndexing`` is set to true, the content of files of any size wi :SignUpUrl ++++++++++ -The relative path URL to which users will be sent for signup. The default setting is below. +The relative path URL to which users will be sent for sign up. The default setting is below. ``curl -X PUT -d '/dataverseuser.xhtml?editMode=CREATE' http://localhost:8080/api/admin/settings/:SignUpUrl`` @@ -2262,6 +2273,31 @@ Set to false to disallow local accounts from being created. See also the section ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:AllowSignUp`` +.. _:AllowRemoteAuthSignUp: + +:AllowRemoteAuthSignUp +++++++++++++++++++++++ + +This is a **compound** setting that enables or disables sign up for new accounts for individual OAuth2 authentication methods (such as Orcid, Google and GitHub). This way it is possible to continue allowing logins via an OAuth2 provider for already existing accounts, without letting new users create accounts with this method. + +By default, if the setting is not present, all configured OAuth sign ups are allowed. If the setting is present, but the value for this specific method is not specified, it is assumed that the sign ups are allowed for it. + +Examples: + +This curl command... + +``curl -X PUT -d '{"default":"false"}' http://localhost:8080/api/admin/settings/:AllowRemoteAuthSignUp`` + +...disables all OAuth sign ups. + +This curl command... + +``curl -X PUT -d '{"default":"true","google":"false"}' http://localhost:8080/api/admin/settings/:AllowRemoteAuthSignUp`` + +...keeps sign ups open for all the OAuth login providers except google. (That said, note that the ``"default":"true"`` part in this example is redundant, since it would default to true anyway for all the methods other than google.) + +See also :doc:`oauth2`. + :FileFixityChecksumAlgorithm ++++++++++++++++++++++++++++ @@ -2714,9 +2750,9 @@ Part of the database settings to configure the BagIt file handler. This is the p ++++++++++++++++++ Your Dataverse installation can export archival "Bag" files to an extensible set of storage systems (see :ref:`BagIt Export` above for details about this and for further explanation of the other archiving related settings below). -This setting specifies which storage system to use by identifying the particular Java class that should be run. Current options include DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, and GoogleCloudSubmitToArchiveCommand. +This setting specifies which storage system to use by identifying the particular Java class that should be run. Current configuration options include DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchiveCommand, and S3SubmitToArchiveCommand. -``curl -X PUT -d 'LocalSubmitToArchiveCommand' http://localhost:8080/api/admin/settings/:ArchiverClassName`` +For examples, see the specific configuration above in :ref:`BagIt Export`. :ArchiverSettings +++++++++++++++++ diff --git a/doc/sphinx-guides/source/installation/installation-main.rst b/doc/sphinx-guides/source/installation/installation-main.rst index 4b000f1ef9e..8559d6ce194 100755 --- a/doc/sphinx-guides/source/installation/installation-main.rst +++ b/doc/sphinx-guides/source/installation/installation-main.rst @@ -82,6 +82,8 @@ While Postgres can accomodate usernames and database names containing hyphens, i For more information, please see https://docs.payara.fish/documentation/payara-server/password-aliases/password-alias-asadmin-commands.html +.. _importance-of-siteUrl: + **IMPORTANT:** The installer will also ask for an external site URL for the Dataverse installation. It is *imperative* that this value be supplied accurately, or a long list of functions will be inoperable, including: - email confirmation links @@ -134,6 +136,11 @@ Dataset Cannot Be Published Check to make sure you used a fully qualified domain name when installing the Dataverse Software. You can change the ``dataverse.fqdn`` JVM option after the fact per the :doc:`config` section. +Got ERR_ADDRESS_UNREACHABLE While Navigating on Interface or API Calls +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you are receiving an ``ERR_ADDRESS_UNREACHABLE`` while navigating the GUI or making an API call, make sure the ``siteUrl`` JVM option is defined. For details on how to set ``siteUrl``, please refer to :ref:`dataverse.siteUrl` from the :doc:`config` section. For context on why setting this option is necessary, refer to :ref:`dataverse.fqdn` from the :doc:`config` section. + Problems Sending Email ^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/sphinx-guides/source/installation/oauth2.rst b/doc/sphinx-guides/source/installation/oauth2.rst index 0dfdb0393e0..8dffde87cc2 100644 --- a/doc/sphinx-guides/source/installation/oauth2.rst +++ b/doc/sphinx-guides/source/installation/oauth2.rst @@ -78,6 +78,11 @@ This template can be used for configuring this setting (**this is not something - :download:`orcid-sandbox.json <../_static/installation/files/root/auth-providers/orcid-sandbox.json>` +Disabling Sign Up +~~~~~~~~~~~~~~~~~ + +See :ref:`:AllowRemoteAuthSignUp`. + Converting Local Users to OAuth ------------------------------- diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 3cf876a2251..59de507a264 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -291,8 +291,8 @@ If the installed location of the convert executable is different from ``/usr/bin R - -The Dataverse Software uses `R `_ to handle -tabular data files. The instructions below describe a **minimal** R +The Dataverse Software uses `R `_ to handle +tabular data files. The instructions below describe a **minimal** R Project installation. It will allow you to ingest R (.RData) files as tabular data and to export tabular data as .RData files. R can be considered an optional component, meaning that if you don't have R installed, you will still be able to run and diff --git a/doc/sphinx-guides/source/user/dataset-management.rst b/doc/sphinx-guides/source/user/dataset-management.rst index 77a760ef838..ec3bb392ce5 100755 --- a/doc/sphinx-guides/source/user/dataset-management.rst +++ b/doc/sphinx-guides/source/user/dataset-management.rst @@ -193,8 +193,8 @@ Additional download options available for tabular data (found in the same drop-d - The original file uploaded by the user; - Saved as R data (if the original file was not in R format); - Variable Metadata (as a `DDI Codebook `_ XML file); -- Data File Citation (currently in either RIS, EndNote XML, or BibTeX format); -- All of the above, as a zipped bundle. +- Data File Citation (currently in either RIS, EndNote XML, or BibTeX format). + Differentially Private (DP) Metadata can also be accessed for restricted tabular files if the data depositor has created a DP Metadata Release. See :ref:`dp-release-create` for more information. diff --git a/doc/sphinx-guides/source/versions.rst b/doc/sphinx-guides/source/versions.rst index 1cbd785b5dd..e0a344de9a1 100755 --- a/doc/sphinx-guides/source/versions.rst +++ b/doc/sphinx-guides/source/versions.rst @@ -6,7 +6,8 @@ Dataverse Software Documentation Versions This list provides a way to refer to the documentation for previous versions of the Dataverse Software. In order to learn more about the updates delivered from one version to another, visit the `Releases `__ page in our GitHub repo. -- 5.12 +- 5.12.1 +- `5.12 `__ - `5.11.1 `__ - `5.11 `__ - `5.10.1 `__ diff --git a/doc/sphinx_bootstrap_theme/bootstrap/layout.html b/doc/sphinx_bootstrap_theme/bootstrap/layout.html index 9d17996292b..d3ccd463814 100755 --- a/doc/sphinx_bootstrap_theme/bootstrap/layout.html +++ b/doc/sphinx_bootstrap_theme/bootstrap/layout.html @@ -106,7 +106,7 @@ {%- if hasdoc('copyright') %} {% trans path=pathto('copyright'), copyright=copyright|e %}

Copyright © {{ copyright }}.

{% endtrans %} {%- else %} - {% trans copyright=copyright|e %}

Developed at the Institute for Quantitative Social Science  |  Code available at  |  Created using Sphinx {{ sphinx_version }}
Last updated on {{ last_updated }}  |  Dataverse v. {{ version }}  |  View the latest version of Dataverse Guides

+ {% trans copyright=copyright|e %}

Developed at the Institute for Quantitative Social Science  |  Code available at  |  Created using Sphinx {{ sphinx_version }}
Last updated on {{ last_updated }}  |  Dataverse v. {{ version }}  |  View the latest version of Dataverse Guides

Copyright © {{ copyright }}

{% endtrans %} {%- endif %} {%- endif %} diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar deleted file mode 100644 index a23530b895c..00000000000 Binary files a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.md5 b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.md5 deleted file mode 100644 index d8b1ce2fa75..00000000000 --- a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -f578d8ec91811d5d72981355cb7a1f0f diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.sha1 b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.sha1 deleted file mode 100644 index 4c7d114634b..00000000000 --- a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -523abaf48b4423eb874dbc086b876aa917930a04 diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom deleted file mode 100644 index 2915745c27d..00000000000 --- a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom +++ /dev/null @@ -1,77 +0,0 @@ - - - - xoai - com.lyncode - 4.1.0 - - 4.0.0 - - XOAI Commons - xoai-common - 4.1.0-header-patch - - - - com.lyncode - xml-io - - - com.lyncode - test-support - - - - commons-codec - commons-codec - - - - commons-io - commons-io - - - - com.google.guava - guava - - - - xml-apis - xml-apis - - - - org.hamcrest - hamcrest-all - - - - org.codehaus.woodstox - stax2-api - - - - javax.xml.stream - stax-api - - - - org.apache.commons - commons-lang3 - - - - stax - stax-api - - - - junit - junit - test - - - - - diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.md5 b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.md5 deleted file mode 100644 index 15f47f4140a..00000000000 --- a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.md5 +++ /dev/null @@ -1 +0,0 @@ -346e9f235523e52256006bbe8eba60bb diff --git a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.sha1 b/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.sha1 deleted file mode 100644 index 88668a4d49c..00000000000 --- a/local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.sha1 +++ /dev/null @@ -1 +0,0 @@ -cd83d08c097d6aa1b27b20ef4742c7e4fa47e6b5 diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar deleted file mode 100644 index 28e5da7b0d6..00000000000 Binary files a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.md5 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.md5 deleted file mode 100644 index 67dda34a6c9..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -546f1ab3f3f654280f88e429ba3471ae diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.sha1 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.sha1 deleted file mode 100644 index 50e8f2f42cd..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -f4da7ebc3fda69e1e7db12bda6d7b5fb4aecc7a4 diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar deleted file mode 100644 index bdec990e2c6..00000000000 Binary files a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.md5 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.md5 deleted file mode 100644 index 77416d80f87..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -ec87ba7cb8e7396fc903acdbacd31ff6 diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.sha1 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.sha1 deleted file mode 100644 index a6157b4dc0b..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -370c2955550a42b11fe7b9007771c506f5769639 diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar deleted file mode 100644 index 331c9a80cd1..00000000000 Binary files a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.md5 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.md5 deleted file mode 100644 index 27381662d09..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -544e9b97062d054370695b9b09d4bb1c diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.sha1 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.sha1 deleted file mode 100644 index 37dd4e47c2c..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -67c505461f3c190894bb036cc866eb640c2f6a48 diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom deleted file mode 100644 index 87d67b8c4a7..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom +++ /dev/null @@ -1,54 +0,0 @@ - - - - xoai - com.lyncode - 4.1.0-header-patch - - - 4.0.0 - - XOAI Data Provider - xoai-data-provider - 4.1.0-header-patch - - - - com.lyncode - xoai-common - ${project.version} - - - - log4j - log4j - - - - com.google.guava - guava - - - - com.lyncode - builder-commons - - - - org.apache.commons - commons-lang3 - - - - org.mockito - mockito-all - test - - - - junit - junit - test - - - diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.md5 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.md5 deleted file mode 100644 index 5959ea476c7..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.md5 +++ /dev/null @@ -1 +0,0 @@ -a1b49c13fcf448de9628798f8682fcaa diff --git a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.sha1 b/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.sha1 deleted file mode 100644 index 87fd86c23e0..00000000000 --- a/local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.sha1 +++ /dev/null @@ -1 +0,0 @@ -41be98af31f8d17d83ab6c38bd7939ba212eab8d diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar deleted file mode 100644 index 4382b3ded5d..00000000000 Binary files a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.md5 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.md5 deleted file mode 100644 index c9e720e6039..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -21bc45a29b715720f4b77f51bf9f1754 diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.sha1 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.sha1 deleted file mode 100644 index 756955d2840..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -b544162e82d322116b87d99f2fbb6ddd4c4745e1 diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar deleted file mode 100644 index 314dad81872..00000000000 Binary files a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.md5 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.md5 deleted file mode 100644 index 27c55f9af58..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -88dc05805672ebe01ded1197a582cd60 diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.sha1 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.sha1 deleted file mode 100644 index 1098e506b93..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -fe41289cb74c56e9282dd09c22df2eda47c68a0d diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar deleted file mode 100644 index 781fc1ce1e2..00000000000 Binary files a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar and /dev/null differ diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.md5 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.md5 deleted file mode 100644 index 84891040047..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.md5 +++ /dev/null @@ -1 +0,0 @@ -52f8b446f78009757d593312778f428c diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.sha1 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.sha1 deleted file mode 100644 index dbded3dd83f..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -feb6903ad32d4b42461b7ca1b3fae6146740bb31 diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom deleted file mode 100644 index c45e15a91f9..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom +++ /dev/null @@ -1,67 +0,0 @@ - - - - xoai - com.lyncode - 4.1.0-header-patch - - 4.0.0 - - XOAI Service Provider - xoai-service-provider - 4.1.0-header-patch - - - - com.lyncode - xoai-common - ${project.version} - - - - com.lyncode - xml-io - - - - log4j - log4j - - - - org.apache.commons - commons-lang3 - - - - org.apache.httpcomponents - httpclient - - - - org.codehaus.woodstox - wstx-asl - - - - - com.lyncode - xoai-data-provider - ${project.version} - test - - - - org.mockito - mockito-all - test - - - - junit - junit - test - - - - diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.md5 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.md5 deleted file mode 100644 index 5e51f198572..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.md5 +++ /dev/null @@ -1 +0,0 @@ -b97b8ee92daa5fc4fd87004465f9ad2b diff --git a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.sha1 b/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.sha1 deleted file mode 100644 index 2c6dc74f02b..00000000000 --- a/local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.sha1 +++ /dev/null @@ -1 +0,0 @@ -f772583549263bd72ea4d5268d9db0a84c27cb9f diff --git a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom b/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom deleted file mode 100644 index 89a14d88c51..00000000000 --- a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom +++ /dev/null @@ -1,273 +0,0 @@ - - 4.0.0 - pom - - - xoai-common - xoai-data-provider - xoai-service-provider - - - - org.sonatype.oss - oss-parent - 7 - - - com.lyncode - xoai - 4.1.0-header-patch - - XOAI : OAI-PMH Java Toolkit - http://www.lyncode.com - - - 1.9.5 - 15.0 - 3.1 - 1.2.14 - 4.2.1 - 4.0.0 - - 1.0.2 - 1.0.3 - 1.0.4 - - - - - The Apache Software License, Version 2.0 - http://www.apache.org/licenses/LICENSE-2.0.txt - repo - - - - - scm:git:git@github.com:lyncode/xoai.git - scm:git:git@github.com:lyncode/xoai.git - git@github.com:lyncode/xoai.git - xoai-4.1.0 - - - - - ossrh - https://oss.sonatype.org/content/repositories/snapshots - - - ossrh - https://oss.sonatype.org/service/local/staging/deploy/maven2/ - - - - - - - - org.apache.maven.plugins - maven-gpg-plugin - 1.5 - - - org.apache.maven.plugins - maven-javadoc-plugin - 2.8.1 - - - org.apache.maven.plugins - maven-source-plugin - 2.2.1 - - - org.apache.maven.plugins - maven-release-plugin - 2.5 - - - org.apache.maven.plugins - maven-compiler-plugin - 3.1 - - - - - - org.apache.maven.plugins - maven-release-plugin - - true - false - release - deploy - - - - org.apache.maven.plugins - maven-compiler-plugin - - 1.6 - 1.6 - false - false - true - - - - org.apache.maven.plugins - maven-javadoc-plugin - true - - - attach-javadocs - - jar - - - - - - org.apache.maven.plugins - maven-source-plugin - true - - - attach-sources - - jar - - - - - - - - - - - com.lyncode - xml-io - ${lyncode.xml-io} - - - - com.lyncode - test-support - ${lyncode.test-support} - - - - - log4j - log4j - ${log4j.version} - - - - org.apache.commons - commons-lang3 - ${commons.lang3.version} - - - - org.apache.httpcomponents - httpclient - ${http-commons.version} - - - - org.codehaus.woodstox - wstx-asl - ${woodstox.version} - - - - org.codehaus.woodstox - stax2-api - 3.0.4 - - - - commons-codec - commons-codec - 1.3 - - - org.hamcrest - hamcrest-all - 1.3 - - - xalan - xalan - 2.7.2 - - - dom4j - dom4j - 1.6.1 - - - - javax.xml.stream - stax-api - 1.0-2 - - - jaxen - jaxen - 1.1.4 - - - junit - junit - 4.11 - - - commons-io - commons-io - 2.4 - - - - xml-apis - xml-apis - 1.0.b2 - - - - stax - stax-api - 1.0.1 - - - - org.mockito - mockito-all - ${mockito.version} - - - - com.google.guava - guava - ${guava.version} - - - - com.lyncode - builder-commons - ${lyncode.builder-commons} - - - - - - - - DSpace @ Lyncode - dspace@lyncode.com - Lyncode - http://www.lyncode.com - - - - diff --git a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.md5 b/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.md5 deleted file mode 100644 index d2fdadd114f..00000000000 --- a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.md5 +++ /dev/null @@ -1 +0,0 @@ -b50966bebe8cfdcb58478cf029b08aa3 diff --git a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.sha1 b/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.sha1 deleted file mode 100644 index b142cd649e8..00000000000 --- a/local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.sha1 +++ /dev/null @@ -1 +0,0 @@ -28a5d65399cbc25b29b270caebbb86e292c5ba18 diff --git a/modules/dataverse-parent/pom.xml b/modules/dataverse-parent/pom.xml index 8fe611d7716..c1ba693da1b 100644 --- a/modules/dataverse-parent/pom.xml +++ b/modules/dataverse-parent/pom.xml @@ -129,7 +129,7 @@ - 5.12 + 5.12.1 11 UTF-8 @@ -161,6 +161,9 @@ 1.21 4.5.13 4.4.14 + + + 5.0.0-RC1 1.15.0 @@ -301,7 +304,7 @@ Local repository for hosting jars not available from network repositories. file://${project.basedir}/local_lib - - - - - - - - - - - + - com.lyncode - xoai-common - 4.1.0-header-patch - - - com.lyncode + io.gdcc xoai-data-provider - 4.1.0-header-patch - - - log4j - log4j - - + ${gdcc.xoai.version} - com.lyncode + io.gdcc xoai-service-provider - 4.1.0-header-patch - - - log4j - log4j - - - - - - - - ch.qos.reload4j - reload4j - ${reload4j.version} - runtime + ${gdcc.xoai.version} diff --git a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java index 9027be1350b..abe3cc3e6d7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java @@ -33,6 +33,7 @@ import javax.xml.stream.XMLStreamWriter; import edu.harvard.iq.dataverse.util.BundleUtil; +import edu.harvard.iq.dataverse.util.DateUtil; import org.apache.commons.text.StringEscapeUtils; import org.apache.commons.lang3.StringUtils; @@ -713,25 +714,24 @@ private String flattenHtml(String html) { private Date getDateFrom(DatasetVersion dsv) { Date citationDate = null; - SimpleDateFormat sdf = new SimpleDateFormat("yyyy"); - if (!dsv.getDataset().isHarvested()) { - citationDate = dsv.getCitationDate(); + + if (dsv.getDataset().isHarvested()) { + citationDate = DateUtil.parseDate(dsv.getProductionDate()); if (citationDate == null) { - if (dsv.getDataset().getCitationDate() != null) { - citationDate = dsv.getDataset().getCitationDate(); - } else { // for drafts - citationDate = dsv.getLastUpdateTime(); - } + citationDate = DateUtil.parseDate(dsv.getDistributionDate()); } - } else { - try { - citationDate= sdf.parse(dsv.getDistributionDate()); - } catch (ParseException ex) { - // ignore - } catch (Exception ex) { - // ignore + } + + if (citationDate == null) { + if (dsv.getCitationDate() != null) { + citationDate = dsv.getCitationDate(); + } else if (dsv.getDataset().getCitationDate() != null) { + citationDate = dsv.getDataset().getCitationDate(); + } else { // for drafts + citationDate = dsv.getLastUpdateTime(); } } + if (citationDate == null) { //As a last resort, pick the current date logger.warning("Unable to find citation date for datasetversion: " + dsv.getId()); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java index 01f627ea23b..2410da04072 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java @@ -217,37 +217,37 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] } } } - if (redirect_url_str!=null) { + } + if (redirect_url_str != null) { - logger.fine("Data Access API: redirect url: " + redirect_url_str); - URI redirect_uri; + logger.fine("Data Access API: redirect url: " + redirect_url_str); + URI redirect_uri; - try { - redirect_uri = new URI(redirect_url_str); - } catch (URISyntaxException ex) { - logger.info("Data Access API: failed to create redirect url (" + redirect_url_str + ")"); - redirect_uri = null; - } - if (redirect_uri != null) { - // increment the download count, if necessary: - if (di.getGbr() != null && !(isThumbnailDownload(di) || isPreprocessedMetadataDownload(di))) { - try { - logger.fine("writing guestbook response, for a download redirect."); - Command cmd = new CreateGuestbookResponseCommand(di.getDataverseRequestService().getDataverseRequest(), di.getGbr(), di.getGbr().getDataFile().getOwner()); - di.getCommand().submit(cmd); - MakeDataCountEntry entry = new MakeDataCountEntry(di.getRequestUriInfo(), di.getRequestHttpHeaders(), di.getDataverseRequestService(), di.getGbr().getDataFile()); - mdcLogService.logEntry(entry); - } catch (CommandException e) { - } + try { + redirect_uri = new URI(redirect_url_str); + } catch (URISyntaxException ex) { + logger.info("Data Access API: failed to create redirect url (" + redirect_url_str + ")"); + redirect_uri = null; + } + if (redirect_uri != null) { + // increment the download count, if necessary: + if (di.getGbr() != null && !(isThumbnailDownload(di) || isPreprocessedMetadataDownload(di))) { + try { + logger.fine("writing guestbook response, for a download redirect."); + Command cmd = new CreateGuestbookResponseCommand(di.getDataverseRequestService().getDataverseRequest(), di.getGbr(), di.getGbr().getDataFile().getOwner()); + di.getCommand().submit(cmd); + MakeDataCountEntry entry = new MakeDataCountEntry(di.getRequestUriInfo(), di.getRequestHttpHeaders(), di.getDataverseRequestService(), di.getGbr().getDataFile()); + mdcLogService.logEntry(entry); + } catch (CommandException e) { } - - // finally, issue the redirect: - Response response = Response.seeOther(redirect_uri).build(); - logger.fine("Issuing redirect to the file location."); - throw new RedirectionException(response); } - throw new ServiceUnavailableException(); + + // finally, issue the redirect: + Response response = Response.seeOther(redirect_uri).build(); + logger.fine("Issuing redirect to the file location."); + throw new RedirectionException(response); } + throw new ServiceUnavailableException(); } if (di.getConversionParam() != null) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2LoginBackingBean.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2LoginBackingBean.java index 225352dec43..c5be41a014a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2LoginBackingBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2LoginBackingBean.java @@ -1,9 +1,11 @@ package edu.harvard.iq.dataverse.authorization.providers.oauth2; import edu.harvard.iq.dataverse.DataverseSession; +import edu.harvard.iq.dataverse.authorization.AuthenticationProvider; import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; import edu.harvard.iq.dataverse.authorization.UserRecordIdentifier; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.ClockUtil; import edu.harvard.iq.dataverse.util.StringUtil; import java.io.BufferedReader; @@ -27,6 +29,8 @@ import static edu.harvard.iq.dataverse.util.StringUtil.toOption; import edu.harvard.iq.dataverse.util.SystemConfig; +import java.text.MessageFormat; +import java.util.ArrayList; import org.omnifaces.util.Faces; /** @@ -45,6 +49,8 @@ public class OAuth2LoginBackingBean implements Serializable { private String responseBody; Optional redirectPage = Optional.empty(); private OAuth2Exception error; + private boolean disabled = false; + private boolean signUpDisabled = false; /** * TODO: Only used in exchangeCodeForToken(). Make local var in method. */ @@ -96,13 +102,28 @@ public void exchangeCodeForToken() throws IOException { AbstractOAuth2AuthenticationProvider idp = oIdp.get(); oauthUser = idp.getUserRecord(code.get(), systemConfig.getOAuth2CallbackUrl()); + // Throw an error if this authentication method is disabled: + // (it's not clear if it's possible at all, for somebody to get here with + // the provider really disabled; but, shouldn't hurt either). + if (isProviderDisabled(idp.getId())) { + disabled = true; + throw new OAuth2Exception(-1, "", MessageFormat.format(BundleUtil.getStringFromBundle("oauth2.callback.error.providerDisabled"), idp.getId())); + } + UserRecordIdentifier idtf = oauthUser.getUserRecordIdentifier(); AuthenticatedUser dvUser = authenticationSvc.lookupUser(idtf); if (dvUser == null) { - // need to create the user - newAccountPage.setNewUser(oauthUser); - Faces.redirect("/oauth2/firstLogin.xhtml"); + // Need to create a new user - unless signups are disabled + // for this authentication method; in which case, throw + // an error: + if (systemConfig.isSignupDisabledForRemoteAuthProvider(idp.getId())) { + signUpDisabled = true; + throw new OAuth2Exception(-1, "", MessageFormat.format(BundleUtil.getStringFromBundle("oauth2.callback.error.signupDisabledForProvider"), idp.getId())); + } else { + newAccountPage.setNewUser(oauthUser); + Faces.redirect("/oauth2/firstLogin.xhtml"); + } } else { // login the user and redirect to HOME of intended page (if any). @@ -271,4 +292,32 @@ public List getProviders() { public boolean isOAuth2ProvidersDefined() { return !authenticationSvc.getOAuth2Providers().isEmpty(); } + + public boolean isDisabled() { + return disabled; + } + + public boolean isSignUpDisabled() { + return signUpDisabled; + } + + private boolean isProviderDisabled(String providerId) { + // Compare this provider id against the list of *enabled* auth providers + // returned by the Authentication Service: + List idps = new ArrayList<>(authenticationSvc.getAuthenticationProviders()); + + // for the tests to work: + if (idps.isEmpty()) { + return false; + } + + for (AuthenticationProvider idp : idps) { + if (idp != null) { + if (providerId.equals(idp.getId())) { + return false; + } + } + } + return true; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/users/AuthenticatedUser.java b/src/main/java/edu/harvard/iq/dataverse/authorization/users/AuthenticatedUser.java index b2b5fa92e76..2cd28d9aac9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/users/AuthenticatedUser.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/users/AuthenticatedUser.java @@ -18,6 +18,7 @@ import java.io.Serializable; import java.sql.Timestamp; import java.util.Date; +import java.util.HashSet; import java.util.List; import java.util.Objects; import java.util.Set; @@ -135,10 +136,10 @@ public class AuthenticatedUser implements User, Serializable { private String mutedNotifications; @Transient - private Set mutedEmailsSet; + private Set mutedEmailsSet = new HashSet<>(); @Transient - private Set mutedNotificationsSet; + private Set mutedNotificationsSet = new HashSet<>(); @PrePersist void prePersist() { diff --git a/src/main/java/edu/harvard/iq/dataverse/export/dublincore/DublinCoreExportUtil.java b/src/main/java/edu/harvard/iq/dataverse/export/dublincore/DublinCoreExportUtil.java index 9278faf1f8f..4409d2340b1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/dublincore/DublinCoreExportUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/dublincore/DublinCoreExportUtil.java @@ -122,7 +122,12 @@ private static void createDC(XMLStreamWriter xmlw, DatasetDTO datasetDto, String writeFullElementList(xmlw, dcFlavor+":"+"language", dto2PrimitiveList(version, DatasetFieldConstant.language)); writeRelPublElement(xmlw, version, dcFlavor); - writeFullElement(xmlw, dcFlavor+":"+"date", dto2Primitive(version, DatasetFieldConstant.productionDate)); + + String date = dto2Primitive(version, DatasetFieldConstant.productionDate); + if (date == null) { + date = datasetDto.getPublicationDate(); + } + writeFullElement(xmlw, dcFlavor+":"+"date", date); writeFullElement(xmlw, dcFlavor+":"+"contributor", dto2Primitive(version, DatasetFieldConstant.depositor)); @@ -172,7 +177,11 @@ private static void createOAIDC(XMLStreamWriter xmlw, DatasetDTO datasetDto, Str writeFullElementList(xmlw, dcFlavor+":"+"language", dto2PrimitiveList(version, DatasetFieldConstant.language)); - writeFullElement(xmlw, dcFlavor+":"+"date", dto2Primitive(version, DatasetFieldConstant.productionDate)); + String date = dto2Primitive(version, DatasetFieldConstant.productionDate); + if (date == null) { + date = datasetDto.getPublicationDate(); + } + writeFullElement(xmlw, dcFlavor+":"+"date", date); writeFullElement(xmlw, dcFlavor+":"+"contributor", dto2Primitive(version, DatasetFieldConstant.depositor)); diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java index 60abc97bccd..5b3e4df331d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java @@ -72,7 +72,6 @@ public class FastGetRecord { - private static final String DATAVERSE_EXTENDED_METADATA = "dataverse_json"; private static final String XML_METADATA_TAG = "metadata"; private static final String XML_METADATA_TAG_OPEN = "<"+XML_METADATA_TAG+">"; private static final String XML_METADATA_TAG_CLOSE = ""; @@ -222,13 +221,7 @@ public void harvestRecord(String baseURL, String identifier, String metadataPref //metadataOut.println(""); /* ? */ metadataFlag = true; - } else if (line.matches(".*<"+XML_METADATA_TAG+" [^>]*>.*")) { - if (metadataPrefix.equals(DATAVERSE_EXTENDED_METADATA)) { - oaiResponseHeader = oaiResponseHeader.concat(line); - metadataWritten = true; - metadataFlag = true; - } - } + } } //System.out.println(line); @@ -380,19 +373,12 @@ public void harvestRecord(String baseURL, String identifier, String metadataPref try { StringReader reader = new StringReader(oaiResponseHeader); xmlr = xmlInputFactory.createXMLStreamReader(reader); - processOAIheader(xmlr, metadataPrefix.equals(DATAVERSE_EXTENDED_METADATA)); + processOAIheader(xmlr); } catch (XMLStreamException ex) { - //Logger.getLogger("global").log(Level.SEVERE, null, ex); if (this.errorMessage == null) { this.errorMessage = "Malformed GetRecord response; baseURL=" + baseURL + ", identifier=" + identifier + ", metadataPrefix=" + metadataPrefix; } - - // delete the temp metadata file; we won't need it: - if (savedMetadataFile != null) { - //savedMetadataFile.delete(); - } - } try { @@ -414,14 +400,8 @@ public void harvestRecord(String baseURL, String identifier, String metadataPref if (!(metadataWritten) && !(this.isDeleted())) { this.errorMessage = "Failed to parse GetRecord response; baseURL=" + baseURL + ", identifier=" + identifier + ", metadataPrefix=" + metadataPrefix; - //savedMetadataFile.delete(); - } - - if (this.isDeleted()) { - //savedMetadataFile.delete(); } - } else { this.errorMessage = "GetRecord request failed. HTTP error code "+responseCode; } @@ -445,16 +425,16 @@ private static String getRequestURL(String baseURL, return requestURL.toString(); } - private void processOAIheader (XMLStreamReader xmlr, boolean extensionMode) throws XMLStreamException, IOException { + private void processOAIheader (XMLStreamReader xmlr) throws XMLStreamException, IOException { // is this really a GetRecord response? xmlr.nextTag(); xmlr.require(XMLStreamConstants.START_ELEMENT, null, "OAI-PMH"); - processOAIPMH(xmlr, extensionMode); + processOAIPMH(xmlr); } - private void processOAIPMH (XMLStreamReader xmlr, boolean extensionMode) throws XMLStreamException, IOException { + private void processOAIPMH (XMLStreamReader xmlr) throws XMLStreamException, IOException { for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) { if (event == XMLStreamConstants.START_ELEMENT) { @@ -477,7 +457,7 @@ else if (xmlr.getLocalName().equals("error")) { } else if (xmlr.getLocalName().equals("GetRecord")) { - processGetRecordSection(xmlr, extensionMode); + processGetRecordSection(xmlr); } } else if (event == XMLStreamConstants.END_ELEMENT) { if (xmlr.getLocalName().equals("OAI-PMH")) return; @@ -485,11 +465,11 @@ else if (xmlr.getLocalName().equals("GetRecord")) { } } - private void processGetRecordSection (XMLStreamReader xmlr, boolean extensionMode) throws XMLStreamException, IOException { + private void processGetRecordSection (XMLStreamReader xmlr) throws XMLStreamException, IOException { for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) { if (event == XMLStreamConstants.START_ELEMENT) { if (xmlr.getLocalName().equals("record")) { - processRecord(xmlr, extensionMode); + processRecord(xmlr); } } else if (event == XMLStreamConstants.END_ELEMENT) { if (xmlr.getLocalName().equals("GetRecord")) return; @@ -498,7 +478,7 @@ private void processGetRecordSection (XMLStreamReader xmlr, boolean extensionMod } - private void processRecord (XMLStreamReader xmlr, boolean extensionMode) throws XMLStreamException, IOException { + private void processRecord (XMLStreamReader xmlr) throws XMLStreamException, IOException { for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) { if (event == XMLStreamConstants.START_ELEMENT) { if (xmlr.getLocalName().equals("header")) { @@ -506,11 +486,6 @@ private void processRecord (XMLStreamReader xmlr, boolean extensionMode) throws this.recordDeleted = true; } processHeader(xmlr); - } else if (xmlr.getLocalName().equals("metadata")) { - if (extensionMode) { - String extendedMetadataApiUrl = xmlr.getAttributeValue(null, "directApiCall"); - processMetadataExtended(extendedMetadataApiUrl); - } } } else if (event == XMLStreamConstants.END_ELEMENT) { if (xmlr.getLocalName().equals("record")) return; @@ -532,67 +507,6 @@ else if (xmlr.getLocalName().equals("setSpec")) {/*do nothing*/} } } - private void processMetadataExtended (String extendedApiUrl) throws IOException { - InputStream in = null; - int responseCode = 0; - HttpURLConnection con = null; - - - - try { - URL url = new URL(extendedApiUrl.replaceAll("&", "&")); // is this necessary? - - con = (HttpURLConnection) url.openConnection(); - con.setRequestProperty("User-Agent", "DataverseHarvester/3.0"); - responseCode = con.getResponseCode(); - } catch (MalformedURLException mue) { - throw new IOException ("Bad API URL: "+extendedApiUrl); - } catch (FileNotFoundException e) { - responseCode = HttpURLConnection.HTTP_UNAVAILABLE; - } - - - - - if (responseCode == 200) { - in = con.getInputStream(); - // TODO: - /* we should probably still support gzip/compress encoding here - ? - String contentEncoding = con.getHeaderField("Content-Encoding"); - - // support for the standard compress/gzip/deflate compression - // schemes: - - if ("compress".equals(contentEncoding)) { - ZipInputStream zis = new ZipInputStream(con.getInputStream()); - zis.getNextEntry(); - in = zis; - } else if ("gzip".equals(contentEncoding)) { - in = new GZIPInputStream(con.getInputStream()); - } else if ("deflate".equals(contentEncoding)) { - in = new InflaterInputStream(con.getInputStream()); - } ... - */ - FileOutputStream tempOut = new FileOutputStream(savedMetadataFile); - - int bufsize; - byte[] buffer = new byte[4 * 8192]; - - while ((bufsize = in.read(buffer)) != -1) { - tempOut.write(buffer, 0, bufsize); - tempOut.flush(); - } - - in.close(); - tempOut.close(); - return; - } - - throw new IOException("Failed to download extended metadata."); - - } - - // (from Gustavo's ddiServiceBean -- L.A.) // /* We had to add this method because the ref getElementText has a bug where it diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java index 71cc23e242b..e7156dfe9aa 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java @@ -5,6 +5,7 @@ */ package edu.harvard.iq.dataverse.harvest.client; +import static java.net.HttpURLConnection.HTTP_OK; import edu.harvard.iq.dataverse.Dataset; import edu.harvard.iq.dataverse.DatasetServiceBean; import edu.harvard.iq.dataverse.Dataverse; @@ -17,7 +18,6 @@ import java.util.Date; import java.util.Iterator; import java.util.List; -//import java.net.URLEncoder; import java.util.logging.FileHandler; import java.util.logging.Level; import java.util.logging.Logger; @@ -28,22 +28,26 @@ import javax.ejb.Stateless; import javax.ejb.Timer; import javax.inject.Named; -//import javax.xml.bind.Unmarshaller; import javax.xml.parsers.ParserConfigurationException; import javax.xml.transform.TransformerException; import org.apache.commons.lang3.mutable.MutableBoolean; -import org.apache.commons.lang3.mutable.MutableLong; import org.xml.sax.SAXException; -import com.lyncode.xoai.model.oaipmh.Header; +import io.gdcc.xoai.model.oaipmh.results.record.Header; import edu.harvard.iq.dataverse.EjbDataverseEngine; import edu.harvard.iq.dataverse.api.imports.ImportServiceBean; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.harvest.client.oai.OaiHandler; import edu.harvard.iq.dataverse.harvest.client.oai.OaiHandlerException; import edu.harvard.iq.dataverse.search.IndexServiceBean; +import java.io.FileOutputStream; import java.io.FileWriter; +import java.io.InputStream; import java.io.PrintWriter; +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; import javax.persistence.EntityManager; import javax.persistence.PersistenceContext; @@ -75,13 +79,12 @@ public class HarvesterServiceBean { IndexServiceBean indexService; private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean"); - private static final SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd"); private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); public static final String HARVEST_RESULT_SUCCESS="success"; public static final String HARVEST_RESULT_FAILED="failed"; - private static final Long INDEXING_CONTENT_BATCH_SIZE = 10000000L; - + public static final String DATAVERSE_PROPRIETARY_METADATA_FORMAT="dataverse_json"; + public static final String DATAVERSE_PROPRIETARY_METADATA_API="/api/datasets/export?exporter="+DATAVERSE_PROPRIETARY_METADATA_FORMAT+"&persistentId="; public HarvesterServiceBean() { @@ -183,24 +186,7 @@ public void doHarvest(DataverseRequest dataverseRequest, Long harvestingClientId hdLogger.log(Level.INFO, "COMPLETED HARVEST, server=" + harvestingClientConfig.getArchiveUrl() + ", metadataPrefix=" + harvestingClientConfig.getMetadataPrefix()); hdLogger.log(Level.INFO, "Datasets created/updated: " + harvestedDatasetIds.size() + ", datasets deleted: " + deletedIdentifiers.size() + ", datasets failed: " + failedIdentifiers.size()); - // now index all the datasets we have harvested - created, modified or deleted: - /* (TODO: may not be needed at all. In Dataverse4, we may be able to get away with the normal - reindexing after every import. See the rest of the comments about batch indexing throughout - this service bean) - if (this.processedSizeThisBatch > 0) { - hdLogger.log(Level.INFO, "POST HARVEST, reindexing the remaining studies."); - if (this.harvestedDatasetIdsThisBatch != null) { - hdLogger.log(Level.INFO, this.harvestedDatasetIdsThisBatch.size()+" studies in the batch"); - } - hdLogger.log(Level.INFO, this.processedSizeThisBatch + " bytes of content"); - indexService.updateIndexList(this.harvestedDatasetIdsThisBatch); - hdLogger.log(Level.INFO, "POST HARVEST, calls to index finished."); - } else { - hdLogger.log(Level.INFO, "(All harvested content already reindexed)"); - } - */ } - //mailService.sendHarvestNotification(...getSystemEmail(), harvestingDataverse.getName(), logFileName, logTimestamp, harvestErrorOccurred.booleanValue(), harvestedDatasetIds.size(), failedIdentifiers); } catch (Throwable e) { harvestErrorOccurred.setValue(true); String message = "Exception processing harvest, server= " + harvestingClientConfig.getHarvestingUrl() + ",format=" + harvestingClientConfig.getMetadataPrefix() + " " + e.getClass().getName() + " " + e.getMessage(); @@ -235,8 +221,8 @@ private List harvestOAI(DataverseRequest dataverseRequest, HarvestingClien logBeginOaiHarvest(hdLogger, harvestingClient); List harvestedDatasetIds = new ArrayList(); - MutableLong processedSizeThisBatch = new MutableLong(0L); OaiHandler oaiHandler; + HttpClient httpClient = null; try { oaiHandler = new OaiHandler(harvestingClient); @@ -248,22 +234,35 @@ private List harvestOAI(DataverseRequest dataverseRequest, HarvestingClien hdLogger.log(Level.SEVERE, errorMessage); throw new IOException(errorMessage); } - + + if (DATAVERSE_PROPRIETARY_METADATA_FORMAT.equals(oaiHandler.getMetadataPrefix())) { + // If we are harvesting native Dataverse json, we'll also need this + // jdk http client to make direct calls to the remote Dataverse API: + httpClient = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.ALWAYS).build(); + } + try { for (Iterator
idIter = oaiHandler.runListIdentifiers(); idIter.hasNext();) { Header h = idIter.next(); String identifier = h.getIdentifier(); - Date dateStamp = h.getDatestamp(); + Date dateStamp = Date.from(h.getDatestamp()); hdLogger.info("processing identifier: " + identifier + ", date: " + dateStamp); + + if (h.isDeleted()) { + hdLogger.info("Deleting harvesting dataset for " + identifier + ", per ListIdentifiers."); + + deleteHarvestedDatasetIfExists(identifier, oaiHandler.getHarvestingClient().getDataverse(), dataverseRequest, deletedIdentifiers, hdLogger); + continue; + } MutableBoolean getRecordErrorOccurred = new MutableBoolean(false); // Retrieve and process this record with a separate GetRecord call: - Long datasetId = processRecord(dataverseRequest, hdLogger, importCleanupLog, oaiHandler, identifier, getRecordErrorOccurred, processedSizeThisBatch, deletedIdentifiers, dateStamp); - hdLogger.info("Total content processed in this batch so far: "+processedSizeThisBatch); + Long datasetId = processRecord(dataverseRequest, hdLogger, importCleanupLog, oaiHandler, identifier, getRecordErrorOccurred, deletedIdentifiers, dateStamp, httpClient); + if (datasetId != null) { harvestedDatasetIds.add(datasetId); @@ -280,20 +279,6 @@ private List harvestOAI(DataverseRequest dataverseRequest, HarvestingClien //temporary: //throw new IOException("Exception occured, stopping harvest"); } - - // reindexing in batches? - this is from DVN 3; - // we may not need it anymore. - if ( processedSizeThisBatch.longValue() > INDEXING_CONTENT_BATCH_SIZE ) { - - hdLogger.log(Level.INFO, "REACHED CONTENT BATCH SIZE LIMIT; calling index ("+ harvestedDatasetIdsThisBatch.size()+" datasets in the batch)."); - //indexService.updateIndexList(this.harvestedDatasetIdsThisBatch); - hdLogger.log(Level.INFO, "REINDEX DONE."); - - - processedSizeThisBatch.setValue(0L); - harvestedDatasetIdsThisBatch = null; - } - } } catch (OaiHandlerException e) { throw new IOException("Failed to run ListIdentifiers: " + e.getMessage()); @@ -303,57 +288,56 @@ private List harvestOAI(DataverseRequest dataverseRequest, HarvestingClien return harvestedDatasetIds; - } - + } - - private Long processRecord(DataverseRequest dataverseRequest, Logger hdLogger, PrintWriter importCleanupLog, OaiHandler oaiHandler, String identifier, MutableBoolean recordErrorOccurred, MutableLong processedSizeThisBatch, List deletedIdentifiers, Date dateStamp) { + private Long processRecord(DataverseRequest dataverseRequest, Logger hdLogger, PrintWriter importCleanupLog, OaiHandler oaiHandler, String identifier, MutableBoolean recordErrorOccurred, List deletedIdentifiers, Date dateStamp, HttpClient httpClient) { String errMessage = null; Dataset harvestedDataset = null; logGetRecord(hdLogger, oaiHandler, identifier); File tempFile = null; - try { - FastGetRecord record = oaiHandler.runGetRecord(identifier); - errMessage = record.getErrorMessage(); + try { + boolean deleted = false; + + if (DATAVERSE_PROPRIETARY_METADATA_FORMAT.equals(oaiHandler.getMetadataPrefix())) { + // Make direct call to obtain the proprietary Dataverse metadata + // in JSON from the remote Dataverse server: + String metadataApiUrl = oaiHandler.getProprietaryDataverseMetadataURL(identifier); + logger.info("calling "+metadataApiUrl); + tempFile = retrieveProprietaryDataverseMetadata(httpClient, metadataApiUrl); + + } else { + FastGetRecord record = oaiHandler.runGetRecord(identifier); + errMessage = record.getErrorMessage(); + deleted = record.isDeleted(); + tempFile = record.getMetadataFile(); + } if (errMessage != null) { hdLogger.log(Level.SEVERE, "Error calling GetRecord - " + errMessage); - } else if (record.isDeleted()) { - hdLogger.info("Deleting harvesting dataset for "+identifier+", per the OAI server's instructions."); - Dataset dataset = datasetService.getDatasetByHarvestInfo(oaiHandler.getHarvestingClient().getDataverse(), identifier); - if (dataset != null) { - hdLogger.info("Deleting dataset " + dataset.getGlobalIdString()); - datasetService.deleteHarvestedDataset(dataset, dataverseRequest, hdLogger); - // TODO: - // check the status of that Delete - see if it actually succeeded - deletedIdentifiers.add(identifier); - } else { - hdLogger.info("No dataset found for "+identifier+", skipping delete. "); - } - + } else if (deleted) { + hdLogger.info("Deleting harvesting dataset for "+identifier+", per GetRecord."); + + deleteHarvestedDatasetIfExists(identifier, oaiHandler.getHarvestingClient().getDataverse(), dataverseRequest, deletedIdentifiers, hdLogger); } else { hdLogger.info("Successfully retrieved GetRecord response."); - tempFile = record.getMetadataFile(); PrintWriter cleanupLog; harvestedDataset = importService.doImportHarvestedDataset(dataverseRequest, oaiHandler.getHarvestingClient(), identifier, oaiHandler.getMetadataPrefix(), - record.getMetadataFile(), + tempFile, dateStamp, importCleanupLog); hdLogger.fine("Harvest Successful for identifier " + identifier); - hdLogger.fine("Size of this record: " + record.getMetadataFile().length()); - processedSizeThisBatch.add(record.getMetadataFile().length()); + hdLogger.fine("Size of this record: " + tempFile.length()); } } catch (Throwable e) { logGetRecordException(hdLogger, oaiHandler, identifier, e); errMessage = "Caught exception while executing GetRecord on "+identifier; - //logException(e, hdLogger); } finally { if (tempFile != null) { @@ -364,14 +348,12 @@ private Long processRecord(DataverseRequest dataverseRequest, Logger hdLogger, P } } - // TODO: the message below is taken from DVN3; - figure out what it means... - // // If we got an Error from the OAI server or an exception happened during import, then // set recordErrorOccurred to true (if recordErrorOccurred is being used) // otherwise throw an exception (if recordErrorOccurred is not used, i.e null) if (errMessage != null) { - if (recordErrorOccurred != null) { + if (recordErrorOccurred != null) { recordErrorOccurred.setValue(true); } else { throw new EJBException(errMessage); @@ -380,7 +362,55 @@ private Long processRecord(DataverseRequest dataverseRequest, Logger hdLogger, P return harvestedDataset != null ? harvestedDataset.getId() : null; } - + + File retrieveProprietaryDataverseMetadata (HttpClient client, String remoteApiUrl) throws IOException { + + if (client == null) { + throw new IOException("Null Http Client, cannot make a call to obtain native metadata."); + } + + HttpRequest request = HttpRequest.newBuilder() + .uri(URI.create(remoteApiUrl)) + .GET() + .header("User-Agent", "DataverseHarvester/6.0") + .build(); + + HttpResponse response; + + try { + response = client.send(request, HttpResponse.BodyHandlers.ofInputStream()); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + throw new IOException("Failed to connect to the remote dataverse server to obtain native metadata"); + } + + int responseCode = response.statusCode(); + + if (responseCode == HTTP_OK) { + File tempMetadataFile = File.createTempFile("meta", ".tmp"); + + try (InputStream inputStream = response.body(); + FileOutputStream outputStream = new FileOutputStream(tempMetadataFile);) { + inputStream.transferTo(outputStream); + return tempMetadataFile; + } + } + + throw new IOException("Failed to download native metadata from the remote dataverse server."); + } + + private void deleteHarvestedDatasetIfExists(String persistentIdentifier, Dataverse harvestingDataverse, DataverseRequest dataverseRequest, List deletedIdentifiers, Logger hdLogger) { + Dataset dataset = datasetService.getDatasetByHarvestInfo(harvestingDataverse, persistentIdentifier); + if (dataset != null) { + datasetService.deleteHarvestedDataset(dataset, dataverseRequest, hdLogger); + // TODO: + // check the status of that Delete - see if it actually succeeded + deletedIdentifiers.add(persistentIdentifier); + return; + } + hdLogger.info("No dataset found for " + persistentIdentifier + ", skipping delete. "); + } + private void logBeginOaiHarvest(Logger hdLogger, HarvestingClient harvestingClient) { hdLogger.log(Level.INFO, "BEGIN HARVEST, oaiUrl=" +harvestingClient.getHarvestingUrl() @@ -448,47 +478,5 @@ private void logException(Throwable e, Logger logger) { } while ((e = e.getCause()) != null); logger.severe(fullMessage); } - - /* - some dead code below: - this functionality has been moved into OaiHandler. - TODO: test that harvesting is still working and remove. - - private ServiceProvider getServiceProvider(String baseOaiUrl, Granularity oaiGranularity) { - Context context = new Context(); - - context.withBaseUrl(baseOaiUrl); - context.withGranularity(oaiGranularity); - context.withOAIClient(new HttpOAIClient(baseOaiUrl)); - - ServiceProvider serviceProvider = new ServiceProvider(context); - return serviceProvider; - } - */ - - /** - * Creates an XOAI parameters object for the ListIdentifiers call - * - * @param metadataPrefix - * @param set - * @param from - * @return ListIdentifiersParameters - */ - /* - private ListIdentifiersParameters buildParams(String metadataPrefix, String set, Date from) { - ListIdentifiersParameters mip = ListIdentifiersParameters.request(); - mip.withMetadataPrefix(metadataPrefix); - - if (from != null) { - mip.withFrom(from); - } - - if (set != null) { - mip.withSetSpec(set); - } - return mip; - } - */ - - + } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java index d1aaea50793..c0a039e2d2b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java @@ -1,33 +1,27 @@ -/* - * To change this license header, choose License Headers in Project Properties. - * To change this template file, choose Tools | Templates - * and open the template in the editor. - */ package edu.harvard.iq.dataverse.harvest.client.oai; -import com.lyncode.xoai.model.oaipmh.Description; -import com.lyncode.xoai.model.oaipmh.Granularity; -import com.lyncode.xoai.model.oaipmh.Header; -import com.lyncode.xoai.model.oaipmh.MetadataFormat; -import com.lyncode.xoai.model.oaipmh.Set; -import com.lyncode.xoai.serviceprovider.ServiceProvider; -import com.lyncode.xoai.serviceprovider.client.HttpOAIClient; -import com.lyncode.xoai.serviceprovider.exceptions.BadArgumentException; -import com.lyncode.xoai.serviceprovider.exceptions.InvalidOAIResponse; -import com.lyncode.xoai.serviceprovider.exceptions.NoSetHierarchyException; -import com.lyncode.xoai.serviceprovider.model.Context; -import com.lyncode.xoai.serviceprovider.parameters.ListIdentifiersParameters; +import io.gdcc.xoai.model.oaipmh.Granularity; +import io.gdcc.xoai.model.oaipmh.results.record.Header; +import io.gdcc.xoai.model.oaipmh.results.MetadataFormat; +import io.gdcc.xoai.model.oaipmh.results.Set; +import io.gdcc.xoai.serviceprovider.ServiceProvider; +import io.gdcc.xoai.serviceprovider.client.JdkHttpOaiClient; +import io.gdcc.xoai.serviceprovider.exceptions.BadArgumentException; +import io.gdcc.xoai.serviceprovider.exceptions.InvalidOAIResponse; +import io.gdcc.xoai.serviceprovider.exceptions.NoSetHierarchyException; +import io.gdcc.xoai.serviceprovider.exceptions.IdDoesNotExistException; +import io.gdcc.xoai.serviceprovider.model.Context; +import io.gdcc.xoai.serviceprovider.parameters.ListIdentifiersParameters; import edu.harvard.iq.dataverse.harvest.client.FastGetRecord; +import static edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.DATAVERSE_PROPRIETARY_METADATA_API; import edu.harvard.iq.dataverse.harvest.client.HarvestingClient; import java.io.IOException; import java.io.Serializable; -import java.io.UnsupportedEncodingException; import javax.xml.parsers.ParserConfigurationException; import org.apache.commons.lang3.StringUtils; import org.xml.sax.SAXException; import javax.xml.transform.TransformerException; -import java.net.URLEncoder; import java.util.ArrayList; import java.util.Date; import java.util.Iterator; @@ -74,8 +68,9 @@ public OaiHandler(HarvestingClient harvestingClient) throws OaiHandlerException this.harvestingClient = harvestingClient; } - private String baseOaiUrl; //= harvestingClient.getHarvestingUrl(); - private String metadataPrefix; // = harvestingClient.getMetadataPrefix(); + private String baseOaiUrl; + private String dataverseApiUrl; // if the remote server is a Dataverse and we access its native metadata + private String metadataPrefix; private String setName; private Date fromDate; private Boolean setListTruncated = false; @@ -124,7 +119,7 @@ public boolean isSetListTruncated() { return setListTruncated; } - private ServiceProvider getServiceProvider() throws OaiHandlerException { + public ServiceProvider getServiceProvider() throws OaiHandlerException { if (serviceProvider == null) { if (baseOaiUrl == null) { throw new OaiHandlerException("Could not instantiate Service Provider, missing OAI server URL."); @@ -133,8 +128,8 @@ private ServiceProvider getServiceProvider() throws OaiHandlerException { context.withBaseUrl(baseOaiUrl); context.withGranularity(Granularity.Second); - context.withOAIClient(new HttpOAIClient(baseOaiUrl)); - + // builds the client with the default parameters and the JDK http client: + context.withOAIClient(JdkHttpOaiClient.newBuilder().withBaseUrl(baseOaiUrl).build()); serviceProvider = new ServiceProvider(context); } @@ -199,6 +194,16 @@ public List runListMetadataFormats() throws OaiHandlerException { try { mfIter = sp.listMetadataFormats(); + } catch (IdDoesNotExistException idnee) { + // TODO: + // not sure why this exception is now thrown by List Metadata Formats (?) + // but looks like it was added in xoai 4.2. + // It appears that the answer is, they added it because you can + // call ListMetadataFormats on a specific identifier, optionally, + // and therefore it is possible to get back that response. Of course + // it will never be the case when calling it on an entire repository. + // But it's ok. + throw new OaiHandlerException("Id does not exist exception"); } catch (InvalidOAIResponse ior) { throw new OaiHandlerException("No valid response received from the OAI server."); } @@ -261,7 +266,7 @@ private ListIdentifiersParameters buildListIdentifiersParams() throws OaiHandler mip.withMetadataPrefix(metadataPrefix); if (this.fromDate != null) { - mip.withFrom(this.fromDate); + mip.withFrom(this.fromDate.toInstant()); } if (!StringUtils.isEmpty(this.setName)) { @@ -271,6 +276,18 @@ private ListIdentifiersParameters buildListIdentifiersParams() throws OaiHandler return mip; } + public String getProprietaryDataverseMetadataURL(String identifier) { + + if (dataverseApiUrl == null) { + dataverseApiUrl = baseOaiUrl.replaceFirst("/oai", ""); + } + + StringBuilder requestURL = new StringBuilder(dataverseApiUrl); + requestURL.append(DATAVERSE_PROPRIETARY_METADATA_API).append(identifier); + + return requestURL.toString(); + } + public void runIdentify() { // not implemented yet // (we will need it, both for validating the remote server, diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java index 057903d506a..6cdc4e5c277 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java @@ -12,6 +12,7 @@ import edu.harvard.iq.dataverse.export.ExportService; import edu.harvard.iq.dataverse.search.IndexServiceBean; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import java.time.Instant; import java.io.File; import java.io.IOException; import java.sql.Timestamp; @@ -286,15 +287,15 @@ public List findOaiRecordsBySetName(String setName) { return findOaiRecordsBySetName(setName, null, null); } - public List findOaiRecordsBySetName(String setName, Date from, Date until) { + public List findOaiRecordsBySetName(String setName, Instant from, Instant until) { return findOaiRecordsBySetName(setName, from, until, false); } - public List findOaiRecordsNotInThisSet(String setName, Date from, Date until) { + public List findOaiRecordsNotInThisSet(String setName, Instant from, Instant until) { return findOaiRecordsBySetName(setName, from, until, true); } - public List findOaiRecordsBySetName(String setName, Date from, Date until, boolean excludeSet) { + public List findOaiRecordsBySetName(String setName, Instant from, Instant until, boolean excludeSet) { if (setName == null) { setName = ""; @@ -314,35 +315,18 @@ public List findOaiRecordsBySetName(String setName, Date from, Date u logger.fine("Query: "+queryString); TypedQuery query = em.createQuery(queryString, OAIRecord.class); - if (setName != null) { query.setParameter("setName",setName); } - if (from != null) { query.setParameter("from",from,TemporalType.TIMESTAMP); } - // In order to achieve inclusivity on the "until" matching, we need to do - // the following (if the "until" parameter is supplied): - // 1) if the supplied "until" parameter has the time portion (and is not just - // a date), we'll increment it by one second. This is because the time stamps we - // keep in the database also have fractional thousands of a second. - // So, a record may be shown as "T17:35:45", but in the database it is - // actually "17:35:45.356", so "<= 17:35:45" isn't going to work on this - // time stamp! - So we want to try "<= 17:35:45" instead. - // 2) if it's just a date, we'll increment it by a *full day*. Otherwise - // our database time stamp of 2016-10-23T17:35:45.123Z is NOT going to - // match " <= 2016-10-23" - which is really going to be interpreted as - // "2016-10-23T00:00:00.000". - // -- L.A. 4.6 + if (setName != null) { + query.setParameter("setName",setName); + } + // TODO: review and phase out the use of java.util.Date throughout this service. + + if (from != null) { + query.setParameter("from",Date.from(from),TemporalType.TIMESTAMP); + } if (until != null) { - // 24 * 3600 * 1000 = number of milliseconds in a day. - - if (until.getTime() % (24 * 3600 * 1000) == 0) { - // The supplied "until" parameter is a date, with no time - // portion. - logger.fine("plain date. incrementing by one day"); - until.setTime(until.getTime()+(24 * 3600 * 1000)); - } else { - logger.fine("date and time. incrementing by one second"); - until.setTime(until.getTime()+1000); - } - query.setParameter("until",until,TemporalType.TIMESTAMP); + Date untilDate = Date.from(until); + query.setParameter("until",untilDate,TemporalType.TIMESTAMP); } try { diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java index f300f02f70c..6b28c8808a0 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java @@ -67,7 +67,7 @@ public OAISet find(Object pk) { return em.find(OAISet.class, pk); } - public boolean specExists(String spec) { + public boolean setExists(String spec) { boolean specExists = false; OAISet set = findBySpec(spec); diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java index d8619c42dfa..5eacb1addb6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java @@ -5,26 +5,19 @@ */ package edu.harvard.iq.dataverse.harvest.server.web.servlet; -import com.lyncode.xml.exceptions.XmlWriteException; -import com.lyncode.xoai.dataprovider.builder.OAIRequestParametersBuilder; -import com.lyncode.xoai.dataprovider.exceptions.OAIException; -import com.lyncode.xoai.dataprovider.repository.Repository; -import com.lyncode.xoai.dataprovider.repository.RepositoryConfiguration; -import com.lyncode.xoai.dataprovider.model.Context; -import com.lyncode.xoai.dataprovider.model.MetadataFormat; -import com.lyncode.xoai.services.impl.SimpleResumptionTokenFormat; -import com.lyncode.xoai.dataprovider.repository.ItemRepository; -import com.lyncode.xoai.dataprovider.repository.SetRepository; -import com.lyncode.xoai.model.oaipmh.DeletedRecord; -import com.lyncode.xoai.model.oaipmh.Granularity; -import com.lyncode.xoai.model.oaipmh.OAIPMH; -import static com.lyncode.xoai.model.oaipmh.OAIPMH.NAMESPACE_URI; -import static com.lyncode.xoai.model.oaipmh.OAIPMH.SCHEMA_LOCATION; -import com.lyncode.xoai.model.oaipmh.Verb; -import com.lyncode.xoai.xml.XSISchema; - -import com.lyncode.xoai.xml.XmlWriter; -import static com.lyncode.xoai.xml.XmlWriter.defaultContext; +import io.gdcc.xoai.dataprovider.DataProvider; +import io.gdcc.xoai.dataprovider.repository.Repository; +import io.gdcc.xoai.dataprovider.repository.RepositoryConfiguration; +import io.gdcc.xoai.dataprovider.model.Context; +import io.gdcc.xoai.dataprovider.model.MetadataFormat; +import io.gdcc.xoai.dataprovider.request.RequestBuilder; +import io.gdcc.xoai.dataprovider.request.RequestBuilder.RawRequest; +import io.gdcc.xoai.dataprovider.repository.ItemRepository; +import io.gdcc.xoai.dataprovider.repository.SetRepository; +import io.gdcc.xoai.model.oaipmh.DeletedRecord; +import io.gdcc.xoai.model.oaipmh.OAIPMH; + +import io.gdcc.xoai.xml.XmlWriter; import edu.harvard.iq.dataverse.DatasetServiceBean; import edu.harvard.iq.dataverse.DataverseServiceBean; import edu.harvard.iq.dataverse.export.ExportException; @@ -32,24 +25,20 @@ import edu.harvard.iq.dataverse.export.spi.Exporter; import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean; import edu.harvard.iq.dataverse.harvest.server.OAISetServiceBean; -import edu.harvard.iq.dataverse.harvest.server.xoai.XdataProvider; -import edu.harvard.iq.dataverse.harvest.server.xoai.XgetRecord; -import edu.harvard.iq.dataverse.harvest.server.xoai.XitemRepository; -import edu.harvard.iq.dataverse.harvest.server.xoai.XsetRepository; -import edu.harvard.iq.dataverse.harvest.server.xoai.XlistRecords; +import edu.harvard.iq.dataverse.harvest.server.xoai.DataverseXoaiItemRepository; +import edu.harvard.iq.dataverse.harvest.server.xoai.DataverseXoaiSetRepository; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.MailUtil; import edu.harvard.iq.dataverse.util.SystemConfig; +import io.gdcc.xoai.exceptions.OAIException; import org.apache.commons.lang3.StringUtils; -import java.io.ByteArrayOutputStream; import java.io.IOException; -import java.io.OutputStream; -import java.util.Date; -import java.util.HashMap; import java.util.logging.Logger; import javax.ejb.EJB; +import javax.inject.Inject; +import org.eclipse.microprofile.config.inject.ConfigProperty; import javax.mail.internet.InternetAddress; import javax.servlet.ServletConfig; import javax.servlet.ServletException; @@ -57,12 +46,14 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import javax.xml.stream.XMLStreamException; +import org.eclipse.microprofile.config.Config; +import org.eclipse.microprofile.config.ConfigProvider; /** * * @author Leonid Andreev * Dedicated servlet for handling OAI-PMH requests. - * Uses lyncode XOAI data provider implementation for serving content. + * Uses lyncode/Dspace/gdcc XOAI data provider implementation for serving content. * The servlet itself is somewhat influenced by the older OCLC OAIcat implementation. */ public class OAIServlet extends HttpServlet { @@ -79,23 +70,42 @@ public class OAIServlet extends HttpServlet { @EJB SystemConfig systemConfig; + + @Inject + @ConfigProperty(name = "dataverse.oai.server.maxidentifiers", defaultValue="100") + private Integer maxListIdentifiers; + + @Inject + @ConfigProperty(name = "dataverse.oai.server.maxsets", defaultValue="100") + private Integer maxListSets; + + @Inject + @ConfigProperty(name = "dataverse.oai.server.maxrecords", defaultValue="10") + private Integer maxListRecords; private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.web.servlet.OAIServlet"); - protected HashMap attributesMap = new HashMap(); - private static final String OAI_PMH = "OAI-PMH"; - private static final String RESPONSEDATE_FIELD = "responseDate"; - private static final String REQUEST_FIELD = "request"; + // If we are going to stick with this solution - of providing a minimalist + // xml record containing a link to the proprietary json metadata API for + // "dataverse json harvesting", we'll probably want to create minimalistic, + // but valid schemas for this format as well. + // UPDATE: we are keeping this hack on the server side purely for backward + // compatibility with older (pre v6) Dataverses who may still be using the + // format. Once v6 has been around for a while, we will get rid of it completely. + // Starting this version, harvesting clients will not be making GetRecord + // calls at all when using harvesting dataverse_json; instead they will only + // be calling ListIdentifiers, and then making direct calls to the export + // API of the remote Dataverse, to obtain the records in native json. This + // is how we should have implemented this in the first place, really. private static final String DATAVERSE_EXTENDED_METADATA_FORMAT = "dataverse_json"; - private static final String DATAVERSE_EXTENDED_METADATA_INFO = "Custom Dataverse metadata in JSON format (Dataverse4 to Dataverse4 harvesting only)"; - private static final String DATAVERSE_EXTENDED_METADATA_SCHEMA = "JSON schema pending"; - + private static final String DATAVERSE_EXTENDED_METADATA_NAMESPACE = "Custom Dataverse metadata in JSON format (Dataverse4 to Dataverse4 harvesting only)"; + private static final String DATAVERSE_EXTENDED_METADATA_SCHEMA = "JSON schema pending"; private Context xoaiContext; private SetRepository setRepository; private ItemRepository itemRepository; private RepositoryConfiguration repositoryConfiguration; private Repository xoaiRepository; - private XdataProvider dataProvider; + private DataProvider dataProvider; public void init(ServletConfig config) throws ServletException { super.init(config); @@ -106,18 +116,17 @@ public void init(ServletConfig config) throws ServletException { xoaiContext = addDataverseJsonMetadataFormat(xoaiContext); } - setRepository = new XsetRepository(setService); - itemRepository = new XitemRepository(recordService, datasetService); + setRepository = new DataverseXoaiSetRepository(setService); + itemRepository = new DataverseXoaiItemRepository(recordService, datasetService, systemConfig.getDataverseSiteUrl()); repositoryConfiguration = createRepositoryConfiguration(); - + xoaiRepository = new Repository() .withSetRepository(setRepository) .withItemRepository(itemRepository) - .withResumptionTokenFormatter(new SimpleResumptionTokenFormat()) .withConfiguration(repositoryConfiguration); - dataProvider = new XdataProvider(getXoaiContext(), getXoaiRepository()); + dataProvider = new DataProvider(getXoaiContext(), getXoaiRepository()); } private Context createContext() { @@ -145,6 +154,7 @@ private void addSupportedMetadataFormats(Context context) { metadataFormat = MetadataFormat.metadataFormat(formatName); metadataFormat.withNamespace(exporter.getXMLNameSpace()); metadataFormat.withSchemaLocation(exporter.getXMLSchemaLocation()); + } catch (ExportException ex) { metadataFormat = null; } @@ -153,12 +163,11 @@ private void addSupportedMetadataFormats(Context context) { } } } - //return context; } private Context addDataverseJsonMetadataFormat(Context context) { MetadataFormat metadataFormat = MetadataFormat.metadataFormat(DATAVERSE_EXTENDED_METADATA_FORMAT); - metadataFormat.withNamespace(DATAVERSE_EXTENDED_METADATA_INFO); + metadataFormat.withNamespace(DATAVERSE_EXTENDED_METADATA_NAMESPACE); metadataFormat.withSchemaLocation(DATAVERSE_EXTENDED_METADATA_SCHEMA); context.withMetadataFormat(metadataFormat); return context; @@ -169,26 +178,30 @@ private boolean isDataverseOaiExtensionsSupported() { } private RepositoryConfiguration createRepositoryConfiguration() { - // TODO: - // some of the settings below - such as the max list numbers - - // need to be configurable! + Config config = ConfigProvider.getConfig(); + String repositoryName = config.getOptionalValue("dataverse.oai.server.repositoryname", String.class).orElse(""); - String dataverseName = dataverseService.getRootDataverseName(); - String repositoryName = StringUtils.isEmpty(dataverseName) || "Root".equals(dataverseName) ? "Test Dataverse OAI Archive" : dataverseName + " Dataverse OAI Archive"; - InternetAddress internetAddress = MailUtil.parseSystemAddress(settingsService.getValueForKey(SettingsServiceBean.Key.SystemEmail)); - - RepositoryConfiguration repositoryConfiguration = new RepositoryConfiguration() + if (repositoryName == null || repositoryName.isEmpty()) { + String dataverseName = dataverseService.getRootDataverseName(); + repositoryName = StringUtils.isEmpty(dataverseName) || "Root".equals(dataverseName) ? "Test Dataverse OAI Archive" : dataverseName + " Dataverse OAI Archive"; + } + // The admin email address associated with this installation: + // (Note: if the setting does not exist, we are going to assume that they + // have a reason not to want to advertise their email address, so no + // email will be shown in the output of Identify. + InternetAddress systemEmailAddress = MailUtil.parseSystemAddress(settingsService.getValueForKey(SettingsServiceBean.Key.SystemEmail)); + + RepositoryConfiguration repositoryConfiguration = RepositoryConfiguration.defaults() + .withEnableMetadataAttributes(true) .withRepositoryName(repositoryName) .withBaseUrl(systemConfig.getDataverseSiteUrl()+"/oai") - .withCompression("gzip") // ? - .withCompression("deflate") // ? - .withAdminEmail(internetAddress != null ? internetAddress.getAddress() : null) + .withCompression("gzip") + .withCompression("deflate") + .withAdminEmail(systemEmailAddress != null ? systemEmailAddress.getAddress() : null) .withDeleteMethod(DeletedRecord.TRANSIENT) - .withGranularity(Granularity.Second) - .withMaxListIdentifiers(100) - .withMaxListRecords(100) - .withMaxListSets(100) - .withEarliestDate(new Date()); + .withMaxListIdentifiers(maxListIdentifiers) + .withMaxListRecords(maxListRecords) + .withMaxListSets(maxListSets); return repositoryConfiguration; } @@ -221,9 +234,9 @@ protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } - + - private void processRequest(HttpServletRequest request, HttpServletResponse response) + private void processRequest(HttpServletRequest httpServletRequest, HttpServletResponse response) throws ServletException, IOException { try { @@ -233,150 +246,22 @@ private void processRequest(HttpServletRequest request, HttpServletResponse resp "Sorry. OAI Service is disabled on this Dataverse node."); return; } + + RawRequest rawRequest = RequestBuilder.buildRawRequest(httpServletRequest.getParameterMap()); - OAIRequestParametersBuilder parametersBuilder = newXoaiRequest(); - - for (Object p : request.getParameterMap().keySet()) { - String parameterName = (String)p; - String parameterValue = request.getParameter(parameterName); - parametersBuilder = parametersBuilder.with(parameterName, parameterValue); - - } - - OAIPMH handle = dataProvider.handle(parametersBuilder); + OAIPMH handle = dataProvider.handle(rawRequest); response.setContentType("text/xml;charset=UTF-8"); - - if (isGetRecord(request) && !handle.hasErrors()) { - writeGetRecord(response, handle); - } else if (isListRecords(request) && !handle.hasErrors()) { - writeListRecords(response, handle); - } else { - XmlWriter xmlWriter = new XmlWriter(response.getOutputStream()); + + try (XmlWriter xmlWriter = new XmlWriter(response.getOutputStream(), repositoryConfiguration);) { xmlWriter.write(handle); - xmlWriter.flush(); - xmlWriter.close(); } - } catch (IOException ex) { - logger.warning("IO exception in Get; "+ex.getMessage()); - throw new ServletException ("IO Exception in Get", ex); - } catch (OAIException oex) { - logger.warning("OAI exception in Get; "+oex.getMessage()); - throw new ServletException ("OAI Exception in Get", oex); - } catch (XMLStreamException xse) { - logger.warning("XML Stream exception in Get; "+xse.getMessage()); - throw new ServletException ("XML Stream Exception in Get", xse); - } catch (XmlWriteException xwe) { - logger.warning("XML Write exception in Get; "+xwe.getMessage()); - throw new ServletException ("XML Write Exception in Get", xwe); - } catch (Exception e) { - logger.warning("Unknown exception in Get; "+e.getMessage()); - throw new ServletException ("Unknown servlet exception in Get.", e); + } catch (XMLStreamException | OAIException e) { + throw new ServletException (e); } } - // Custom methods for the potentially expensive GetRecord and ListRecords requests: - - private void writeListRecords(HttpServletResponse response, OAIPMH handle) throws IOException { - OutputStream outputStream = response.getOutputStream(); - - outputStream.write(oaiPmhResponseToString(handle).getBytes()); - - Verb verb = handle.getVerb(); - - if (verb == null) { - throw new IOException("An error or a valid response must be set"); - } - - if (!verb.getType().equals(Verb.Type.ListRecords)) { - throw new IOException("writeListRecords() called on a non-ListRecords verb"); - } - - outputStream.write(("<" + verb.getType().displayName() + ">").getBytes()); - - outputStream.flush(); - - ((XlistRecords) verb).writeToStream(outputStream); - - outputStream.write(("").getBytes()); - outputStream.write(("\n").getBytes()); - - outputStream.flush(); - outputStream.close(); - - } - - private void writeGetRecord(HttpServletResponse response, OAIPMH handle) throws IOException, XmlWriteException, XMLStreamException { - OutputStream outputStream = response.getOutputStream(); - - outputStream.write(oaiPmhResponseToString(handle).getBytes()); - - Verb verb = handle.getVerb(); - - if (verb == null) { - throw new IOException("An error or a valid response must be set"); - } - - if (!verb.getType().equals(Verb.Type.GetRecord)) { - throw new IOException("writeListRecords() called on a non-GetRecord verb"); - } - - outputStream.write(("<" + verb.getType().displayName() + ">").getBytes()); - - outputStream.flush(); - - ((XgetRecord) verb).writeToStream(outputStream); - - outputStream.write(("").getBytes()); - outputStream.write(("\n").getBytes()); - - outputStream.flush(); - outputStream.close(); - - } - - // This function produces the string representation of the top level, - // "service" record of an OAIPMH response (i.e., the header that precedes - // the actual "payload" record, such as , , - // , etc. - - private String oaiPmhResponseToString(OAIPMH handle) { - try { - ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream(); - XmlWriter writer = new XmlWriter(byteOutputStream, defaultContext()); - - writer.writeStartElement(OAI_PMH); - writer.writeDefaultNamespace(NAMESPACE_URI); - writer.writeNamespace(XSISchema.PREFIX, XSISchema.NAMESPACE_URI); - writer.writeAttribute(XSISchema.PREFIX, XSISchema.NAMESPACE_URI, "schemaLocation", - NAMESPACE_URI + " " + SCHEMA_LOCATION); - - writer.writeElement(RESPONSEDATE_FIELD, handle.getResponseDate(), Granularity.Second); - writer.writeElement(REQUEST_FIELD, handle.getRequest()); - writer.writeEndElement(); - writer.flush(); - writer.close(); - - String ret = byteOutputStream.toString().replaceFirst("", ""); - - return ret; - } catch (Exception ex) { - logger.warning("caught exception trying to convert an OAIPMH response header to string: " + ex.getMessage()); - ex.printStackTrace(); - return null; - } - } - - private boolean isGetRecord(HttpServletRequest request) { - return "GetRecord".equals(request.getParameter("verb")); - - } - - private boolean isListRecords(HttpServletRequest request) { - return "ListRecords".equals(request.getParameter("verb")); - } - protected Context getXoaiContext () { return xoaiContext; } @@ -385,11 +270,6 @@ protected Repository getXoaiRepository() { return xoaiRepository; } - protected OAIRequestParametersBuilder newXoaiRequest() { - return new OAIRequestParametersBuilder(); - } - - public boolean isHarvestingServerEnabled() { return systemConfig.isOAIServerEnabled(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xitem.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItem.java similarity index 63% rename from src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xitem.java rename to src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItem.java index bd7a35ddb79..ecfd2f82369 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xitem.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItem.java @@ -1,18 +1,14 @@ -/* - * To change this license header, choose License Headers in Project Properties. - * To change this template file, choose Tools | Templates - * and open the template in the editor. - */ package edu.harvard.iq.dataverse.harvest.server.xoai; -import com.lyncode.xoai.dataprovider.model.Item; -import com.lyncode.xoai.dataprovider.model.Set; -import com.lyncode.xoai.model.oaipmh.About; +import io.gdcc.xoai.dataprovider.model.Item; +import io.gdcc.xoai.dataprovider.model.Set; +import io.gdcc.xoai.model.oaipmh.results.record.Metadata; +import io.gdcc.xoai.model.oaipmh.results.record.About; import edu.harvard.iq.dataverse.Dataset; import edu.harvard.iq.dataverse.harvest.server.OAIRecord; import edu.harvard.iq.dataverse.util.StringUtil; +import java.time.Instant; import java.util.ArrayList; -import java.util.Date; import java.util.List; @@ -20,13 +16,13 @@ * * @author Leonid Andreev * - * This is an implemention of an Lyncode XOAI Item; + * This is an implemention of a Lyncode/DSpace/gdcc XOAI Item. * You can think of it as an XOAI Item wrapper around the * Dataverse OAIRecord entity. */ -public class Xitem implements Item { +public final class DataverseXoaiItem implements Item { - public Xitem(OAIRecord oaiRecord) { + public DataverseXoaiItem(OAIRecord oaiRecord) { super(); this.oaiRecord = oaiRecord; oaisets = new ArrayList<>(); @@ -34,7 +30,7 @@ public Xitem(OAIRecord oaiRecord) { oaisets.add(new Set(oaiRecord.getSetName())); } } - + private OAIRecord oaiRecord; public OAIRecord getOaiRecord() { @@ -51,19 +47,21 @@ public Dataset getDataset() { return dataset; } - public Xitem withDataset(Dataset dataset) { + public DataverseXoaiItem withDataset(Dataset dataset) { this.dataset = dataset; return this; } + private Metadata metadata; + @Override - public List getAbout() { - return null; + public Metadata getMetadata() { + return metadata; } - - @Override - public Xmetadata getMetadata() { - return new Xmetadata((String)null); + + public DataverseXoaiItem withMetadata(Metadata metadata) { + this.metadata = metadata; + return this; } @Override @@ -72,8 +70,8 @@ public String getIdentifier() { } @Override - public Date getDatestamp() { - return oaiRecord.getLastUpdateTime(); + public Instant getDatestamp() { + return oaiRecord.getLastUpdateTime().toInstant(); } private List oaisets; @@ -82,12 +80,10 @@ public Date getDatestamp() { public List getSets() { return oaisets; - } @Override public boolean isDeleted() { return oaiRecord.isRemoved(); } - } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItemRepository.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItemRepository.java new file mode 100644 index 00000000000..faf3cf9ddc4 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiItemRepository.java @@ -0,0 +1,270 @@ +package edu.harvard.iq.dataverse.harvest.server.xoai; + +import io.gdcc.xoai.dataprovider.exceptions.handler.IdDoesNotExistException; +import io.gdcc.xoai.dataprovider.filter.ScopedFilter; +import io.gdcc.xoai.dataprovider.model.Item; +import io.gdcc.xoai.dataprovider.model.ItemIdentifier; +import io.gdcc.xoai.dataprovider.model.Set; +import io.gdcc.xoai.dataprovider.model.MetadataFormat; +import io.gdcc.xoai.dataprovider.repository.ItemRepository; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetServiceBean; +import edu.harvard.iq.dataverse.export.ExportException; +import edu.harvard.iq.dataverse.export.ExportService; +import edu.harvard.iq.dataverse.harvest.server.OAIRecord; +import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean; +import edu.harvard.iq.dataverse.util.StringUtil; +import io.gdcc.xoai.dataprovider.exceptions.handler.HandlerException; +import io.gdcc.xoai.dataprovider.exceptions.handler.NoMetadataFormatsException; +import io.gdcc.xoai.dataprovider.repository.ResultsPage; +import io.gdcc.xoai.model.oaipmh.ResumptionToken; +import io.gdcc.xoai.model.oaipmh.results.record.Metadata; +import io.gdcc.xoai.xml.EchoElement; +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.time.Instant; +import java.util.List; +import java.util.logging.Logger; + +/** + * + * @author Leonid Andreev + * Implements an XOAI "Item Repository". Retrieves Dataverse "OAIRecords" + * representing harvestable local datasets and translates them into + * XOAI "items". + */ + +public class DataverseXoaiItemRepository implements ItemRepository { + private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.xoai.DataverseXoaiItemRepository"); + + private final OAIRecordServiceBean recordService; + private final DatasetServiceBean datasetService; + private final String serverUrl; + + public DataverseXoaiItemRepository (OAIRecordServiceBean recordService, DatasetServiceBean datasetService, String serverUrl) { + this.recordService = recordService; + this.datasetService = datasetService; + this.serverUrl = serverUrl; + } + + @Override + public ItemIdentifier getItem(String identifier) throws IdDoesNotExistException { + // This method is called when ListMetadataFormats request specifies + // the identifier, requesting the formats available for this specific record. + // In our case, under the current implementation, we need to simply look + // up if the record exists; if it does, all the OAI formats that we serve + // should be available for this record. + + List oaiRecords = recordService.findOaiRecordsByGlobalId(identifier); + if (oaiRecords != null && !oaiRecords.isEmpty()) { + for (OAIRecord oaiRecord : oaiRecords) { + // We can return the first *active* record we find for this identifier. + if (!oaiRecord.isRemoved()) { + return new DataverseXoaiItem(oaiRecord); + } + } + } + + throw new IdDoesNotExistException(); + } + + @Override + public Item getItem(String identifier, MetadataFormat metadataFormat) throws HandlerException { + logger.fine("getItem; calling findOaiRecordsByGlobalId, identifier " + identifier); + + if (metadataFormat == null) { + throw new NoMetadataFormatsException("Metadata Format is Required"); + } + + List oaiRecords = recordService.findOaiRecordsByGlobalId(identifier); + if (oaiRecords != null && !oaiRecords.isEmpty()) { + DataverseXoaiItem xoaiItem = null; + for (OAIRecord oaiRecord : oaiRecords) { + if (xoaiItem == null) { + xoaiItem = new DataverseXoaiItem(oaiRecord); + xoaiItem = addMetadata(xoaiItem, metadataFormat); + } else { + // Adding extra set specs to the XOAI Item, if this oaiRecord + // is part of multiple sets: + if (!StringUtil.isEmpty(oaiRecord.getSetName())) { + xoaiItem.getSets().add(new Set(oaiRecord.getSetName())); + } + } + } + if (xoaiItem != null) { + return xoaiItem; + } + } + + throw new IdDoesNotExistException(); + } + + @Override + public ResultsPage getItemIdentifiers(List filters, MetadataFormat metadataFormat, int maxResponseLength, ResumptionToken.Value resumptionToken) throws HandlerException { + + return (ResultsPage)getRepositoryRecords(metadataFormat, maxResponseLength, resumptionToken, false); + + } + + @Override + public ResultsPage getItems(List filters, MetadataFormat metadataFormat, int maxResponseLength, ResumptionToken.Value resumptionToken) throws HandlerException { + + return (ResultsPage)getRepositoryRecords(metadataFormat, maxResponseLength, resumptionToken, true); + } + + private ResultsPage getRepositoryRecords ( + MetadataFormat metadataFormat, + int maxResponseLength, + ResumptionToken.Value resumptionToken, + boolean fullItems) throws HandlerException { + + int offset = Long.valueOf(resumptionToken.getOffset()).intValue(); + String setSpec = resumptionToken.getSetSpec(); + Instant from = resumptionToken.getFrom(); + Instant until = resumptionToken.getUntil(); + + boolean hasMore = false; + + logger.fine("calling " + (fullItems ? "getItems" : "getItemIdentifiers") + + "; offset=" + offset + + ", length=" + maxResponseLength + + ", setSpec=" + setSpec + + ", from=" + from + + ", until=" + until); + + List oaiRecords = recordService.findOaiRecordsBySetName(setSpec, from, until); + + List xoaiItems = new ArrayList<>(); + + if (oaiRecords != null && !oaiRecords.isEmpty()) { + logger.fine("total " + oaiRecords.size() + " records returned"); + + for (int i = offset; i < offset + maxResponseLength && i < oaiRecords.size(); i++) { + OAIRecord record = oaiRecords.get(i); + DataverseXoaiItem xoaiItem = new DataverseXoaiItem(record); + + if (fullItems) { + // If we are cooking "full" Items (for the ListRecords verb), + // add the metadata to the item object (if not a deleted + // record, if available, etc.): + xoaiItem = addMetadata(xoaiItem, metadataFormat); + } + + xoaiItems.add(xoaiItem); + } + + // Run a second pass, looking for records in this set that occur + // in *other* sets. Then we'll add these multiple sets to the + // formatted output in the header: + addExtraSets(xoaiItems, setSpec, from, until); + + hasMore = offset + maxResponseLength < oaiRecords.size(); + + ResultsPage result = new ResultsPage(resumptionToken, hasMore, xoaiItems, oaiRecords.size()); + logger.fine("returning result with " + xoaiItems.size() + " items."); + return result; + } + + return new ResultsPage(resumptionToken, false, xoaiItems, 0); + } + + private void addExtraSets(Object xoaiItemsList, String setSpec, Instant from, Instant until) { + + List xoaiItems = (List)xoaiItemsList; + + List oaiRecords = recordService.findOaiRecordsNotInThisSet(setSpec, from, until); + + if (oaiRecords == null || oaiRecords.isEmpty()) { + return; + } + + // Make a second pass through the list of xoaiItems already found for this set, + // and add any other sets in which this item occurs: + + int j = 0; + for (int i = 0; i < xoaiItems.size(); i++) { + // fast-forward the second list, until we find a oaiRecord with this identifier, + // or until we are past this oaiRecord (both lists are sorted alphabetically by + // the identifier: + DataverseXoaiItem xitem = xoaiItems.get(i); + + while (j < oaiRecords.size() && xitem.getIdentifier().compareTo(oaiRecords.get(j).getGlobalId()) > 0) { + j++; + } + + while (j < oaiRecords.size() && xitem.getIdentifier().equals(oaiRecords.get(j).getGlobalId())) { + xoaiItems.get(i).getSets().add(new Set(oaiRecords.get(j).getSetName())); + j++; + } + } + } + + private DataverseXoaiItem addMetadata(DataverseXoaiItem xoaiItem, MetadataFormat metadataFormat) { + // This may be a "deleted" record - i.e., a oaiRecord kept in + // the OAI set for a dataset that's no longer in this Dataverse. + // (it serves to tell the remote client to delete it from their + // holdings too). + // If this is the case here, there's nothing we need to do for this item. + // If not, if it's a live record, let's try to look up the dataset and + // open the pre-generated metadata stream. + + if (!xoaiItem.isDeleted()) { + Dataset dataset = datasetService.findByGlobalId(xoaiItem.getIdentifier()); + if (dataset != null) { + try { + Metadata metadata = getDatasetMetadata(dataset, metadataFormat.getPrefix()); + xoaiItem.withDataset(dataset).withMetadata(metadata); + } catch (ExportException | IOException ex) { + // This is not supposed to happen in normal operations; + // since by design only the datasets for which the metadata + // records have been pre-generated ("exported") should be + // served as "OAI Record". But, things happen. If for one + // reason or another that cached metadata file is no longer there, + // we are not going to serve any metadata for this oaiRecord, + // BUT we are going to include it marked as "deleted" + // (because skipping it could potentially mess up the + // counts and offsets, in a resumption token scenario. + xoaiItem.getOaiRecord().setRemoved(true); + } + } else { + // If dataset (somehow) no longer exists (again, this is + // not supposed to happen), we will serve the oaiRecord, + // marked as "deleted" and without any metadata. + // We can't just skip it, because that could mess up the + // counts and offsets, in a resumption token scenario. + xoaiItem.getOaiRecord().setRemoved(true); + } + } + return xoaiItem; + } + + private Metadata getDatasetMetadata(Dataset dataset, String metadataPrefix) throws ExportException, IOException { + Metadata metadata; + + if ("dataverse_json".equals(metadataPrefix)) { + // Solely for backward compatibility, for older Dataverse harvesting clients + // that may still be relying on harvesting "dataverse_json"; + // we will want to eventually get rid of this hack! + // @Deprecated(since = "5.0") + metadata = new Metadata( + new EchoElement("custom metadata")) + .withAttribute("directApiCall", customDataverseJsonApiUri(dataset.getGlobalId().asString())); + + } else { + InputStream pregeneratedMetadataStream; + pregeneratedMetadataStream = ExportService.getInstance().getExport(dataset, metadataPrefix); + + metadata = Metadata.copyFromStream(pregeneratedMetadataStream); + } + return metadata; + } + + private String customDataverseJsonApiUri(String identifier) { + String ret = serverUrl + + "/api/datasets/export?exporter=dataverse_json&persistentId=" + + identifier; + + return ret; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XsetRepository.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiSetRepository.java similarity index 56% rename from src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XsetRepository.java rename to src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiSetRepository.java index 8e58e1bbf9a..b4e275b6059 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XsetRepository.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/DataverseXoaiSetRepository.java @@ -1,15 +1,9 @@ -/* - * To change this license header, choose License Headers in Project Properties. - * To change this template file, choose Tools | Templates - * and open the template in the editor. - */ package edu.harvard.iq.dataverse.harvest.server.xoai; -import com.lyncode.xoai.model.xoai.Element; -import com.lyncode.xoai.dataprovider.repository.SetRepository; -import com.lyncode.xoai.dataprovider.handlers.results.ListSetsResult; -import com.lyncode.xoai.dataprovider.model.Set; -import com.lyncode.xoai.model.xoai.XOAIMetadata; +import io.gdcc.xoai.model.xoai.Element; +import io.gdcc.xoai.dataprovider.repository.SetRepository; +import io.gdcc.xoai.dataprovider.model.Set; +import io.gdcc.xoai.model.xoai.XOAIMetadata; import edu.harvard.iq.dataverse.harvest.server.OAISet; import edu.harvard.iq.dataverse.harvest.server.OAISetServiceBean; @@ -21,13 +15,12 @@ * * @author Leonid Andreev */ -public class XsetRepository implements SetRepository { - private static Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.xoai.XsetRepository"); +public class DataverseXoaiSetRepository implements SetRepository { + private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.xoai.DataverseXoaiSetRepository"); private OAISetServiceBean setService; - public XsetRepository (OAISetServiceBean setService) { - super(); + public DataverseXoaiSetRepository (OAISetServiceBean setService) { this.setService = setService; } @@ -42,7 +35,6 @@ public void setSetService(OAISetServiceBean setService) { @Override public boolean supportSets() { - logger.fine("calling supportSets()"); List dataverseOAISets = setService.findAllNamedSets(); if (dataverseOAISets == null || dataverseOAISets.isEmpty()) { @@ -52,7 +44,7 @@ public boolean supportSets() { } @Override - public ListSetsResult retrieveSets(int offset, int length) { + public List getSets() { logger.fine("calling retrieveSets()"); List dataverseOAISets = setService.findAllNamedSets(); List XOAISets = new ArrayList(); @@ -62,25 +54,21 @@ public ListSetsResult retrieveSets(int offset, int length) { OAISet dataverseSet = dataverseOAISets.get(i); Set xoaiSet = new Set(dataverseSet.getSpec()); xoaiSet.withName(dataverseSet.getName()); - XOAIMetadata xMetadata = new XOAIMetadata(); + XOAIMetadata xoaiMetadata = new XOAIMetadata(); Element element = new Element("description"); element.withField("description", dataverseSet.getDescription()); - xMetadata.getElements().add(element); - xoaiSet.withDescription(xMetadata); + xoaiMetadata.getElements().add(element); + xoaiSet.withDescription(xoaiMetadata); XOAISets.add(xoaiSet); } } - return new ListSetsResult(offset + length < XOAISets.size(), XOAISets.subList(offset, Math.min(offset + length, XOAISets.size()))); + return XOAISets; } @Override public boolean exists(String setSpec) { - //for (Set s : this.sets) - // if (s.getSpec().equals(setSpec)) - // return true; - - return false; + return setService.setExists(setSpec); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XdataProvider.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XdataProvider.java deleted file mode 100644 index 8ba8fe96bec..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XdataProvider.java +++ /dev/null @@ -1,116 +0,0 @@ -package edu.harvard.iq.dataverse.harvest.server.xoai; - - -import com.lyncode.builder.Builder; -import com.lyncode.xoai.dataprovider.exceptions.*; -import com.lyncode.xoai.dataprovider.handlers.*; -import com.lyncode.xoai.exceptions.InvalidResumptionTokenException; -import com.lyncode.xoai.dataprovider.model.Context; -import com.lyncode.xoai.model.oaipmh.Identify; -import com.lyncode.xoai.model.oaipmh.OAIPMH; -import com.lyncode.xoai.model.oaipmh.Request; -import com.lyncode.xoai.dataprovider.parameters.OAICompiledRequest; -import com.lyncode.xoai.dataprovider.parameters.OAIRequest; -import com.lyncode.xoai.dataprovider.repository.Repository; -import com.lyncode.xoai.services.api.DateProvider; -import com.lyncode.xoai.services.impl.UTCDateProvider; -import static com.lyncode.xoai.dataprovider.parameters.OAIRequest.Parameter.*; - -import java.util.logging.Logger; - -/** - * - * @author Leonid Andreev - */ -public class XdataProvider { - private static Logger log = Logger.getLogger(XdataProvider.class.getCanonicalName()); - - public static XdataProvider dataProvider (Context context, Repository repository) { - return new XdataProvider(context, repository); - } - - private Repository repository; - private DateProvider dateProvider; - - private final IdentifyHandler identifyHandler; - private final XgetRecordHandler getRecordHandler; - private final ListSetsHandler listSetsHandler; - private final XlistRecordsHandler listRecordsHandler; - private final ListIdentifiersHandler listIdentifiersHandler; - private final ListMetadataFormatsHandler listMetadataFormatsHandler; - private final ErrorHandler errorsHandler; - - public XdataProvider (Context context, Repository repository) { - this.repository = repository; - this.dateProvider = new UTCDateProvider(); - - this.identifyHandler = new IdentifyHandler(context, repository); - this.listSetsHandler = new ListSetsHandler(context, repository); - this.listMetadataFormatsHandler = new ListMetadataFormatsHandler(context, repository); - this.listRecordsHandler = new XlistRecordsHandler(context, repository); - this.listIdentifiersHandler = new ListIdentifiersHandler(context, repository); - //this.getRecordHandler = new GetRecordHandler(context, repository); - this.getRecordHandler = new XgetRecordHandler(context, repository); - this.errorsHandler = new ErrorHandler(); - } - - public OAIPMH handle (Builder builder) throws OAIException { - return handle(builder.build()); - } - - public OAIPMH handle (OAIRequest requestParameters) throws OAIException { - log.fine("Handling OAI request"); - Request request = new Request(repository.getConfiguration().getBaseUrl()) - .withVerbType(requestParameters.get(Verb)) - .withResumptionToken(requestParameters.get(ResumptionToken)) - .withIdentifier(requestParameters.get(Identifier)) - .withMetadataPrefix(requestParameters.get(MetadataPrefix)) - .withSet(requestParameters.get(Set)) - .withFrom(requestParameters.get(From)) - .withUntil(requestParameters.get(Until)); - - OAIPMH response = new OAIPMH() - .withRequest(request) - .withResponseDate(dateProvider.now()); - try { - OAICompiledRequest parameters = compileParameters(requestParameters); - - switch (request.getVerbType()) { - case Identify: - Identify identify = identifyHandler.handle(parameters); - identify.getDescriptions().clear(); // We don't want to use the default description - response.withVerb(identify); - break; - case ListSets: - response.withVerb(listSetsHandler.handle(parameters)); - break; - case ListMetadataFormats: - response.withVerb(listMetadataFormatsHandler.handle(parameters)); - break; - case GetRecord: - response.withVerb(getRecordHandler.handle(parameters)); - break; - case ListIdentifiers: - response.withVerb(listIdentifiersHandler.handle(parameters)); - break; - case ListRecords: - response.withVerb(listRecordsHandler.handle(parameters)); - break; - } - } catch (HandlerException e) { - log.fine("HandlerException when executing "+request.getVerbType()+": " + e.getMessage()); - response.withError(errorsHandler.handle(e)); - } - - return response; - } - - private OAICompiledRequest compileParameters(OAIRequest requestParameters) throws IllegalVerbException, UnknownParameterException, BadArgumentException, DuplicateDefinitionException, BadResumptionToken { - try { - return requestParameters.compile(); - } catch (InvalidResumptionTokenException e) { - throw new BadResumptionToken("The resumption token is invalid"); - } - } - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecord.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecord.java deleted file mode 100644 index d86f555d105..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecord.java +++ /dev/null @@ -1,52 +0,0 @@ -/* - * To change this license header, choose License Headers in Project Properties. - * To change this template file, choose Tools | Templates - * and open the template in the editor. - */ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xoai.model.oaipmh.GetRecord; -import com.lyncode.xoai.model.oaipmh.Record; -import java.io.IOException; -import java.io.OutputStream; - -/** - * - * @author Leonid Andreev - * - * This is the Dataverse extension of XOAI GetRecord, - * optimized to stream individual records to the output directly - */ - -public class XgetRecord extends GetRecord { - private static final String RECORD_FIELD = "record"; - private static final String RECORD_START_ELEMENT = "<"+RECORD_FIELD+">"; - private static final String RECORD_CLOSE_ELEMENT = ""; - private static final String RESUMPTION_TOKEN_FIELD = "resumptionToken"; - private static final String EXPIRATION_DATE_ATTRIBUTE = "expirationDate"; - private static final String COMPLETE_LIST_SIZE_ATTRIBUTE = "completeListSize"; - private static final String CURSOR_ATTRIBUTE = "cursor"; - - - public XgetRecord(Xrecord record) { - super(record); - } - - public void writeToStream(OutputStream outputStream) throws IOException { - - if (this.getRecord() == null) { - throw new IOException("XgetRecord: null Record"); - } - Xrecord xrecord = (Xrecord) this.getRecord(); - - outputStream.write(RECORD_START_ELEMENT.getBytes()); - outputStream.flush(); - - xrecord.writeToStream(outputStream); - - outputStream.write(RECORD_CLOSE_ELEMENT.getBytes()); - outputStream.flush(); - - } - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecordHandler.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecordHandler.java deleted file mode 100644 index ba28894482a..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XgetRecordHandler.java +++ /dev/null @@ -1,92 +0,0 @@ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xml.exceptions.XmlWriteException; -import com.lyncode.xoai.dataprovider.exceptions.BadArgumentException; -import com.lyncode.xoai.dataprovider.exceptions.CannotDisseminateFormatException; -import com.lyncode.xoai.dataprovider.parameters.OAICompiledRequest; -import com.lyncode.xoai.dataprovider.exceptions.CannotDisseminateRecordException; -import com.lyncode.xoai.dataprovider.exceptions.HandlerException; -import com.lyncode.xoai.dataprovider.exceptions.IdDoesNotExistException; -import com.lyncode.xoai.dataprovider.exceptions.NoMetadataFormatsException; -import com.lyncode.xoai.dataprovider.exceptions.OAIException; -import com.lyncode.xoai.dataprovider.handlers.VerbHandler; -import com.lyncode.xoai.dataprovider.handlers.helpers.ItemHelper; -import com.lyncode.xoai.dataprovider.model.Context; -import com.lyncode.xoai.dataprovider.model.Item; -import com.lyncode.xoai.dataprovider.model.MetadataFormat; -import com.lyncode.xoai.dataprovider.model.Set; -import com.lyncode.xoai.model.oaipmh.*; -import com.lyncode.xoai.dataprovider.repository.Repository; -import com.lyncode.xoai.xml.XSLPipeline; -import com.lyncode.xoai.xml.XmlWriter; -import edu.harvard.iq.dataverse.Dataset; - -import javax.xml.stream.XMLStreamException; -import javax.xml.transform.TransformerException; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.util.logging.Logger; - -/* - * @author Leonid Andreev -*/ -public class XgetRecordHandler extends VerbHandler { - private static Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.xoai.XgetRecordHandler"); - public XgetRecordHandler(Context context, Repository repository) { - super(context, repository); - } - - @Override - public GetRecord handle(OAICompiledRequest parameters) throws OAIException, HandlerException { - - MetadataFormat format = getContext().formatForPrefix(parameters.getMetadataPrefix()); - Item item = getRepository().getItemRepository().getItem(parameters.getIdentifier()); - - if (getContext().hasCondition() && - !getContext().getCondition().getFilter(getRepository().getFilterResolver()).isItemShown(item)) - throw new IdDoesNotExistException("This context does not include this item"); - - if (format.hasCondition() && - !format.getCondition().getFilter(getRepository().getFilterResolver()).isItemShown(item)) - throw new CannotDisseminateRecordException("Format not applicable to this item"); - - - Xrecord record = this.createRecord(parameters, item); - GetRecord result = new XgetRecord(record); - - return result; - } - - private Xrecord createRecord(OAICompiledRequest parameters, Item item) - throws BadArgumentException, CannotDisseminateRecordException, - OAIException, NoMetadataFormatsException, CannotDisseminateFormatException { - MetadataFormat format = getContext().formatForPrefix(parameters.getMetadataPrefix()); - Header header = new Header(); - - Dataset dataset = ((Xitem)item).getDataset(); - Xrecord xrecord = new Xrecord().withFormatName(parameters.getMetadataPrefix()).withDataset(dataset); - header.withIdentifier(item.getIdentifier()); - - ItemHelper itemHelperWrap = new ItemHelper(item); - header.withDatestamp(item.getDatestamp()); - for (Set set : itemHelperWrap.getSets(getContext(), getRepository().getFilterResolver())) - header.withSetSpec(set.getSpec()); - if (item.isDeleted()) - header.withStatus(Header.Status.DELETED); - - xrecord.withHeader(header); - xrecord.withMetadata(item.getMetadata()); - - return xrecord; - } - - private XSLPipeline toPipeline(Item item) throws XmlWriteException, XMLStreamException { - ByteArrayOutputStream output = new ByteArrayOutputStream(); - XmlWriter writer = new XmlWriter(output); - Metadata metadata = item.getMetadata(); - metadata.write(writer); - writer.close(); - return new XSLPipeline(new ByteArrayInputStream(output.toByteArray()), true); - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XitemRepository.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XitemRepository.java deleted file mode 100644 index b4c60a3171d..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XitemRepository.java +++ /dev/null @@ -1,248 +0,0 @@ -/* - * To change this license header, choose License Headers in Project Properties. - * To change this template file, choose Tools | Templates - * and open the template in the editor. - */ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xoai.dataprovider.exceptions.IdDoesNotExistException; -import com.lyncode.xoai.dataprovider.exceptions.OAIException; -import com.lyncode.xoai.dataprovider.filter.ScopedFilter; -import com.lyncode.xoai.dataprovider.handlers.results.ListItemIdentifiersResult; -import com.lyncode.xoai.dataprovider.handlers.results.ListItemsResults; -import com.lyncode.xoai.dataprovider.model.Item; -import com.lyncode.xoai.dataprovider.model.ItemIdentifier; -import com.lyncode.xoai.dataprovider.model.Set; -import com.lyncode.xoai.dataprovider.repository.ItemRepository; -import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.DatasetServiceBean; -import edu.harvard.iq.dataverse.harvest.server.OAIRecord; -import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean; -import edu.harvard.iq.dataverse.util.StringUtil; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.logging.Logger; - -/** - * - * @author Leonid Andreev - * Implements an XOAI "Item Repository". Retrieves Dataverse "OAIRecords" - * representing harvestable local datasets and translates them into - * XOAI "items". - */ - -public class XitemRepository implements ItemRepository { - private static Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.server.xoai.XitemRepository"); - - private OAIRecordServiceBean recordService; - private DatasetServiceBean datasetService; - - public XitemRepository (OAIRecordServiceBean recordService, DatasetServiceBean datasetService) { - super(); - this.recordService = recordService; - this.datasetService = datasetService; - } - - private List list = new ArrayList(); - - - @Override - public Item getItem(String identifier) throws IdDoesNotExistException, OAIException { - logger.fine("getItem; calling findOaiRecordsByGlobalId, identifier " + identifier); - List oaiRecords = recordService.findOaiRecordsByGlobalId(identifier); - if (oaiRecords != null && !oaiRecords.isEmpty()) { - Xitem xoaiItem = null; - for (OAIRecord record : oaiRecords) { - if (xoaiItem == null) { - Dataset dataset = datasetService.findByGlobalId(record.getGlobalId()); - if (dataset != null) { - xoaiItem = new Xitem(record).withDataset(dataset); - } - } else { - // Adding extra set specs to the XOAI Item, if this record - // is part of multiple sets: - if (!StringUtil.isEmpty(record.getSetName())) { - xoaiItem.getSets().add(new Set(record.getSetName())); - } - } - } - if (xoaiItem != null) { - return xoaiItem; - } - } - - throw new IdDoesNotExistException(); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length) throws OAIException { - return getItemIdentifiers(filters, offset, length, null, null, null); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length, Date from) throws OAIException { - return getItemIdentifiers(filters, offset, length, null, from, null); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiersUntil(List filters, int offset, int length, Date until) throws OAIException { - return getItemIdentifiers(filters, offset, length, null, null, until); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length, Date from, Date until) throws OAIException { - return getItemIdentifiers(filters, offset, length, null, from, until); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length, String setSpec) throws OAIException { - return getItemIdentifiers(filters, offset, length, setSpec, null, null); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length, String setSpec, Date from) throws OAIException { - return getItemIdentifiers(filters, offset, length, setSpec, from, null); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiersUntil(List filters, int offset, int length, String setSpec, Date until) throws OAIException { - return getItemIdentifiers(filters, offset, length, setSpec, null, until); - } - - @Override - public ListItemIdentifiersResult getItemIdentifiers(List filters, int offset, int length, String setSpec, Date from, Date until) throws OAIException { - logger.fine("calling getItemIdentifiers; offset=" + offset - + ", length=" + length - + ", setSpec=" + setSpec - + ", from=" + from - + ", until=" + until); - - List oaiRecords = recordService.findOaiRecordsBySetName(setSpec, from, until); - - logger.fine("total " + oaiRecords.size() + " returned"); - - List xoaiItems = new ArrayList<>(); - if (oaiRecords != null && !oaiRecords.isEmpty()) { - - for (int i = offset; i < offset + length && i < oaiRecords.size(); i++) { - OAIRecord record = oaiRecords.get(i); - xoaiItems.add(new Xitem(record)); - } - - // Run a second pass, looking for records in this set that occur - // in *other* sets. Then we'll add these multiple sets to the - // formatted output in the header: - addExtraSets(xoaiItems, setSpec, from, until); - - boolean hasMore = offset + length < oaiRecords.size(); - ListItemIdentifiersResult result = new ListItemIdentifiersResult(hasMore, xoaiItems); - logger.fine("returning result with " + xoaiItems.size() + " items."); - return result; - } - - return new ListItemIdentifiersResult(false, xoaiItems); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length) throws OAIException { - return getItems(filters, offset, length, null, null, null); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length, Date from) throws OAIException { - return getItems(filters, offset, length, null, from, null); - } - - @Override - public ListItemsResults getItemsUntil(List filters, int offset, int length, Date until) throws OAIException { - return getItems(filters, offset, length, null, null, until); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length, Date from, Date until) throws OAIException { - return getItems(filters, offset, length, null, from, until); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length, String setSpec) throws OAIException { - return getItems(filters, offset, length, setSpec, null, null); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length, String setSpec, Date from) throws OAIException { - return getItems(filters, offset, length, setSpec, from, null); - } - - @Override - public ListItemsResults getItemsUntil(List filters, int offset, int length, String setSpec, Date until) throws OAIException { - return getItems(filters, offset, length, setSpec, null, until); - } - - @Override - public ListItemsResults getItems(List filters, int offset, int length, String setSpec, Date from, Date until) throws OAIException { - logger.fine("calling getItems; offset=" + offset - + ", length=" + length - + ", setSpec=" + setSpec - + ", from=" + from - + ", until=" + until); - - List oaiRecords = recordService.findOaiRecordsBySetName(setSpec, from, until); - - logger.fine("total " + oaiRecords.size() + " returned"); - - List xoaiItems = new ArrayList<>(); - if (oaiRecords != null && !oaiRecords.isEmpty()) { - - for (int i = offset; i < offset + length && i < oaiRecords.size(); i++) { - OAIRecord oaiRecord = oaiRecords.get(i); - Dataset dataset = datasetService.findByGlobalId(oaiRecord.getGlobalId()); - if (dataset != null) { - Xitem xItem = new Xitem(oaiRecord).withDataset(dataset); - xoaiItems.add(xItem); - } - } - - addExtraSets(xoaiItems, setSpec, from, until); - - boolean hasMore = offset + length < oaiRecords.size(); - ListItemsResults result = new ListItemsResults(hasMore, xoaiItems); - logger.fine("returning result with " + xoaiItems.size() + " items."); - return result; - } - - return new ListItemsResults(false, xoaiItems); - } - - private void addExtraSets(Object xoaiItemsList, String setSpec, Date from, Date until) { - - List xoaiItems = (List)xoaiItemsList; - - List oaiRecords = recordService.findOaiRecordsNotInThisSet(setSpec, from, until); - - if (oaiRecords == null || oaiRecords.isEmpty()) { - return; - } - - // Make a second pass through the list of xoaiItems already found for this set, - // and add any other sets in which this item occurs: - - int j = 0; - for (int i = 0; i < xoaiItems.size(); i++) { - // fast-forward the second list, until we find a record with this identifier, - // or until we are past this record (both lists are sorted alphabetically by - // the identifier: - Xitem xitem = xoaiItems.get(i); - - while (j < oaiRecords.size() && xitem.getIdentifier().compareTo(oaiRecords.get(j).getGlobalId()) > 0) { - j++; - } - - while (j < oaiRecords.size() && xitem.getIdentifier().equals(oaiRecords.get(j).getGlobalId())) { - xoaiItems.get(i).getSets().add(new Set(oaiRecords.get(j).getSetName())); - j++; - } - } - - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecords.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecords.java deleted file mode 100644 index 15bd005cacf..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecords.java +++ /dev/null @@ -1,86 +0,0 @@ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xml.exceptions.XmlWriteException; -import static com.lyncode.xoai.model.oaipmh.Granularity.Second; -import com.lyncode.xoai.model.oaipmh.ListRecords; -import com.lyncode.xoai.model.oaipmh.Record; -import com.lyncode.xoai.model.oaipmh.ResumptionToken; -import com.lyncode.xoai.xml.XmlWriter; -import static com.lyncode.xoai.xml.XmlWriter.defaultContext; -import java.io.ByteArrayOutputStream; - -import java.io.IOException; -import java.io.OutputStream; -import javax.xml.stream.XMLStreamException; - -/** - * - * @author Leonid Andreev - * - * This is the Dataverse extension of XOAI ListRecords, - * optimized to stream individual records using fast dumping - * of pre-exported metadata fragments (and by-passing expensive - * XML parsing and writing). - */ -public class XlistRecords extends ListRecords { - private static final String RECORD_FIELD = "record"; - private static final String RECORD_START_ELEMENT = "<"+RECORD_FIELD+">"; - private static final String RECORD_CLOSE_ELEMENT = ""; - private static final String RESUMPTION_TOKEN_FIELD = "resumptionToken"; - private static final String EXPIRATION_DATE_ATTRIBUTE = "expirationDate"; - private static final String COMPLETE_LIST_SIZE_ATTRIBUTE = "completeListSize"; - private static final String CURSOR_ATTRIBUTE = "cursor"; - - public void writeToStream(OutputStream outputStream) throws IOException { - if (!this.records.isEmpty()) { - for (Record record : this.records) { - outputStream.write(RECORD_START_ELEMENT.getBytes()); - outputStream.flush(); - - ((Xrecord)record).writeToStream(outputStream); - - outputStream.write(RECORD_CLOSE_ELEMENT.getBytes()); - outputStream.flush(); - } - } - - if (resumptionToken != null) { - - String resumptionTokenString = resumptionTokenToString(resumptionToken); - if (resumptionTokenString == null) { - throw new IOException("XlistRecords: failed to output resumption token"); - } - outputStream.write(resumptionTokenString.getBytes()); - outputStream.flush(); - } - } - - private String resumptionTokenToString(ResumptionToken token) { - try { - ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream(); - XmlWriter writer = new XmlWriter(byteOutputStream, defaultContext()); - - writer.writeStartElement(RESUMPTION_TOKEN_FIELD); - - if (token.getExpirationDate() != null) - writer.writeAttribute(EXPIRATION_DATE_ATTRIBUTE, token.getExpirationDate(), Second); - if (token.getCompleteListSize() != null) - writer.writeAttribute(COMPLETE_LIST_SIZE_ATTRIBUTE, "" + token.getCompleteListSize()); - if (token.getCursor() != null) - writer.writeAttribute(CURSOR_ATTRIBUTE, "" + token.getCursor()); - if (token.getValue() != null) - writer.write(token.getValue()); - - writer.writeEndElement(); // resumptionToken; - writer.flush(); - writer.close(); - - String ret = byteOutputStream.toString(); - - return ret; - } catch (XMLStreamException | XmlWriteException e) { - return null; - } - } - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecordsHandler.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecordsHandler.java deleted file mode 100644 index 8fe13bc4044..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XlistRecordsHandler.java +++ /dev/null @@ -1,168 +0,0 @@ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xml.exceptions.XmlWriteException; -import com.lyncode.xoai.dataprovider.exceptions.BadArgumentException; -import com.lyncode.xoai.dataprovider.exceptions.CannotDisseminateFormatException; -import com.lyncode.xoai.dataprovider.exceptions.CannotDisseminateRecordException; -import com.lyncode.xoai.dataprovider.exceptions.DoesNotSupportSetsException; -import com.lyncode.xoai.dataprovider.exceptions.HandlerException; -import com.lyncode.xoai.dataprovider.exceptions.NoMatchesException; -import com.lyncode.xoai.dataprovider.exceptions.NoMetadataFormatsException; -import com.lyncode.xoai.dataprovider.exceptions.OAIException; -import com.lyncode.xoai.dataprovider.handlers.VerbHandler; -import com.lyncode.xoai.dataprovider.handlers.results.ListItemsResults; -import com.lyncode.xoai.dataprovider.handlers.helpers.ItemHelper; -import com.lyncode.xoai.dataprovider.handlers.helpers.ItemRepositoryHelper; -import com.lyncode.xoai.dataprovider.handlers.helpers.SetRepositoryHelper; -import com.lyncode.xoai.dataprovider.model.Context; -import com.lyncode.xoai.dataprovider.model.Item; -import com.lyncode.xoai.dataprovider.model.MetadataFormat; -import com.lyncode.xoai.dataprovider.model.Set; -import com.lyncode.xoai.dataprovider.parameters.OAICompiledRequest; -import com.lyncode.xoai.dataprovider.repository.Repository; -import com.lyncode.xoai.model.oaipmh.Header; -import com.lyncode.xoai.model.oaipmh.ListRecords; -import com.lyncode.xoai.model.oaipmh.Metadata; -import com.lyncode.xoai.model.oaipmh.Record; -import com.lyncode.xoai.model.oaipmh.ResumptionToken; -import com.lyncode.xoai.xml.XSLPipeline; -import com.lyncode.xoai.xml.XmlWriter; -import edu.harvard.iq.dataverse.Dataset; - -import javax.xml.stream.XMLStreamException; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.util.List; - -/** - * - * @author Leonid Andreev - * - * This is Dataverse's own implementation of ListRecords Verb Handler - * (used instead of the ListRecordsHandler provided by XOAI). - * It is customized to support the optimizations that allows - * Dataverse to directly output pre-exported metadata records to the output - * stream, bypassing expensive XML parsing and writing. - */ -public class XlistRecordsHandler extends VerbHandler { - private static java.util.logging.Logger logger = java.util.logging.Logger.getLogger("XlistRecordsHandler"); - private final ItemRepositoryHelper itemRepositoryHelper; - private final SetRepositoryHelper setRepositoryHelper; - - public XlistRecordsHandler(Context context, Repository repository) { - super(context, repository); - this.itemRepositoryHelper = new ItemRepositoryHelper(getRepository().getItemRepository()); - this.setRepositoryHelper = new SetRepositoryHelper(getRepository().getSetRepository()); - } - - @Override - public ListRecords handle(OAICompiledRequest parameters) throws OAIException, HandlerException { - XlistRecords res = new XlistRecords(); - int length = getRepository().getConfiguration().getMaxListRecords(); - - if (parameters.hasSet() && !getRepository().getSetRepository().supportSets()) - throw new DoesNotSupportSetsException(); - - int offset = getOffset(parameters); - ListItemsResults result; - if (!parameters.hasSet()) { - if (parameters.hasFrom() && !parameters.hasUntil()) - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getFrom()); - else if (!parameters.hasFrom() && parameters.hasUntil()) - result = itemRepositoryHelper.getItemsUntil(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getUntil()); - else if (parameters.hasFrom() && parameters.hasUntil()) - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getFrom(), parameters.getUntil()); - else - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix()); - } else { - if (!setRepositoryHelper.exists(getContext(), parameters.getSet())) { - // throw new NoMatchesException(); - } - if (parameters.hasFrom() && !parameters.hasUntil()) - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getSet(), parameters.getFrom()); - else if (!parameters.hasFrom() && parameters.hasUntil()) - result = itemRepositoryHelper.getItemsUntil(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getSet(), parameters.getUntil()); - else if (parameters.hasFrom() && parameters.hasUntil()) - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getSet(), parameters.getFrom(), - parameters.getUntil()); - else - result = itemRepositoryHelper.getItems(getContext(), offset, - length, parameters.getMetadataPrefix(), - parameters.getSet()); - } - - List results = result.getResults(); - if (results.isEmpty()) throw new NoMatchesException(); - for (Item i : results) - res.withRecord(this.createRecord(parameters, i)); - - - ResumptionToken.Value currentResumptionToken = new ResumptionToken.Value(); - if (parameters.hasResumptionToken()) { - currentResumptionToken = parameters.getResumptionToken(); - } else if (result.hasMore()) { - currentResumptionToken = parameters.extractResumptionToken(); - } - - XresumptionTokenHelper resumptionTokenHelper = new XresumptionTokenHelper(currentResumptionToken, - getRepository().getConfiguration().getMaxListRecords()); - res.withResumptionToken(resumptionTokenHelper.resolve(result.hasMore())); - - return res; - } - - - private int getOffset(OAICompiledRequest parameters) { - if (!parameters.hasResumptionToken()) - return 0; - if (parameters.getResumptionToken().getOffset() == null) - return 0; - return parameters.getResumptionToken().getOffset().intValue(); - } - - private Record createRecord(OAICompiledRequest parameters, Item item) - throws BadArgumentException, CannotDisseminateRecordException, - OAIException, NoMetadataFormatsException, CannotDisseminateFormatException { - MetadataFormat format = getContext().formatForPrefix(parameters.getMetadataPrefix()); - Header header = new Header(); - - Dataset dataset = ((Xitem)item).getDataset(); - Xrecord xrecord = new Xrecord().withFormatName(parameters.getMetadataPrefix()).withDataset(dataset); - header.withIdentifier(item.getIdentifier()); - - ItemHelper itemHelperWrap = new ItemHelper(item); - header.withDatestamp(item.getDatestamp()); - for (Set set : itemHelperWrap.getSets(getContext(), getRepository().getFilterResolver())) - header.withSetSpec(set.getSpec()); - if (item.isDeleted()) - header.withStatus(Header.Status.DELETED); - - xrecord.withHeader(header); - xrecord.withMetadata(item.getMetadata()); - - return xrecord; - } - - - private XSLPipeline toPipeline(Item item) throws XmlWriteException, XMLStreamException { - ByteArrayOutputStream output = new ByteArrayOutputStream(); - XmlWriter writer = new XmlWriter(output); - Metadata metadata = item.getMetadata(); - metadata.write(writer); - writer.close(); - return new XSLPipeline(new ByteArrayInputStream(output.toByteArray()), true); - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xmetadata.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xmetadata.java deleted file mode 100644 index 225b9b13777..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xmetadata.java +++ /dev/null @@ -1,27 +0,0 @@ - -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xml.exceptions.XmlWriteException; -import com.lyncode.xoai.model.oaipmh.Metadata; -import com.lyncode.xoai.xml.XmlWriter; - -/** - * - * @author Leonid Andreev - */ -public class Xmetadata extends Metadata { - - - public Xmetadata(String value) { - super(value); - } - - - @Override - public void write(XmlWriter writer) throws XmlWriteException { - // Do nothing! - // - rather than writing Metadata as an XML writer stram, we will write - // the pre-exported *and pre-validated* content as a byte stream, directly. - } - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xrecord.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xrecord.java deleted file mode 100644 index 7e115c78f06..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/Xrecord.java +++ /dev/null @@ -1,184 +0,0 @@ -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xoai.model.oaipmh.Header; -import com.lyncode.xoai.model.oaipmh.Record; -import com.lyncode.xoai.xml.XmlWriter; -import static com.lyncode.xoai.xml.XmlWriter.defaultContext; - -import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.export.ExportException; -import edu.harvard.iq.dataverse.export.ExportService; -import static edu.harvard.iq.dataverse.util.SystemConfig.FQDN; -import static edu.harvard.iq.dataverse.util.SystemConfig.SITE_URL; -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.net.InetAddress; -import java.net.UnknownHostException; -import org.apache.poi.util.ReplacingInputStream; - -/** - * - * @author Leonid Andreev - * - * This is the Dataverse extension of XOAI Record, - * optimized to directly output a pre-exported metadata record to the - * output stream, thus by-passing expensive parsing and writing by - * an XML writer, as in the original XOAI implementation. - */ - -public class Xrecord extends Record { - private static final String METADATA_FIELD = "metadata"; - private static final String METADATA_START_ELEMENT = "<"+METADATA_FIELD+">"; - private static final String METADATA_END_ELEMENT = ""; - private static final String HEADER_FIELD = "header"; - private static final String STATUS_ATTRIBUTE = "status"; - private static final String IDENTIFIER_FIELD = "identifier"; - private static final String DATESTAMP_FIELD = "datestamp"; - private static final String SETSPEC_FIELD = "setSpec"; - private static final String DATAVERSE_EXTENDED_METADATA_FORMAT = "dataverse_json"; - private static final String DATAVERSE_EXTENDED_METADATA_API = "/api/datasets/export"; - - protected Dataset dataset; - protected String formatName; - - - public Dataset getDataset() { - return dataset; - } - - public Xrecord withDataset(Dataset dataset) { - this.dataset = dataset; - return this; - } - - - public String getFormatName() { - return formatName; - } - - - public Xrecord withFormatName(String formatName) { - this.formatName = formatName; - return this; - } - - public void writeToStream(OutputStream outputStream) throws IOException { - outputStream.flush(); - - String headerString = itemHeaderToString(this.header); - - if (headerString == null) { - throw new IOException("Xrecord: failed to stream item header."); - } - - outputStream.write(headerString.getBytes()); - - // header.getStatus() is only non-null when it's indicating "deleted". - if (header.getStatus() == null) { // Deleted records should not show metadata - if (!isExtendedDataverseMetadataMode(formatName)) { - outputStream.write(METADATA_START_ELEMENT.getBytes()); - - outputStream.flush(); - - if (dataset != null && formatName != null) { - InputStream inputStream = null; - try { - inputStream = new ReplacingInputStream( - ExportService.getInstance().getExport(dataset, formatName), - "", - "" - ); - } catch (ExportException ex) { - inputStream = null; - } - - if (inputStream == null) { - throw new IOException("Xrecord: failed to open metadata stream."); - } - writeMetadataStream(inputStream, outputStream); - } - outputStream.write(METADATA_END_ELEMENT.getBytes()); - } else { - outputStream.write(customMetadataExtensionRef(this.dataset.getGlobalIdString()).getBytes()); - } - } - outputStream.flush(); - } - - private String itemHeaderToString(Header header) { - try { - ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream(); - XmlWriter writer = new XmlWriter(byteOutputStream, defaultContext()); - - writer.writeStartElement(HEADER_FIELD); - - if (header.getStatus() != null) { - writer.writeAttribute(STATUS_ATTRIBUTE, header.getStatus().value()); - } - writer.writeElement(IDENTIFIER_FIELD, header.getIdentifier()); - writer.writeElement(DATESTAMP_FIELD, header.getDatestamp()); - for (String setSpec : header.getSetSpecs()) { - writer.writeElement(SETSPEC_FIELD, setSpec); - } - writer.writeEndElement(); // header - writer.flush(); - writer.close(); - - String ret = byteOutputStream.toString(); - - return ret; - } catch (Exception ex) { - return null; - } - } - - private void writeMetadataStream(InputStream inputStream, OutputStream outputStream) throws IOException { - int bufsize; - byte[] buffer = new byte[4 * 8192]; - - while ((bufsize = inputStream.read(buffer)) != -1) { - outputStream.write(buffer, 0, bufsize); - outputStream.flush(); - } - - inputStream.close(); - } - - private String customMetadataExtensionRef(String identifier) { - String ret = "<" + METADATA_FIELD - + " directApiCall=\"" - + getDataverseSiteUrl() - + DATAVERSE_EXTENDED_METADATA_API - + "?exporter=" - + DATAVERSE_EXTENDED_METADATA_FORMAT - + "&persistentId=" - + identifier - + "\"" - + "/>"; - - return ret; - } - - private boolean isExtendedDataverseMetadataMode(String formatName) { - return DATAVERSE_EXTENDED_METADATA_FORMAT.equals(formatName); - } - - private String getDataverseSiteUrl() { - String hostUrl = System.getProperty(SITE_URL); - if (hostUrl != null && !"".equals(hostUrl)) { - return hostUrl; - } - String hostName = System.getProperty(FQDN); - if (hostName == null) { - try { - hostName = InetAddress.getLocalHost().getCanonicalHostName(); - } catch (UnknownHostException e) { - return null; - } - } - hostUrl = "https://" + hostName; - return hostUrl; - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XresumptionTokenHelper.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XresumptionTokenHelper.java deleted file mode 100644 index 7f9eac2cbe8..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/xoai/XresumptionTokenHelper.java +++ /dev/null @@ -1,61 +0,0 @@ - -package edu.harvard.iq.dataverse.harvest.server.xoai; - -import com.lyncode.xoai.dataprovider.handlers.helpers.ResumptionTokenHelper; -import com.lyncode.xoai.model.oaipmh.ResumptionToken; -import static java.lang.Math.round; -import static com.google.common.base.Predicates.isNull; - -/** - * - * @author Leonid Andreev - * Dataverse's own version of the XOAI ResumptionTokenHelper - * Fixes the issue with the offset cursor: the OAI validation spec - * insists that it starts with 0, while the XOAI implementation uses 1 - * as the initial offset. - */ -public class XresumptionTokenHelper { - - private ResumptionToken.Value current; - private long maxPerPage; - private Long totalResults; - - public XresumptionTokenHelper(ResumptionToken.Value current, long maxPerPage) { - this.current = current; - this.maxPerPage = maxPerPage; - } - - public XresumptionTokenHelper withTotalResults(long totalResults) { - this.totalResults = totalResults; - return this; - } - - public ResumptionToken resolve (boolean hasMoreResults) { - if (isInitialOffset() && !hasMoreResults) return null; - else { - if (hasMoreResults) { - ResumptionToken.Value next = current.next(maxPerPage); - return populate(new ResumptionToken(next)); - } else { - ResumptionToken resumptionToken = new ResumptionToken(); - resumptionToken.withCursor(round((current.getOffset()) / maxPerPage)); - if (totalResults != null) - resumptionToken.withCompleteListSize(totalResults); - return resumptionToken; - } - } - } - - private boolean isInitialOffset() { - return isNull().apply(current.getOffset()) || current.getOffset() == 0; - } - - private ResumptionToken populate(ResumptionToken resumptionToken) { - if (totalResults != null) - resumptionToken.withCompleteListSize(totalResults); - resumptionToken.withCursor(round((resumptionToken.getValue().getOffset() - maxPerPage)/ maxPerPage)); - return resumptionToken; - } - - -} diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index 50e29d2a333..102772bdcf3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -563,7 +563,11 @@ Whether Harvesting (OAI) service is enabled /* * Allow a custom JavaScript to control values of specific fields. */ - ControlledVocabularyCustomJavaScript + ControlledVocabularyCustomJavaScript, + /** + * A compound setting for disabling signup for remote Auth providers: + */ + AllowRemoteAuthSignUp ; @Override @@ -668,7 +672,39 @@ public Long getValueForCompoundKeyAsLong(Key key, String param){ } } + + /** + * Same, but with Booleans + * (returns null if not set; up to the calling method to decide what that should + * default to in each specific case) + * Example: + * :AllowRemoteAuthSignUp {"default":"true","google":"false"} + */ + public Boolean getValueForCompoundKeyAsBoolean(Key key, String param) { + + String val = this.getValueForKey(key); + + if (val == null) { + return null; + } + + try (StringReader rdr = new StringReader(val)) { + JsonObject settings = Json.createReader(rdr).readObject(); + if (settings.containsKey(param)) { + return Boolean.parseBoolean(settings.getString(param)); + } else if (settings.containsKey("default")) { + return Boolean.parseBoolean(settings.getString("default")); + } else { + return null; + } + + } catch (Exception e) { + logger.log(Level.WARNING, "Incorrect setting. Could not convert \"{0}\" from setting {1} to boolean: {2}", new Object[]{val, key.toString(), e.getMessage()}); + return null; + } + + } /** * Return the value stored, or the default value, in case no setting by that * name exists. The main difference between this method and the other {@code get()}s diff --git a/src/main/java/edu/harvard/iq/dataverse/util/DateUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/DateUtil.java index d6f22471f68..669780b9436 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/DateUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/DateUtil.java @@ -16,7 +16,9 @@ */ public class DateUtil { + public static String YEAR_PATTERN = "yyyy"; public static String YEAR_DASH_MONTH_PATTERN = "yyyy-MM"; + public static String YEAR_DASH_MONTH_DASH_DAY_PATTERN = "yyyy-MM-dd"; public static String formatDate(Date dateToformat) { String formattedDate; @@ -63,4 +65,40 @@ public static String formatDate(Timestamp datetimeToformat) { } } + public static Date parseDate(String dateString) { + SimpleDateFormat sdf; + Date date; + + // YYYY-MM-DD + date = parseDate(dateString, YEAR_DASH_MONTH_DASH_DAY_PATTERN); + if (date != null) { + return date; + } + + // YYYY-MM + date = parseDate(dateString, YEAR_DASH_MONTH_PATTERN); + if (date != null) { + return date; + } + + // YYYT + date = parseDate(dateString, YEAR_PATTERN); + return date; + + } + + public static Date parseDate(String dateString, String format) { + + try { + SimpleDateFormat sdf = new SimpleDateFormat(format); + Date date = sdf.parse(dateString); + return date; + } catch (ParseException ex) { + // ignore + } catch (Exception ex) { + // ignore + } + return null; + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java index e76e2c5696b..d64a1f7cce1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java @@ -32,6 +32,12 @@ public static InternetAddress parseSystemAddress(String systemEmail) { public static String getSubjectTextBasedOnNotification(UserNotification userNotification, Object objectOfNotification) { List rootDvNameAsList = Arrays.asList(BrandingUtil.getInstallationBrandName()); + String datasetDisplayName = ""; + + if (objectOfNotification != null && (objectOfNotification instanceof Dataset) ) { + datasetDisplayName = ((Dataset)objectOfNotification).getDisplayName(); + } + switch (userNotification.getType()) { case ASSIGNROLE: return BundleUtil.getStringFromBundle("notification.email.assign.role.subject", rootDvNameAsList); @@ -46,23 +52,23 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti case REJECTFILEACCESS: return BundleUtil.getStringFromBundle("notification.email.rejected.file.access.subject", rootDvNameAsList); case DATASETCREATED: - return BundleUtil.getStringFromBundle("notification.email.dataset.created.subject", Arrays.asList(rootDvNameAsList.get(0), ((Dataset)objectOfNotification).getDisplayName())); + return BundleUtil.getStringFromBundle("notification.email.dataset.created.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case CREATEDS: - return BundleUtil.getStringFromBundle("notification.email.create.dataset.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.create.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case SUBMITTEDDS: - return BundleUtil.getStringFromBundle("notification.email.submit.dataset.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.submit.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case PUBLISHEDDS: - return BundleUtil.getStringFromBundle("notification.email.publish.dataset.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.publish.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case PUBLISHFAILED_PIDREG: - return BundleUtil.getStringFromBundle("notification.email.publishFailure.dataset.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.publishFailure.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case RETURNEDDS: - return BundleUtil.getStringFromBundle("notification.email.returned.dataset.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.returned.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case WORKFLOW_SUCCESS: - return BundleUtil.getStringFromBundle("notification.email.workflow.success.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.workflow.success.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case WORKFLOW_FAILURE: - return BundleUtil.getStringFromBundle("notification.email.workflow.failure.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.workflow.failure.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case STATUSUPDATED: - return BundleUtil.getStringFromBundle("notification.email.status.change.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.status.change.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case CREATEACC: return BundleUtil.getStringFromBundle("notification.email.create.account.subject", rootDvNameAsList); case CHECKSUMFAIL: @@ -115,13 +121,13 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti case APIGENERATED: return BundleUtil.getStringFromBundle("notification.email.apiTokenGenerated.subject", rootDvNameAsList); case INGESTCOMPLETED: - return BundleUtil.getStringFromBundle("notification.email.ingestCompleted.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.ingestCompleted.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case INGESTCOMPLETEDWITHERRORS: - return BundleUtil.getStringFromBundle("notification.email.ingestCompletedWithErrors.subject", rootDvNameAsList); + return BundleUtil.getStringFromBundle("notification.email.ingestCompletedWithErrors.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case DATASETMENTIONED: return BundleUtil.getStringFromBundle("notification.email.datasetWasMentioned.subject", rootDvNameAsList); } return ""; } -} \ No newline at end of file +} diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index 7abd0d02065..80af2df081c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -1228,4 +1228,15 @@ public Map getCurationLabels() { } return labelMap; } + + public boolean isSignupDisabledForRemoteAuthProvider(String providerId) { + Boolean ret = settingsService.getValueForCompoundKeyAsBoolean(SettingsServiceBean.Key.AllowRemoteAuthSignUp, providerId); + + // we default to false - i.e., "not disabled" if the setting is not present: + if (ret == null) { + return false; + } + + return !ret; + } } diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index 8a4fdeb9e28..b19e80020ba 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -473,6 +473,8 @@ oauth2.convertAccount.failedDeactivated=Your existing account cannot be converte # oauth2/callback.xhtml oauth2.callback.page.title=OAuth Callback oauth2.callback.message=Authentication Error - Dataverse could not authenticate your login with the provider that you selected. Please make sure you authorize your account to connect with Dataverse. For more details about the information being requested, see the User Guide. +oauth2.callback.error.providerDisabled=This authentication method ({0}) is currently disabled. Please log in using one of the supported methods. +oauth2.callback.error.signupDisabledForProvider=Sorry, signup for new accounts using {0} authentication is currently disabled. # deactivated user accounts deactivated.error=Sorry, your account has been deactivated. @@ -732,28 +734,28 @@ dashboard.card.datamove.dataset.command.error.indexingProblem=Dataset could not #MailServiceBean.java notification.email.create.dataverse.subject={0}: Your dataverse has been created -notification.email.create.dataset.subject={0}: Your dataset has been created +notification.email.create.dataset.subject={0}: Dataset "{1}" has been created notification.email.dataset.created.subject={0}: Dataset "{1}" has been created notification.email.request.file.access.subject={0}: Access has been requested for a restricted file notification.email.grant.file.access.subject={0}: You have been granted access to a restricted file notification.email.rejected.file.access.subject={0}: Your request for access to a restricted file has been rejected -notification.email.submit.dataset.subject={0}: Your dataset has been submitted for review -notification.email.publish.dataset.subject={0}: Your dataset has been published -notification.email.publishFailure.dataset.subject={0}: Failed to publish your dataset -notification.email.returned.dataset.subject={0}: Your dataset has been returned -notification.email.workflow.success.subject={0}: Your dataset has been processed +notification.email.submit.dataset.subject={0}: Dataset "{1}" has been submitted for review +notification.email.publish.dataset.subject={0}: Dataset "{1}" has been published +notification.email.publishFailure.dataset.subject={0}: Failed to publish your dataset "{1}" +notification.email.returned.dataset.subject={0}: Dataset "{1}" has been returned +notification.email.workflow.success.subject={0}: Dataset "{1}" has been processed notification.email.workflow.success=A workflow running on {0} (view at {1} ) succeeded: {2} -notification.email.workflow.failure.subject={0}: Failed to process your dataset +notification.email.workflow.failure.subject={0}: Failed to process your dataset "{1}" notification.email.workflow.failure=A workflow running on {0} (view at {1} ) failed: {2} -notification.email.status.change.subject={0}: Dataset Status Change +notification.email.status.change.subject={0}: Dataset "{1}" Status Change notification.email.status.change=The Status of the dataset ({0}) has changed to {1} notification.email.workflow.nullMessage=No additional message sent from the workflow. notification.email.create.account.subject={0}: Your account has been created notification.email.assign.role.subject={0}: You have been assigned a role notification.email.revoke.role.subject={0}: Your role has been revoked notification.email.verifyEmail.subject={0}: Verify your email address -notification.email.ingestCompleted.subject={0}: Dataset status -notification.email.ingestCompletedWithErrors.subject={0}: Dataset status +notification.email.ingestCompleted.subject={0}: Dataset "{1}" status +notification.email.ingestCompletedWithErrors.subject={0}: Dataset "{1}" status notification.email.greeting=Hello, \n notification.email.greeting.html=Hello,
# Bundle file editors, please note that "notification.email.welcome" is used in a unit test diff --git a/src/main/resources/META-INF/microprofile-config.properties b/src/main/resources/META-INF/microprofile-config.properties index 16298d83118..be02bb1b090 100644 --- a/src/main/resources/META-INF/microprofile-config.properties +++ b/src/main/resources/META-INF/microprofile-config.properties @@ -8,3 +8,10 @@ dataverse.db.host=localhost dataverse.db.port=5432 dataverse.db.user=dataverse dataverse.db.name=dataverse +# OAI SERVER +dataverse.oai.server.maxidentifiers=100 +dataverse.oai.server.maxrecords=10 +dataverse.oai.server.maxsets=100 +# the OAI repository name, as shown by the Identify verb, +# can be customized via the setting below: +#dataverse.oai.server.repositoryname= diff --git a/src/main/webapp/oauth2/callback.xhtml b/src/main/webapp/oauth2/callback.xhtml index f0d66b2fa74..8d09f06da2d 100644 --- a/src/main/webapp/oauth2/callback.xhtml +++ b/src/main/webapp/oauth2/callback.xhtml @@ -21,14 +21,18 @@