Merge pull request #4010 from IQSS/develop

v4.7.1
IQSS · Jul 14, 2017 · 62f8987 · 62f8987
2 parents 7a56c58 + 22090e2
commit 62f8987
Show file tree

Hide file tree

Showing 93 changed files with 4,256 additions and 2,488 deletions.
diff --git a/doc/sphinx-guides/source/_static/installation/files/etc/shibboleth/shibboleth2.xml b/doc/sphinx-guides/source/_static/installation/files/etc/shibboleth/shibboleth2.xml
@@ -41,6 +41,7 @@ https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPConfiguration
             <Handler type="Status" Location="/Status" acl="127.0.0.1"/>
 
             <!-- Session diagnostic service. -->
+	    <!-- showAttributeValues must be set to true to see attributes at /Shibboleth.sso/Session . -->
             <Handler type="Session" Location="/Session" showAttributeValues="true"/>
 
             <!-- JSON feed of discovery information. -->

diff --git a/doc/sphinx-guides/source/admin/index.rst b/doc/sphinx-guides/source/admin/index.rst
@@ -19,4 +19,8 @@ These "superuser" tasks are managed via the new page called the Dashboard. A use
    metadataexport
    timers
    geoconnect-worldmap
+   user-administration
+   solr-search-index
+   monitoring
+   maintenance
    troubleshooting
diff --git a/doc/sphinx-guides/source/admin/maintenance.rst b/doc/sphinx-guides/source/admin/maintenance.rst
@@ -0,0 +1,9 @@
+Maintenance
+===========
+
+.. contents:: Contents:
+	:local:
+
+When you have scheduled down time for your production servers, we provide a :download:`sample maintenance page <../_static/installation/files/etc/maintenance/maintenance.xhtml>` for you to use. To download, right-click and select "Save Link As".
+
+The maintenance page is intended to be a static page served by Apache to provide users with a nicer, more informative experience when the site is unavailable.
diff --git a/doc/sphinx-guides/source/admin/monitoring.rst b/doc/sphinx-guides/source/admin/monitoring.rst
@@ -0,0 +1,11 @@
+Monitoring
+===========
+
+.. contents:: Contents:
+	:local:
+
+In production you'll want to monitor the usual suspects such as CPU, memory, free disk space, etc.
+
+https://github.com/IQSS/dataverse/issues/2595 contains some information on enabling monitoring of Glassfish, which is disabled by default.
+
+There is a database table called ``actionlogrecord`` that captures events that may be of interest. See https://github.com/IQSS/dataverse/issues/2729 for more discussion around this table.
diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst
@@ -0,0 +1,47 @@
+Solr Search Index
+=================
+
+Dataverse requires Solr to be operational at all times. If you stop Solr, you should see a error about this on the home page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse application will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take.
+
+.. contents:: Contents:
+	:local:
+
+Full Reindex
+-------------
+
+There are two ways to perform a full reindex of the Dataverse search index. Starting with a "clear" ensures a completely clean index but involves downtime. Reindexing in place doesn't involve downtime but does not ensure a completely clean index.
+
+Clear and Reindex
++++++++++++++++++
+
+Clearing Data from Solr
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Please note that the moment you issue this command, it will appear to end users looking at the home page that all data is gone! This is because the home page is powered by the search index.
+
+``curl http://localhost:8080/api/admin/index/clear``
+
+Start Async Reindex
+~~~~~~~~~~~~~~~~~~~
+
+Please note that this operation may take hours depending on the amount of data in your system. This known issue is being tracked at https://github.com/IQSS/dataverse/issues/50
+
+``curl http://localhost:8080/api/admin/index``
+
+Reindex in Place
++++++++++++++++++
+
+An alternative to completely clearing the search index is to reindex in place.
+
+Clear Index Timestamps
+~~~~~~~~~~~~~~~~~~~~~~
+
+``curl -X DELETE http://localhost:8080/api/admin/index/timestamps``
+
+Start or Continue Async Reindex
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If indexing stops, this command should pick up where it left off based on which index timestamps have been set, which is why we start by clearing these timestamps above. These timestamps are stored in the ``dvobject`` database table.
+
+``curl http://localhost:8080/api/admin/index/continue``
+
diff --git a/doc/sphinx-guides/source/admin/troubleshooting.rst b/doc/sphinx-guides/source/admin/troubleshooting.rst
@@ -5,9 +5,19 @@ Troubleshooting
 
 This new (as of v.4.6) section of the Admin guide is for tips on how to diagnose and fix system problems. 
 
-.. contents:: |toctitle|
+.. contents:: Contents:
 	:local:
 
+Glassfish
+---------
+
+``server.log`` is the main place to look when you encounter problems. Hopefully an error message has been logged. If there's a stack trace, it may be of interest to developers, especially they can trace line numbers back to a tagged version.
+
+For debugging purposes, you may find it helpful to increase logging levels as mentioned in the :doc:`/developers/debugging` section of the Developer Guide.
+
+Our guides focus on using the command line to manage Glassfish but you might be interested in an admin GUI at http://localhost:4848
+
+
 Deployment fails, "EJB Timer Service not available"
 ---------------------------------------------------
 

diff --git a/doc/sphinx-guides/source/admin/user-administration.rst b/doc/sphinx-guides/source/admin/user-administration.rst
@@ -0,0 +1,39 @@
+User Administration
+===================
+
+This section focuses on user administration tools and tasks. 
+
+.. contents:: Contents:
+	:local:
+
+Manage Users Table
+------------------
+
+The Manage Users table gives the network administrator a list of all user accounts in table form. It lists username, full name, email address, and whether or not the user has Superuser status.
+
+Usernames are listed alphabetically and clicking on a username takes you to the account page that contains detailed information on that account.
+
+You can access the Manage Users table by clicking the "Manage Users" button on the Dashboard, which is linked from the header of all Dataverse pages (if you're loggied in as an administrator).
+
+Confirm Email
+-------------
+
+Dataverse encourages builtin/local users to verify their email address upon signup or email change so that sysadmins can be assured that users can be contacted.
+
+The app will send a standard welcome email with a URL the user can click, which, when activated, will store a ``lastconfirmed`` timestamp in the ``authenticateduser`` table of the database. Any time this is "null" for a user (immediately after signup and/or changing of their Dataverse email address), their current email on file is considered to not be verified. The link that is sent expires after a time (the default is 24 hours), but this is configurable by a superuser via the ``:MinutesUntilConfirmEmailTokenExpires`` config option.
+
+Should users' URL token expire, they will see a "Verify Email" button on the account information page to send another URL.
+
+Sysadmins can determine which users have verified their email addresses by looking for the presence of the value ``emailLastConfirmed`` in the JSON output from listing users (see the "Admin" section of the :doc:`/api/native-api`). As mentioned in the :doc:`/user/account` section of the User Guide, the email addresses for Shibboleth users are re-confirmed on every login.
+
+Deleting an API Token
+---------------------
+
+If an API token is compromised it should be deleted. Users can generate a new one for themselves as explained in the :doc:`/user/account` section of the User Guide, but you may want to preemptively delete tokens from the database.
+
+Using the API token 7ae33670-be21-491d-a244-008149856437 as an example:
+
+``delete from apitoken where tokenstring = '7ae33670-be21-491d-a244-008149856437';``
+
+You should expect the output ``DELETE 1`` after issuing the command above.
+
diff --git a/doc/sphinx-guides/source/api/intro.rst b/doc/sphinx-guides/source/api/intro.rst
@@ -21,7 +21,7 @@ We use the term "native" to mean that the API is not based on any standard. For
 Authentication
 --------------
 
-Most Dataverse APIs require the use of an API token. Instructions for getting a token are described in the :doc:`/user/account` section of the User Guide.
+Most Dataverse APIs require the use of an API token. (In code we sometimes call it a "key" because it's shorter.) Instructions for getting a token are described in the :doc:`/user/account` section of the User Guide.
 
 There are two ways to pass your API token to Dataverse APIs. The preferred method is to send the token in the ``X-Dataverse-key`` HTTP header, as in the following curl example::
 

diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst
@@ -24,6 +24,7 @@ If ``$id`` is omitted, a root dataverse is created. ``$id`` can either be a data
 
 Download the :download:`JSON example <../_static/api/dataverse-complete.json>` file and modified to create dataverses to suit your needs. The fields ``name``, ``alias``, and ``dataverseContacts`` are required. The controlled vocabulary for ``dataverseType`` is
 
+- ``DEPARTMENT``
 - ``JOURNALS``
 - ``LABORATORY``
 - ``ORGANIZATIONS_INSTITUTIONS``
@@ -559,9 +560,110 @@ Creates a global role in the Dataverse installation. The data POSTed are assumed
 
     POST http://$SERVER/api/admin/roles
 
-List all users::
+List users with the options to search and "page" through results. Only accessible to superusers. Optional parameters:
+
+* ``searchTerm`` A string that matches the beginning of a user identifier, first name, last name or email address.
+* ``itemsPerPage`` The number of detailed results to return.  The default is 25.  This number has no limit. e.g. You could set it to 1000 to return 1,000 results
+* ``selectedPage`` The page of results to return.  The default is 1. 
+
+    GET http://$SERVER/api/admin/list-users
+
+
+Sample output appears below. 
+
+* When multiple pages of results exist, the ``selectedPage`` parameters may be specified. 
+* Note, the resulting ``pagination`` section includes ``pageCount``, ``previousPageNumber``, ``nextPageNumber``, and other variables that may be used to re-create the UI.          
+
+.. code-block:: json
+
+    {
+        "status":"OK",
+        "data":{
+            "userCount":27,
+            "selectedPage":1,
+            "pagination":{
+                "isNecessary":true,
+                "numResults":27,
+                "numResultsString":"27",
+                "docsPerPage":25,
+                "selectedPageNumber":1,
+                "pageCount":2,
+                "hasPreviousPageNumber":false,
+                "previousPageNumber":1,
+                "hasNextPageNumber":true,
+                "nextPageNumber":2,
+                "startResultNumber":1,
+                "endResultNumber":25,
+                "startResultNumberString":"1",
+                "endResultNumberString":"25",
+                "remainingResults":2,
+                "numberNextResults":2,
+                "pageNumberList":[
+                    1,
+                    2
+                ]
+            },
+            "bundleStrings":{
+                "userId":"ID",
+                "userIdentifier":"Username",
+                "lastName":"Last Name ",
+                "firstName":"First Name ",
+                "email":"Email",
+                "affiliation":"Affiliation",
+                "position":"Position",
+                "isSuperuser":"Superuser",
+                "authenticationProvider":"Authentication",
+                "roles":"Roles",
+                "createdTime":"Created Time",
+                "lastLoginTime":"Last Login Time",
+                "lastApiUseTime":"Last API Use Time"
+            },
+            "users":[
+                {
+                    "id":8,
+                    "userIdentifier":"created1",
+                    "lastName":"created1",
+                    "firstName":"created1",
+                    "email":"[email protected]",
+                    "affiliation":"hello",
+                    "isSuperuser":false,
+                    "authenticationProvider":"BuiltinAuthenticationProvider",
+                    "roles":"Curator",
+                    "createdTime":"2017-06-28 10:36:29.444"
+                },
+                {
+                    "id":9,
+                    "userIdentifier":"created8",
+                    "lastName":"created8",
+                    "firstName":"created8",
+                    "email":"[email protected]",
+                    "isSuperuser":false,
+                    "authenticationProvider":"BuiltinAuthenticationProvider",
+                    "roles":"Curator",
+                    "createdTime":"2000-01-01 00:00:00.0"
+                },
+                {
+                    "id":1,
+                    "userIdentifier":"dataverseAdmin",
+                    "lastName":"Admin",
+                    "firstName":"Dataverse",
+                    "email":"[email protected]",
+                    "affiliation":"Dataverse.org",
+                    "position":"Admin",
+                    "isSuperuser":true,
+                    "authenticationProvider":"BuiltinAuthenticationProvider",
+                    "roles":"Admin, Contributor",
+                    "createdTime":"2000-01-01 00:00:00.0",
+                    "lastLoginTime":"2017-07-03 12:22:35.926",
+                    "lastApiUseTime":"2017-07-03 12:55:57.186"
+                },
+                **... 22 more user documents ...**      
+            ]
+        }
+    }
+
+.. note:: "List all users" ``GET http://$SERVER/api/admin/authenticatedUsers`` is deprecated, but supported.
 
-    GET http://$SERVER/api/admin/authenticatedUsers
 
 List user whose ``identifier`` (without the ``@`` sign) is passed::
 

diff --git a/doc/sphinx-guides/source/conf.py b/doc/sphinx-guides/source/conf.py
@@ -64,9 +64,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '4.7'
+version = '4.7.1'
 # The full version, including alpha/beta/rc tags.
-release = '4.7'
+release = '4.7.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst
@@ -221,6 +221,60 @@ Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete
 
 The solution is to put the file back to how it was before Netbeans touched it. If anyone knows of an open Netbeans bug about this, please let us know.
 
+Configuring / Troubleshooting Mail Host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Out of the box, no emails will be sent from your development environment. This is because you have set the ``:SystemEmail`` setting and make sure you've configured your SMTP correctly.
+
+You can configure ``:SystemEmail`` like this:
+
+``curl -X PUT -d 'Davisverse SWAT Team <[email protected]>' http://localhost:8080/api/admin/settings/:SystemEmail``
+
+Unfortunately for developers not at Harvard, the installer script gives you by default an SMTP server of ``mail.hmdc.harvard.edu`` but you can specify an alternative SMTP server when you run the installer.
+
+You can check the current SMTP server with the ``asadmin`` command:
+
+``asadmin get server.resources.mail-resource.mail/notifyMailSession.host``
+
+This command helps verify what host your domain is using to send mail. Even if it's the correct hostname, you may still need to adjust settings. If all else fails, there are some free SMTP service options available such as Gmail and MailGun. Let's find where we can configure it.
+
+1. First, navigate to your Glassfish admin console: http://localhost:4848
+2. From the left-side panel, select **JavaMail Sessions**
+3. You should see one session named **mail/notifyMailSession** -- click on that.
+
+From this window you can modify certain fields of your Dataverse's notifyMailSession, which is the JavaMail session for outgoing system email (such as on user signup or data publication). Two of the most important fields we need are:
+
+- **Mail Host:** The DNS name of the default mail server (e.g. smtp.gmail.com)
+- **Default User:** The username provided to your Mail Host when you connect to it (e.g. [email protected])
+
+Most of the other defaults can safely be left as is. **Default Sender Address** indicates the address that your installation's emails are sent from.
+
+If your user credentials for the SMTP server require a password, you'll need to configure some **Additional Properties** at the bottom.
+
+**IMPORTANT:** Before continuing, it's highly recommended that your Default User account does NOT use a password you share with other accounts, as one of the additional properties includes entering the Default User's password (without concealing it on screen). For smtp.gmail.com you can safely use an `app password <https://support.google.com/accounts/answer/185833?hl=en>`_ or create an extra Gmail account for use with your Dataverse dev environment.
+
+Authenticating yourself to a Mail Host can be tricky. As an example, we'll walk through setting up our JavaMail Session to use smtp.gmail.com as a host by way of SSL on port 465. Use the Add Property button to generate a blank property for each name/value pair.
+
+======================================	==============================
+				Name 								Value
+======================================	==============================
+mail.smtp.auth							true
+mail.smtp.password						[user's (*app*) password\*]
+mail.smtp.port							465
+mail.smtp.socketFactory.port			465
+mail.smtp.socketFactory.fallback		false
+mail.smtp.socketFactory.class			javax.net.ssl.SSLSocketFactory
+======================================	==============================
+
+**\*WARNING**: Entering a password here will *not* conceal it on-screen. It’s recommended to use an *app password* (for smtp.gmail.com users) or utilize a dedicated/non-personal user account with SMTP server auths so that you do not risk compromising your password.
+
+Save these changes at the top of the page and restart your Glassfish server to try it out.
+
+These properties can be tailored to your own preferred mail service, but if all else fails these settings work fine with Dataverse development environments for your localhost.
+
++ If you're seeing a "Relay access denied" error in your Glassfish logs when your app attempts to send an email, double check your user/password credentials for the Mail Host you're using.
++ If you're seeing a "Connection refused" / similar error upon email sending, try another port.
+
 Rebuilding Your Dev Environment
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

diff --git a/doc/sphinx-guides/source/index.rst b/doc/sphinx-guides/source/index.rst
@@ -3,10 +3,10 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Dataverse 4.7 Guides
-====================
+Dataverse 4.7.1 Guides
+======================
 
-These guides are for the most recent version of Dataverse. For the guides for **version 4.6.2** please go `here <http://guides.dataverse.org/en/4.6.2/>`_.
+These guides are for the most recent version of Dataverse. For the guides for **version 4.7** please go `here <http://guides.dataverse.org/en/4.7/>`_.
 
 .. toctree::
   :glob: