Elasticsearch setup

Elasticsearch Background

Herd uses Elasticsearch to index BDef-level data for search purposes. The data that is indexed includes: BDef name, display name, and descriptions; BDef column descriptions; BDef tag associations. The search index is exposed in various Index Search endpoints. See the 'Index Search' portion of the API documentation in your installed Herd instance for more details on how to search.

The instructions below are in all the Herd Demo Install templates including and after version 0.65.0 -- but details are included for information and in case modifications are required in some environments.

Elasticsearch Setup

Important: Please install an Elasticsearch software version corresponding to the Elasticsearch library version in the Herd pom.xml file. The version number can be found in the <elasticsearch.version> property in the top level pom.xml (https://github.com/FINRAOS/herd/blob/master/pom.xml) file in the herd repository.

Before creating new Elasticsearch indices, confirm that the Elasticsearch settings in the configuration table (cnfgn) are correct for your system setup. Please see the Configuration Values page for more details.

Each index has an associated settings and mappings JSON file that must be configured. To configure the mappings and settings to the default values add the following configuration values to the configuration table (cnfgn) in the database:

elasticsearch.bdef.mappings.json

elasticsearch.bdef.settings.json

elasticsearch.tag.mappings.json

elasticsearch.tag.settings.json

These default settings are well-tuned and suitable for most environments but can be altered as necessary. When altering, be sure not to break the queries in use by Index Search endpoints.

Note: The values above are CLOBs (Character large objects) and are stored in the cnfgn_value_cl column in the cnfgn table.

Creating and activating indexes

To create and activate new Herd Elasticsearch indexes use the search indexes Herd REST API. Below are examples of creating and activating indexes using the search index HERD REST API. Replace localhost and port number to the appropriate values for your Herd server hostname and port.

# Create tag index
# Record tag index name from the result JSON
curl -X POST \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  -d '{"searchIndexType": "TAG"}' \
  http://localhost:8080/herd-app/rest/searchIndexes

# Create business object definition index
# Record business object definition index name from the result JSON
curl -X POST \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  -d '{"searchIndexType": "BUS_OBJCT_DFNTN"}' \
  http://localhost:8080/herd-app/rest/searchIndexes

# Activate tag index
# Use the recorded tag index name for the seachIndexName
# Example:
curl -X POST \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  -d '{"searchIndexKey": {"searchIndexName": "tag_1520597704738"}}' \
  http://localhost:8080/herd-app/rest/searchIndexActivations

# Activate business object definition index
# Use the recorded business object definition index name for the seachIndexName
# Example:
curl -X POST \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  -d '{"searchIndexKey": {"searchIndexName": "bdef_1520597692303"}}' \
  http://localhost:8080/herd-app/rest/searchIndexActivations

Encrypted communication setup

If you want encrypted communication between the Herd server and the Elasticsearch cluster you must set the elasticsearch.rest.client.scheme configuration value to https, and you must add the following configuration values to the configuration table (cnfgn) in the database:

elasticsearch.rest.client.username
elasticsearch.rest.client.usercredentialname
credstash.aws.region.name
credstash.table.name
credstash.encryption.context