elasticsearch_reader_api configuration overrides id_reader/ elasticsearch_reader configs

[ciorg]

In this job set up:

"name": "test-job",
"lifecycle": "once",
"workers": 1,
"assets": [
        "elasticsearch:2.1.0"
    ],
"apis":[
  {
    "_name": "elasticsearch_reader_api",
     "connection": "homeslice",
    "index": "test-index",
    "field": "_key",
    "query": "key:someKey",
    "size": 10000
  }
],
    "operations": [
        {
            "_op": "id_reader",
            "connection": "otherConnection",
            "index": "second-test-index",
            "field": "_key",
            "size": 10000
        },
        {
            "_op": "custom-api-reader-op",
            "api_name": "elasticsearch_reader_api"        
      },
    ]

The settings for the elasticsearch_reader_api in the apis property override the id_reader configs. It seems like the elasticsearch_reader_api should have to be called in the _op config for the api configs to be applied.

[peterdemartini]

If there is a conflict, it should throw an error.

[ciorg]

Talked with jared about this. Will add documentation to specify how to handle multiple elasticsearch_reader_api configs in a job.

[peterdemartini]

I am going to reopen this because it should absolutely fail when submitting the job. You should NOT be able to specify connection, index, field, or any other conflicting config value on both the api and the reader. If there was validation for this, the problem is immediately obvious and there isn't a need to debug why the job isn't working right. Documentation can be used to document business logic but never to enforce it.

@jsnoble I thought we talked about this behavior?