[BUG] Wrong HTTP Response code in _bulk API

[manojsm51]

Describe the bug

Hi,

I tried ingesting data using _bulk API to OpenSearch which has RED cluster state.
It throws below primary shard is not active error as expected.

{
  "took" : 60002,
  "errors" : true,
  "items" : [
    {
      "create" : {
        "_index" : "test-index",
        "_id" : "1",
        "status" : 503,
        "error" : {
          "type" : "unavailable_shards_exception",
          "reason" : "[test-index][4] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[test-index][4]] containing [index {[test-index][1], source[{ \"field1\" : \"value1\", \"field2\" : \"value2\" }]}]]"
        }
      }
    }
  ]
}

But in HTTP response headers, it throws 200 instead of 503.

* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
> POST /_bulk?pretty HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 562
>



< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 2673

Related component

Other

To Reproduce

1. Make data nodes down (just to make sure cluster is in RED state)
2. Send _bulk request to master/other available nodes (use curl with verbose -v)
3. Check the output of the request

Expected behavior

HTTP response code of _bulk API should reflect the original response code of each internal request. i.e., 503 in this case

Additional Details

Plugins
None

Host/Environment (please complete the following information):

• OS: SLES
• Version: 15-SP6
• OpenSearch version: 2.18.0

[manojsm51]

Found that HTTP Status is always set to 200 OK.
https://github.com/opensearch-project/OpenSearch/blob/2.18.0/server/src/main/java/org/opensearch/action/bulk/BulkResponse.java#L165

Is this correct?
Shouldn't it change for hasFailures()?

[andrross]

HTTP response code of _bulk API should reflect the original response code of each internal request. i.e., 503 in this case

@manojsm51 Batch style APIs are tricky. How do you handle the case where operations within the batch have different status codes?

In any case where the request was valid and we processed individual items then the overall request status will be 200 and the result of each item will be described in the response payload, even if there is only one item or all items have the same status. This is the behavior you have described here and is correct.

In cases where the batch cannot be processed at all (such as improper syntax in the request body), then you'll get a non-200 status code in the response.

[manojsm51]

Hi @andrross ,

Response code 200 indicates the request is accepted though each operation within the batch did not succeed.
Atleast providing 207 code will indicate something is not right about the request and may give a chance to look deeper inside the payload.
By simply looking at the header, I could wrongly decide that my _bulk API request succeeds and may ignore the actual error.

[andrross]

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/207:

The HTTP 207 Multi-Status successful response status code indicates a mixture of responses. This response is used exclusively in the context of Web Distributed Authoring and Versioning (WebDAV).

The response body is a text/xml or application/xml HTTP entity with a multistatus root element that lists individual response codes.

The 207 code does not seem appropriate here.

By simply looking at the header, I could wrongly decide that my _bulk API request succeeds and may ignore the actual error.

The _bulk response body does include a top-level errors field as a convenience to determine if any of the individual operations had an error.