[curl] allow request body, content-type to be provided as strings

[dylan-shipwell]

Story

As a terraform project author responsible for integrating systems
I want to be able to submit arbitrary http REST requests, templated from data within the current terraform project's terraform state
So that I can administer delivering http requests from terraform when terraform detects a change is necessary via its drift detection or a resource definition was changed prompting a Create Modify or Delete terraform plan result.

Situation

While attempting to fulfill this goal, I was offered this limitation by the provider version "1.18.2", resource "restapi_object":

data = "%24class=Foo"
Error: data attribute is invalid JSON: invalid character '%' looking for beginning of value

data = "dispnumber=567567567&extension=6"
Error: data attribute is invalid JSON: invalid character 'd' looking for beginning of value

data = urlencode("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ ")
Error: data attribute is invalid JSON: invalid character '1' after top-level value

Background

In my case, I was trying to pass a request body as a literal (x-www-form-urlencoded required by server).

resource "restapi_object" "Foo2" {
  provider = restapi.restapi_headers
  path = "/api/objects"
  data = "dispnumber=567567567&extension=6"
}

Assessment

My hypothesis is that this provider intentionally json-decodes and then json-encodes the given request body.

Allowing users to pass data to libcurl as required might be a goal of this provider, presently I didn't find an obvious way to do that for specifying the request body. doc here, there is no variable to specify that data is a string that I see.

Recommendation

Currently I would advise users: this provider may fail to submit the given request body to the server due to preventing terraform from producing a plan.

Future looking I would advise authors be familiar with curl's interface and avoid adding unnecessary constraints for specifying the curl configuration necessary to submit requests.

I understand that terraform-provider-restapi may need to also have its own configuration that is not curl-facing for defining how remote resources are identified, drift-checked. In terms of avoiding unnecessary constraints, offering configuration to disable as many non essential features such as drift detection (relying only on terraform project files compared to tfstate to compel api updates), and such as record identification either specified (rely on the user to specify the record identifier directly, allows cases where the record id is known in advanced) or offering a plugable namespace of transformation functions (such as accepting the string "regex_replace" with companion configuration inputs, and "sha512sum" for invoking a transforming function on the response body to transform into a stable record unique identifier that can be used for things like drift detection) specifically to allow incremental feature addition for more complex edge cases over time.

I am not very familiar with libcurl, I found today a sample of the libcurl interface that appears to support delivering post bodies that are not in json format, that sample code provided by libcurl is call for assigning the request body from a string as follows
(excerpt from https://curl.se/libcurl/c/CURLOPT_POSTFIELDS.html)

    const char *data = "data to send";
 
    curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
 
    /* size of the POST data if strlen() is not good enough */
    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 12L);
 
    /* pass in a pointer to the data - libcurl does not copy */
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);

Thank You for consideration.

Errata

My apologies if I misunderstood the objective of this provider, today it seems like a better way to invoke libcurl is via a constructed environment plus a terraform "local-exec" null resource, but unfortunately this forces a glue program be in the constructed environment to convert terraform's forced stdin-must-be-single-depth-json and stdout-must-be-single-depth-json ^{[edit: oops i was confusing null_resource "local-exec" from data "external"]} constraints into curl and back. Which is a steep enough barrier to entry that this same thread ^ advises using this project as a provider instead. so we're in a catch-22 right now