Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP Release 1.2.8-Bugs/Enhancements #7

Open
wants to merge 7 commits into
base: release/1.2
Choose a base branch
from
Open

HTTP Release 1.2.8-Bugs/Enhancements #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2dfbd12
Update Json batch support
psainics Aug 17, 2023
4d52810
HTTP Sink enhancement: Retry and PUT changes with placeholder
sgarg-CS Aug 30, 2023
9d9f90e
Http_enhancements
Shubhangi-cs Oct 27, 2023
4c9cf74
commit
Shubhangi-cs Nov 16, 2023
b47346e
Http_enhancements
Shubhangi-cs Oct 27, 2023
8b4a89e
changes
Shubhangi-cs Nov 16, 2023
406f2a1
Changes
Shubhangi-cs Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 55 additions & 4 deletions docs/HTTP-batchsink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,28 @@ Sink plugin to send the messages from the pipeline to an external http endpoint.
Properties
----------

**url:** The URL to post data to. (Macro enabled)
**url:** The URL to post data to. Additionally, a placeholder like #columnName can be added to the URL that can be
substituted with column value at the runtime. E.g. https://customer-url/user/#user_id. Here user_id column should exist
in input schema. (Macro enabled)

**method:** The HTTP request method. Defaults to POST. (Macro enabled)

**batchSize:** Batch size. Defaults to 1. (Macro enabled)

**Write JSON As Array:** Whether to write the JSON as an array. Defaults to false. (Macro enabled)

When set to true, the payload will be written as an array of JSON objects.
Example - If batch size is 2, then the payload will be `[{"key":"val"}, {"key":"val"}]`

When false, the payload will be JSON objects separated by a delimiter.
Example - If batch size is 2, delimiter is "\n" , then the payload will be `{"key":"val"}\n{"key":"val"}`

**Json Batch Key** Optional key to be used for wrapping json array as object
Leave empty for no wrapping of the array. Ignored if Write JSON As Array is set to false. (Macro Enabled)

Example - If batch size is 2 and json batch key is "data", then the payload will be
`{"data": [{"key":"val"}, {"key":"val"}]}` instead of `[{"key":"val"}, {"key":"val"}]`

**messageFormat:** Format to send messsage in. Options are JSON, Form, Custom. Defaults to JSON. (Macro enabled)

**body:** Optional custom message. This is required if the message format is set to 'Custom'.
Expand All @@ -33,13 +49,49 @@ delimited by a colon (":") and each pair is delimited by a newline ("\n"). (Macr

**disableSSLValidation:** If user enables SSL validation, they will be expected to add the certificate to the trustStore on each machine. Defaults to true. (Macro enabled)

**numRetries:** The number of times the request should be retried if the request fails. Defaults to 3. (Macro enabled)
### Error Handling

**HTTP Errors Handling:** Defines the error handling strategy to use for certain HTTP response codes.
The left column contains a regular expression for HTTP status code. The right column contains an action which
is done in case of match. If HTTP status code matches multiple regular expressions, the first specified in mapping
is matched.

Example:

| HTTP Code Regexp | Error Handling |
| ----------------- |:-----------------------:|
| 2.. | Success |
| 401 | Retry and fail |
| 4.. | Fail |
| 5.. | Retry and send to error |
| .* | Fail |

Note: pagination types "Link in response header", "Link in response body", "Token in response body" do not support
"Send to error", "Skip", "Retry and send to error", "Retry and skip" options.

**Non-HTTP Error Handling:** Error handling strategy to use when the HTTP response cannot be transformed to an output record.
Possible values are:
Stop on error - Fails pipeline due to erroneous record.
Send to error - Sends erroneous record's text to error port and continues.
Skip on error - Ignores erroneous records.

**Retry Policy:** Policy used to calculate delay between retries. Default Retry Policy is Exponential.

**Linear Retry Interval:** Interval in seconds between retries. Is only used if retry policy is "linear".

**Max Retry Duration:** Maximum time in seconds retries can take. Default value is 600 seconds (10 minute).

**connectTimeout:** The time in milliseconds to wait for a connection. Set to 0 for infinite. Defaults to 60000 (1 minute). (Macro enabled)

**readTimeout:** The time in milliseconds to wait for a read. Set to 0 for infinite. Defaults to 60000 (1 minute). (Macro enabled)

**failOnNon200Response** Whether to fail the pipeline on non-200 response from the http end point. Defaults to true. (Macro enabled)
### HTTP Proxy

**Proxy URL:** Proxy URL. Must contain a protocol, address and port.

**Username:** Proxy username.

**Password:** Proxy password.

Example
-------
Expand All @@ -56,7 +108,6 @@ This example performs HTTP POST request to http://example.com/data.
"charset": "UTF-8",
"followRedirects": "true",
"disableSSLValidation": "true",
"numRetries": 0,
"connectTimeout": 60000,
"readTimeout": 60000
}
Expand Down
32 changes: 28 additions & 4 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<name>HTTP Plugins</name>
<groupId>io.cdap</groupId>
<artifactId>http-plugins</artifactId>
<version>1.2.8-SNAPSHOT</version>
<version>1.2.8</version>

<licenses>
<license>
Expand Down Expand Up @@ -80,13 +80,14 @@
<commons.version>3.9</commons.version>
<common.codec.version>1.12</common.codec.version>
<gson.version>2.8.5</gson.version>
<googleauth.version>0.4.0</googleauth.version>
<hadoop.version>2.3.0</hadoop.version>
<httpcomponents.version>4.5.9</httpcomponents.version>
<hydrator.version>2.3.0-SNAPSHOT</hydrator.version>
<hydrator.version>2.10.0-SNAPSHOT</hydrator.version>
<jackson.version>2.9.9</jackson.version>
<junit.version>4.11</junit.version>
<jython.version>2.7.1</jython.version>
<mockito.version>1.10.19</mockito.version>
<mockito.version>2.24.0</mockito.version>
<spark2.version>2.1.3</spark2.version>
<unxml.version>0.9</unxml.version>
<wiremock.version>1.49</wiremock.version>
Expand Down Expand Up @@ -324,6 +325,17 @@
<artifactId>httpclient</artifactId>
<version>${httpcomponents.version}</version>
</dependency>
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>${googleauth.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
Expand Down Expand Up @@ -416,10 +428,22 @@
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>2.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito2</artifactId>
<version>2.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.tomakehurst</groupId>
<artifactId>wiremock</artifactId>
Expand Down
Loading