Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Streaming] Fix issues when not using try-with-resource #49

Open
wants to merge 92 commits into
base: main
Choose a base branch
from

Conversation

newtork
Copy link
Contributor

@newtork newtork commented Sep 4, 2024

Make the streaming API lenient towards try-with-resources.

Before:

  • with try-with-resource: HTTP connections are closed in normal cases, response errors, http errors
  • without try-with-resource: HTTP connections are NOT closed

After:

  • with try-with-resource: HTTP connections are closed in normal cases, response errors, http errors
  • without try-with-resource: HTTP connections are closed in normal cases, and http errors. Errors in response payload does not lead to closing of HTTP connection.
- InputStream <> BufferedReader <> Stream
+ InputStream <> BufferedReader <> [HandledIterator] <> Stream

The suggested HandledIterator understands when BufferedReader#readLine() returns null, then the underlying InputStream shall be closed [left]. Also when the consuming Stream is closed or finished iteration [right].

See JavaDoc of String BufferedReader#readLine() to understand null contract.

image

See JavaDoc and code of Stream BufferedReader#lines() for comparison.

image

Copy link
Contributor

@CharlesDuboisSAP CharlesDuboisSAP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove try-with-resources in the controller and readme since it doesn't do anything anymore

@CharlesDuboisSAP CharlesDuboisSAP dismissed their stale review September 10, 2024 13:58

The close handler is still there

Copy link
Member

@MatKuhr MatKuhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments. I would consider un-nesting the HandledIterator and merge it with the StreamConverter into just one class.

Copy link
Contributor

@CharlesDuboisSAP CharlesDuboisSAP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get a timeout on http://localhost:8080/streamChatCompletionDeltas

final var entity = new InputStreamEntity(inputStream, ContentType.TEXT_PLAIN);

final var sut = IterableStreamConverter.lines(entity);
assertThatCode(sut::count)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assertThatCode(sut::count)
assertThatThrownBy(sut::count)

The main difference with assertThatCode(ThrowingCallable) is that this method fails if no exception was thrown.

package com.sap.ai.sdk.foundationmodels.openai;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

@@ -42,20 +36,8 @@ Stream<D> handleResponse(@Nonnull final ClassicHttpResponse response)
@SuppressWarnings("PMD.CloseResource")
private Stream<D> parseResponse(@Nonnull final ClassicHttpResponse response)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stream is closed by the user of the Stream

This is false now. Could you update this comment and duplicate it in IterableStreamConverter.lines()
Alternatively just merge parseResponse inside of handleResponse since parseResponse only contains a return and nothing else

Comment on lines +65 to +68
} catch (final Exception e) {
isDone = true;
stopHandler.run();
throw new IllegalStateException("Iterator stopped unexpectedly.", e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What can be thrown here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants