You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the async API is designed to use the AWS EMR service when it needs Spark to process a query. For example, the EMRServerlessClientFactory class is used for getting an instance of AWSEMRServerless.
There should be an abstraction here so that async API uses an abstract service to start and manage Spark jobs. EMR would be an implementation of this abstract service. Another possible implementation is to use Docker.
Is your feature request related to a problem?
No
What solution would you like?
An abstract service for managing Spark jobs, along with an EMR implementation of this service.
What alternatives have you considered?
None, open to discussion.
Do you have any additional context?
In this OpenSearch Spark PR, the aws-java-sdk-emrserverless Jar file is updated to replace the AWSEMRServerless implementation. There should be a cleaner way of replacing usage of the EMR service.
The text was updated successfully, but these errors were encountered:
Currently the async API is designed to use the AWS EMR service when it needs Spark to process a query. For example, the
EMRServerlessClientFactory
class is used for getting an instance ofAWSEMRServerless
.https://github.com/opensearch-project/sql/blob/main/async-query/src/main/java/org/opensearch/sql/spark/transport/config/AsyncExecutorServiceModule.java#L250
https://github.com/opensearch-project/sql/blob/main/async-query-core/src/main/java/org/opensearch/sql/spark/client/EMRServerlessClientFactoryImpl.java#L62
There should be an abstraction here so that async API uses an abstract service to start and manage Spark jobs. EMR would be an implementation of this abstract service. Another possible implementation is to use Docker.
Is your feature request related to a problem?
No
What solution would you like?
An abstract service for managing Spark jobs, along with an EMR implementation of this service.
What alternatives have you considered?
None, open to discussion.
Do you have any additional context?
In this OpenSearch Spark PR, the
aws-java-sdk-emrserverless
Jar file is updated to replace theAWSEMRServerless
implementation. There should be a cleaner way of replacing usage of the EMR service.The text was updated successfully, but these errors were encountered: