Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explore DataSourceV2 API #3

Open
ericsun95 opened this issue Nov 10, 2020 · 3 comments
Open

Explore DataSourceV2 API #3

ericsun95 opened this issue Nov 10, 2020 · 3 comments
Labels
enhancement New feature or request question Further information is requested

Comments

@ericsun95
Copy link
Owner

Explore DataSourceV2 API and come up with a plan.
@ericsun95 ericsun95 added enhancement New feature or request question Further information is requested labels Nov 10, 2020
@ericsun95
Copy link
Owner Author

ericsun95 commented Nov 14, 2020 •
edited
Loading

This is the doc describing datasource v2 in general. Implementations should at least use ReadSupport or WriteSupport interfaces for readable or writable data sources, respectively. https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-DataSourceV2.html

@ericsun95
Copy link
Owner Author

ReadSupport defines a single createReader method that creates a DataSourceReader.

DataSourceReader createReader(DataSourceOptions options)
DataSourceReader createReader(StructType schema, DataSourceOptions options)

@ericsun95 ericsun95 mentioned this issue Nov 16, 2020
Merged
@ericsun95
Copy link
Owner Author

Add a initial support for DataSourceV2. Will upgrade more for the other relevant optimization.

@ericsun95 ericsun95 pinned this issue Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ericsun95