Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read EAV - w/ batching #145

Open
wibeasley opened this issue Mar 21, 2017 · 1 comment
Open

Read EAV - w/ batching #145

wibeasley opened this issue Mar 21, 2017 · 1 comment
Assignees

Comments

@wibeasley
Copy link
Member

(Spun off from #138 and #133.)

@nutterb, I'm starting to come into your camp on this topic. Similar to how your batching occurs inside the function. (Also #51.)

When I was experimenting with eav exports, I think the batching has to be written in that function --specifically before the type.convert() call. Even using metadata cues, a variable can still resolve to different data types across subsequent batches. This easily happens with sparse columns where a variable is completely NA in a batch. But it can even happen if one batch resolves to an integer, and another to a double.

@wibeasley wibeasley self-assigned this Mar 21, 2017
@wibeasley
Copy link
Member Author

Current ideas after experimenting w/ the eav playground:

  1. The eav oneshot fx has an optional conversion optional (and it's on, by default)
  2. The batch version calls oneshot repeatedly without the conversion.
  3. Only after stacking all the oneshot results, the type.convert() is called.
  4. The eav accepts a metadata dataset as a parameter. If it's missing, then it will retrieve the metadata itself. Consequently, the metadata API should be called only once for a dataset with k batches.

wibeasley added a commit that referenced this issue Jan 20, 2020
wibeasley added a commit that referenced this issue Oct 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant