This repository has been archived by the owner on Nov 1, 2024. It is now read-only.
Support TorchArrow column with binary type (SQL VARBINARY, pyarrow.binary) #178
Comments
YLGH
pushed a commit
to YLGH/torcharrow
that referenced
this issue
May 7, 2022
Summary: Pull Request resolved: pytorch/torchrec#178 Instead of naively showing input as just the pooling factor and output as just the embedding dimension this diff changes planner stats to use the actual size of input & output in terms of megabytes per iteration **input**: global_batch_size * pooling factor * sizeof(dtype of input) **output**: global_batch_size * (output size (1 in pooled)) * sizeof(dytpe of emb) * emb_dim This provides a sense of scale for data coming in and out, and additionally makes plans with multiple sharding types directly comparable. Also fixes a bug with TWCW, we incorrectly specified the ranks as entire world size when it should be limited to the local world of the host that the parameter is sharded on. Reviewed By: dstaay-fb Differential Revision: D35153224 fbshipit-source-id: c1e7d717ec0c1d074f7e059d843fba2d287eee56
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
cc: @wenleix
The text was updated successfully, but these errors were encountered: