Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix type of the a batch returned by make_batch_reader when TransformSpec's function returns column with all values being None #750

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Fix type of the a batch returned by make_batch_reader when TransformSpec's function returns column with all values being None #750

wants to merge 7 commits into from

Commits on Sep 14, 2022

  1. Fix type of the a batch returned by make_batch_reader when Transforme… 

    …rSpec's function sets an entire column to None.

Resolves uber#744

We implemented a Unischema->Pyarrow-schema conversion and explicitly
set the pyarrow schema when converting a pandas dataframe returned by
transform spec function into a pyarrow table. This way, pyarrow does not
have to guess the type of data from the data itself (which it obviously
could not do before, since all values were None).
    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    80815cb View commit details
    Browse the repository at this point in the history

  2. Handle timestamp type correctly.

    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    02bc27c View commit details
    Browse the repository at this point in the history

  3. Test fix

    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    bb3dbec View commit details
    Browse the repository at this point in the history

  4. Fix failing "test_transform_spec_returns_all_none_values" test. 

    The test tests properly columns with scalars only. Will need to verify
correct behavior with columns that are lists separately. Will extend the
tests in the following commits.
    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    9b2bb69 View commit details
    Browse the repository at this point in the history

  5. Add tests: without tranform_spec and with list of strings with some, … 

    …all elements being None.

Change the type of numpy dtype to np.object instead of np.unicode_
    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    1fcf22f View commit details
    Browse the repository at this point in the history

  6. Type fix

    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    6ef18b3 View commit details
    Browse the repository at this point in the history

  7. Mypy error ignore

    Yevgeni Litvin committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    ad9defb View commit details
    Browse the repository at this point in the history