Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Native S3 transfer driver #1010

Merged
merged 67 commits into from
Apr 5, 2024
Merged

Native S3 transfer driver #1010

merged 67 commits into from
Apr 5, 2024

Commits on Mar 26, 2024

  1. Initial code for an S3 transfer class

    @gcglinton
    gcglinton committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    483e25e View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Add bandwidth capping to S3 transfer class

    @gcglinton
    gcglinton committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    fc1aa50 View commit details
    Browse the repository at this point in the history

  2. Update core sr3 code to work with S3 transfer

    @gcglinton
    gcglinton committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    c27094d View commit details
    Browse the repository at this point in the history

  3. many changes to s3 transfer class 

    - Add all the methods I think it needs
- Re-order them alphabetically
- Work on how to configure it
    @gcglinton
    gcglinton committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    3bd7491 View commit details
    Browse the repository at this point in the history

  4. #982 statehost config option should be used to see pick state directory

    @petersilva
    petersilva committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    9e450b5 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. Try to catch bad connections in S3 Transfer

    @gcglinton
    gcglinton committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    15a9022 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. Got S3 connecting and LSing 

    Worked with Reid and Peter to get the S3 transfer class connecting to S3, and recusively LS-ing objects in a bucket.
It also inherently gets `__credentials`, and `cd` working too.
    @gcglinton
    gcglinton committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    315470c View commit details
    Browse the repository at this point in the history

  2. Add more methods so send works 

    It does send, the paths are weird, and I don't know what they're supposed to do.

I suspect there's some changes to make.
    @gcglinton
    gcglinton committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    12f271b View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Add metadata to uploaded S3 files 

    Also cleaned up some test code, and did some re-formatting.
    @gcglinton
    gcglinton committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    9ee6447 View commit details
    Browse the repository at this point in the history

  2. Catch object metadata on ls 

    Also fix statically assigned bucket and key names in object size fetch on puts
    @gcglinton
    gcglinton committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    f9e0f05 View commit details
    Browse the repository at this point in the history

  3. Start of unit Tests for S3 Transfer class 

    Just the basics for now, but more to come.
    @gcglinton
    gcglinton committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    3d9764b View commit details
    Browse the repository at this point in the history

  4. Add dependencies for S3 transfer tests

    @gcglinton
    gcglinton committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    7087ffc View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. Expanded unit tests for S3 transfer class

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    bd8924d View commit details
    Browse the repository at this point in the history

  2. Add last two method tests 

    They're not 100% complete, but overall coverage of the class is at 90%, which seems like it's good enough.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    386a9f7 View commit details
    Browse the repository at this point in the history

  3. housekeeping check missing in wiski

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2417c60 View commit details
    Browse the repository at this point in the history

  4. allow gather to abort further gathers by changing return value to a t… 

    …uple
    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    ecbb5b3 View commit details
    Browse the repository at this point in the history

  5. implement #974 poll refactor to used scheduled flows

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4fc56d4 View commit details
    Browse the repository at this point in the history

  6. updating documentation of gather routine's new variable return formats

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    b9463ca View commit details
    Browse the repository at this point in the history

  7. Adding initial run on polls with scheduled intervals

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    512fec5 View commit details
    Browse the repository at this point in the history

  8. fix bug where using localtime instead of utc can cause infinite loop

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    822b47d View commit details
    Browse the repository at this point in the history

  9. notice when appointments are in the past

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    6c365be View commit details
    Browse the repository at this point in the history

  10. refactoring Flow/run() to simplify

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2dd35ee View commit details
    Browse the repository at this point in the history

  11. refactoring run further to make it shorter

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4162df0 View commit details
    Browse the repository at this point in the history

  12. more run refactoring, trying to get it shorter

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    8df3e55 View commit details
    Browse the repository at this point in the history

  13. move on_start to save some vip check logic

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    c068255 View commit details
    Browse the repository at this point in the history

  14. fixing indenting of code block moved in last commit

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    aaa908b View commit details
    Browse the repository at this point in the history

  15. remove do stub, same as parent now

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    47c810f View commit details
    Browse the repository at this point in the history

  16. documenting use of scheduled_ settings over sleep for polls

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0f5efe8 View commit details
    Browse the repository at this point in the history

  17. poll implement default interval 300s

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0496b63 View commit details
    Browse the repository at this point in the history

  18. have convert change poll sleep to scheduled_interval, and related fut… 

    …zing
    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    95e1fc8 View commit details
    Browse the repository at this point in the history

  19. abort gather when housekeeping or stopping

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    614a73f View commit details
    Browse the repository at this point in the history

  20. providing a type hint for ls entry point

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    095de1c View commit details
    Browse the repository at this point in the history

  21. more detailed dicts

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0659bf9 View commit details
    Browse the repository at this point in the history

  22. fix #1002 sftp binary accelletor fails for file names with colons in … 

    …them
    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    210fa9b View commit details
    Browse the repository at this point in the history

  23. #997 Move retry logic to on_start

    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    22a3b92 View commit details
    Browse the repository at this point in the history

  24. #997 Correctly launch child through os.execl. Corrects instance numbers.

    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    40481f0 View commit details
    Browse the repository at this point in the history

  25. #997 Correct method names

    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    58ddcf4 View commit details
    Browse the repository at this point in the history

  26. #997 Add a socket timeout to mimic sarracenia flow and correct poor e… 

    …rror handling of broken sockets
    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4bd6cd0 View commit details
    Browse the repository at this point in the history

  27. #997 Add method to cleanup socket descriptor files

    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5565b24 View commit details
    Browse the repository at this point in the history

  28. #997 Add timeout checks for parent as well. Avoids sr3 sanity restarts.

    @andreleblanc11 @petersilva
    andreleblanc11 authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    cb642b1 View commit details
    Browse the repository at this point in the history

  29. had_vip must be set to true to ensure novip file is written

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    7d9cadc View commit details
    Browse the repository at this point in the history

  30. still run housekeeping even in poll and wVip

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    8baabdd View commit details
    Browse the repository at this point in the history

  31. documentation touchups sleep -> scheduled interval

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5915943 View commit details
    Browse the repository at this point in the history

  32. retry cleanup was broken by work with andre for am... fixed

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2df2230 View commit details
    Browse the repository at this point in the history

  33. more information about ls entry point

    @petersilva
    petersilva authored and petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    93124d0 View commit details
    Browse the repository at this point in the history

  34. more information about ls entry point for transfer classes

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    3849a8d View commit details
    Browse the repository at this point in the history

  35. fixing typo from Reid

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    002006d View commit details
    Browse the repository at this point in the history

  36. fix for unit tests of retry class

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    25df985 View commit details
    Browse the repository at this point in the history

  37. fixing problem seen in tests noticed by andre

    @petersilva
    petersilva committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    556d4a2 View commit details
    Browse the repository at this point in the history

  38. Initial code for an S3 transfer class

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    44d941f View commit details
    Browse the repository at this point in the history

  39. Add bandwidth capping to S3 transfer class

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    503b3cd View commit details
    Browse the repository at this point in the history

  40. Update core sr3 code to work with S3 transfer

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e63f0e4 View commit details
    Browse the repository at this point in the history

  41. many changes to s3 transfer class 

    - Add all the methods I think it needs
- Re-order them alphabetically
- Work on how to configure it
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9f8eae1 View commit details
    Browse the repository at this point in the history

  42. Try to catch bad connections in S3 Transfer

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    f5cea72 View commit details
    Browse the repository at this point in the history

  43. Got S3 connecting and LSing 

    Worked with Reid and Peter to get the S3 transfer class connecting to S3, and recusively LS-ing objects in a bucket.
It also inherently gets `__credentials`, and `cd` working too.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    16f3c06 View commit details
    Browse the repository at this point in the history

  44. Add more methods so send works 

    It does send, the paths are weird, and I don't know what they're supposed to do.

I suspect there's some changes to make.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    ecf81a1 View commit details
    Browse the repository at this point in the history

  45. Add metadata to uploaded S3 files 

    Also cleaned up some test code, and did some re-formatting.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    67c57cc View commit details
    Browse the repository at this point in the history

  46. Catch object metadata on ls 

    Also fix statically assigned bucket and key names in object size fetch on puts
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5795ca4 View commit details
    Browse the repository at this point in the history

  47. Start of unit Tests for S3 Transfer class 

    Just the basics for now, but more to come.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    15c575e View commit details
    Browse the repository at this point in the history

  48. Add dependencies for S3 transfer tests

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    cfbbab6 View commit details
    Browse the repository at this point in the history

  49. Expanded unit tests for S3 transfer class

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    937a957 View commit details
    Browse the repository at this point in the history

  50. Add last two method tests 

    They're not 100% complete, but overall coverage of the class is at 90%, which seems like it's good enough.
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9cc1d8a View commit details
    Browse the repository at this point in the history

  51. Merge branch 'Issue939_S3Transfer' of https://github.com/MetPX/sarrac… 

    …enia into Issue939_S3Transfer
    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    947320a View commit details
    Browse the repository at this point in the history

  52. Add type hints to S3 trasnfer class, per #1009

    @gcglinton
    gcglinton committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5052384 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. revert retur type hints on transfer/ls because older python3.6 cannot… 

    … handle them
    @petersilva
    petersilva committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    0f73d22 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'issue983_3_per_config_statehost' into Issue939_S3Transfer

    @petersilva
    petersilva committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    1d60e78 View commit details
    Browse the repository at this point in the history

  3. removing prototype return values again (do not work on older python)

    @petersilva
    petersilva committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    c0961fb View commit details
    Browse the repository at this point in the history