Finalize Azure Transfer tests #1051

gcglinton · 2024-05-16T03:11:21Z

Managed to write the last few tests for the Azure Transfer driver, using Testcontainers, specifically the Azurite one.

It spins up a container for every test function in the file, and then destroys it, so the state is clean, and well defined between tests.

It takes quite a long time to run, compared to the S3 tests, or any others, because it's having to do a lot of work in the background for each test. It might be possible to optimize things, but using different "containers" (in Azurite, not a Docker container), re-using the same Testcontainer, but that would need a bit more work to setup for each test.

edit: Going to see if that last idea is viable...

Found while adding remaining unit tests

Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.

github-actions · 2024-05-16T03:14:48Z

Test Results

243 tests 242 ✅ 1m 4s ⏱️
1 suites 1 💤
1 files 0 ❌

Results for commit d1d4ed7.

♻️ This comment has been updated with latest results.

Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.

gcglinton · 2024-05-16T04:02:23Z

Was able to re-work the tests per my last paragraph.

It now only creates a single instance of Azurite for the whole test module.

Each test function then creates it's own storage container (in Azurite, not a docker container), and creates the same blobs, so that the environment is uniform across tests.

It took the runtime down from 2m40s, to just around 1 minute.

gcglinton · 2024-05-16T12:37:45Z

Closes #1015, and related to #1016.

…ateMax log levels

Doc feedback

issue #982 detail in cleanup was not working when statehost set

…=None

more robust validation of return value from destfnscript

Add directory to send, etc. errors

only run after_post after we actually post

…sg_missing_size fix #1095 crash in sender when size missing from message

Issue1089 fileAgeMin and fileAgeMax should be honoured by all components

petersilva · 2024-06-07T12:23:47Z

So... I checked out the branch, and merged from development, and pip3 install -r requirements.txt, and it fails very long...

The summary is:

tests/sarracenia/transfer/azure_test.py ......FEE..E.E.EE.

the very long complaints are lots of occurrences of:


../../.local/lib/python3.10/site-packages/urllib3/connectionpool.py:703: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.local/lib/python3.10/site-packages/urllib3/connectionpool.py:398: in _make_request
    conn.request(method, url, **httplib_request_kw)
../../.local/lib/python3.10/site-packages/urllib3/connection.py:239: in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
/usr/lib/python3.10/http/client.py:1283: in request
    self._send_request(method, url, body, headers, encode_chunked)
/usr/lib/python3.10/http/client.py:1329: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
/usr/lib/python3.10/http/client.py:1278: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
/usr/lib/python3.10/http/client.py:1038: in _send_output
    self.send(msg)
/usr/lib/python3.10/http/client.py:976: in send
    self.connect()

other occurrences of:

E           requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

../../.local/lib/python3.10/site-packages/requests/adapters.py:501: ConnectionError

        ) from e

E docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

../../.local/lib/python3.10/site-packages/docker/api/client.py:230: DockerException

E FileNotFoundError: [Errno 2] No such file or directory

../../.local/lib/python3.10/site-packages/docker/transport/unixconn.py:26: FileNotFoundError


what file is missing where is the question I guess.  running this on ubuntu 22.04.

Found while adding remaining unit tests

Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.

Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.

…/sarracenia into FinalizeAzureTransferTests

petersilva

works better with docker installed ;-)

…enabled

petersilva and others added 20 commits April 30, 2024 20:13

unit test engineering for wmo00 plugin... started

bad2c62

wmo00 simplest roundtrip

119054c

range upper bound is excluded

6869ff1

yesterday on the first is the last

3cd7955

firt shot at release guide... but it doesn't work for pypi.org yet

10a5f85

documenting a failure mode

3b4dd23

Merge branch 'development' into ute_wmo00

0971bb6

improve durationToSeconds + test it

fafff63

adding new_fileWrite() as per Reid suggestion

cd6f1a6

Adding durationToString tests

cf18565

feedback from doing a pre-release

646394f

Fix spelling and add comments

c618304

Fix #1008 by implementing selective user declaration

3e382dd

Fix a typo

7efc80b

refactoring parse_file routine to make easier to test parse_line routine

7eee4e9

better listing of file names in error messages

741a516

Add post_baseUrl to delete on post, changes to logging

472427e

add base 2 support to counts and sizes

6d5aa4b

Fix issues with Azure Transfer Driver

d133543

Found while adding remaining unit tests

Add remaining Azure transfer driver unit tests

36f8c81

Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.

petersilva and others added 2 commits May 15, 2024 23:39

for sarracenia/config.py tests and some fixes resulting from tests

faf0754

Change scope of azurite container

e14033a

Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.

gcglinton marked this pull request as ready for review May 16, 2024 12:36

gcglinton mentioned this pull request May 16, 2024

azure transfer driver modeled on the s3 one #1015

Open

gcglinton added 3 commits May 16, 2024 11:51

Update unit test requirements file

ff35602

Add flowcb test files

1d744e1

Add flowcb/poll test files

f17bbc0

reidsunderland and others added 16 commits June 6, 2024 14:14

French translation, credit to @andreleblanc11 and Google. Changed 2 R…

2646de7

…ateMax log levels

Merge branch 'development' into doc_feedback

461cafc

logFormat works now!

8047e8b

Merge pull request #1094 from MetPX/doc_feedback

3211eff

Doc feedback

Merge pull request #1096 from MetPX/issue982_2_statehost_cleanup_broken

01e9904

issue #982 detail in cleanup was not working when statehost set

Add directory to errors

5d300fa

better handle destfn rename failures by setting and catching new_name…

7efabbf

…=None

Merge pull request #1085 from MetPX/daniel_made_me_do_it

f983ae5

more robust validation of return value from destfnscript

even more voluble sender error log messages

d21fe2e

second shot at 1089, implementing fileAgeMin and Max

d028e93

only run after_post after we actually post

d31c610

Merge pull request #1099 from MetPX/issue1098

cbfa0fe

Add directory to send, etc. errors

fix #1095 crash in sender when size missing from message

b808222

Merge pull request #1101 from MetPX/issue1066

590e591

only run after_post after we actually post

Merge pull request #1102 from MetPX/issue1095_sender_not_crash_when_m…

b0377c6

…sg_missing_size fix #1095 crash in sender when size missing from message

Merge pull request #1100 from MetPX/issue1089

2a28e91

Issue1089 fileAgeMin and fileAgeMax should be honoured by all components

gcglinton added 4 commits June 7, 2024 10:05

Fix issues with Azure Transfer Driver

f0a8d02

Found while adding remaining unit tests

Add remaining Azure transfer driver unit tests

84d4e5e

Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.

Change scope of azurite container

f1ad22f

Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.

Merge branch 'FinalizeAzureTransferTests' of https://github.com/MetPX…

d1d4ed7

…/sarracenia into FinalizeAzureTransferTests

petersilva approved these changes Jun 11, 2024

View reviewed changes

petersilva merged commit 6b46501 into development Jun 11, 2024
4 checks passed

petersilva deleted the FinalizeAzureTransferTests branch June 19, 2024 12:49

reidsunderland and others added 6 commits July 21, 2024 21:27

Fix #1064 by adding try/except

f459754

refactor into runCallbackPoll method, allow full crash when debug is …

9bb5764

…enabled

support modular entry_points in flow itself, not just callbacks

09a28f0

Add scheduled flow with http metadata

a3d9093

just comment out this debug log

ea58219

add variable expansion for the path

648ac62

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finalize Azure Transfer tests #1051

Finalize Azure Transfer tests #1051

gcglinton commented May 16, 2024 •

edited

Loading

github-actions bot commented May 16, 2024 •

edited

Loading

gcglinton commented May 16, 2024

gcglinton commented May 16, 2024

petersilva commented Jun 7, 2024

petersilva left a comment

Finalize Azure Transfer tests #1051

Finalize Azure Transfer tests #1051

Conversation

gcglinton commented May 16, 2024 • edited Loading

github-actions bot commented May 16, 2024 • edited Loading

Test Results

gcglinton commented May 16, 2024

gcglinton commented May 16, 2024

petersilva commented Jun 7, 2024

petersilva left a comment

Choose a reason for hiding this comment

gcglinton commented May 16, 2024 •

edited

Loading

github-actions bot commented May 16, 2024 •

edited

Loading