-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Finalize Azure Transfer tests #1051
Conversation
Found while adding remaining unit tests
Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.
Test Results243 tests 242 ✅ 1m 4s ⏱️ Results for commit d1d4ed7. ♻️ This comment has been updated with latest results. |
Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.
Was able to re-work the tests per my last paragraph. It now only creates a single instance of Azurite for the whole test module. Each test function then creates it's own storage container (in Azurite, not a docker container), and creates the same blobs, so that the environment is uniform across tests. It took the runtime down from 2m40s, to just around 1 minute. |
…ateMax log levels
Doc feedback
issue #982 detail in cleanup was not working when statehost set
more robust validation of return value from destfnscript
Add directory to send, etc. errors
only run after_post after we actually post
…sg_missing_size fix #1095 crash in sender when size missing from message
Issue1089 fileAgeMin and fileAgeMax should be honoured by all components
So... I checked out the branch, and merged from development, and pip3 install -r requirements.txt, and it fails very long... The summary is:
the very long complaints are lots of occurrences of:
other occurrences of:
E docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) ../../.local/lib/python3.10/site-packages/docker/api/client.py:230: DockerException
E FileNotFoundError: [Errno 2] No such file or directory ../../.local/lib/python3.10/site-packages/docker/transport/unixconn.py:26: FileNotFoundError
|
Found while adding remaining unit tests
Using the Azureite testcontainers to build a local dummy instance of Azure Storage accounts. There's a slightly hacky fix for the rmdir method, as documented.
Was previously spinning up/down Docker containers for each test, now it only creates one, but uses a different storage account "container" for each test. Way faster this way. Also, fixed a warning in test_rmdir from using blob, instead of blob.name.
…/sarracenia into FinalizeAzureTransferTests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
works better with docker installed ;-)
Managed to write the last few tests for the Azure Transfer driver, using Testcontainers, specifically the Azurite one.
It spins up a container for every test function in the file, and then destroys it, so the state is clean, and well defined between tests.
It takes quite a long time to run, compared to the S3 tests, or any others, because it's having to do a lot of work in the background for each test. It might be possible to optimize things, but using different "containers" (in Azurite, not a Docker container), re-using the same Testcontainer, but that would need a bit more work to setup for each test.
edit: Going to see if that last idea is viable...