[DPE-4487] Add Integration Tests for Azure Storage #89
Conversation
theoctober19th
changed the title
There was a problem hiding this comment.
The code does what it should, but I have a comment to maybe make the tests more maintainable and reusable, possibly also improving in consistency.
Right now we handle s3 setup and azure setup a bit differently, so for instance: s3 configuration are fed into the spark-submit command, while azure are fed using kubectl commands. Also a number of business logic in the iceberg test is just copied and pasted. I'm wondering whether we could rewrite the tests such that it reads:
(setup_user_context && setup_object_storage_s3 && test_iceberg_example_in_pod && cleanup_user_success) || cleanup_user_failure_in_pod
(setup_user_context && setup_object_storage_azure && test_iceberg_example_in_pod && cleanup_user_success) || cleanup_user_failure_in_pod
The custom part (between the two) is just the setup_object_storage_*part, where we both setup using the CLI but also we inject the right configuration using the spark-client.service-account-registry that is embedded in the OCI image.
Then, the iceberg test should be just using these configuration already injected and add the spark-submit command with the iceberg configuration only
It is not too critical, but I would honestly spend some time right now to do this, such that when adding other backends (blob storage, abfs, etc) it should be easier and more straight forward. Also this should be super easy to translate this into more structured tests, where the azure or s3 configuration is setup by the integration-hub
Thanks @deusebio. I've just updated the PR to make the SQL tests for both S3 and Azure storage. |
|
great!!! Thanks! Feel free to merge! I'm very happy with this PR, I believe it provides also an improved structure of functionalites! |
No description provided.