[COST-5125] Trino migration command improvements

[samdoran]

Jira Ticket

COST-5125

Description

Add better logging and fine tune the retry behavior.

The overall progess is reported during exeuction as well as indenting log messages for operations on each schema.

[2024-06-26 18:50:33,174] INFO None 57531 Finding all schemas for migration...
[2024-06-26 18:50:33,338] INFO None 57531 Running against the following schemas: {'org1234567', 'test_ocp_table_has_column'}
[2024-06-26 18:50:33,338] INFO None 57531 Modifying tables for schema org1234567 (1 / 2)
[2024-06-26 18:50:33,338] INFO None 57531     org1234567: Altering table reporting_ocpusagelineitem_daily_summary...
[2024-06-26 18:50:33,650] INFO None 57531     org1234567: Altered table reporting_ocpusagelineitem_daily_summary []
[2024-06-26 18:50:33,651] INFO None 57531 Modifying tables for schema test_ocp_table_has_column (2 / 2)
[2024-06-26 18:50:33,651] INFO None 57531     test_ocp_table_has_column: Altering table reporting_ocpusagelineitem_daily_summary...
[2024-06-26 18:50:33,723] INFO None 57531     test_ocp_table_has_column: Altered table reporting_ocpusagelineitem_daily_summary []
[2024-06-26 18:50:33,723] INFO None 57531 Validating...
[2024-06-26 18:50:33,723] INFO None 57531 Finding all schemas for migration...
[2024-06-26 18:50:33,829] INFO None 57531 Migration successful

If the migration will run against more than ten schemas, it lists the count in the log message instead of all the schemas.

Testing

1. Checkout Branch
2. Load some data
3. Run the following command twice. On the second run, it should not make changes.

python koku/manage.py migrate_trino_tables --add-columns '[ {"table": "reporting_ocpusagelineitem_daily_summary", "column": "some_new_column", "datatype": "varchar"} ]'

4. Run the following command twice

python koku/manage.py migrate_trino_tables --drop-columns '[ {"table": "reporting_ocpusagelineitem_daily_summary", "column": "some_new_column", "datatype": "varchar"} ]'

5. Run the follow command to drop an external table

python koku/manage.py migrate_trino_tables --drop-tables aws_line_items

6. Run the following command to attempt to drop a managed table

python koku/manage.py migrate_trino_tables --drop-tables aws_openshift_daily_resource_matched_temp

Release Notes

• proposed release note

* [COST-5125](https://issues.redhat.com/browse/COST-5125) Improve logging and fine tune behavior of Trino migration management command

[samdoran]

/retest

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.2%. Comparing base (011ec69) to head (d2897b4).

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5191   +/-   ##
=====================================
  Coverage   94.2%   94.2%           
=====================================
  Files        376     376           
  Lines      31230   31230           
  Branches    3725    3725           
=====================================
  Hits       29410   29410           
+ Misses      1161    1160    -1     
- Partials     659     660    +1