could not convert string to float when extracting RDS extended support pricing for engine aurora website

[lcscarvalho]

hey team,

I’m receiving the following error when trying to execute this script:

2024-07-12 17:54:46,363 - INFO - ThreadPoolExecutor-0_0 - Extracting RDS extended support priciing for engine aurora from https://aws.amazon.com/rds/aurora/pricing
2024-07-12 17:54:46,684 - ERROR - MainThread - Error in processing account. Exception: could not convert string to float: ''
Traceback (most recent call last):
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 306, in <module>
    main()
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 266, in main
    future.result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 149, in get_rds_extended_support_instances
    save_to_csv(rds_extended_support_instances)
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 189, in save_to_csv
    global db_engine_price_map

    df['Year 1 Price'] = df.apply(lambda row: get_y1_price(row), axis=1)
  File "/home/cloudshell-user/.local/lib/python3.9/site-packages/pandas/core/frame.py", line 10374, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/home/cloudshell-user/.local/lib/python3.9/site-packages/pandas/core/apply.py", line 916, in apply
    return self.apply_standard()
  File "/home/cloudshell-user/.local/lib/python3.9/site-packages/pandas/core/apply.py", line 1063, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/home/cloudshell-user/.local/lib/python3.9/site-packages/pandas/core/apply.py", line 1081, in apply_series_generator
    results[i] = self.func(v, *self.args, **self.kwargs)
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 189, in <lambda>
    df['Year 1 Price'] = df.apply(lambda row: get_y1_price(row), axis=1)
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/find_rds_extended_support_instances.py", line 163, in get_y1_price
    (provisioned_price_map, _) = get_rds_extended_support_pricing(row['Engine'].split('-')[0])
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/utils/rds_mappings.py", line 194, in get_rds_extended_support_pricing
    aurora_provisioned_price_map = get_price_map(aurora_provisioned)
  File "/home/cloudshell-user/rds-extended-support-cost-estimator-main/scripts/utils/rds_mappings.py", line 168, in get_price_map
    yr_1_2_price = float((cols[1].text.strip()).strip('$'))
ValueError: could not convert string to float: ''

[sin-ak]

Apologies - I was on PTO and missed seeing this. I'm actively working on fixing this. Will resolve this issue once I have it fixed.

[sin-ak]

There are 2 main issues here -
1/ I was scrapping https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html to create a mapping of db instance types to the vCPUs used. For eg: db.m6i.large = 2 vCPUs. Earlier, AWS documentation had a single table listing all db instance classes and their vCPUs. However, recently, that was changed, and all tables are now browken up into their own sub tables, hence breaking my code. I have a fix for this issue now.
2/ After I fixed the above issue, i found another issue in extracting the Extended Support Price from AWS documentation for various RDS engines. That was again a single big table, listing all regions and the respective extended support price for Year1 & Year2 and Year3. AWS pricing page changed and now there is a drop down to select a region and the price for extended support for only that region is shown. This HTML code was not geting parsed properly by beautifulsoup python library (i still dont know why). So i replaced the code with using the official AWS Pricing List Query API. However, that means I now have to use boto3 and make an API call that counts against the user's API limits. I have not found a way around this unfortunately. Using the earlier approach of scrapping the AWS pricing page allowed me to avoid making an AWS API call.

I have fixed both these issues and am doing some more tests for different RDS engine types. I will push the fix soon.

[sin-ak]

Pushed fixes for the above mentioned changes now. Please git pull latest script and let me know if you encounter any errors.

[sin-ak]

Closing