Update/integration iqair devices #3962
Conversation
📝 WalkthroughWalkthroughThis pull request introduces significant modifications to the Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (4)
src/workflows/airqo_etl_utils/airqo_utils.py (4)
Line range hint
398-537: Consider removing deprecated methodextract_devices_data_The method
extract_devices_data_appears to be an outdated version ofextract_devices_dataand may no longer be in use. Removing unused or deprecated code enhances maintainability and reduces confusion.
Line range hint
552-573: Use appropriate logging methods outside exception handlingThe use of
logger.exception()is intended for logging exceptions within anexceptblock. In this context, where no exception is being raised, consider usinglogger.error()orlogger.warning()instead.Apply the following changes:
- if not devices: - logger.exception("Failed to fetch devices.") + if not devices: + logger.error("Failed to fetch devices.") ... - if device_number and read_key is None: - logger.exception(f"{device_number} does not have a read key") + if device_number and read_key is None: + logger.warning(f"{device_number} does not have a read key") ... - except Exception as e: - logger.exception(f"An error occured: {e} - device {device['name']}") + except Exception as e: + logger.exception(f"An error occurred: {e} - device {device['name']}")Note: Corrected the typo in "occurred" in the exception message.
Line range hint
600-612: PotentialNameErrordue to undefinedmeta_datafor IQAir devicesWhen processing IQAir devices (
network == "iqair"), the variablemeta_datamay not be defined before it's used to assignlatitudeandlongitude. This could lead to aNameError. To prevent this, ensure thatlatitudeandlongitudeare fetched correctly for IQAir devices.Apply the following changes:
- if device_category in AirQoDataUtils.Device_Field_Mapping: - data["latitude"] = device.get("latitude", None) - data["longitude"] = device.get("longitude", None) - else: - data["latitude"] = meta_data.get("latitude", None) - data["longitude"] = meta_data.get("longitude", None) + data["latitude"] = device.get("latitude", None) + data["longitude"] = device.get("longitude", None)This change ensures that
latitudeandlongitudeare consistently obtained from thedevicedictionary, preventing potential errors.
Line range hint
559-575: Ensure consistent data types fordataacross device networksThe variable
datais used differently for 'airqo' and 'iqair' networks:
- For 'airqo',
datais a list that accumulates data over multiple dates.- For 'iqair',
datais assigned directly as a DataFrame.This inconsistency may cause issues during concatenation with
devices_data. It's advisable to standardizedataas a DataFrame in both cases.Suggested adjustments:
- Initialize
dataas an empty DataFrame before the loops.- For 'airqo', after collecting data, convert it to a DataFrame.
- Ensure that
datais consistently a DataFrame before concatenation.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
src/workflows/airqo_etl_utils/airqo_utils.py(2 hunks)src/workflows/dags/airqo_measurements.py(1 hunks)
🧰 Additional context used
📓 Learnings (1)
src/workflows/dags/airqo_measurements.py (1)
Learnt from: AnthonyByansi
PR: airqo-platform/AirQo-api#3262
File: src/workflows/dags/airqo_measurements.py:0-0
Timestamp: 2024-11-12T09:15:21.461Z
Learning: The dataset name `AirQodataset` was corrected to `AirQo-dataset` in multiple places within the `airqo_measurements.py` file to ensure consistency.
🔇 Additional comments (1)
src/workflows/dags/airqo_measurements.py (1)
496-499: Method call correctly updated to extract_devices_data
The function now appropriately calls AirQoDataUtils.extract_devices_data, ensuring correct data extraction as per the updated method.
Description
This PR switches all extract device data implementation to the extract device data by networks.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation