cache calculation log reduction

[RagingPuppiesX]

Hi,
we are using the DBT_LS with cache and every time there's a need to parse the dags and refresh the Dagbag (and we do that a lot in our deployment to discover some dag enrichments) or by regular scheduler refresh, we are getting tons of:

[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.335+0000�[0m] {�[34mgraph.py:�[0m377} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Trying to parse the dbt project using dbt ls cache cosmos_cache__dbt_dag_name__transform_data...�[0m
[2024-12-16, 11:21:54 EST] {graph.py:377} INFO - Trying to parse the dbt project using dbt ls cache cosmos_cache__dbt_dag_name__transform_data...
[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.335+0000�[0m] {�[34mgraph.py:�[0m377} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Trying to parse the dbt project using dbt ls cache cosmos_cache__dbt_dag_name__transform_data...�[0m
[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.369+0000�[0m] {�[34mcache.py:�[0m259} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Cosmos performance: time to calculate cache identifier cosmos_cache__dbt_dag_name__transform_data for current version: 0.009718475863337517�[0m
[2024-12-16, 11:21:54 EST] {cache.py:259} INFO - Cosmos performance: time to calculate cache identifier cosmos_cache__dbt_dag_name__transform_data for current version: 0.009718475863337517
[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.369+0000�[0m] {�[34mgraph.py:�[0m394} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Cosmos performance [airflow-worker-545bdc65b6-2v9rp|687864]: The cache size for cosmos_cache__dbt_dag_name__transform_data is 1821�[0m
[2024-12-16, 11:21:54 EST] {graph.py:394} INFO - Cosmos performance [airflow-worker-545bdc65b6-2v9rp|687864]: The cache size for cosmos_cache__dbt_dag_name__transform_data is 1821
[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.369+0000�[0m] {�[34mgraph.py:�[0m394} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Cosmos performance [airflow-worker-545bdc65b6-2v9rp|687864]: The cache size for cosmos_cache__dbt_dag_name__transform_data is 1821�[0m
[2024-12-16, 11:21:54 EST] {log.py:232} WARNING - [�[34m2024-12-16T16:21:54.370+0000�[0m] {�[34mgraph.py:�[0m402} INFO�[0m - �[35m(astronomer-cosmos)�[0m - Cosmos performance: Cache hit for cosmos_cache__dbt_dag_name__transform_data - cd198bf000f19184f09e4664f3edd74c,8bde3fd800e757b42bd1b9c852aca154�[0m
[2024-12-16, 11:21:54 EST] {graph.py:402} INFO - Cosmos performance: Cache hit for cosmos_cache__dbt_dag_name__transform_data - cd198bf000f19184f09e4664f3edd74c,8bde3fd800e757b42bd1b9c852aca154
[2024-12-16, 11:21:54 EST] {graph.py:335} INFO - Total nodes: 1

i could not find a simple way to remove the caching logs, no configuration for that what so ever, and also it is a bit unclear under which logger it is located...

[dosubot]

Hey @RagingPuppiesX! 👋 I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

To reduce or remove the caching logs generated by DBT_LS with cache during DAG parsing and Dagbag refresh, you can disable the caching feature by setting the environment variable AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS=0. This will turn off the caching of the dbt ls output, which is responsible for generating these logs:

export AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS=0

Additionally, you can control the propagation of logs in the Cosmos module by setting the AIRFLOW__COSMOS__PROPAGATE_LOGS environment variable to False, which may further help in reducing log verbosity:

export AIRFLOW__COSMOS__PROPAGATE_LOGS=False

These configurations should help in reducing or eliminating the associated logs ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[RagingPuppiesX]

AIRFLOW__COSMOS__PROPAGATE_LOGS is not only to add the astronomer-cosmos string? as mentioned in the docs

[dosubot]

The AIRFLOW__COSMOS__PROPAGATE_LOGS environment variable is not specifically for adding the astronomer-cosmos string to logs. Its purpose is to control whether logs are propagated in the Cosmos module. However, this option was deprecated in version 1.6.0a1 because the issue it addressed is no longer relevant with Cosmos's new logging approach ^[1][2].

The astronomer-cosmos string is added to log messages when the rich_logging option is enabled, which is a separate configuration. This can be done by setting the AIRFLOW__COSMOS__ENRICH_LOGGING environment variable to True ^[3][4].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[RagingPuppiesX]

so what's my alternative if AIRFLOW__COSMOS__PROPAGATE_LOGS is deprecated?

[dosubot]

The alternative to using the deprecated AIRFLOW__COSMOS__PROPAGATE_LOGS environment variable is to use the rich_logging configuration option. This can be enabled by setting the rich_logging option to True in the [cosmos] section of the airflow.cfg file:

[cosmos]
rich_logging = True

Alternatively, you can set the AIRFLOW__COSMOS__ENRICH_LOGGING environment variable to True:

export AIRFLOW__COSMOS__ENRICH_LOGGING=True

This new logging approach in Cosmos version 1.6.0 adds "(astronomer-cosmos)" to each log message and resolves the need for the deprecated propagate_logs option ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[RagingPuppiesX]

and how does help me with my original issue? i don't want to see the caching logs everywhere

[dosubot]

To address your original issue of reducing or eliminating caching logs, you can disable caching entirely by setting the following environment variable:

export AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS=0

This will prevent the caching logs from being generated during DAG parsing and Dagbag refresh. The rich_logging configuration is unrelated to suppressing these specific logs; it only affects the format of log messages by adding a prefix. Disabling caching is the most direct way to stop these logs from appearing ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[pankajkoti]

hi @RagingPuppiesX , the logger is located here:

astronomer-cosmos/cosmos/cache.py

Line 46 in 9820935

logger = get_logger(__name__)

and these logs come from the same module.

At the moment Cosmos does not expose config to disable individual loggers, but in your DAG could try disabling or increasing the log level for this logger?

import logging

logging.getLogger("cosmos.cache").setLevel(logging.CRITICAL)

or

import logging

logging.getLogger("cosmos.cache").disabled = True

And let us know if this helps?

[RagingPuppiesX]

will try, thanks a lot!