airflow helm chart installation not working

[Abhinav1598]

Official Helm Chart version

1.6.0 (latest released)

Apache Airflow version

2.3.1 (latest released)

Kubernetes Version

1.21

Helm Chart configuration

No response

Docker Image customisations

No response

What happened

airflow not installing properly as per the command given in documentation
https://airflow.apache.org/docs/helm-chart/stable/index.html#installing-the-chart

What you think should happen instead

No response

How to reproduce

helm repo add apache-airflow https://airflow.apache.org
helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Abhinav1598]

helm install airflow apache-airflow/airflow --namespace airflow --debug
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /data/roshan/kubeconfig-prod
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /data/roshan/kubeconfig-prod
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /home/e4338/.cache/helm/repository/airflow-1.6.0.tgz

client.go:290: [debug] Starting delete for "airflow-broker-url" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:290: [debug] Starting delete for "airflow-fernet-key" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:290: [debug] Starting delete for "airflow-redis-password" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 30 resource(s)
client.go:290: [debug] Starting delete for "airflow-run-airflow-migrations" Job
client.go:128: [debug] creating 1 resource(s)
client.go:519: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 5m0s
client.go:547: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:586: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: failed post-install: timed out waiting for the condition
helm.go:88: [debug] failed post-install: timed out waiting for the condition

[potiuk]

I think this is specific to your configuration. You need to take a look at the logs of your migration job and see what kind of errors you see - and explain what kind of custom configuration you have.

You can use k9s tool to see the logs of your migration job (highly recommended).

Converting that into discussion until more details are provided.

[potiuk]

We would love to help you with that - but without you doing investigation we are not able. Help us to help you.

[mvaude]

I have the same problem and behavior from the helm chart when I try to deploy it.
In the database logs, I can see this error log in loop:

ERROR:  relation "log" does not exist at character 13

I tried with the default database (bitnami image of postgres11 ?) and an external one running version 14.

I found this question on stackoverflow without answer reporting the same error.

[potiuk]

In which error log do you seee it Can you also show what other logs are saying? are your pods running? or failing? Which ones? Do you see errors in other logs? Can you please use k9s to look at the logs and see if you see any other problems. Is your airflow working? The SO question is clearly about warning not error and mentions about possibilty of ignoring it so likely this is a warning only. Is this the same for you?

Just posting a single entry from "unknown" log does not make us closer to being able to help you :)

I think you should be aware that if you need help, you should spend a a bit time to prepare all the information that will make it easy to help you. I know it requires some investment, but if you reach out for help, I think you should make sure that whoever helps you (in their free time - this is a free forum) has all the information that can help with it.

[ArnauFauquer]

hello, I think the problem is in the "pods" of psql:

2022-06-10 12:13:05.148 GMT [785] STATEMENT: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES ('2022-06-10T12:13:05.129835+00:00'::timestamptz, NULL, NULL, 'cli_check', NULL, 'airflow', '{"host_name": "airflow-triggerer-d9b96c859-nk676", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id 2022-06-10 12:15:31.913 GMT [1007] ERROR: relation "log" does not exist at character 13

In my case, the redis and statsd pods are raised with any problem.

the scheduler, triggerer, webserver, and worker pods keep waiting for the init container
2022-06-10 12:16:42,553] {db.py:702} INFO - Waiting for migrations... 59 second(s) Traceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 8, in <module> sys.exit(main()) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main args.func(args) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 136, in check_migrations db.check_migrations(timeout=args.migration_wait_timeout) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 704, in check_migrations f"There are still unapplied migrations after {timeout} seconds. Migration" TimeoutError: There are still unapplied migrations after 60 seconds. MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'b1b348e02d07'}

[potiuk]

That's not enough. There are no logs here indicating a problem. looks like - for whatever reason your migrations did not run. you need to look at the logs and your configuration to see what is wrong there. Also you might simply hit the problem fixed here (if you have not updated your configuration ): #23723 - try with 2.3.2 which should fix it (and I recommend you also to fix any deprecations warning you might see in your logs before attempting to diagnose it).

[lordvcs]

@potiuk I have the same issue and I am willing to share the logs or any other details. I tried getting the logs to pods using kubectl logs <podname> command but it doesnt return anything(because the pods are being constantly deleted and recreated???). I tried using k9s as u had suggested on a different issue, but all I can see is this

Please let me know if there is anything else I can do to view some logs

[potiuk]

If you cannot see airflow pod logs, then I guess you should look at all the logs of your kubernetes cluster.

Unfortunately, this is something that people who manage those clusters (you) would only know how to do., Any other application that you manage with kubernets might have the same problem and there are various ways you can debug and see the reasons why containers was not possible to create.

I do remember when I first looked at various kubernetes-run applications, It was a bit difficult to learn where to look for the logs, but if you chose kubernetes as your deployment, you basically need to learn it. For me k9s helped me a lot when I tried to find out similar problems. Sometimes it required a bit of patience to get the right logs (happened to me sometimes that I had to be patient and try few times). Sometimes it did not.

But how to debug your k8s cluster is beyond my skills to teach you. First of all I have no idea for your particular cluster choice.

Apparently you chose specific K8S version as a deployment mechanism, so you should learn the ropes on how to debug it. I have no idea which cluster you have and what kind of mechanisms it exposes to dump and view all logs, moreover I have no idee what best practices you already use to monitor other apps of yours. If Airflow is your first app to run on Kubernetes, then likely you need to do some training, online course, or maybe just deep dive into documentation and management of your K8S cluster mechanisms. This is not something we can teach you here, especially that ther are different implementations of Kubernetes and you need to know how to manage your own k8s.

I can tell you for example that when you use "kind" cluster (one that we love for development) you can export all logs via "kind export logs" command - and it will dump all the logs of kubernetes. We are using it sometimes to debug similar issues. But This is "kind" - special kubernetes for debugging and development. Not sure if that is useful to you, but maybe similar mechanisms exist in your cluster. Don't know.

You seem to be using different Kubernetes than I usually use. You forgot to mention that you are using a "lightweight" k3s implementation. And k9s has already been useful, because it revealed to me this crucial information, since you posted your k9s screenshot. K3S might, or might not work with airflow in it's default settings. It's by definition "slimmed down", "lightweight" and "iot" -ready k8S implementationa and it might or might not work - depending what "lightweight" means and what has been trimmed donwn and disabled by default.

But you apparently made a conscious choice to use it rather than "standard" kubernetes, installation, so I think the responsibility of at least providing us the information what does not work is on you.

Maybe it simply misses some options - or maybe you need to configure it properly to enable certain features. I do not know much about k3s but getting access to the logs of k3s is crucial so that you can investigate your problem and what you miss. Or maybe you will find that k3s simply lacks some of capabilties. Maybe you just miss resources and you need to increase memory? Or maybe you need to define networking to access something? Or maybe you need to define persitent storage driver? Or maybe ..... there are plenty of possibilities. But without deep dive in logs of the k8s YOU chose, those are only guesses. And you are the only person who can do that and answer it, I am afraid.

I recoommend you to do one of the two things:

• Start with https://k3s.io/ and find out how to get access to all logs, learn to analyse them and see what is the problem
• Look at the "standard" kubernetes installation, not a lightweight one - this will likely work out-of-the box, but you still you will have to learn the same kind of things.

I would love to help more, but simply - you are the only person in the world who can help yourself.