Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I can use helm cli to deploy the crane to a GKE cluster but not via terraform #802

Open
csharpknife2002 opened this issue May 25, 2023 · 2 comments
Labels
kind/bug Something isn't working

Comments

@csharpknife2002
Copy link

csharpknife2002 commented May 25, 2023

Describe the bug
In the same GKE cluster, I can use helm cli to install crane successfuly via the following cmd:

helm install gocrane -n crane-test --create-namespace  crane/crane --set craned.containerArgs.prometheus-address=http://chen-prometheus-server.prometheus.svc.cluster.local:8080  --debug

But it always failed when I use terraform scripts, (I have no problem to deploy prometheus, granfana and other apps via TF to the same cluster


data "terraform_remote_state" "gke" {
  backend = "gcs" 
  config = {
    bucket  = "xxx-terraform-states"
    prefix  = "cluster"
  }
}

provider "google" {
  project = data.terraform_remote_state.gke.outputs.project_id
  region  = data.terraform_remote_state.gke.outputs.region
}

data "google_client_config" "default" {}

data "google_container_cluster" "my_cluster" {
  name     = data.terraform_remote_state.gke.outputs.kubernetes_cluster_name
  location = data.terraform_remote_state.gke.outputs.zone
  project = data.terraform_remote_state.gke.outputs.project_id
}

provider "helm" {
  kubernetes {
    host = data.terraform_remote_state.gke.outputs.kubernetes_cluster_host
    token = data.google_client_config.default.access_token
    cluster_ca_certificate = base64decode(data.google_container_cluster.my_cluster.master_auth[0].cluster_ca_certificate)
  }
  debug = true
}

provider "kubernetes" {
  host                   = "https://${data.terraform_remote_state.gke.outputs.kubernetes_cluster_host}"
  token                  = "${data.google_client_config.default.access_token}"
  cluster_ca_certificate = "${base64decode(data.google_container_cluster.my_cluster.master_auth.0.cluster_ca_certificate)}"
}

provider "kubectl" {
  load_config_file       = false
  host                   = "https://${data.terraform_remote_state.gke.outputs.kubernetes_cluster_host}"
  token                  = "${data.google_client_config.default.access_token}"
  cluster_ca_certificate = "${base64decode(data.google_container_cluster.my_cluster.master_auth.0.cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "gocrane_ns" {
  metadata {
    name = "crane-system"
  }
}

resource "helm_release" "grafana-gocrane" {
  name  = "grafana-gocrane"
  repository = "https://grafana.github.io/helm-charts"
  chart = "grafana"

  timeout = 120
  cleanup_on_fail = true
  force_update    = false
  namespace       = kubernetes_namespace.gocrane_ns.metadata.0.name
  # version = "6.11.0"


  depends_on = [ kubernetes_namespace.gocrane_ns]

  values = [
    file("${path.module}/grafana_override_values.yaml")
  ]
}

resource "helm_release" "gocrane" {
  name  = "gocrane"
  repository = "https://gocrane.github.io/helm-charts"
  chart = "crane"

  timeout = 300
  cleanup_on_fail = true
  force_update    = false
  namespace       = kubernetes_namespace.gocrane_ns.metadata.0.name
  

  set {
    name = "craned.containerArgs.prometheus-address"
    value = "http://chen-prometheus-server.prometheus.svc.cluster.local:8080"
  }

  depends_on = [ helm_release.grafana-gocrane ]
}

resource "helm_release" "fadvisor" {
  name  = "fadvisor"
  repository = "https://gocrane.github.io/helm-charts"
  chart = "fadvisor"

  timeout = 120
  cleanup_on_fail = true
  force_update    = false
  namespace       = kubernetes_namespace.gocrane_ns.metadata.0.name

  set {
    name = "craned.containerArgs.prometheus-address"
    value = "http://chen-prometheus-server.prometheus.svc.cluster.local:8080"
  }

  depends_on = [helm_release.gocrane]
}



it always failed with the following logs, regardless how long I set the timeout. (with the cli, the installatio take a few seconds)

Warning: Helm release "gocrane" was created but has a failed status. Use the helm command to investigate the error, correct it, then run Terraform again.

│ 
│   with helm_release.gocrane,
│   on main.tf line 101, in resource "helm_release" "gocrane":
│  101: resource "helm_release" "gocrane" {
│ 
╵
╷
│ Error: timed out waiting for the condition
│ 
│   with helm_release.gocrane,
│   on main.tf line 101, in resource "helm_release" "gocrane":
│  101: resource "helm_release" "gocrane" 

Reproduce steps

Expected behavior

Screenshots
image

Environment (please complete the following information):

  • K8S Version: [e.g. 1.19]
    Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.8-gke.500", GitCommit:"f117e29cb87cfb7e1de32ab4e163fb01ac5d0af9", GitTreeState:"clean", BuildDate:"2023-03-23T10:22:38Z", GoVersion:"go1.19.7 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
  • Crane Version: [e.g. 0.1.0]
  • 0.10.0
  • Browser [e.g. chrome, safari]
@csharpknife2002 csharpknife2002 added the kind/bug Something isn't working label May 25, 2023
@qmhu
Copy link
Member

qmhu commented May 26, 2023

Is the chart download blocked by network?

@clin4
Copy link

clin4 commented Jun 14, 2023

@qmhu , no, the download was success. Actually I didn't see any differences between use helm cli and terraform. I got all the things as expected, Except terraform will fail after a while, which I have not clue what it is failing for... (or what it is waiting for...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants