Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unify cluster alter commands #628

Merged
merged 1 commit into from
Aug 19, 2024
Merged

unify cluster alter commands #628

merged 1 commit into from
Aug 19, 2024

Conversation

jubrad
Copy link
Contributor

@jubrad jubrad commented Aug 8, 2024

Unifying cluster alter commands. This is a pre-req for adding graceful reconfiguration.

@jubrad jubrad requested review from bobbyiliev and a team as code owners August 8, 2024 21:29
@jubrad jubrad requested review from arusahni and removed request for a team August 8, 2024 21:29
@jubrad jubrad force-pushed the unify-cluster-alter branch 3 times, most recently from 628a016 to 268c887 Compare August 9, 2024 14:24
@jubrad
Copy link
Contributor Author

jubrad commented Aug 13, 2024

Test this with the following setup:

./terraformrc

provider_installation {
  dev_overrides {
    "MaterializeInc/materialize" = "/Users/justin/go/bin/"
  }
  direct {}
}
terraform {
  required_providers {
    materialize = {
      source = "MaterializeInc/materialize"
    }
  }
}

provider "materialize" {
  password       = "dontlookatmypassworddood"
  default_region = "aws/us-east-1"
  database       = "materialize"
  cloud_endpoint = "https://api.staging.cloud.materialize.com"
  base_endpoint  = "https://admin.staging.cloud.materialize.com"
  endpoint       = "https://admin.staging.cloud.materialize.com"
}

resource "materialize_cluster" "test" {
  name               = "test"
  size               = "25cc"
  replication_factor = 2
}

Then changed the size/replication factor a few ways..
I can see in the sql lifecycle on the console that it ran successfully in a single statement, where using the latest release was multiple statements.
image

@jubrad jubrad mentioned this pull request Aug 14, 2024
Comment on lines 239 to 242
} else {
// Reset the schedule settings if not enabled.
q = fmt.Sprintf("ALTER CLUSTER %s RESET (SCHEDULE);", b.QualifiedName())
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Thanks a lot @jubrad!

The only issue that I've noticed while manually testing this is that if you were to change the scheduling from enabled true to false, it generates the following alter statement which will fail:

ALTER CLUSTER "scheduling_cluster" SET ();

Repro:

resource "materialize_cluster" "scheduling_cluster" {
  name = "scheduling_cluster"
  size = "50cc"
  scheduling {
    on_refresh {
      enabled                 = true # Apply and then change to false and apply again
      hydration_time_estimate = "3 hour"
    }
  }
}

@jubrad jubrad force-pushed the unify-cluster-alter branch from 268c887 to cbcc42e Compare August 19, 2024 17:21
@jubrad jubrad requested a review from bobbyiliev August 19, 2024 17:21
@jubrad jubrad force-pushed the unify-cluster-alter branch from cbcc42e to 6acf49d Compare August 19, 2024 17:24
Comment on lines +194 to +200
func (b *ClusterBuilder) AlterClusterScheduling(s SchedulingConfig) error {
q := strings.Builder{}
q.WriteString(fmt.Sprintf(`ALTER CLUSTER %s`, b.QualifiedName()))
q.WriteString(fmt.Sprintf(` SET (%s)`, b.GenSchedulingConfigSql(s)))
q.WriteString(`;`)
return b.ddl.exec(q.String())
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bobbyiliev the plan now is that we do AlterClusterScheduling separate from AlterCluster and always first.

This will allow us to from a cluster with scheduling and no replication factor to a cluster without scheduling and a replication factor, and vice versa.

Note, we only really set the scheduling option on the ClusterBuilder directly in the create phase where everything can be done in a single statement.

@jubrad jubrad force-pushed the unify-cluster-alter branch from 6acf49d to 4c935b9 Compare August 19, 2024 18:04
Copy link
Contributor

@bobbyiliev bobbyiliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that this looks good! Thank you for putting this all together @jubrad 👏

@jubrad jubrad merged commit a5cc884 into main Aug 19, 2024
4 checks passed
@bobbyiliev bobbyiliev deleted the unify-cluster-alter branch August 25, 2024 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants