5xx errors from UI from sporadic code location updates

[skwon615]

Hello! We deploy Dagster open source on ECS, where each gRPC server code location is deployed from a separate ECS service. We recently migrated from v1.6.9 to v1.7.5, and started hitting issues where some (but not all) deployments on a specific code location will trigger Dagster to reload all definitions (across all code locations). When that happens, we get 502/504 errors when accessing the UI for a few minutes.

I checked that it's not a problem with any individual code location; if I reload all code locations (31 total) individually, everything still works. We did notice that during the above, a new ECS task for dagit & daemon gets spun up; but I'm not sure if that's expected to cause the 5xx issues. Also, if it matters, we typically use the dagster api grpc command to start the gRPC servers, though we started moving some (but not all) code locations to use dagster code-server start instead.

Can you help us figure out what's going on? Thanks in advance!

[garethbrickman]

Relates to #14709

[skwon615]

Thanks @garethbrickman - does that mean using dagster code-server start everywhere + increasing the timeout thresholds should be enough to fix this going forward?

[garethbrickman]

I'm not positive but those are definitely worth doing!