Configure cloud and BYOC log routing

docs/ARCHITECTURE.md

@@ -49,7 +49,7 @@ flowchart TB
     subgraph datastores["State"]
         PG[("PostgreSQL<br/>teams, templates, builds, snapshots")]
         RD[("Redis<br/>running sandboxes, routing catalog, caches")]
-        CH[("ClickHouse<br/>metrics, events, optional logs")]
+        CH[("ClickHouse<br/>metrics, events, customer logs by default")]
         OS[("Object storage GCS/S3<br/>template + snapshot artifacts")]
     end
 
@@ -123,10 +123,9 @@ The control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, p
   (templates, builds, snapshots, teams) live in Postgres.
 - **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so
   client-proxy can wake paused sandboxes on incoming traffic.
-- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to
-  Loki, with a LaunchDarkly-gated ClickHouse read path for local-cluster logs during the log
-  storage migration. LaunchDarkly feature flags also gate placement parameters, rate limits, and
-  rollouts.
+- Reads ClickHouse for sandbox/team metrics and, when `CLICKHOUSE_LOGS_READ_ENABLED` is true or
+  unset, sandbox and template-build logs. When it is false, those logs use Loki only. Log backend
+  selection is not controlled by LaunchDarkly; the read environment setting is the sole switch.
 
 ### Orchestrator (`packages/orchestrator`)
 
@@ -163,9 +162,18 @@ Key mechanisms (all under `pkg/sandbox/`):
 - **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to
   the sandbox's slot IP and requested port, enforcing per-sandbox traffic access tokens.
 - Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via
-  OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy
-  collector remains the fallback primary destination, and configured shadow destinations can mirror
-  writes during collector/storage migrations without changing sandbox behavior.
+  OTel. Sandbox, template-build, and envd log writes select one fixed destination using
+  `CLICKHOUSE_LOGS_WRITE_ONLY`: true sends directly to the existing local otel-router
+  `/logs/clickhouse` writer endpoint, while false sends exactly once to `LOGS_COLLECTOR_ADDRESS`;
+  logs-collector owns the Loki and ClickHouse fan-out. There is no read fallback. Read and write
+  settings are independent, so all four combinations are valid for BYOC migrations.
+
+The application code defaults are BYOC-safe: Loki reads and exactly-once writes to logs-collector.
+The collector/system owns the dual write to Loki and ClickHouse. Cloud staging/foxtrot/juliett/
+tango deployments explicitly override both settings to `true` for ClickHouse-only reads and
+writes. BYOC deployments explicitly set `CLICKHOUSE_LOGS_READ_ENABLED=false` and
+`CLICKHOUSE_LOGS_WRITE_ONLY=false`. Setting `CLICKHOUSE_LOGS_READ_ENABLED=true` enables BYOC
+ClickHouse reads; setting `CLICKHOUSE_LOGS_WRITE_ONLY=true` changes BYOC writes to ClickHouse-only.
 
 ### Envd (`packages/envd`)
 
@@ -256,7 +264,7 @@ planes today.
 |---|---|---|
 | **PostgreSQL** | `packages/db` (goose migrations, sqlc) | Durable control-plane state: `teams`, `users`, `tiers` (quota defaults), `project_limits` (per-team quota overrides pushed in by the owning service; the `team_limits` view reads it in preference to `tiers`), `envs` (templates), `env_builds` (build rows: vcpu, ram_mb, status, versions), `env_aliases`, `snapshots` (paused sandboxes), `team_api_keys`, `access_tokens`, `volumes`, `clusters` |
 | **Redis** | API, client-proxy, orchestrator | Ephemeral runtime state: running-sandbox store (source of truth), sandbox→node routing catalog, team/template/snapshot caches, rate limiting, P2P chunk peer registry |
-| **ClickHouse** | `packages/clickhouse` | Time-series/analytics: `metrics_gauge`/`metrics_sum` (written by the OTel collector), `sandbox_events`, `sandbox_host_stats` (written by orchestrator), team metrics, and optionally `sandbox_logs` during the log migration. Read by API and dashboard-api |
+| **ClickHouse** | `packages/clickhouse` | Time-series/analytics: `metrics_gauge`/`metrics_sum` (written by the OTel collector), `sandbox_events`, `sandbox_host_stats` (written by orchestrator), team metrics, and customer `sandbox_logs` when `CLICKHOUSE_LOGS_READ_ENABLED` is true. Read by API and dashboard-api |
 | **Object storage** (GCS/S3/local, `packages/shared/pkg/storage`) | orchestrator, template-manager | Template & snapshot artifacts, keyed by build ID: `{buildID}/memfile`, `{buildID}/rootfs.ext4`, `{buildID}/snapfile`, `{buildID}/metadata.json` + `.header` index files |
 
 A template and a paused-sandbox snapshot have the **same artifact shape** — a snapshot is just a
@@ -466,9 +474,12 @@ flowchart TB
 - PostgreSQL is external (connection string via secrets); Redis runs as a Nomad job or as a
   managed service; ClickHouse runs on its own pool.
 - Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics)
-  and Grafana Cloud/stack. Logs default to the legacy Vector → Loki path; dynamic log routing can
-  select a primary collector and shadow collectors, and local-cluster log reads can be switched to
-  ClickHouse with `logs-read-config` after `sandbox_logs` is populated.
+  and Grafana Cloud/stack. Application log writers route directly to the existing local otel-router
+  `/logs/clickhouse` endpoint when `CLICKHOUSE_LOGS_WRITE_ONLY` is true. When false, the sole
+  application destination is `LOGS_COLLECTOR_ADDRESS`; logs-collector owns the dual write to
+  Loki and ClickHouse. The otel-router listener is an existing external deployment contract, not created by
+  this repository; true requires the colocated listener and intentionally fails/drops rather than
+  falling back to Loki when unavailable.
 
 ## Repository layout
 

iac/provider-aws/main.tf

@@ -126,6 +126,8 @@ locals {
 
     LOKI_URL                     = "http://loki.service.consul:${local.loki_port}"
     CLICKHOUSE_CONNECTION_STRING = local.clickhouse_connection_string
+    CLICKHOUSE_LOGS_READ_ENABLED = "false"
+    CLICKHOUSE_LOGS_WRITE_ONLY   = "false"
 
     LOGS_COLLECTOR_ADDRESS       = "http://localhost:${local.logs_proxy_port}"
     OTEL_COLLECTOR_GRPC_ENDPOINT = "localhost:${local.otel_collector_port}"
@@ -173,6 +175,7 @@ locals {
     OTEL_COLLECTOR_GRPC_ENDPOINT = "localhost:${local.otel_collector_port}"
     ALLOW_SANDBOX_INTERNAL_CIDRS = var.allow_sandbox_internal_cidrs
     CLICKHOUSE_CONNECTION_STRING = local.clickhouse_connection_string
+    CLICKHOUSE_LOGS_WRITE_ONLY   = "false"
     REDIS_POOL_SIZE              = "10"
     REDIS_CLUSTER_URL            = local.redis_cluster_url
     REDIS_TLS_CA_BASE64          = local.redis_tls_ca_base64
@@ -210,6 +213,7 @@ locals {
     ORCHESTRATOR_SERVICES        = "template-manager"
     REDIS_POOL_SIZE              = "10"
     CLICKHOUSE_CONNECTION_STRING = local.clickhouse_connection_string
+    CLICKHOUSE_LOGS_WRITE_ONLY   = "false"
     GIN_MODE                     = "release"
     LAUNCH_DARKLY_API_KEY        = module.init.launch_darkly_api_key
   }, var.template_manager_env_vars)

packages/api/internal/cfg/model.go

@@ -41,6 +41,8 @@ type Config struct {
 
 	ClickhouseConnectionString  string   `env:"CLICKHOUSE_CONNECTION_STRING"`
 	ClickhouseConnectionStrings []string `env:"CLICKHOUSE_CONNECTION_STRINGS" envSeparator:";"`
+	ClickhouseLogsReadEnabled   bool     `env:"CLICKHOUSE_LOGS_READ_ENABLED"  envDefault:"false"`
+	ClickhouseLogsWriteOnly     bool     `env:"CLICKHOUSE_LOGS_WRITE_ONLY"    envDefault:"false"`
 
 	LokiPassword string `env:"LOKI_PASSWORD"`
 	LokiURL      string `env:"LOKI_URL,required"`

packages/api/internal/cfg/model_test.go

@@ -19,6 +19,38 @@ func TestParse(t *testing.T) {
 	t.Setenv("VOLUME_TOKEN_SIGNING_KEY", fmt.Sprintf("HMAC:%s", base64.StdEncoding.EncodeToString([]byte("secret"))))
 	t.Setenv("VOLUME_TOKEN_SIGNING_KEY_NAME", "my-key-name")
 
+	for _, variable := range []string{"CLICKHOUSE_LOGS_READ_ENABLED", "CLICKHOUSE_LOGS_WRITE_ONLY"} {
+		t.Run(variable+" parses strictly", func(t *testing.T) { //nolint:paralleltest // mutates process environment
+			removeEnv(t, "CLICKHOUSE_LOGS_READ_ENABLED")
+			removeEnv(t, variable)
+			config, err := Parse()
+			require.NoError(t, err)
+			assert.False(t, config.ClickhouseLogsReadEnabled)
+			assert.False(t, config.ClickhouseLogsWriteOnly)
+
+			t.Setenv(variable, "true")
+			config, err = Parse()
+			require.NoError(t, err)
+			if variable == "CLICKHOUSE_LOGS_READ_ENABLED" {
+				assert.True(t, config.ClickhouseLogsReadEnabled)
+				assert.False(t, config.ClickhouseLogsWriteOnly)
+			} else {
+				assert.False(t, config.ClickhouseLogsReadEnabled)
+				assert.True(t, config.ClickhouseLogsWriteOnly)
+			}
+
+			t.Setenv(variable, "false")
+			config, err = Parse()
+			require.NoError(t, err)
+			assert.False(t, config.ClickhouseLogsReadEnabled)
+			assert.False(t, config.ClickhouseLogsWriteOnly)
+
+			t.Setenv(variable, "invalid")
+			_, err = Parse()
+			assert.Error(t, err)
+		})
+	}
+
 	t.Run("postgres connection string is required", func(t *testing.T) { //nolint:paralleltest // cannot call t.Setenv and t.Parallel
 		removeEnv(t, "POSTGRES_CONNECTION_STRING")
 

packages/api/internal/clusters/cluster.go

@@ -19,7 +19,6 @@ import (
 	"github.com/e2b-dev/infra/packages/api/internal/clusters/discovery"
 	clickhouse "github.com/e2b-dev/infra/packages/clickhouse/pkg"
 	"github.com/e2b-dev/infra/packages/shared/pkg/consts"
-	"github.com/e2b-dev/infra/packages/shared/pkg/featureflags"
 	infogrpc "github.com/e2b-dev/infra/packages/shared/pkg/grpc/orchestrator-info"
 	api "github.com/e2b-dev/infra/packages/shared/pkg/http/edge"
 	"github.com/e2b-dev/infra/packages/shared/pkg/logger"
@@ -77,7 +76,6 @@ func newLocalCluster(
 	clickhouse clickhouse.Clickhouse,
 	queryLogsProvider *loki.LokiQueryProvider,
 	sandboxLogsReader ClickhouseLogsReader,
-	featureFlags *featureflags.Client,
 	config cfg.Config,
 ) *Cluster {
 	clusterID := consts.LocalClusterID
@@ -96,7 +94,7 @@ func newLocalCluster(
 		"",
 		instances,
 		synchronization.NewSynchronize("cluster-instances", "Cluster instances", store),
-		newLocalClusterResourceProvider(clickhouse, queryLogsProvider, sandboxLogsReader, featureFlags, instances, config),
+		newLocalClusterResourceProvider(clickhouse, queryLogsProvider, sandboxLogsReader, instances, config),
 	)
 
 	// Periodically sync cluster instances

packages/api/internal/clusters/clusters_sync.go

@@ -14,7 +14,6 @@ import (
 	"github.com/e2b-dev/infra/packages/db/client"
 	"github.com/e2b-dev/infra/packages/db/queries"
 	"github.com/e2b-dev/infra/packages/shared/pkg/consts"
-	"github.com/e2b-dev/infra/packages/shared/pkg/featureflags"
 	"github.com/e2b-dev/infra/packages/shared/pkg/logger"
 	"github.com/e2b-dev/infra/packages/shared/pkg/logs/loki"
 	"github.com/e2b-dev/infra/packages/shared/pkg/smap"
@@ -51,7 +50,6 @@ func NewPool(
 	queryMetricsProvider clickhouse.Clickhouse,
 	queryLogsProvider *loki.LokiQueryProvider,
 	sandboxLogsReader ClickhouseLogsReader,
-	featureFlags *featureflags.Client,
 	config cfg.Config,
 ) (*Pool, error) {
 	clusters := smap.New[*Cluster]()
@@ -75,7 +73,6 @@ func NewPool(
 				queryLogsProvider:    queryLogsProvider,
 				queryMetricsProvider: queryMetricsProvider,
 				sandboxLogsReader:    sandboxLogsReader,
-				featureFlags:         featureFlags,
 			},
 		),
 	}
@@ -120,7 +117,6 @@ type clustersSyncStore struct {
 	queryMetricsProvider clickhouse.Clickhouse
 	queryLogsProvider    *loki.LokiQueryProvider
 	sandboxLogsReader    ClickhouseLogsReader
-	featureFlags         *featureflags.Client
 	config               cfg.Config
 }
 
@@ -178,7 +174,7 @@ func (d clustersSyncStore) PoolInsert(ctx context.Context, cluster queries.Clust
 
 	// Local cluster
 	if cluster.ID == consts.LocalClusterID {
-		c = newLocalCluster(context.WithoutCancel(ctx), d.tel, d.localDiscovery, d.queryMetricsProvider, d.queryLogsProvider, d.sandboxLogsReader, d.featureFlags, d.config)
+		c = newLocalCluster(context.WithoutCancel(ctx), d.tel, d.localDiscovery, d.queryMetricsProvider, d.queryLogsProvider, d.sandboxLogsReader, d.config)
 		d.clusters.Insert(clusterID, c)
 		logger.L().Info(ctx, "Local cluster initialized successfully", logger.WithClusterID(cluster.ID))
 

packages/api/internal/clusters/resources_local.go

@@ -2,6 +2,7 @@ package clusters
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"net/http"
 	"time"
@@ -17,7 +18,6 @@ import (
 	clickhouse "github.com/e2b-dev/infra/packages/clickhouse/pkg"
 	"github.com/e2b-dev/infra/packages/clickhouse/pkg/sandboxlogs"
 	clickhouseutils "github.com/e2b-dev/infra/packages/clickhouse/pkg/utils"
-	"github.com/e2b-dev/infra/packages/shared/pkg/featureflags"
 	"github.com/e2b-dev/infra/packages/shared/pkg/logs"
 	"github.com/e2b-dev/infra/packages/shared/pkg/logs/loki"
 	"github.com/e2b-dev/infra/packages/shared/pkg/smap"
@@ -64,15 +64,13 @@ type LocalClusterResourceProvider struct {
 	querySandboxMetricsProvider clickhouse.SandboxQueriesProvider
 	queryLogsProvider           *loki.LokiQueryProvider
 	sandboxLogsReader           ClickhouseLogsReader
-	featureFlags                *featureflags.Client
 	instances                   *smap.Map[*Instance]
 }
 
 func newLocalClusterResourceProvider(
 	querySandboxMetricsProvider clickhouse.SandboxQueriesProvider,
 	queryLogsProvider *loki.LokiQueryProvider,
 	sandboxLogsReader ClickhouseLogsReader,
-	featureFlags *featureflags.Client,
 	instances *smap.Map[*Instance],
 	config cfg.Config,
 ) ClusterResource {
@@ -81,20 +79,12 @@ func newLocalClusterResourceProvider(
 		querySandboxMetricsProvider: querySandboxMetricsProvider,
 		queryLogsProvider:           queryLogsProvider,
 		sandboxLogsReader:           sandboxLogsReader,
-		featureFlags:                featureFlags,
 		instances:                   instances,
 	}
 }
 
-// readFromClickhouse reports whether log reads should hit ClickHouse. It is
-// true only when the logs-read-config flag is enabled AND a ClickHouse reader
-// is configured; otherwise reads stay on Loki (default behavior).
-func (l *LocalClusterResourceProvider) readFromClickhouse(ctx context.Context) bool {
-	if l.sandboxLogsReader == nil || l.featureFlags == nil {
-		return false
-	}
-
-	return l.featureFlags.BoolFlag(ctx, featureflags.LogsReadConfigFlag)
+func (l *LocalClusterResourceProvider) readFromClickhouse() bool {
+	return l.config.ClickhouseLogsReadEnabled
 }
 
 // apiLogDirectionToSandboxLogsSortOrder converts an API log direction into the
@@ -203,7 +193,10 @@ func (l *LocalClusterResourceProvider) GetSandboxLogs(ctx context.Context, teamI
 		raw []logs.LogEntry
 		err error
 	)
-	if l.readFromClickhouse(ctx) {
+	if l.readFromClickhouse() {
+		if l.sandboxLogsReader == nil {
+			return api.SandboxLogs{}, &api.APIError{Err: errors.New("ClickHouse logs reader is unavailable"), ClientMsg: "Failed to fetch sandbox logs", Code: http.StatusInternalServerError}
+		}
 		teamUUID, parseErr := uuid.Parse(teamID)
 		if parseErr != nil {
 			return api.SandboxLogs{}, &api.APIError{
@@ -258,12 +251,11 @@ func (l *LocalClusterResourceProvider) GetBuildLogs(
 	direction api.LogsDirection,
 	source *api.LogsSource,
 ) ([]logs.LogEntry, *api.APIError) {
-	// The persistent log backend is Loki by default, ClickHouse when the
-	// logs-read-config flag is enabled and a ClickHouse reader is configured.
+	// The persistent log backend is selected exclusively by CLICKHOUSE_LOGS_READ_ENABLED.
 	start, end := LogQueryWindow(cursor, direction)
 
 	var persistentFetcher logSourceFunc
-	if l.readFromClickhouse(ctx) {
+	if l.readFromClickhouse() {
 		persistentFetcher = l.logsFromClickhouse(ctx, templateID, buildID, start, end, int(limit), offset, level, apiLogDirectionToSandboxLogsSortOrder(&direction))
 	} else {
 		persistentFetcher = l.logsFromLocalLoki(ctx, templateID, buildID, start, end, int(limit), offset, level, apiLogDirectionToLokiProtoDirection(&direction))
@@ -274,6 +266,9 @@ func (l *LocalClusterResourceProvider) GetBuildLogs(
 
 func (l *LocalClusterResourceProvider) logsFromClickhouse(ctx context.Context, templateID string, buildID string, start time.Time, end time.Time, limit int, offset int32, level *logs.LogLevel, order sandboxlogs.SortOrder) logSourceFunc {
 	return func() ([]logs.LogEntry, *api.APIError) {
+		if l.sandboxLogsReader == nil {
+			return nil, &api.APIError{Err: errors.New("ClickHouse logs reader is unavailable"), ClientMsg: "Failed to fetch build logs", Code: http.StatusInternalServerError}
+		}
 		entries, err := l.sandboxLogsReader.QueryBuildLogs(ctx, templateID, buildID, start, end, limit, offset, level, order)
 		if err != nil {
 			recordClickhouseLogReadError(ctx, "build")