[Failing Test]: TestAPMConfig – no such index [traces-apm-default] #5890

ycombinator · 2024-10-30T16:22:51Z

Failing test case

TestAPMConfig

Error message

apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4075#0192dd03-00ea-41c0-a56a-a5edb99318eb
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4077#0192dd14-e4c3-4901-b0d9-f50065d445b5
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4086#0192dda7-21c4-4d6a-9ce7-2c9a9ac4a3e3

OS

Linux, Windows

Stacktrace and notes

=== RUN   TestAPMConfig
    fetcher.go:93: Using existing artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:288: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAPMConfig4001669197
    fixture.go:306: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAPMConfig4001669197
    fixture.go:963: All component specifications where removed
    fixture.go:1004: Placed component specifications: fake
    apm_propagation_test.go:72: Rendered agent config:
        
        outputs:
          default:
            type: fake-output
        inputs:
          - id: fake-apm
            type: fake-apm
        agent.monitoring:
          traces: true
          apm:
            hosts:
              - https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443
            environment: nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu
            secret_token: AeKIQKeIi5tKM2Wt9s
            global_labels:
              test_name: TestAPMConfig
              test_type: Agent integration test
            tls:
              skip_verify: true
    fixture.go:1234: 2024-10-30T14:19:57.114200275Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 77.874125ms
    fixture.go:1234: 2024-10-30T14:19:57.192670596Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 134.287776ms
    log.go:85: Flag --disable-encrypted-store has been deprecated, agent on Mac OS can be run/installed without root privileges, see elastic-agent install --help
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:57.323Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":21903,"agent.version":"9.0.0","agent.unprivileged":true,"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:57.327931667Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 136.304281ms
    fixture.go:1234: 2024-10-30T14:19:57.465385895Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 278.736473ms
    fixture.go:1234: 2024-10-30T14:19:57.745135101Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 526.499465ms
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.026Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.029Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.030Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":62},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.031Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":68},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["fake","fake-apm","fake-isolated-units"],"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.032Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.032Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":74},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:58.272176018Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 1.087157926s
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.478Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":175},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.478Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":130},"message":"Elastic Agent has been started in testing mode and is managed through the control protocol","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.480Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock","log":{"source":"elastic-agent"},"address":"unix:///tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.481Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":238},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:19:58.482Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1234: 2024-10-30T14:19:59.359729146Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/TestAPMConfig4001669197/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent.sock: connect: no such file or directory" retrying: 1.588421624s
    fixture.go:1242: 2024-10-30T14:20:00.948864551Z: StateWatch started
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.951Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.(*testingModeConfigManager).SetConfig","file.name":"application/testing_mode.go","file.line":53},"message":"Testing mode received new configuration from protocol, passing to coordinator","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.953Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0022b4360), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu\", APIKey:\"\", SecretToken:\"AeKIQKeIi5tKM2Wt9s\", Hosts:[]string{\"https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"test_name\":\"TestAPMConfig\", \"test_type\":\"Agent integration test\"}, TLS:config.APMTLS{SkipVerify:true, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.954Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.955Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:00.971Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1270},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":625},"message":"Spawned new component fake-apm-default: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":632},"message":"Spawned new unit fake-apm-default-fake-apm: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"unit":{"id":"fake-apm-default-fake-apm","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.227Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":632},"message":"Spawned new unit fake-apm-default: Starting: spawned pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"STARTING"},"unit":{"id":"fake-apm-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.248Z","log.logger":"component.runtime.fake-apm-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":704},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.248Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":645},"message":"Component state changed fake-apm-default (STARTING->HEALTHY): Healthy: communicating with pid '21913'","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.251Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.257Z","message":"Starting fake APM traces sender with config elastic:{tls:{skip_verify:true}  environment:\"nsi3gya7ccprbvh2xzveb5ytfy0beb0csqxcuftfqu\"  secret_token:\"AeKIQKeIi5tKM2Wt9s\"  hosts:\"https://9c26ecf86b6347b1b8363bce0edebab6.apm.us-west2.gcp.elastic-cloud.com:443\"  global_labels:\"test_name=TestAPMConfig,test_type=Agent integration test\"}","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.257Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.258Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fake-apm-default-fake-apm (STARTING->HEALTHY): Fake APM traces sender has started","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY"},"unit":{"id":"fake-apm-default-fake-apm","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.258Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fake-apm-default (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"fake-apm-default","state":"HEALTHY"},"unit":{"id":"fake-apm-default","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    apm_propagation_test.go:78: agent state: &{Info:{ID:ed947dca-69ee-484f-ab24-ec1b22214be7 Version:9.0.0 Commit:e6f281aa4de18d5ba1aa10aa36da9796c3b22a47 BuildTime:2024-10-30 13:48:12 +0000 UTC Snapshot:true PID:21903 Unprivileged:true} State:HEALTHY Message:Running Components:[{ID:fake-apm-default Name:fake-apm State:HEALTHY Message:Healthy: communicating with pid '21913' Units:[{UnitID:fake-apm-default UnitType:OUTPUT State:HEALTHY Message:Healthy Payload:map[]} {UnitID:fake-apm-default-fake-apm UnitType:INPUT State:HEALTHY Message:Fake APM traces sender has started Payload:map[]}] VersionInfo:{Name:fake Meta:map[input:fake]}}] FleetState:STOPPED FleetMessage:Not enrolled into Fleet UpgradeDetails:}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:01.425Z","message":"config request failed: request failed with 503 Service Unavailable: {\"ok\":false,\"message\":\"The requested resource is currently unavailable.\"}","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"error","time":"2024-10-30T14:20:01Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:12.419Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~0s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:12Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:23.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~1s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:23Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:31.261Z","message":"gathering metrics","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:31Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:34.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~4s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:34Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:45.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~9s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:45Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:56.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~16s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:56Z","ecs.version":"1.6.0"}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:184: received ES response: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"}],"type":"index_not_found_exception","reason":"no such index [traces-apm-default]","resource.type":"index_or_alias","resource.id":"traces-apm-default","index_uuid":"_na_","index":"traces-apm-default"},"status":404}
    apm_propagation_test.go:81: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/apm_propagation_test.go:81
        	            				/home/ubuntu/agent/pkg/testing/machine.go:230
        	            				/home/ubuntu/agent/pkg/testing/fixture.go:608
        	            				/home/ubuntu/agent/pkg/testing/fixture.go:646
        	            				/home/ubuntu/agent/testing/integration/apm_propagation_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestAPMConfig
    fixture.go:1282: Temporary directory "/tmp/TestAPMConfig4001669197" preserved for investigation/debugging
--- FAIL: TestAPMConfig (74.56s)

This test is failing on main from 053b320 onwards. Note that main is pointing to 9.0.0 now. It is not failing on the 8.x branch.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2024-10-30T16:22:53Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

pchila · 2024-10-30T16:27:22Z

Most likely it's not a flaky test, rather APM server is unhealthy on the ESS deployment, as shown by the log from the tracer below

 log.go:96: {"log.level":"info","@timestamp":"2024-10-30T14:20:23.410Z","message":"request failed: request failed with 502 Bad Gateway: {\"ok\":false,\"message\":\"The instance rejected the connection.\"} (next request in ~1s)","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-apm-default","type":"fake-apm"},"log":{"source":"fake-apm-default"},"level":"debug","time":"2024-10-30T14:20:23Z","ecs.version":"1.6.0"}

ycombinator · 2024-10-30T16:29:07Z

Thanks, I agree. It's failed 3 builds in a row, so it's unlikely to be flaky. I'll try to spin up an ESS deployment on 9.0.0 and check the health of APM server.

ycombinator · 2024-10-30T16:42:36Z

Indeed, APM Server on 9.0.0 in ESS CFT is unhealthy right now:

@simitt Is this something the APM Server team is already aware of?

kruskall · 2024-10-30T20:52:32Z

Hey @ycombinator 👋

This might be related to the beats bump, let me quickly check

kruskall · 2024-10-30T23:15:27Z

According to the logs in cloud the error is

Error: error loading config file: stat apm-server.yml: no such file or directory

Downloading the docker image shows the apm-server config and binary owned by 1000:1000. Did anything change in the recent version ? What user is used to run apm-server ?

ycombinator · 2024-10-31T00:11:37Z

Did anything change in the recent version?

Something must've changed very recently because, as of a5de320 (created October 30 00:46 UTC), this test was passing.

ycombinator · 2024-10-31T00:14:30Z

According to the logs in cloud the error is
Error: error loading config file: stat apm-server.yml: no such file or directory
Downloading the docker image shows the apm-server config and binary owned by 1000:1000. Did anything change in the recent version ? What user is used to run apm-server ?

BTW, this looks mighty similar to #5615.

1pkg · 2024-10-31T01:23:28Z

Seems like it's neither file owner nor permissions this time, the output is identical from both
docker run elastic-agent-cloud:9.0.0-SNAPSHOT
and
docker run elastic-agent-cloud:8.16.0-SNAPSHOT

ls -la apm*
-rwxr-xr-x    1 elastic-agent elastic-agent  53936280 Oct 30 17:08 apm-server
-rw-r--r--    1 elastic-agent elastic-agent       542 Oct 30 17:08 apm-server.spec.yml
-rw-r--r--    1 elastic-agent elastic-agent     39322 Oct 30 17:08 apm-server.yml

ycombinator · 2024-10-31T22:45:11Z

This TestAPMConfig test has now started failing on the 8.x branch too, which was not the case yesterday when this issue was created for the main branch.

Checking ESS CFT region, similar to #5890 (comment) (which was pertaining to 9.0.0-SNAPSHOT), now even 8.17.0-SNAPSHOT deployments are showing an unhealthy APM Server:

So clearly some change that got merged on main and deployed to ESS CFT yesterday, and then the same change got merged to 8.x and deployed on ESS CFT today, causing the same failure.

ycombinator · 2024-10-31T22:51:11Z

@kruskall referring to my previous comment, could the changes in question be elastic/apm-server#14469 (for main) and elastic/apm-server#14470 (for 8.x)?

1pkg · 2024-10-31T23:36:12Z

@ycombinator we were suspecting that it could be related to the beats update, your comment from here #5890 (comment) is reaffirming it. With a local tests however I cannot reproduce this regression, let me think of another way to test this.

1pkg · 2024-11-01T00:24:42Z

@ycombinator I was able to identify the root cause of the error finally. It's 100% related to the beats update we did. After the update this condition resolves to false in ESS now.

I will try to check the history to find the changes in libbeat caused this regression.

1pkg · 2024-11-01T00:46:36Z

Okay the changes from https://github.com/elastic/beats/pull/41277/files#diff-aec28a4e07878561b81c713b3da56350499ea5fbb44eac866430a249d25acffdR162 are not parsing the management.enabled flag as we would expect. I will try to create a patch, in the meanwhile I'm not really sure if we want to wait for the patch to be tested and merged or we should go ahead and revert beats version for now.

CC @kruskall @simitt @axw

kruskall · 2024-11-01T00:48:24Z

referring to my previous comment, could the changes in question be elastic/apm-server#14469 (for main) and elastic/apm-server#14470 (for 8.x)?

yep, as @1pkg mentioned, that was the initial assumption but that commit is also in 8.16 which is not affected by the issue. We discussed it today and it could be a change in beats that was not backported to 8.16

kruskall · 2024-11-01T00:49:52Z

Okay the changes from https://github.com/elastic/beats/pull/41277/files#diff-aec28a4e07878561b81c713b3da56350499ea5fbb44eac866430a249d25acffdR162 are not parsing the management.enabled flag as we would expect.

looks like that PR was not backported to 8.16 so that could be the issue 😅
Good catch!

ycombinator · 2024-11-01T09:35:34Z

Okay the changes from https://github.com/elastic/beats/pull/41277/files#diff-aec28a4e07878561b81c713b3da56350499ea5fbb44eac866430a249d25acffdR162 are not parsing the management.enabled flag as we would expect.

cc'ing @leehinman for awareness, as the author of elastic/beats#41277.

leehinman · 2024-11-01T14:26:38Z

@kruskall @simitt @axw @1pkg sorry this isn't working

I had tested apm-server -E management.enabled=true and the server started waiting for config on stdin. What behavior are you seeing? And let me know if you want any help.

leehinman · 2024-11-01T14:54:06Z

@kruskall @simitt @axw @1pkg

I had tested apm-server -E management.enabled=true and the server started waiting for config on stdin. What behavior are you seeing? And let me know if you want any help.

Ugghhh. I see why my test was insufficient. There was still a apm-server.yml file in the directory. So while the management flag was parsed, and it was waiting for config, it also had read the config.

This is because APM is Loading the config file early. https://github.com/elastic/apm-server/blob/cb9f315835a3f1c2ba262d07bcdc118568d2d245/internal/beatcmd/config.go#L74

putting a cfgfile.HandleFlags before the cfgfile.Load fixes it.

1pkg · 2024-11-01T16:22:45Z

@leehinman we call cfgfile.HandleFlags before the cfgfile.Load see https://github.com/elastic/apm-server/blob/cb9f315835a3f1c2ba262d07bcdc118568d2d245/internal/beatcmd/init.go#L65. The problem is however that when we call cfgfile.HandleFlags in init flags are not yet parsed and therefore -E flags do no resolve here as expected https://github.com/elastic/beats/blob/main/libbeat/cfgfile/cfgfile.go#L166. It then caused this check to fail for us https://github.com/elastic/beats/blob/main/libbeat/cfgfile/cfgfile.go#L208.

Before the recent changes fleetmode.Enabled() was parsing lazily which worked fine for us since it was called after the call to flag.Parse in our root cobra cli but now there is a regression.

leehinman · 2024-11-01T16:41:18Z

@leehinman we call cfgfile.HandleFlags before the cfgfile.Load see https://github.com/elastic/apm-server/blob/cb9f315835a3f1c2ba262d07bcdc118568d2d245/internal/beatcmd/init.go#L65. The problem is however that when we call cfgfile.HandleFlags in init flags are not yet parsed and therefore -E flags do no resolve here as expected https://github.com/elastic/beats/blob/main/libbeat/cfgfile/cfgfile.go#L166. It then caused this check to fail for us https://github.com/elastic/beats/blob/main/libbeat/cfgfile/cfgfile.go#L208.

Did you try putting cfgfile.HandleFlags just prior to cfgfile.Load? In my test it works because flags.Parse has been called at that point. Calling it twice doesn't cause any problems. We had previously tried moving flags.Parse into initFlags in apm-server but that caused other regressions.

1pkg · 2024-11-01T16:51:30Z

@leehinman yes I think the ordering for working with flags parsing in APM Server is off right now. I just opened the fix to move cfgfile.HandleFlags after the cobra cli calls flag.Parse(). See elastic/apm-server#14512

1pkg · 2024-11-05T00:21:13Z

@leehinman yes I think the ordering for working with flags parsing in APM Server is off right now. I just opened the fix to move cfgfile.HandleFlags after the cobra cli calls flag.Parse(). See elastic/apm-server#14512

The APM Server fix was merged, the tests should be passing again.

CC @ycombinator @pchila

ycombinator · 2024-11-11T19:42:41Z

Thanks @1pkg. I've put up #5993 to revert the test skip.

ycombinator added flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team labels Oct 30, 2024

ycombinator mentioned this issue Oct 30, 2024

Skip TestAPMConfig #5891

Merged

ycombinator changed the title ~~[Flaky Test]: TestAPMConfig – no such index [traces-apm-default]~~ [Failing Test]: TestAPMConfig – no such index [traces-apm-default] Oct 31, 2024

ycombinator removed the flaky-test Unstable or unreliable test cases. label Oct 31, 2024

ycombinator assigned pchila Oct 31, 2024

mergify bot mentioned this issue Oct 31, 2024

[8.x](backport #5891) Skip TestAPMConfig #5908

Merged

ycombinator assigned leehinman and unassigned pchila Nov 1, 2024

ycombinator mentioned this issue Nov 11, 2024

Revert "Skip TestAPMConfig " #5993

Merged

mergify bot mentioned this issue Nov 12, 2024

[8.x](backport #5993) Revert "Skip TestAPMConfig " #5998

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Failing Test]: TestAPMConfig – no such index [traces-apm-default] #5890

[Failing Test]: TestAPMConfig – no such index [traces-apm-default] #5890

ycombinator commented Oct 30, 2024 •

edited

Loading

elasticmachine commented Oct 30, 2024

pchila commented Oct 30, 2024

ycombinator commented Oct 30, 2024

ycombinator commented Oct 30, 2024

kruskall commented Oct 30, 2024 •

edited

Loading

kruskall commented Oct 30, 2024

ycombinator commented Oct 31, 2024 •

edited

Loading

ycombinator commented Oct 31, 2024 •

edited

Loading

1pkg commented Oct 31, 2024

ycombinator commented Oct 31, 2024 •

edited

Loading

ycombinator commented Oct 31, 2024

1pkg commented Oct 31, 2024

1pkg commented Nov 1, 2024

1pkg commented Nov 1, 2024

kruskall commented Nov 1, 2024 •

edited

Loading

kruskall commented Nov 1, 2024

ycombinator commented Nov 1, 2024

leehinman commented Nov 1, 2024

leehinman commented Nov 1, 2024

1pkg commented Nov 1, 2024 •

edited

Loading

leehinman commented Nov 1, 2024

1pkg commented Nov 1, 2024

1pkg commented Nov 5, 2024

ycombinator commented Nov 11, 2024

[Failing Test]: TestAPMConfig – no such index [traces-apm-default] #5890

[Failing Test]: TestAPMConfig – no such index [traces-apm-default] #5890

Comments

ycombinator commented Oct 30, 2024 • edited Loading

Failing test case

Error message

Build

OS

Stacktrace and notes

elasticmachine commented Oct 30, 2024

pchila commented Oct 30, 2024

ycombinator commented Oct 30, 2024

ycombinator commented Oct 30, 2024

kruskall commented Oct 30, 2024 • edited Loading

kruskall commented Oct 30, 2024

ycombinator commented Oct 31, 2024 • edited Loading

ycombinator commented Oct 31, 2024 • edited Loading

1pkg commented Oct 31, 2024

ycombinator commented Oct 31, 2024 • edited Loading

ycombinator commented Oct 31, 2024

1pkg commented Oct 31, 2024

1pkg commented Nov 1, 2024

1pkg commented Nov 1, 2024

kruskall commented Nov 1, 2024 • edited Loading

kruskall commented Nov 1, 2024

ycombinator commented Nov 1, 2024

leehinman commented Nov 1, 2024

leehinman commented Nov 1, 2024

1pkg commented Nov 1, 2024 • edited Loading

leehinman commented Nov 1, 2024

1pkg commented Nov 1, 2024

1pkg commented Nov 5, 2024

ycombinator commented Nov 11, 2024

ycombinator commented Oct 30, 2024 •

edited

Loading

kruskall commented Oct 30, 2024 •

edited

Loading

ycombinator commented Oct 31, 2024 •

edited

Loading

ycombinator commented Oct 31, 2024 •

edited

Loading

ycombinator commented Oct 31, 2024 •

edited

Loading

kruskall commented Nov 1, 2024 •

edited

Loading

1pkg commented Nov 1, 2024 •

edited

Loading