ECS fluentbit crashes with SIGSEGV when using HTTP output #828

santhoshratala · 2024-05-23T08:54:44Z

Describe the question/issue

When using HTTP output in our fluentbit config, we've noticed that the fluentbit image crashes randomly with [engine] caught signal (SIGSEGV).

we've been using it to push logs to our Opensearch cluster using ingestion pipeline.

Reference Doc:-
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-fluentbit.html

Fluent Bit Version Info

v2.32.2

Cluster Details

ECS with Fargate
Task CPU=512 & Memory=1024
Sidecar deployment for Fluent Bit

Configuration

ECS Task Definition

module "container_firelens" {
  source                       = "cloudposse/ecs-container-definition/aws"
  version                      = "0.58.1"

  container_name               = "container-firelens"
  container_image              = "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-debug-latest"
  essential                    = "true"
  container_memory_reservation = 128

  firelens_configuration = {
    type = "fluentbit"
    options = {
      "enable-ecs-log-metadata" : "true"
    }
  }

  map_environment = {
    "aws_fluent_bit_init_s3_1" = aws_s3_object.s3_fluentbit_conf_object[0].arn # Pass fluentbit config from S3 bucket
    "FLB_LOG_LEVEL"            = "debug"
    "S3_BUCKET"                = module.s3_fluebit_conf_bucket[0].s3_bucket_id
    "S3_KEY_PREFIX"            = "debug"
  }

  log_configuration = {
    logDriver = "awslogs"
    options = {
      "awslogs-group" : "${aws_cloudwatch_log_group.firelens[0].name}",
      "awslogs-region" : "ap-southeast-1",
      "awslogs-stream-prefix" : "aws"
    }
    secretOptions = null
  }
}

module "container_app" {
  source  = "cloudposse/ecs-container-definition/aws"
  version = "0.58.1"

  container_name  = "container-${local.name}"
  container_image = "${module.ecr_repo.repository_url}:latest"
  environment     = local.container_env_list

  log_configuration = {
    logDriver = "awsfirelens"
    options = {
      "Name" : "cloudwatch_logs",
      "log_group_name" : "${aws_cloudwatch_log_group.app.name}",
      "region" : "ap-southeast-1",
      "log_stream_prefix" : "fluentbit-",
    }
    secretOptions = null
  }

  port_mappings = [
    {
      "hostPort" : var.application_port,
      "protocol" : "tcp",
      "containerPort" : var.application_port
    }
  ]
}

Fluent Bit Configuration File

[SERVICE]
    Grace 30
    Flush 1

[FILTER]
    Name record_modifier
    Match *
    Record app_name ${app_prefix}
    Record env ${env}

[OUTPUT]
    Name http
    Match *
    Host ${ingestion_pipeline_endpoint}
    Port 443
    URI /ingest
    Format json
    aws_auth true
    aws_region ap-southeast-1
    aws_service osis
    aws_role_arn ${http_ingestion_role_arn}
    tls On
    json_date_key timestamp
    json_date_format iso8601
    net.keepalive Off

Fluent Bit Log Output

Stacktrace Output

[New LWP 61]
[New LWP 30]
[New LWP 39]
[New LWP 38]
[New LWP 40]
[New LWP 42]
[New LWP 41]
[New LWP 43]
[New LWP 45]
[New LWP 47]
[New LWP 46]
[New LWP 50]
[New LWP 48]
[New LWP 51]
[New LWP 49]
[New LWP 54]
[New LWP 52]
[New LWP 56]
[New LWP 53]
[New LWP 58]
[New LWP 59]
[New LWP 60]
[New LWP 55]
[New LWP 57]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f6fea443ca0 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f6f5eb63700 (LWP 61))]

Thread 24 (Thread 0x7f6f624e5700 (LWP 57)):
#0  0x00007f6fea4ff84c in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000a4f221 in _mk_event_wait_2 (loop=0x7f6f6160a000, timeout=-1) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event_epoll.c:445
        ctx = 0x7f6f61600000
        ret = 0
#2  0x0000000000a4f636 in mk_event_wait (loop=0x7f6f6160a000) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event.c:194
No locals.
#3  0x00000000004f7464 in flb_engine_start (config=0x7f6fe7819e40) at /tmp/fluent-bit-1.9.10/src/flb_engine.c:775
        __flb_event_priority_live_foreach_iter = 1
        __flb_event_priority_live_foreach_n_events = 0
        ret = 0
        ts = 1716444560062650557
        tmp = "24.0K\000\000\000\000\000\000\000\000\000\000"
        t_flush = {tm = {tv_sec = 1, tv_nsec = 0}}
        event = 0x0
        evl = 0x7f6f6160a000
        evl_bktq = 0x7f6f61613000
        sched = 0x7f6f61633120
        dns_ctx = {lookups = {prev = 0x7f6f624daa70, next = 0x7f6f624daa70}, lookups_drop = {prev = 0x7f6f624daa80, next = 0x7f6f624daa80}}
#4  0x00000000004d40c6 in flb_lib_worker (data=0x7f6fe7818000) at /tmp/fluent-bit-1.9.10/src/flb_lib.c:626
        ret = 0
        ctx = 0x7f6fe7818000
        config = 0x7f6fe7819e40
#5  0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 23 (Thread 0x7f6f638e7700 (LWP 55)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19116 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117094504035) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6f8d9f3cc7 in runtime.notesleep (n=0x2c000080148) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6f8da23c2c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c0000821a0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c0000821a0
#5  0x00007f6f8da24745 in runtime.startlockedm (gp=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2607
        mp = 0x7f6f8eaa8880 <runtime.m0>
        pp = <optimized out>
#6  0x00007f6f8da262cc in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3412
        mp = 0x2c000080000
        pp = <optimized out>
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6f8da2682d in runtime.park_m (gp=0x2c000007380) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x2c000080000
#8  0x00007f6f8da4d4a5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 22 (Thread 0x7f6f5f564700 (LWP 60)):
#0  0x00007f6fea4ff84c in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000a4f221 in _mk_event_wait_2 (loop=0x7f6f5f9ee8b8, timeout=-1) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event_epoll.c:445
        ctx = 0x7f6f5f9e4690
        ret = 0
#2  0x0000000000a4f636 in mk_event_wait (loop=0x7f6f5f9ee8b8) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event.c:194
No locals.
#3  0x00000000004e76ee in output_thread (data=0x7f6f5fa26500) at /tmp/fluent-bit-1.9.10/src/flb_output_thread.c:250
        __flb_event_priority_live_foreach_iter = 1
        __flb_event_priority_live_foreach_n_events = 0
        n = 8
        ret = 0
        running = 1
        stopping = 0
        thread_id = 0
        tmp = "flb-out-http.1-w0", '\000' <repeats 46 times>
        event_local = {fd = 70, type = 32768, mask = 1, status = 2 '\002', data = 0x0, handler = 0x0, _head = {prev = 0x0, next = 0x0}, _priority_head = {prev = 0x0, next = 0x0}, priority = 6 '\006'}
        event = 0x0
        sched = 0x7f6f5d803000
        task = 0x7f6f5f98b950
        u_conn = 0x7f6f5d858000
        ins = 0x7f6fe78ba300
        out_flush = 0x7f6f5d80b000
        th_ins = 0x7f6f5fa26500
        params = 0x0
        dns_ctx = {lookups = {prev = 0x7f6f5f559960, next = 0x7f6f5f559960}, lookups_drop = {prev = 0x7f6f5f559970, next = 0x7f6f5f559970}}
#4  0x0000000000500b19 in step_callback (data=0x7f6f5f9bb200) at /tmp/fluent-bit-1.9.10/src/flb_worker.c:43
        worker = 0x7f6f5f9bb200
#5  0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 21 (Thread 0x7f6f60bfe700 (LWP 59)):
#0  0x00007f6fea4ff84c in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000a4f221 in _mk_event_wait_2 (loop=0x7f6f6160aaf8, timeout=-1) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event_epoll.c:445
        ctx = 0x7f6f616002e0
        ret = 0
#2  0x0000000000a4f636 in mk_event_wait (loop=0x7f6f6160aaf8) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event.c:194
No locals.
#3  0x00000000004e76ee in output_thread (data=0x7f6f61653000) at /tmp/fluent-bit-1.9.10/src/flb_output_thread.c:250
        __flb_event_priority_live_foreach_iter = 1
        __flb_event_priority_live_foreach_n_events = 0
        n = 0
        ret = 0
        running = 1
        stopping = 0
        thread_id = 0
        tmp = "flb-out-null.0-w0", '\000' <repeats 46 times>
        event_local = {fd = 47, type = 32768, mask = 1, status = 2 '\002', data = 0x0, handler = 0x0, _head = {prev = 0x0, next = 0x0}, _priority_head = {prev = 0x0, next = 0x0}, priority = 6 '\006'}
        event = 0x0
        sched = 0x7f6f5fc03000
        task = 0x0
        u_conn = 0x0
        ins = 0x7f6fe78ba000
        out_flush = 0x0
        th_ins = 0x7f6f61653000
        params = 0x0
        dns_ctx = {lookups = {prev = 0x7f6f60bf3960, next = 0x7f6f60bf3960}, lookups_drop = {prev = 0x7f6f60bf3970, next = 0x7f6f60bf3970}}
#4  0x0000000000500b19 in step_callback (data=0x7f6f61629e40) at /tmp/fluent-bit-1.9.10/src/flb_worker.c:43
        worker = 0x7f6f61629e40
#5  0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 20 (Thread 0x7f6f615ff700 (LWP 58)):
#0  0x00007f6fea4ff84c in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000a4f221 in _mk_event_wait_2 (loop=0x7f6f6160a018, timeout=-1) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event_epoll.c:445
        ctx = 0x7f6f61600010
        ret = 0
#2  0x0000000000a4f636 in mk_event_wait (loop=0x7f6f6160a018) at /tmp/fluent-bit-1.9.10/lib/monkey/mk_core/mk_event.c:194
No locals.
#3  0x00000000004d488b in log_worker_collector (data=0x7f6f61615000) at /tmp/fluent-bit-1.9.10/src/flb_log.c:131
        __i = 2
        __ctx = 0x7f6f61600010
        run = 1
        event = 0x0
        log = 0x7f6f61615000
#4  0x0000000000500b19 in step_callback (data=0x7f6f616290c0) at /tmp/fluent-bit-1.9.10/src/flb_worker.c:43
        worker = 0x7f6f616290c0
#5  0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 19 (Thread 0x7f6f64ce9700 (LWP 53)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19116 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117094504035) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6f8d9f3cc7 in runtime.notesleep (n=0x2c00004e948) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6f8da23c2c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c000006ea0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c000006ea0
#5  0x00007f6f8da24745 in runtime.startlockedm (gp=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2607
        mp = 0x7f6f8eaa8880 <runtime.m0>
        pp = <optimized out>
#6  0x00007f6f8da262cc in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3412
        mp = 0x2c00004e800
        pp = <optimized out>
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6f8da2682d in runtime.park_m (gp=0x2c0000071e0) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x2c00004e800
#8  0x00007f6f8da4d4a5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 18 (Thread 0x7f6f62ee6700 (LWP 56)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19116 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117094504035) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6f8d9f3cc7 in runtime.notesleep (n=0x7f6f8ead8558 <runtime.newmHandoff+24>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6f8da23b11 in runtime.templateThread () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2313
No locals.
#4  0x00007f6f8da22655 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#5  0x00007f6f8da2259a in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0x2c0000824e0
        osStack = false
#6  0x00007f6f8da4d425 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#7  0x00007f6f8e08077f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#8  0x00007f6f8d899970 in ?? ()
No symbol table info available.
#9  0x00007f6f8ca07060 in ?? ()
No symbol table info available.
#10 0x00007f6f8d89987f in ?? ()
No symbol table info available.
#11 0x00007f6f8d89987e in ?? ()
No symbol table info available.
#12 0x000002c0000824e0 in ?? ()
No symbol table info available.
#13 0x00007f6f8da4d420 in ?? () from /fluent-bit/kinesis.so
No locals.
#14 0x00007f6f8e0801a4 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6f62edba60, tls = <optimized out>, fn = 0x0}
#15 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7f6f656ea700 (LWP 52)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19191 in runtime.futexsleep (addr=<optimized out>, val=<optimized out>, ns=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:75
No locals.
#2  0x00007f6f8d9f3e05 in runtime.notetsleep_internal (n=0x7f6f8eaa8d80 <runtime.sched+256>, ns=60000000000, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:202
        now = <optimized out>
        gp = 0x2c000006b60
        deadline = 22049804818730
#3  0x00007f6f8d9f3f54 in runtime.notetsleep (n=0xfffffffffffffdfc, ns=0, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:225
        gp = 0x2c000006b60
#4  0x00007f6f8da2ae2e in runtime.sysmon () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:5352
        sleep = 0
        syscallWake = <optimized out>
        next = 9223372036854775807
        now = 21989804816272
        lastpoll = <optimized out>
        lasttrace = 0
        idle = 362
        delay = 10000
#5  0x00007f6f8da22655 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#6  0x00007f6f8da2259a in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0x2c000006b60
        osStack = 252
#7  0x00007f6f8da4d425 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#8  0x00007f6f8e08077f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#9  0x00007f6f8d8998d0 in ?? ()
No symbol table info available.
#10 0x00007f6f8ca07000 in ?? ()
No symbol table info available.
#11 0x00007f6f8d8997df in ?? ()
No symbol table info available.
#12 0x00007f6f8d8997de in ?? ()
No symbol table info available.
#13 0x000002c000006b60 in ?? ()
No symbol table info available.
#14 0x00007f6f8da4d420 in ?? () from /fluent-bit/kinesis.so
No locals.
#15 0x00007f6f8e0801a4 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6f656df9e8, tls = <optimized out>, fn = 0x0}
#16 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#17 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7f6f642e8700 (LWP 54)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19116 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117094504035) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6f8d9f3cc7 in runtime.notesleep (n=0x2c00004ed48) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6f8da23c2c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c0000076c0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x2c0000076c0
#5  0x00007f6f8da254e9 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0x2c00004ec00
        pp = 0x2c000038000
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6f8da26311 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0x2c00004ec00
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6f8da2682d in runtime.park_m (gp=0x2c000082820) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x2c00004ec00
#8  0x00007f6f8da4d4a5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 15 (Thread 0x7f6f8fee7700 (LWP 49)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e976 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117844388387) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fba519067 in runtime.notesleep (n=0x1c000080148) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fba5493ec in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c0000821a0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c0000821a0
#5  0x00007f6fba54aca9 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0x1c000080000
        pp = 0x1c000038000
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6fba54bad1 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0x1c000080000
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fba54bfed in runtime.park_m (gp=0x1c000007d40) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x1c000080000
#8  0x00007f6fba572885 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 14 (Thread 0x7f6f8d8a4700 (LWP 51)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6f8da19116 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117094504035) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6f8d9f3cc7 in runtime.notesleep (n=0x7f6f8eaa89c8 <runtime.m0+328>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6f8da23c2c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6f8eaa84e0 <runtime.g0>
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6f8eaa84e0 <runtime.g0>
#5  0x00007f6f8da254e9 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0x7f6f8eaa8880 <runtime.m0>
        pp = 0x2c00003a500
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6f8da26311 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0x7f6f8eaa8880 <runtime.m0>
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6f8da272b0 in runtime.goexit0 (gp=0x2c000006340) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3705
        mp = 0x7f6f8eaa8880 <runtime.m0>
        pp = 0x2c00003a500
#8  0x00007f6f8da4d4a5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x00007f6f8da521c7 in runtime.newproc (fn=0x0) at <autogenerated>:1
No locals.
#10 0x00007f6f8eaa84e0 in crypto/tls.emptyConfig () from /fluent-bit/kinesis.so
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 13 (Thread 0x7f6f908e8700 (LWP 48)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e976 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117844388387) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fba519067 in runtime.notesleep (n=0x1c00004ed48) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fba5493ec in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c0000076c0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c0000076c0
#5  0x00007f6fba54aca9 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0x1c00004ec00
        pp = 0x1c00003a500
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6fba54bad1 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0x1c00004ec00
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fba54bfed in runtime.park_m (gp=0x1c000007380) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x1c00004ec00
#8  0x00007f6fba572885 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 12 (Thread 0x7f6f8f4e6700 (LWP 50)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e976 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117844388387) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fba519067 in runtime.notesleep (n=0x7f6fbb2eca98 <runtime.newmHandoff+24>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fba5492d1 in runtime.templateThread () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2313
No locals.
#4  0x00007f6fba547e15 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#5  0x00007f6fba547d5a in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0x1c000007a00
        osStack = false
#6  0x00007f6fba572805 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#7  0x00007f6fba9f6c3f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#8  0x00007f6fba410970 in ?? ()
No symbol table info available.
#9  0x00007f6fb9607060 in ?? ()
No symbol table info available.
#10 0x00007f6fba41087f in ?? ()
No symbol table info available.
#11 0x00007f6fba41087e in ?? ()
No symbol table info available.
#12 0x000001c000007a00 in ?? ()
No symbol table info available.
#13 0x00007f6fba572800 in ?? () from /fluent-bit/cloudwatch.so
No locals.
#14 0x00007f6fba9f6664 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6f8f4dba60, tls = <optimized out>, fn = 0x0}
#15 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7f6f91cea700 (LWP 46)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e9f1 in runtime.futexsleep (addr=<optimized out>, val=<optimized out>, ns=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:75
No locals.
#2  0x00007f6fba5191a5 in runtime.notetsleep_internal (n=0x7f6fbb2bd8c0 <runtime.sched+256>, ns=60000000000, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:202
        now = <optimized out>
        gp = 0x1c000006b60
        deadline = 22049351777932
#3  0x00007f6fba5192f4 in runtime.notetsleep (n=0xfffffffffffffdfc, ns=0, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:225
        gp = 0x1c000006b60
#4  0x00007f6fba5505ee in runtime.sysmon () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:5352
        sleep = 0
        syscallWake = <optimized out>
        next = 9223372036854775807
        now = 21989351776396
        lastpoll = <optimized out>
        lasttrace = 0
        idle = 362
        delay = 10000
#5  0x00007f6fba547e15 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#6  0x00007f6fba547d5a in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0x1c000006b60
        osStack = 252
#7  0x00007f6fba572805 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#8  0x00007f6fba9f6c3f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#9  0x00007f6fba4108d0 in ?? ()
No symbol table info available.
#10 0x00007f6fb9607000 in ?? ()
No symbol table info available.
#11 0x00007f6fba4107df in ?? ()
No symbol table info available.
#12 0x00007f6fba4107de in ?? ()
No symbol table info available.
#13 0x000001c000006b60 in ?? ()
No symbol table info available.
#14 0x00007f6fba572800 in ?? () from /fluent-bit/cloudwatch.so
No locals.
#15 0x00007f6fba9f6664 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6f91cdf9e8, tls = <optimized out>, fn = 0x0}
#16 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#17 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f6f912e9700 (LWP 47)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e976 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117844388387) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fba519067 in runtime.notesleep (n=0x1c00004e948) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fba5493ec in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c000006ea0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x1c000006ea0
#5  0x00007f6fba549f05 in runtime.startlockedm (gp=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2607
        mp = 0x7f6fbb2bd3c0 <runtime.m0>
        pp = <optimized out>
#6  0x00007f6fba54ba8c in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3412
        mp = 0x1c00004e800
        pp = <optimized out>
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fba54bfed in runtime.park_m (gp=0x1c0000071e0) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0x1c00004e800
#8  0x00007f6fba572885 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 9 (Thread 0x7f6fba41b700 (LWP 45)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fba53e976 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140117844388387) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fba519067 in runtime.notesleep (n=0x7f6fbb2bd508 <runtime.m0+328>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fba5493ec in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6fbb2bd020 <runtime.g0>
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6fbb2bd020 <runtime.g0>
#5  0x00007f6fba549f05 in runtime.startlockedm (gp=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2607
        mp = 0x1c00004e000
        pp = <optimized out>
#6  0x00007f6fba54ba8c in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3412
        mp = 0x7f6fbb2bd3c0 <runtime.m0>
        pp = <optimized out>
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fba54ca70 in runtime.goexit0 (gp=0x1c000006340) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3705
        mp = 0x7f6fbb2bd3c0 <runtime.m0>
        pp = 0x1c000038000
#8  0x00007f6fba572885 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x00007f6fba577447 in runtime.newproc (fn=0x0) at <autogenerated>:1
No locals.
#10 0x00007f6fbb2bd020 in crypto/tls.emptyConfig () from /fluent-bit/cloudwatch.so
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 8 (Thread 0x7f6fbbcfb700 (LWP 43)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc2f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140118584411235) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fe66d6f07 in runtime.notesleep (n=0x7f6fe73f1cf8 <runtime.newmHandoff+24>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fe6706c51 in runtime.templateThread () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2313
No locals.
#4  0x00007f6fe6705795 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#5  0x00007f6fe67056da in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0xc0000076c0
        osStack = false
#6  0x00007f6fe6730245 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#7  0x00007f6fe6b9d38f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#8  0x00007f6fe65d6970 in ?? ()
No symbol table info available.
#9  0x00007f6fe5607060 in ?? ()
No symbol table info available.
#10 0x00007f6fe65d687f in ?? ()
No symbol table info available.
#11 0x00007f6fe65d687e in ?? ()
No symbol table info available.
#12 0x000000c0000076c0 in ?? ()
No symbol table info available.
#13 0x00007f6fe6730240 in ?? () from /fluent-bit/firehose.so
No locals.
#14 0x00007f6fe6b9cdb4 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6fbbcf0a60, tls = <optimized out>, fn = 0x0}
#15 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f6fbd0fd700 (LWP 41)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc2f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140118584411235) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fe66d6f07 in runtime.notesleep (n=0xc00004ed48) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fe6706d6c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc000007380
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc000007380
#5  0x00007f6fe6708629 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0xc00004ec00
        pp = 0xc00003a500
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6fe6709451 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0xc00004ec00
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fe670996d in runtime.park_m (gp=0xc000084000) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0xc00004ec00
#8  0x00007f6fe67302c5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (Thread 0x7f6fbc6fc700 (LWP 42)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc2f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140118584411235) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fe66d6f07 in runtime.notesleep (n=0xc00008c148) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fe6706d6c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc0000844e0
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc0000844e0
#5  0x00007f6fe6708629 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0xc00008c000
        pp = 0xc00003a500
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6fe6709451 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0xc00008c000
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fe670996d in runtime.park_m (gp=0xc000007a00) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0xc00008c000
#8  0x00007f6fe67302c5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (Thread 0x7f6fbdafe700 (LWP 40)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc2f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140118584411235) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fe66d6f07 in runtime.notesleep (n=0xc00004e948) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fe6706d6c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc000007040
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0xc000007040
#5  0x00007f6fe6707885 in runtime.startlockedm (gp=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2607
        mp = 0x7f6fe73c20c0 <runtime.m0>
        pp = <optimized out>
#6  0x00007f6fe670940c in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3412
        mp = 0xc00004e800
        pp = <optimized out>
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fe670996d in runtime.park_m (gp=0xc0000841a0) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3529
        mp = 0xc00004e800
#8  0x00007f6fe67302c5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 0x7f6fe65e1700 (LWP 38)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc2f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140118584411235) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:69
No locals.
#2  0x00007f6fe66d6f07 in runtime.notesleep (n=0x7f6fe73c2208 <runtime.m0+328>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:160
        gp = <optimized out>
        ns = -1
#3  0x00007f6fe6706d6c in runtime.mPark () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6fe73c1d20 <runtime.g0>
#4  runtime.stopm () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:2335
        gp = 0x7f6fe73c1d20 <runtime.g0>
#5  0x00007f6fe6708629 in runtime.findRunnable (gp=<optimized out>, inheritTime=<optimized out>, tryWakeP=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3025
        mp = 0x7f6fe73c20c0 <runtime.m0>
        pp = 0xc000038000
        now = 0
        pollUntil = 0
        wasSpinning = true
        allpSnapshot = <optimized out>
        idlepMaskSnapshot = <optimized out>
        timerpMaskSnapshot = <optimized out>
#6  0x00007f6fe6709451 in runtime.schedule () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3378
        mp = 0x7f6fe73c20c0 <runtime.m0>
        pp = 0x0
        gp = <optimized out>
        inheritTime = <optimized out>
        tryWakeP = <optimized out>
#7  0x00007f6fe670a3f0 in runtime.goexit0 (gp=0xc000006340) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:3705
        mp = 0x7f6fe73c20c0 <runtime.m0>
        pp = 0xc000038000
#8  0x00007f6fe67302c5 in runtime.mcall () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:452
No locals.
#9  0x00007f6fe6734e87 in runtime.newproc (fn=0x0) at <autogenerated>:1
No locals.
#10 0x00007f6fe73c1d20 in crypto/tls.emptyConfig () from /fluent-bit/firehose.so
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (Thread 0x7f6fbe4ff700 (LWP 39)):
#0  runtime.futex () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:555
No locals.
#1  0x00007f6fe66fc371 in runtime.futexsleep (addr=<optimized out>, val=<optimized out>, ns=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/os_linux.go:75
No locals.
#2  0x00007f6fe66d7045 in runtime.notetsleep_internal (n=0x7f6fe73c25c0 <runtime.sched+256>, ns=60000000000, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:202
        now = <optimized out>
        gp = 0xc000006b60
        deadline = 22048990439730
#3  0x00007f6fe66d7194 in runtime.notetsleep (n=0xfffffffffffffdfc, ns=0, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/lock_futex.go:225
        gp = 0xc000006b60
#4  0x00007f6fe670df6e in runtime.sysmon () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:5352
        sleep = 0
        syscallWake = <optimized out>
        next = 9223372036854775807
        now = 21988990438358
        lastpoll = <optimized out>
        lasttrace = 0
        idle = 362
        delay = 10000
#5  0x00007f6fe6705795 in runtime.mstart1 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1499
        fn = {void (void)} 0x0
        gp = <optimized out>
#6  0x00007f6fe67056da in runtime.mstart0 () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/proc.go:1456
        gp = 0xc000006b60
        osStack = 252
#7  0x00007f6fe6730245 in runtime.mstart () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/asm_amd64.s:394
No locals.
#8  0x00007f6fe6b9d38f in crosscall_amd64 () at gcc_amd64.S:42
No locals.
#9  0x00007f6fe65d68d0 in ?? ()
No symbol table info available.
#10 0x00007f6fe5607000 in ?? ()
No symbol table info available.
#11 0x00007f6fe65d67df in ?? ()
No symbol table info available.
#12 0x00007f6fe65d67de in ?? ()
No symbol table info available.
#13 0x000000c000006b60 in ?? ()
No symbol table info available.
#14 0x00007f6fe6730240 in ?? () from /fluent-bit/firehose.so
No locals.
#15 0x00007f6fe6b9cdb4 in threadentry (v=<optimized out>) at gcc_linux_amd64.c:94
        ts = {g = 0x7f6fbe4f49e8, tls = <optimized out>, fn = 0x0}
#16 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#17 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f6fec52aac0 (LWP 30)):
#0  0x00007f6fea4cfc2b in nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f6fea4cfb4a in sleep () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000000456d6f in flb_main (argc=9, argv=0x7ffc752f2828) at /tmp/fluent-bit-1.9.10/src/fluent-bit.c:1198
        opt = -1
        ret = 0
        json = 0x1 <error: Cannot access memory at address 0x1>
        last_plugin = -1
        cfg_file = 0x7f6fe78183c0 "\005"
        cf = 0x7f6fe781f000
        tmp = 0x7f6fe781f000
        service = 0x7f6fe7822000
        s = 0x7f6fe9725060
        long_opts = {{name = 0xa55020 "storage_path", has_arg = 1, flag = 0x0, val = 98}, {name = 0xa5502d "config", has_arg = 1, flag = 0x0, val = 99}, {name = 0xa54eb7 "daemon", has_arg = 0, flag = 0x0, val = 100}, {name = 0xa55034 "dry-run", has_arg = 0, flag = 0x0, val = 68}, {name = 0xa54ebe "flush", has_arg = 1, flag = 0x0, val = 102}, {name = 0xa5503c "http", has_arg = 0, flag = 0x0, val = 72}, {name = 0xa54eca "log_file", has_arg = 1, flag = 0x0, val = 108}, {name = 0xa55041 "port", has_arg = 1, flag = 0x0, val = 80}, {name = 0xa54cce "custom", has_arg = 1, flag = 0x0, val = 67}, {name = 0xa54cd5 "input", has_arg = 1, flag = 0x0, val = 105}, {name = 0xa54ec4 "match", has_arg = 1, flag = 0x0, val = 109}, {name = 0xa54ce2 "output", has_arg = 1, flag = 0x0, val = 111}, {name = 0xa54cdb "filter", has_arg = 1, flag = 0x0, val = 70}, {name = 0xa54dfc "parser", has_arg = 1, flag = 0x0, val = 82}, {name = 0xa55046 "prop", has_arg = 1, flag = 0x0, val = 112}, {name = 0xa5504b "plugin", has_arg = 1, flag = 0x0, val = 101}, {name = 0xa54ed3 "tag", has_arg = 1, flag = 0x0, val = 116}, {name = 0xa55052 "sp-task", has_arg = 1, flag = 0x0, val = 84}, {name = 0xa5505a "version", has_arg = 0, flag = 0x0, val = 86}, {name = 0xa55062 "verbose", has_arg = 0, flag = 0x0, val = 118}, {name = 0xa5506a "workdir", has_arg = 1, flag = 0x0, val = 119}, {name = 0xa55072 "quiet", has_arg = 0, flag = 0x0, val = 113}, {name = 0xa55078 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0xa5507d "help-json", has_arg = 0, flag = 0x0, val = 74}, {name = 0xa55087 "coro_stack_size", has_arg = 1, flag = 0x0, val = 115}, {name = 0xa55097 "sosreport", has_arg = 0, flag = 0x0, val = 83}, {name = 0xa54ed7 "http_server", has_arg = 0, flag = 0x0, val = 72}, {name = 0xa550a1 "http_listen", has_arg = 1, flag = 0x0, val = 76}, {name = 0xa550ad "http_port", has_arg = 1, flag = 0x0, val = 80}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
#3  0x0000000000456df6 in main (argc=9, argv=0x7ffc752f2828) at /tmp/fluent-bit-1.9.10/src/fluent-bit.c:1217
No locals.

Thread 1 (Thread 0x7f6f5eb63700 (LWP 61)):
#0  0x00007f6fea443ca0 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f6fea445148 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000000455abe in flb_signal_handler (signal=11) at /tmp/fluent-bit-1.9.10/src/fluent-bit.c:581
        len = 22
        ts = "[2024/05/23 06:09:49] \000\000\000\000\000\000\000\000\000"
        s = "[engine] caught signal ("
        now = 1716444589
        cur = 0x7f6fea7bc600 <_tmbuf>
#3  0x00007f6fe673415d in runtime.sigfwd () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:331
No locals.
#4  0x00007f6f5eb57808 in ?? ()
No symbol table info available.
#5  0x00007f6fe6716c05 in runtime.sigfwdgo (sig=<optimized out>, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:1135
        fwdFn = 140116307047680
        flags = 2
        gp = 0x0
#6  0x00007f6fe6715347 in runtime.sigtrampgo (sig=0, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:432
        gp = <optimized out>
        setStack = <optimized out>
#7  0x00007f6fe67341c9 in runtime.sigtramp () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:354
No locals.
#8  0x00007f6fba57671d in runtime.sigfwd () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:331
No locals.
#9  0x00007f6f5eb57918 in ?? ()
No symbol table info available.
#10 0x00007f6fba559285 in runtime.sigfwdgo (sig=<optimized out>, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:1135
        fwdFn = 140116307047680
        flags = 2
        gp = 0x0
#11 0x00007f6fba5579c7 in runtime.sigtrampgo (sig=0, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:432
        gp = <optimized out>
        setStack = <optimized out>
#12 0x00007f6fba576789 in runtime.sigtramp () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:354
No locals.
#13 0x00007f6f8da5135d in runtime.sigfwd () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:331
No locals.
#14 0x00007f6f5eb57a28 in ?? ()
No symbol table info available.
#15 0x00007f6f8da33ac5 in runtime.sigfwdgo (sig=<optimized out>, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>, ~r0=<optimized out>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:1135
        fwdFn = 140116307047680
        flags = 2
        gp = 0x0
#16 0x00007f6f8da32207 in runtime.sigtrampgo (sig=0, info=0x7f6f5eb57c30, ctx=0x7f6fea443ca0 <raise+272>) at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/signal_unix.go:432
        gp = <optimized out>
        setStack = <optimized out>
#17 0x00007f6f8da513c9 in runtime.sigtramp () at /home/.gimme/versions/go1.20.7.linux.amd64/src/runtime/sys_linux_amd64.s:354
No locals.
#18 <signal handler called>
No symbol table info available.
#19 0x00000000004fdaf8 in __mk_list_del (prev=0x0, next=0x0) at /tmp/fluent-bit-1.9.10/lib/monkey/include/monkey/mk_core/mk_list.h:141
No locals.
#20 0x00000000004fdb30 in mk_list_del (entry=0x7f6f5dc53428) at /tmp/fluent-bit-1.9.10/lib/monkey/include/monkey/mk_core/mk_list.h:147
No locals.
#21 0x00000000004fea99 in prepare_destroy_conn (u_conn=0x7f6f5dc53380) at /tmp/fluent-bit-1.9.10/src/flb_upstream.c:466
        u = 0x7f6f617fa580
        uq = 0x7f6f617fa638
#22 0x00000000004feb1c in prepare_destroy_conn_safe (u_conn=0x7f6f5dc53380) at /tmp/fluent-bit-1.9.10/src/flb_upstream.c:492
        ret = 4
        locked = 0
        u = 0x7f6f617fa580
#23 0x00000000004ff411 in cb_upstream_conn_ka_dropped (data=0x7f6f5dc53380) at /tmp/fluent-bit-1.9.10/src/flb_upstream.c:752
        conn = 0x7f6f5dc53380
#24 0x00000000004e7ec4 in output_thread (data=0x7f6f5fa26600) at /tmp/fluent-bit-1.9.10/src/flb_output_thread.c:300
        __flb_event_priority_live_foreach_iter = 0
        __flb_event_priority_live_foreach_n_events = 1
        n = 8
        ret = 0
        running = 1
        stopping = 0
        thread_id = 1
        tmp = "flb-out-http.1-w1", '\000' <repeats 46 times>
        event_local = {fd = 66, type = 32768, mask = 1, status = 2 '\002', data = 0x0, handler = 0x0, _head = {prev = 0x0, next = 0x0}, _priority_head = {prev = 0x0, next = 0x0}, priority = 6 '\006'}
        event = 0x7f6f5dc53380
        sched = 0x7f6f5dc03000
        task = 0x7f6f5f98b950
        u_conn = 0x7f6f5dc53000
        ins = 0x7f6fe78ba300
        out_flush = 0x7f6f5dc0b000
        th_ins = 0x7f6f5fa26600
        params = 0x0
        dns_ctx = {lookups = {prev = 0x7f6f5eb58960, next = 0x7f6f5eb58960}, lookups_drop = {prev = 0x7f6f5eb58970, next = 0x7f6f5eb58970}}
#25 0x0000000000500b19 in step_callback (data=0x7f6f5f9bb2c0) at /tmp/fluent-bit-1.9.10/src/flb_worker.c:43
        worker = 0x7f6f5f9bb2c0
#26 0x00007f6fec10544b in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#27 0x00007f6fea4ff52f in clone () from /lib64/libc.so.6
No symbol table info available.

The text was updated successfully, but these errors were encountered:

This was referenced Oct 14, 2024

ECS Fargate fluent-bit side-car container exit with code 139 #840

Closed

[upstream] connection #-1 to vpc-es-abn-opensearch-xxxxxxxxx.es.amazonaws.com:443 timed out after 10 seconds #833

Closed

singholt mentioned this issue Oct 21, 2024

upstream: always remove keepalive close event on shutdown if registered amazon-contributing/upstream-to-fluent-bit#3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECS fluentbit crashes with SIGSEGV when using HTTP output #828

ECS fluentbit crashes with SIGSEGV when using HTTP output #828

santhoshratala commented May 23, 2024

ECS fluentbit crashes with SIGSEGV when using HTTP output #828

ECS fluentbit crashes with SIGSEGV when using HTTP output #828

Comments

santhoshratala commented May 23, 2024

Describe the question/issue

Fluent Bit Version Info

Cluster Details

Configuration

Fluent Bit Log Output