Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot see error message for insert #184

Open
dobesv opened this issue Jun 30, 2020 · 0 comments
Open

Cannot see error message for insert #184

dobesv opened this issue Jun 30, 2020 · 0 comments

Comments

@dobesv
Copy link

dobesv commented Jun 30, 2020

Environments

  • fluentd version: 1.11.1
  • plugin version: 2.2.0

Configuration

<label @bigquery>
  <filter **>
    @type record_modifier
    <record>
      browserVersion "${record['browser'] && record['browser']['version']}"
      operationSystemVersion "${record['operatingSystem'] && record['operatingSystem']['version']}"
      browser "${record['browser'] && record['browser']['name']}"
      operatingSystem "${record['operatingSystem'] && record['operatingSystem']['name']}"
    </record>
  </filter>
  <match **>
    @id bigquery
    @type bigquery_insert
    @log_level info

    auth_method json_key
    json_key /etc/fluent/secrets/bigquery-service-account.json

    project "#{ENV['BIGQUERY_PROJECT']}"
    dataset "#{ENV['BIGQUERY_DATASET']}"
    table   events

    skip_invalid_rows true
    ignore_unknown_values true

    <inject>
      time_key timestamp
      time_type string
      time_format %FT%T%.%N:z
      utc true
    </inject>

    <buffer>
      @type file
      path /var/log/fluentd-buffers/bigquery.buffer
      total_limit_size 8GB
      overflow_action block
    </buffer>
    auto_create_table false
  </match>
</label>

Expected Behavior

If there's a problem inserting records into BigQuery, I can see in the logs some helpful information about what the problem is.

Actual Behavior

The logs indicate there is an error, but there's no detail provided. Instead the errors are replaced with #.

Log (if you have)

2020-06-30 20:04:42 +0000 [warn]: [bigquery] insert errors project_id="xxx" dataset="xx" table="events" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007f4b574a03e0 @errors=[#, #], @index=1>]"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant