spring-batch-bigquery

spring-batch-bigquery

Spring Batch extension which contains an ItemWriter implementation for BigQuery based on Java BigQuery. It supports writing CSV, JSON using load jobs.

Configuration of BigQueryCsvItemWriter

Next to the configuration of Spring Batch one needs to configure the BigQueryCsvItemWriter.

@Bean
BigQueryCsvItemWriter<MyDto> bigQueryCsvWriter() {
    WriteChannelConfiguration writeConfiguration = WriteChannelConfiguration
        .newBuilder(TableId.of("csv_dataset", "csv_table"))
        .setAutodetect(true)
        .setFormatOptions(FormatOptions.csv())
        .build();

    BigQueryCsvItemWriter<MyDto> writer = new BigQueryCsvItemWriterBuilder<MyDto>()
        .bigQuery(mockedBigQuery)
        .writeChannelConfig(writeConfiguration)
        .build();
}

Additional examples could be found in here.

Configuration properties

Table 1. Properties for an item writer

Property	Required	Description
bigQuery	yes	BigQuery object that provided by BigQuery Java Library. Responsible for connection with BigQuery.
writeChannelConfig	yes	BigQuery write channel config provided by BigQuery Java Library. Responsible for configuring data type, data channel, jobs that will be sent to BigQuery.
rowMapper	no	Your own converter that specifies how to convert input CSV / JSON to a byte array.
datasetInfo	no	Your way to customize how to create BigQuery dataset.
jobConsumer	no	Your custom handler for BigQuery Job provided by BigQuery Java Library.