Skip to content

Add Integration test to increase the coverage on Read by format #1439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Add Integration test to increase the coverage on Read by format #1439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4531e3c
Add testWindowFunctionPartitionBy in ReadByFormatIntegrationTestBase …
yalimu-g Nov 17, 2025
0b9a73a
Merge branch 'GoogleCloudDataproc:master' into issue1435
yalimu-g Nov 19, 2025
173e0f5
Format file
yalimu-g Nov 19, 2025
cb37807
Use dataset with struct type
yalimu-g Nov 19, 2025
7dff794
Un-comment the tests
yalimu-g Nov 19, 2025
8905419
Use a different dataset for partition.
yalimu-g Nov 19, 2025
bb86566
Remove unused imports
yalimu-g Nov 19, 2025
1db870d
Format file
yalimu-g Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions ...est/java/com/google/cloud/spark/bigquery/integration/ReadByFormatIntegrationTestBase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
package com.google.cloud.spark.bigquery.integration;

import static com.google.common.truth.Truth.assertThat;
import static org.apache.spark.sql.functions.*;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assume.assumeThat;
import static org.junit.Assume.assumeTrue;
Expand All @@ -42,6 +43,8 @@
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.expressions.Window;
import org.apache.spark.sql.expressions.WindowSpec;
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
Expand Down Expand Up @@ -316,6 +319,31 @@ public void testTimestampNTZReadFromBigQuery() {
assertThat(row.get(0)).isEqualTo(dateTime);
}

@Test
public void testWindowFunctionPartitionBy() {
WindowSpec windowSpec =
Window.partitionBy(
concat(
col("fullVisitorId"), col("visitStartTime"), col("trafficSource.referralPath")))
.orderBy(lit("window_ordering"));
Dataset<Row> dataset =
spark
.read()
.format("bigquery")
.option("table", TestConstants.ANALYTICS_TABLE)
.option("readDataFormat", dataFormat)
.load()
.withColumn("row_num", row_number().over(windowSpec));

assertThat(dataset.columns().length).isEqualTo(17);
assertThat(
Arrays.stream(dataset.schema().fields())
.filter(field -> field.name().equals("row_num"))
.count())
.isEqualTo(1);
assertThat(dataset.head().get(16)).isEqualTo(1);
}

static <K, V> Map<K, V> scalaMapToJavaMap(scala.collection.Map<K, V> map) {
ImmutableMap.Builder<K, V> result = ImmutableMap.<K, V>builder();
map.foreach(entry -> result.put(entry._1(), entry._2()));
Expand Down
2 changes: 2 additions & 0 deletions ...ector-common/src/test/java/com/google/cloud/spark/bigquery/integration/TestConstants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,8 @@ public class TestConstants {
static DataType BQ_NUMERIC = DataTypes.createDecimalType(38, 9);
static DataType BQ_BIGNUMERIC = DataTypes.createDecimalType(38, 38);
public static int BIG_NUMERIC_COLUMN_POSITION = 11;
static final String ANALYTICS_TABLE =
"bigquery-public-data.google_analytics_sample.ga_sessions_20170801";

public static StructType ALL_TYPES_TABLE_SCHEMA =
new StructType(
Expand Down
Loading