Skip to content

Dekaf: Add proper field selection/extraction tests, and validate that field names are valid avro#2008

Merged
jshearer merged 2 commits intomasterfrom
dekaf/extraction_tests_and_avro_validation
Mar 18, 2025
Merged

Dekaf: Add proper field selection/extraction tests, and validate that field names are valid avro#2008
jshearer merged 2 commits intomasterfrom
dekaf/extraction_tests_and_avro_validation

Conversation

@jshearer
Copy link
Contributor

@jshearer jshearer commented Mar 13, 2025

Description:

  • Add field extraction tests. I had originally written these using the validate crate, but this time it's using the logic straight from build instead, and is much cleaner
  • Add a condition to Dekaf's validate implementation that forbids fields with non-avro-compliant names, and returns a helpful error message with a link to our projection docs in case you really want that field.

Fixes #2004


This change is Reviewable

This adds back the tests I had originally written using the `validate` crate, but this time it's using the logic straight from `build` instead.
@jshearer jshearer force-pushed the dekaf/extraction_tests_and_avro_validation branch 2 times, most recently from 5b16cb1 to 6dec227 Compare March 14, 2025 14:58
@jshearer jshearer force-pushed the dekaf/extraction_tests_and_avro_validation branch from 6dec227 to 6d53b99 Compare March 14, 2025 20:15
@jshearer jshearer marked this pull request as ready for review March 18, 2025 15:00
Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jshearer jshearer merged commit f5700d7 into master Mar 18, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dekaf: Reject field names that can't be represented in an AVRO schema

2 participants