-
Notifications
You must be signed in to change notification settings - Fork 398
[FFL-1361] Evaluation in binding in ruby #5022
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
BenchmarksBenchmark execution time: 2025-11-13 23:32:21 Comparing candidate commit 6b17509 in PR branch Found 1 performance improvements and 1 performance regressions! Performance is the same for 42 metrics, 2 unstable metrics. scenario:profiling - gvl benchmark samples
scenario:tracing - Tracing.log_correlation
|
da05c91 to
1857c1f
Compare
|
👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description. If changes need to be present in CHANGELOG.md you can state it this way **Change log entry**
Yes. A brief summary to be placed into the CHANGELOG.md(possible answers Yes/Yep/Yeah) Or you can opt out like that **Change log entry**
None.(possible answers No/Nope/None) Visited at: 2025-11-11 09:18:47 UTC |
| module Datadog | ||
| module OpenFeature | ||
| module Binding | ||
| # Flat result structure matching NativeEvaluator ResolutionDetails interface |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically I found in #5007 that this is the output format from datadog-ffe-ffi
NativeEvaluator Successful Result:
context = Datadog::OpenFeature::Binding::EvaluationContext.new('test_user')
native = Datadog::OpenFeature::Binding::NativeEvaluator.new(libdatadog_config)
result = native.get_assignment('test_flag', context)
# Output:
result.class # => Datadog::OpenFeature::Binding::ResolutionDetails
result.value # => "control_value"
result.variant # => "control"
result.error_code # => nil
result.error_message # => nil
result.reason # => :static
result.allocation_key # => "rollout"
result.do_log # => falseNativeEvaluator Error Result:
result = native.get_assignment('missing_flag', context)
# Output:
result.value # => nil (no default value handling)
result.error_code # => :flag_not_found
result.error_message # => "flag is missing in configuration, it is either unrecognized or disabled"
result.reason # => :errorAnd I adjusted this to match that format. It's not required to stick to this format so it's something we can discuss and modify, but we should coordinate with FFI in libdatadog
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not too familiar with specifying .rbs files so please let me know if there's anything I can improve here.
| require_relative 'binding/configuration' | ||
|
|
||
| # Define alias for backward compatibility after InternalEvaluator is loaded | ||
| Datadog::OpenFeature::Binding::Evaluator = Datadog::OpenFeature::Binding::InternalEvaluator |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The aspiration is that in #5007 when it is ready, we can just switch this to
Datadog::OpenFeature::Binding::Evaluator = Datadog::OpenFeature::Binding::NativeEvaluator
And then we're using the libdatadog FFE powered evaluator
0a25c37 to
d5bbcd5
Compare
| module Datadog | ||
| module OpenFeature | ||
| module Binding | ||
| # Variation types supported by UFC |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, what does UFC stand for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops sorry it's old terminology (Universal Flag Configuration), refers to the format of this file: https://github.com/DataDog/dd-trace-rb/blob/47869fc74f64c8bb3d357393365e5ddca6afcb25/spec/fixtures/ufc/flags-v1.json
Universal because this approach of representing targeting rules in terms of splits with shard ranges and salts is flexible and accommodates most targeting use cases. I can take this comment out if it's confusing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ufc also figures in fixture file names, I believe it also should be fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
personally I find this comment confusing, as it includes an outdated and unknown abbreviature
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I miscommunicated. "Old" means it has been carried over from an older system, so the term has continuity with the other places this same format is used, e.g. this is the model for the generated UFC and the term is also used in other SDKs.
So it's not quite outdated, but you're right it's not widely known outside of those of us who have worked on it. But changing the name here would make terminology here inconsistent with the other places that use the UFC. I imagine spelling out the acronym in more places might help, so I can start with that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some comments and a readme bf1fbb4
| @allocations = allocations || [] | ||
| end | ||
|
|
||
| def self.from_json(key, flag_data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find this function name confusing - it actually builds a new Flag from a Hash, not JSON. Another thing is that if it builds the flag from flag_data, imo flag_data should be the first argument
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed 4290ae1
| def self.from_json(key, flag_data) | ||
| new( | ||
| key: key, | ||
| enabled: flag_data['enabled'] || false, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more ruby idiomatic:
flag_data.fetch('enabled', false)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and when using flag_data['enabled'] || false, the || will also be evaluated if flag_data['enabled'] is actually set to false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| variations: parse_variations(flag_data['variations'] || {}), | ||
| allocations: parse_allocations(flag_data['allocations'] || []) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here - please use .fetch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also in 25a2e91
| new( | ||
| key: key, | ||
| enabled: flag_data['enabled'] || false, | ||
| variation_type: flag_data['variationType'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we want to ensure variation_type presence, it's better to use fetch here as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
|
||
| def self.parse_variations(variations_data) | ||
| variations_data.transform_values do |variation_data| | ||
| Variation.from_json(variation_data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
variation_data is Hash, not a JSON string, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 4290ae1
| Variation.from_hash(variation_data) |
| shards: parse_shards(split_data['shards'] || []), | ||
| variation_key: split_data['variationKey'], | ||
| extra_logging: split_data['extraLogging'] || {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.fetch might be better here too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dd-trace-rb/lib/datadog/open_feature/binding/configuration.rb
Lines 150 to 152 in 73820ab
| shards: parse_shards(split_data.fetch('shards', [])), | |
| variation_key: split_data.fetch('variationKey'), | |
| extra_logging: split_data.fetch('extraLogging', {}) |
| end | ||
|
|
||
| # Represents a shard configuration for traffic splitting | ||
| class Shard |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure about this name - it's called Shard, but has a property called total_shards?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another case of exactly matching the key name in the JSON
dd-trace-rb/spec/fixtures/ufc/flags-v1.json
Line 831 in 49dafd3
| "shards": [ |
| # Alias for backward compatibility | ||
| def end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this could be an alias method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| end | ||
|
|
||
| # Represents a targeting rule | ||
| class Rule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TargetingRule?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also matching the field name in the universal flag configuration
dd-trace-rb/spec/fixtures/ufc/flags-v1.json
Line 813 in 49dafd3
| "rules": [ |
|
|
||
| private | ||
|
|
||
| def self.parse_condition_value(value_data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the purpose of this method? It doesn't seem that it parses anything - in any case it simply returns the value_data, but for some reason it checks it's type first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you're right. We don't need it 7a4ffe0
| end | ||
|
|
||
| # Assignment reasons returned in ResolutionDetails | ||
| module AssignmentReason |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it also would improve readability and code navigation, if those modules would be extracted as separate files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e7a0762 to
47383e2
Compare
6f4c1fd to
94c24c2
Compare
4e749c5 to
bf1fbb4
Compare
b1a978f to
9e7efb0
Compare
ivoanjo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been summoned by you asking a review of profiling-rb... although I'm not sure why? Were you looking for an extra review?
I've given it a pass, but I stopped short at some of the more deeper domain classes.
Ok so this is a lot code!. The PR description states
Eventually, we want to use a C binding with datadog-ffe-ffi for flag evaluations. In this PR, I aimed to implement an evaluator in Ruby that can be easily swapped out to use the binding when it is ready.
and then you reply at some point with
Much of this is temporary. It will be replaced by a C binding to an evaluator in libdatadog and then the Ruby evaluation code can be deleted.
So this again gets me thinking: What's the plan here again? Are we actually planning on ever shipping the pure-Ruby version to customers? Or is this just for internal testing?
Because, again, this being a lot of code, the answers to those questions are very important in determining how much time we should spend on making sure this PR is solid. Do my doubts here make sense?
| "DD_ENV" => {version: ["A"]}, | ||
| "DD_ERROR_TRACKING_HANDLED_ERRORS" => {version: ["A"]}, | ||
| "DD_ERROR_TRACKING_HANDLED_ERRORS_INCLUDE" => {version: ["A"]}, | ||
| "DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED" => {version: ["A"]}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usually the EXPERIMENTAL part comes after the feature name -- consider maybe DD_FLAGGING_PROVIDER_EXPERIMENTAL_ENABLED or something like that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't pick the name, it's in align with other libs 😥
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're still on time to change that, may be worth it; if not, well, it's experimental anyway 😭
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a lot of copy-paste to reuse the current transport code... I know there's already a lot going on in this PR, but I would gently suggest considering just writing a new transport from scratch rather than copying multiple really confusingly-written files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👋🏼 I'm who wrote copy/paste transport. To be honest I wasn't feeling comfortable to go with self-written due to the limitations of my time on this project. At the same time I don't see much a different approach in most components, all of them use Core to write the transport, as did I.
I don't mind a better transport if I could, but that's not the case. Do you think it's a blocker?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(We've chatted privately in more detail, but just recording it here that the answer is not at all ;D)
|
@ivoanjo @y9v I had a plan to split the work to component and then transport and binding as a separate PRs, but by the accident you got pinged with a full blast PR. I would apply certain suggestions to my dummy binding PR and open it for review, in addition I will extract certain pieces already suggested. Sorry that you get though this and thanks for lots of suggestions! ❤️ |
Remove convert_error_code method and SYMBOL_TO_OPENFEATURE_ERROR_CODE mapping from provider.rb as they are not required for the internal evaluator implementation task. The provider should pass through error codes directly from the engine without additional conversion logic.
* Fix configuration_spec.rb path issues and method calls - Correct fixture path from ../../../../ to ../../../ - Replace undefined from_json with from_hash method - Replace be_in matcher with include matcher for RSpec compatibility * Fix allocation matching test method signatures and expectations - Remove extra parameter from get_assignment calls (4 params → 3 params) - Update error expectations from :Ok to nil for successful evaluations - Fix flag_metadata access to use hash keys instead of method calls - Update test descriptions to reflect internal evaluator behavior * Fix flag lookup logic in Configuration#get_flag - Search by flag.key property instead of hash key for proper UFC compatibility - Enables correct flag resolution where hash key != flag.key * Update assignment reason expectations in provider tests - Change TARGETING_MATCH to STATIC for simple allocations without rules - Aligns with libdatadog assignment reason logic (no rules + single empty shard = STATIC) All binding tests now pass (54 examples, 0 failures). Remaining provider/engine integration issues need separate investigation.
* Fix OpenFeature.engine returning nil by enabling remote configuration
- Component requires both open_feature.enabled and remote.enabled
- Update test to configure both settings for proper initialization
* Fix provider test expected values to match flag configurations
- Update integer expectation: 42 → 21 to match actual config value
- Update number expectation: 9000 → 1000 to match actual config value
- Update float expectation: 36.6 → 12.5 to match actual config value
- Update object expectation: [1,2,3] → {"key":"value"} to match actual config
* Fix variation type mismatches in provider test configurations
- Change "variationType": "NUMBER" → "NUMERIC" for UFC compliance
- Change "variationType": "FLOAT" → "NUMERIC" for UFC compliance
- Change "variationType": "OBJECT" → "JSON" for UFC compliance
- Aligns with VariationType constants and internal evaluator type mapping
All OpenFeature tests now pass: 128 examples, 0 failures, 1 pending.
Complete end-to-end integration working from provider → engine → internal evaluator.
f0d3309 to
f7fe760
Compare
- Remove duplicate :agent_info in components.rb - Fix OpenFeature capability check to match other components pattern - Revert evaluation_engine.rb to target branch version (not part of internal evaluator functionality) - Revert evaluation_engine.rbs to target branch version - Revert component_spec.rb to target branch version - Revert resolution_details.rbs to target branch version (since we're not touching resolution_details.rb)
The InternalEvaluator was expecting flags at the root level but tests were using the nested Universal Flag Configuration (UFC) format with data.attributes.flags structure. This caused evaluation_engine_spec.rb and other tests to fail with nil values instead of expected flag results. Changes: - Update InternalEvaluator to extract flags from data.attributes.flags - Standardize all test specs to use consistent nested UFC format: - evaluation_engine_spec.rb (already used nested format) - internal_evaluator_spec.rb (updated to use full JSON structure) - rule_evaluation_spec.rb (converted from flat to nested format) - allocation_matching_spec.rb (converted from flat to nested format) This resolves 8 test failures and maintains consistency across the OpenFeature implementation by using the standard UFC JSON structure throughout the codebase.
The OpenFeature internal evaluator was expecting nested JSON structure (data.attributes.flags) but this has been simplified to use flags directly at the root level for consistency and clarity.
Changes:
- Update InternalEvaluator parser to expect flags at root level instead of data.attributes.flags
- Simplify flags-v1.json fixture to use flat {"flags": {...}} structure
- Update all test specs to use simplified JSON format and consistent flag_config naming:
- evaluation_engine_spec.rb: changed from nested ufc to flat flag_config
- binding specs: standardized on flag_config variable naming
- configuration_spec.rb: renamed ufc_attributes to flag_config
- Update all test case fixtures to match simplified format
- Update all test files to use flat JSON flag configuration format instead of nested data.attributes structure - Standardize variable naming from 'ufc' to 'flag_config' across all test files for consistency - Convert flag_metadata attributes to camelCase: allocationKey, variationType, doLog - Update documentation to reference system-tests origin and remove UFC terminology - Maintain backward compatibility with internal evaluator while simplifying test structure This consolidation aligns the OpenFeature implementation with the simplified flag configuration format and ensures consistent metadata attribute naming across the codebase.
- Rename result creation methods to clearly describe the three evaluation cases - Fix FlagDisabled and DefaultAllocationNull to return success results with nil values - Update test logic to properly handle three distinct result types - Convert flag_metadata attributes to camelCase: allocationKey, variationType, doLog - Set error_message=nil for successful evaluations (was empty string) - Update test expectations to match new result structure
0ef9b0d to
91e5f1b
Compare
…alidation - Remove ERROR_CODE_MAPPING constant and return string error codes directly to match ResolutionDetails type signature - Replace 19 common UFC test cases with libdatadog versions containing detailed resolution metadata (variant, allocationKey, variationType, doLog) - Enrich 3 Ruby-specific test cases with expected resolution metadata - Update test assertions to validate exact metadata values from test cases instead of just structural validation - Update type signatures in .rbs file to reflect removed constants
- Fix attribute lookup bug where boolean false values were treated as missing due to || operator treating falsy values as nil
- Remove unnecessary symbol key fallback in get_attribute_from_context and get_targeting_key since OpenFeature SDK guarantees string keys via transform_keys(&:to_s)
- Update test expectations from symbol error codes (:flag_not_found) to string error codes ('FLAG_UNRECOGNIZED_OR_DISABLED') for consistency
- Remove outdated "UFC" terminology references - Remove unnecessary "Rust" implementation references - Remove vague TODO comments for structured logging
- Add error? method to check if result has an error (error_code present) - Add result? method to check if result has successful variant - Add log? method as Ruby-friendly alternative to do_log - Add do_log? alias for backward compatibility - Update type signatures for new methods
- Remove duplicate constants from configuration.rbs - Add separate RBS files for constant modules (assignment_reason, condition_operator, error_codes, variation_type) - Recreate internal_evaluator.rbs to match current implementation - Delete evaluator.rbs (no corresponding Ruby file) - Update ext.rbs to only include existing constants
- Cache string coercion in condition evaluation to prevent repeated conversions - Add comprehensive JSON structure validation as single source of truth - Remove redundant Array/Hash defensive wrappers from configuration classes - Replace variation type mapping fallback with explicit validation - Add RBS type signatures for all binding modules - Reorder evaluation case comments sequentially (1, 2, 3)
cdf5175 to
e1171ed
Compare
- Remove conditional logic from get_attribute_from_context and get_targeting_key - Update evaluation engine to pass evaluation_context&.fields to internal evaluator - Add test verifying hash contexts work with OpenFeature SDK fields - Improves performance by eliminating respond_to? checks in hot paths
Changes the internal evaluator to accept string literals instead of symbols for expected_type parameters, improving consistency with external APIs.
Changes:
- Update InternalEvaluator.type_matches? to handle strings ('boolean', 'string', etc.)
- Add symbol-to-string mapping in EvaluationEngine for backward compatibility
- Update RBS type signatures to reflect string parameters
- Update binding tests to use string expected_type parameters
- Fix private class method access modifiers using private_class_method - Refactor validation logic to avoid non-local exits from iterators - Add YARD-style annotation to get_assignment method - Apply Standard RB formatting fixes to spec files - Update test expectations to work with default_value behavior - Ensure all error cases return default_value instead of nil
| # PLEASE DELETE when Datadog:OpenFeature:Provider is updated to pass strings | ||
| # Map symbol types to strings for internal evaluator | ||
| TYPE_MAP = { | ||
| boolean: 'boolean', | ||
| string: 'string', | ||
| integer: 'integer', | ||
| number: 'float', | ||
| float: 'float', | ||
| object: 'object' | ||
| }.freeze |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is here to minimize the footprint of the changes to the .get_assignment API. Doing this the right way (in lib/datadog/open_feature/provider.rb) would involve touching more test and rbs files which might make a rebase even trickier. This should be deleted after a rebase
Or if it's easier to just copy the InternalEvaluator files into a fresh branch and hook them up to the EvaluationEngine that's fine too
14b676f to
e572afc
Compare
What does this PR do?
Implements an feature flag evaluator in Ruby that aligns as closely as possible with with the libdatadog FFE (Feature Flag Evaluation) interface contract, according to what's in DataDog/libdatadog#1282. This is one layer of stacked PRs:
main^- #4998 (Add OpenFeature component)
^- #5024 (Add feature flags events exposure)
^- #5022 (Add InternalEvaluator) <-- you are here!
^- #5007 (Add NativeEvaluator -- binding for the libdatadog FFI)
Motivation:
Eventually, we want to use a C binding with
datadog-ffe-ffifor flag evaluations. In this PR, I aimed to implement an evaluator in Ruby that can be easily swapped out to use the binding when it is ready.Change log entry
Additional Notes:
Ran tests with
And I see one failure that is not from the internal evaluator
How to test the change?