Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Local evaluation testing #108

Closed
wants to merge 5 commits into from
Closed

test: Local evaluation testing #108

wants to merge 5 commits into from

Conversation

khvn26
Copy link
Member

@khvn26 khvn26 commented Aug 13, 2024

DO NOT MERGE. This contains an automated SDK test for Edge V2 migration.

To run, clone, then:

pytest tests/test_local_eval.py 

After the migration, run:

pytest tests/test_local_eval.py --migrated

@github-actions github-actions bot added testing and removed testing labels Aug 13, 2024
Copy link
Contributor

@matthewelwell matthewelwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed this. It all looks ok to me and seems to check mostly the right things, the main issue (as per my comment) is that we're not checking that the values are the same before and after the migration (which I think we should expect?).

I'd say that we should implement some way to verify this, it could just be a local json file or something, or a pickled python object written to a file maybe?

return _get_flag_mock(
value=get_matching_str(
"environment default",
'overridden by "split_segment"',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can, I'd like to be more specific here. Whether the identity belongs to the percentage split segment shouldn't change before and after migration. I think this is going to mean that we're just checking that it takes one of the 2 possible values.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to static values.

@github-actions github-actions bot added testing and removed testing labels Aug 13, 2024
@khvn26 khvn26 requested a review from matthewelwell August 13, 2024 20:17
@khvn26 khvn26 force-pushed the test/test-local-eval branch from 9e1508c to e11c727 Compare August 13, 2024 20:18
@github-actions github-actions bot added testing and removed testing labels Aug 13, 2024
Copy link
Contributor

@matthewelwell matthewelwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to confirm that I'm happy with the tests being performed (not as a suggestion we should merge).

@github-actions github-actions bot added testing and removed testing labels Aug 28, 2024
@github-actions github-actions bot added testing and removed testing labels Aug 28, 2024
@khvn26 khvn26 closed this Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants